r?ronesshaperHanyget_seq_lengthis_compileabler _ignore_causal_mask_sdpatrainingr<get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiontypefinfomin_unmask_unattended) r@r3rCrDrEpast_seen_tokensusing_compilable_cacher<rLrM causal_mask min_dtypes r*_update_causal_mask)PLBartPreTrainedModel._update_causal_mask]s ;; + +/? ?.%,,77!<^!L" ! '!<JJ*003\5G5G5JK-44" " ! ;; + +/B B)~/D.I.I.K.K%% @O?Z?99;`aCRC^!?!?di ;; + +v 5>T%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD  E*..I0CCK[Kr)rLrMr<rNc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer<rHr)diagonalrHrKr)dimr>r\r]fullrHtriuarangereshapeexpandclonerSto masked_fill) r3rLrMr<rDrNkwargsrarb mask_length padding_masks r*rZKPLBartPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positions}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r)encoder_hidden_statesencoder_attention_mask input_shapecUbUbURRS:XaSU;aUnU$SnU$URRS:Xa[UURUSS9nU$URRS:Xa/[ U[ R 5(a[UUSSS9nU$[X$RUSS9nU$) Nr6rr7rK)tgt_lenr8F) query_lengthr9r:)r@rwrxryr4s r*_update_cross_attn_mask-PLBartPreTrainedModel._update_cross_attn_masks ! ,1G1S{{//3FFCDH^C^)?&.&%/ei&.&%-11V;*M*!'''O*&$&%115EE4ellCC-H.%0_"'.*&% *D*,?,?UW*&&%r)r")r$r%r&r'r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnrr>r?rArrrc staticmethodintr<rZSizer}r(r"r)r*r-r-;sN &*#-/CDNellD01||.J u||['@!ABJllJ J  JX4 444{{ 4  4  44n!&$U\\4%78!&!&ellD&8 9!&ZZ !& || !&r)r-c\rSrSrSrg) PLBartEncoderir"Nr#r"r)r*rrr+r)rc\rSrSrSrg) PLBartDecoderi r"Nr#r"r)r*rr r+r)rc&8^\rSrSrSS/rS\4U4SjjrSrSrSr S r S r \ SS \ \RS \ \RS \ \RS\ \R S\ \R S\ \RS\ \R S\ \\R$S\ \S\ \R$S\ \R$S\ \S\ \S\ \S\ \S\ \RS\\\R \44"Sjj5rSrU=r$) PLBartModeliencoder.embed_tokens.weightdecoder.embed_tokens.weightr/cr>[TU]U5 URURp2UR(a [ R "UR5OSn[X1RX$S9Ul [XR5Ul [XR5Ul UR5 g)Ng?) embed_scale)super__init__ pad_token_id vocab_sizescale_embeddingmathsqrtd_modelr sharedrencoderrdecoder init_weights)r@r/ padding_idxrr __class__s r*rPLBartModel.__init__s}  "("5"5v7H7HZ393I3Idii/s / NNKq $V[[9 $V[[9  r)cUR$N)rr@s r*get_input_embeddings PLBartModel.get_input_embeddingss {{r)c|XlURURlURURlgr)rr embed_tokensr)r@values r*set_input_embeddings PLBartModel.set_input_embeddings!s' $(KK !$(KK !r)cURR(aaURURRUR 5 URUR RUR 5 ggr)r/tie_word_embeddings_tie_or_clone_weightsrrrrrs r* _tie_weightsPLBartModel._tie_weights&sP ;; * *  & &t||'@'@$++ N  & &t||'@'@$++ N +r)cUR$r)rrs r* get_encoderPLBartModel.get_encoder+ ||r)cUR$r)rrs r* get_decoderPLBartModel.get_decoder.rr) input_idsr3decoder_input_idsdecoder_attention_mask head_maskdecoder_head_maskcross_attn_head_maskencoder_outputsrEr4decoder_inputs_embeds use_cacheoutput_attentionsoutput_hidden_states return_dictrDreturncU bU OURRn UbUOURRnU bU OURRn UbUOURRnUc"U c[ XRR 5nUcURUUUU U UUS9nORU(aK[U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nURUUUSUUUU U U U UUUS9 nU(dUU-$[URURURUR UR"URURUR S9$) a: decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. N)rr3rr4rrrrrr)last_hidden_state hidden_states attentions) rr3rwrxrrrEr4rrrrrD)rrEdecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_staterwencoder_attentions)r/rrruse_return_dictrrrr=r lenrrrrErrr)r@rr3rrrrrrrEr4rrrrrrDdecoder_outputss r*forwardPLBartModel.forward1sb2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK)F 29kk>V>V W   ""ll#-#+"3%9'+OO_!M!M-"1!"4474H14Loa0RV14_1E1I?1-tO,,'1"1!"4#1'!5+//!5#)'  "_4 4!-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r))rrr)NNNNNNNNNNNNNNNN)r$r%r&r'_tied_weights_keysrrrrrrrrrr> LongTensorr?list FloatTensorrboolrtuplerrr( __classcell__rs@r*rrs79VW | 0 O 15598<9=,08<7;=A+/59=A$(,0/3&*59#k E,,-k !!1!12k $E$4$45 k !) 6 k ELL) k $E$4$45k 'u||4k "$u'8'8"9:k "%k   1 12k  ((9(9:k D>k $D>k 'tnk d^!k "!!1!12#k $ uU\\"$66 7%k k r)rzv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introc(^\rSrSrSrS/r/SQrS\4U4SjjrSr Sr S$S \ S \ \ S \ S \R4U4SjjjrS \ S S 4Sjr\S%S\ \R(S\ \R(S\ \R(S\ \R*S\ \R*S\ \R(S\ \R*S\ \\R.S\ \S\ \R.S\ \R.S\ \R*S\ \ S\ \ S\ \ S\ \ S \ \R(S \\\R*\44$S!jj5rS\R*4S"jrS#rU=r$)&PLBartForConditionalGenerationir0final_logits_bias)rrzlm_head.weightr/cv>[TU]U5 [U5UlUR S[ R "SURRR455 [R"URURRRSS9Ul UR5 g)NrrF)bias)rrrr0register_bufferr>zerosrnum_embeddingsrLinearrlm_headr)r@r/rs r*r'PLBartForConditionalGeneration.__init__s   (  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r)c6URR5$r)r0rrs r*r*PLBartForConditionalGeneration.get_encoderzz%%''r)c6URR5$r)r0rrs r*r*PLBartForConditionalGeneration.get_decoderrr)Nnew_num_tokenspad_to_multiple_of mean_resizingrcx>[TU]XU5nURURRS5 U$)Nr)rresize_token_embeddings_resize_final_logits_biasweightrS)r@rrrnew_embeddingsrs r*r6PLBartForConditionalGeneration.resize_token_embeddingss<8]jk &&~'<'<'B'B1'EFr)c,URRSnX::aURSS2SU24nON[R"SX- 4URRS9n[R "URU/SS9nUR SU5 g)NrKrri)rjr)rrSr>rrHcatr)r@rold_num_tokensnew_bias extra_biass r*r8PLBartForConditionalGeneration._resize_final_logits_biass//55b9  +--a..@AHa)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r)rr3rrrrrrrEr4rlabelsrrrrrDcUbUOURRnU b%Uc"U c[XRR5nUR UUUUUUUUU U U U UUUUS9nUR US5nUUR RUR5-nSnU bF[5nU"URSURR5U RS55nU(dU4USS-nUbU4U-$U$[UUURURURUR UR"UR$UR&S9 $)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N)r3rrrrrrrEr4rrrrrrDrrKr) losslogitsrErrrrrwr)r/rrrr0rrrqrHrviewrrrErrrrrwr)r@rr3rrrrrrrEr4rrrrrrrDoutputs lm_logitsmasked_lm_lossloss_fctoutputs r*r&PLBartForConditionalGeneration.forwards{V&1%rr?rrrrrrrrr(rrs@r*rrsL  ':&;#i|((dh!7?}\` <<<15598<9=,08<7;=A+/59=A)-$(,0/3&*59%x E,,-x !!1!12x $E$4$45 x !) 6 x ELL) x $E$4$45x 'u||4x "$u'8'8"9:x "%x   1 12x  ((9(9:x &x D>x $D>x 'tn!x "d^#x $!!1!12%x & uU\\"O3 4'x x tDELLDDr)rc\rSrSrSrg)PLBartClassificationHeadiGr"Nr#r"r)r*r r Gr+r)r c(^\rSrSrU4SjrSrU=r$)PLBartForSequenceClassificationiKc :>[5R"S0UD6 g)aL decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr"rr super_kwargsrs r*r'PLBartForSequenceClassification.forwardLsB ','r)r")r$r%r&r'rr(rrs@r*rrKs !(!(r)rc2^\rSrSr\U4Sj5rSrU=r$)PLBartForCausalLMipc :>[5R"S0UD6 g)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nr"rrs r*rPLBartForCausalLM.forwardqs: ','r)r")r$r%r&r'rrr(rrs@r*rrps((r)r)rrrrr-)4__doc__rtypingrrr>torch.utils.checkpointrtorch.nnr cache_utilsr generationr modeling_attn_mask_utilsr r r modeling_outputsr rrmodeling_utilsrutilsrrbart.modeling_bartrrrrr(bigbird_pegasus.modeling_bigbird_pegasusrmbart.modeling_mbartrconfiguration_plbartrintegrations.flex_attentionrrr r-rrrrr rr__all__r"r)r*r(s$ " % )  .A_5. !!U 7 G&OG&G&T K  K N 'N N b _D%:O_D  _DD 5 "(&M"(J((B r)