tjJdZddlZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4m5Z5ddl6m7Z7e-r e/j8e9Z:dej;deZ?Gddej>Z@ dEdejAdej;d ej;d!ej;d"ej;dzd#eBdzd$eBd%e(e*fd&ZCGd'd(ejAZDGd)d*eZEGd+d,eZFGd-d.ejAZGe+Gd/d0e&ZHGd1d2eHZIGd3d4eHZJe+Gd5d6eHZKe+d78Gd9d:eHeZLe+d;8Gd<d=eHZMe+Gd>d?eHZNGd@dAeHZOGdBdCeHeZPgdDZQdS)FzPyTorch MBART model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availableis_torchdynamo_compilingloggingtorch_compilable_check)merge_with_config_defaults)OutputRecordercapture_outputs) MBartConfig input_ids pad_token_idc|}|td||dk|||ddz d}|d|}|ddddf|ddddf<||dddf<|S)z Shift input ids one token to the right, and wrap the last non pad token (the token) Note that MBart does not have a single `decoder_start_token_id` in contrast to other Bart-like models. Nz1self.model.config.pad_token_id has to be defined.ir'dimr)clone ValueError masked_fill_nesum unsqueezegathersqueeze)r)r*prev_output_tokens index_of_eosdecoder_start_tokenss f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mbart/modeling_mbart.pyshift_tokens_rightr;@s #**LMMM##$6$$> MMM&)),77;;;BBQFQQRTUUL-44Q EEMMOO 2111crc6 : @ @ B Bqqq!""u3qqq!t cbeZdZdZdedeffd Z d dejded ejdzffd ZxZ S) MBartLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcjd|_t||jz|dSN)offsetsuper__init__)selfr?r@ __class__s r:rFz(MBartLearnedPositionalEmbedding.__init__Zs3  $+5}EEEEEr<rNr)past_key_values_length position_idsc0|V|jdd\}}tj|||ztj|jj|d}n|d}t ||j zS)z3`input_ids' shape is expected to be [bsz x seqlen].NrC)dtypedevicer.r) shapetorcharangelongweightrMexpandr4rEforwardrD)rGr)rIrJbszseq_lenrHs r:rTz'MBartLearnedPositionalEmbedding.forward`s  $?2A2.LC <&(>(HPUPZcgcncufS"oo L(11!44Lww|dk9:::r<)rN) __name__ __module__ __qualname____doc__intrFrOTensorrT __classcell__rHs@r:r>r>UsFsF3FFFFFFmq ; ; ;?B ;V[VbeiVi ; ; ; ; ; ; ; ; ; ;r<r>c VeZdZdZd dededededzffd Zd ejffd Z xZ S) MBartScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?r?r@ padding_idx embed_scaleNc\t|||||_dSN)rErFrc)rGr?r@rbrcrHs r:rFz!MBartScaledWordEmbedding.__init__vs-  DDD&r<r)cVt||jzSre)rErTrc)rGr)rHs r:rTz MBartScaledWordEmbedding.forwardzs!wwy))D,<<>L=((6?([[L,|U33K''1--88::K  $$r<ceZdZdZ ddededed ed ed ed edzd edzffd Z dde j de j dzde dzde j dzde e dee j e j dzff dZxZS)MBartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperriFTN embed_dim num_headsrp is_decoderbias is_causalconfig layer_idxc zt||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ ||_ |/|j r(t d|jjdt!j||||_t!j||||_t!j||||_t!j||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rszInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)rErFrrrphead_dimrr0rorrrlogger warning_oncerHrWrLineark_projv_projq_projout_proj) rGrrrprrrrrrHs r:rFzMBartAttention.__init__sY "" !Y.  MI %$. 8 83dn33%.333 }d* $""      ,4>+B,,,    i 94@@@ i 94@@@ i 94@@@  )YTBBB r< hidden_stateskey_value_statespast_key_valuesrnrqreturnc |du}|jdd}g|d|jR}|||dd} d} |Ht |t r1|j|j } |r|j } n |j } n|} |r|n|} |r3|1| r/| j |j j } | j |j j}n|| } || }g| jddd|jR}| |dd} ||dd}|E| | ||j \} }|r$t |t rd|j|j <t%j|jjt,}||| | ||f|jsdn|j|jd|\}}|jg|dR}||}||fS) z#Input shape: Batch x Time x ChannelNr.r'rCFTri)rpro)rNrrviewry isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationrrvrproreshaper|r)rGrrrrnrqis_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacer~r}s r:rTzMBartAttention.forwards.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66LF-crc2FBF FFH#22<(?(M K ,.E) ) %8$7      % $}>CC$,L % %  % % ! \*k);;;;;;FFHH mmK00 L((r<)riFTFNNNNN)rWrXrYrZr[rhboolr(rFrOr\r rrtuplerTr]r^s@r:rrs]GG  %) $%C%C%C%C %C  %C  %C%Cd"%C:%C%C%C%C%C%CT15(,.2 H)H)|H) ,-H) H)  t+ H)-.H) u|U\D00 1H)H)H)H)H)H)H)H)r<rcfeZdZdeffd ZdejdejdeedejfdZ xZ S)MBartEncoderLayerrct|j|_t |j|j|j||_tj |j|_ |j |_ t|j |_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)rrrpr)rErFd_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrpr activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normrGrrHs r:rFzMBartEncoderLayer.__init__s 'n4,     %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r<rrnrqrc |}||}|jd||d|\}}tj||j|j}||z}|}||}|||}tj||j |j}| |}tj||j|j}||z}|j tj kr9tj|j jdz }tj|| |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rrnrti)minmax)rrrrzrprvrrrrrrLrOfloat16finforclamp)rGrrnrqresidual_ clamp_values r:rTzMBartEncoderLayer.forward$s[!11-@@ )4> ')     q  --mt|VZVc-dd  =0  --m<< **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0  %- / /+m&9::>EK!K K<[YYYMr<) rWrXrYr(rFrOr\rrrTr]r^s@r:rrs={======$"|" "+, "  """"""""r<rceZdZddededzffd Z ddejdejdzdejdzd ejdzd edzd e dzd e e d ejfdZ xZ S)MBartDecoderLayerNrrc t|j|_t |j|j|jdd|||_|j|_t|j |_ |j |_ tj|j|_t |j|j|jd|||_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rrrprrrr)rprrr)rErFrrrdecoder_attention_headsrrrpr rrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrr)rGrrrHs r:rFzMBartDecoderLayer.__init__Js  'n4,   ~ #F$>?"(";$&L$@$@!* N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r<Trrnencoder_hidden_statesencoder_attention_maskr use_cacherqrc |}||}|jd|||d|\}} tj||j|j}||z}|]|}||}|jd||||d|\}} tj||j|j}||z}|}||}| | |}tj||j |j}| |}tj||j|j}||z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )rrrnrtN)rrrnrr) rrrrzrprvrrrrrrr) rGrrnrrrrrqrrs r:rTzMBartDecoderLayer.forwardis*!11-@@ *4> '+)      q  --mt|VZVc-dd  =0  ! ,$H 88GGM0t0 +!65 /        M1M11-4z)MBartEncoder.__init__..s"$e$e$e1%6v%>%>$e$e$er<F)rErFrpencoder_layerdrop layerdroprr*rbmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrtr` vocab_size embed_tokensr>embed_positionsr ModuleListrangeencoder_layersrrrlayernorm_embedding layer_normgradient_checkpointing post_init)rGrrrcrHs ` r:rFzMBartEncoder.__init__s(    ~ 1N !.$*$B!.4.DMdi ***# 4  y$*:     ?  *   m$e$e$e$efNcHdHd$e$e$eff  #%< #:#: ,v~66&+# r<cp|jr,t|jddr|dSdSdS)NrF)rgetattrrgradient_checkpointing_enablerGs r:._backward_compatibility_gradient_checkpointingz;MBartEncoder._backward_compatibility_gradient_checkpointingsP  / 1GDKIach4i4i 1  . . 0 0 0 0 0 1 1 1 1r<Nr)rn inputs_embedsrqrc j|du|duz rtd|||}||d}|||jz}||}t j||j|j }t|j ||}t|j D];\}}d} |j r!tjg} | |jkrd} | s |||fi|}<||}t%|S) a  Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds).r.rt)rr rnFT)last_hidden_state)r0rrtorMrrrzrprvrr enumeraterrOrandr rr) rGr)rnr rq embed_posridx encoder_layerto_dropdropout_probabilitys r:rTzMBartEncoder.forward sk>  -t"; < [YZZ Z   --i88M((w)?@@ % ]5I(J(JJ 00??  --mt|VZVc-dd 2;')    #,DK"8"8   CG} #&+jnn#&77"G  - !"!!!!  66 ????r<r)rWrXrYrZrr%r_can_record_outputsr(rFrr$r&rO LongTensorr\ FloatTensorrrrrrTr]r^s@r:rrs,+$n^1UUU {8111  .2.226 @@@@#d*@@ t+@@(4/ @@ +, @@  @@@@@@_ @@@@@@@@@@r<rc.eZdZdZeeeddeedddZdeffd Z e e dd e j d zd e jd zd e jd zd e j d zded zde jd zded zdeedeezfdZxZS) MBartDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MBartDecoderLayer`] Args: config: MBartConfig embed_tokens (nn.Embedding): output embedding r'rrr)rrcross_attentionsrctj|_j|_j|_j|_j rtj j nd}tjj |j||_t!jj |_t%jfdt)jD|_|_t%jj |_t%jj |_d|_|dS)Nrarc2g|]}t|S))r)r)rirs r:r z)MBartDecoder.__init__..ns($r$r$rPQ%6v%K%K%K$r$r$rr<F)rErFrpdecoder_layerdropr r*rbr max_target_positionsrrrrr`rrr>rrrrdecoder_layersrrrrrrr)rGrrcrHs ` r:rFzMBartDecoder.__init__^s+    ~ 1!.$*$B!393IRdi///s 4  v~t/?[    ?  * N  m$r$r$r$rUZ[a[pUqUq$r$r$rss  #%<#?#? ,v~66&+# r<Nr)rnrrrr rrqrc |du|duz rtd|||}|r[|Y| |jjr6t t |jt |jnt |j}|dd\} } ||nd} tj | |j | z} |/ts!| | z} tj | | |j }t|tr|jn|}t|j|||}t!|j|||}||| | } || |j z}||}t(j||j|j }t1|jD];\}}|jr tjg}||jkr,||||f|||d |}<||}t;|| S) a( Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time)rr.rr)rr rnr)rr rnr)rJrt)rrr)r"r)r0rris_encoder_decoderr r rwget_seq_lengthrOrPrMr!onesrrrrrr#rrrzrprvr$rr%r rr)rGr)rnrrrr rrq batch_size seq_lengthrIrJmask_seq_lengthself_attn_cache causal_maskrr' decoder_layerr*s r:rTzMBartDecoder.forwardxsn  -t"; < ustt t   --i88M  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  "/!3!3!5!5crc!: JETE`!?!?!A!A!Afg|J}7KLLLOee  !*B*D*D !4zAO"Z OML`aaaN/+>?? !O 0 0   );')+    ";;'1"7 " " " ++I7M\h+ii %  8L(M(MM 00??  --mt|VZVc-dd "+DK"8"8   C} &+jnn#&77)M%(> /# MM 66 8++    r<)NNNNNNN)rWrXrYrZrr%rr+r(rFr$r&rOr,r\r-r rrrrrrTr]r^s@r:r/r/Ost+$n^1UUU*N>~^^^ {4 .2.2:>:>(,26!%} } #d*}  t+}  %047 } !& 04 7 }  } (4/} $;} +,}  : :} } } _ } } } } } r<r/cjeZdZdddZdeffd ZdZdZee dde j dzd e j dzd e j dzd e j dzd e e e jdzd edzde jdzde jdzdedzdedzdeedee e jzfdZxZS) MBartModelz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrc\t||j|j}}|jrt j|jnd}t||j|||_ t||_ t||_ |dS)Nrar)rErFr*rrrrrr`sharedrencoderr/decoderr)rGrrbrrcrHs r:rFzMBartModel.__init__s    "("5v7HZ 393IRdi///s .z6>;doppp #F++ #F++  r<c|jSre)rDrs r:get_input_embeddingszMBartModel.get_input_embeddingss {r<cX||_|j|j_|j|j_dSre)rDrErrFrGrms r:set_input_embeddingszMBartModel.set_input_embeddingss' $(K !$(K !!!r<Nr)rndecoder_input_idsdecoder_attention_maskencoder_outputsrr decoder_inputs_embedsr return_dictrqrc | | n |jj} ||t||jj}||jd |||| d| }ne| rct |t sNt |dt|dkr|dndt|dkr|dnd}|jd |||d|||| | d| } | s| |zSt| j | j | j | j | j|j |j |j S) a5 decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 25004 for *en_XX*, and 25003 for *de_DE*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)r)rnr rPrr'rC)r"rr)r)rnrrrr rrP)r"rdecoder_hidden_statesdecoder_attentionsr0encoder_last_hidden_staterencoder_attentionsr)rrPr;r*rErrlenrFrr"rrrr0) rGr)rnrLrMrNrr rOrrPrqdecoder_outputss r:rTzMBartModel.forwardsJ&1%)F 29dk>V W W   "*dl#-+'   OO O_!M!M -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO'$,  '1"1!"4#1+/#       5"_4 4!-?+;"1"?.9,=&5&G"1"?.9    r< NNNNNNNNNN)rWrXrY_tied_weights_keysr(rFrHrKrrrOr,r\rr-r rrrrrTr]r^s@r:rBrBs(7'6 {      000 .2.259:>BF(,26:>!%#'S S #d*S  t+S !+d2 S !& 04 7 S uU%6784? S S (4/S  %047S $;S D[S +,S  eE$56 6S S S ^S S S S S r<rBz The MBART Model with a language modeling head. Can be used for summarization, after fine-tuning the pretrained models. ) custom_introceZdZdZdgZddiZdeffd Z dd ed edzd e d e j ffd Z d ed dfdZ e ddejdzdejdzdejdzdejdzdeeejdzdedzdejdzdejdzdejdzde dzde dzdeed eeejzfdZdejfdZxZS)rrrlm_head.weightzmodel.shared.weightrclt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)Nrr'Fr)rErFrBrregister_bufferrOzerosrDr?rrrlm_headrrs r:rFz&MBartForConditionalGeneration.__init__xs    ''  0%+q$*BSBb>c2d2deeey1B1QX]^^^  r<NTnew_num_tokenspad_to_multiple_of mean_resizingrct|||}||jjd|S)Nr)rEresize_token_embeddings_resize_final_logits_biasrRrN)rGrarbrcnew_embeddingsrHs r:rez5MBartForConditionalGeneration.resize_token_embeddingssG88I[]jkk &&~'<'B1'EFFFr<c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nr.r'rr,r)rrNrOr_rMcatr^)rGraold_num_tokensnew_bias extra_biass r:rfz7MBartForConditionalGeneration._resize_final_logits_biass/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r<r)rnrLrMrNrr rOlabelsrrPrqc \| | n |jj} | <| rtdd} ||t | |jj}|j|f|||||||| | d | } || d|jz}d}| Kt}|| d|jj | d}| s|f| ddz}||f|zn|St||| j | j| j| j| j| j| j S) u6 decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 25004 for *en_XX*, and 25003 for *de_DE*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Translation: ```python >>> from transformers import AutoTokenizer, MBartForConditionalGeneration >>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-en-ro") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-en-ro") >>> example_english_phrase = "42 is the answer" >>> inputs = tokenizer(example_english_phrase, return_tensors="pt") >>> # Translate >>> generated_ids = model.generate(**inputs, num_beams=4, max_length=5) >>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] '42 este răspuns' ``` Mask filling example: ```python >>> from transformers import AutoTokenizer, MBartForConditionalGeneration >>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25") >>> # de_DE is the language symbol id for German >>> TXT = " Meine Freunde sind nett aber sie essen zu viel Kuchen. de_DE" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt")["input_ids"] >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['nett', 'sehr', 'ganz', 'nicht', 'so'] ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) rnrLrNrMrr rOrrPrr.r' losslogitsrrRrSr0rTrrU)rrPrwarningr;r*rr`rrrrrrrRrSr0rTrrU)rGr)rnrLrMrNrr rOrmrrPrqoutputs lm_logitsmasked_lm_lossloss_fctoutputs r:rTz%MBartForConditionalGeneration.forwards`&1%BF(,26:>*.!%#'z z #d*z  t+z !+d2 z !& 04 7 z uU%6784? z z (4/z  %047z  4'z $;z D[z +,z  5!23 3z z z ^z xDELDDDDDDDDr<rz MBart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c0eZdZdeffd Zee ddejdzdej dzdejdzdejdzde ej dzd ej dzd ej dzd ejdzd e dzd e edeezfdZxZS)MBartForSequenceClassificationrc tj|fi|t||_t |j|j|j|j|_| dSre) rErFrBrrr num_labelsclassifier_dropoutclassification_headr)rGrrqrHs r:rFz'MBartForSequenceClassification.__init__sq**6***'' #: N N    % $ $   r<Nr)rnrLrMrNr rOrmrrqrc J|d} ||td|jj|j|f||||||| d| } | d} ||jj| j} ttj | d dkd| | ddf}t|jd| jdzdkd|| dd | d ddd ddf}||}d}|||j}|jjp|jjdkr d |j_nS|jjdkr7|jtjks|jtjkr d |j_n d |j_|jjd kr\t/}|jjdkr1|||}n|||}n|jjd krLt3}||d |jj|d }n*|jjd krt5}|||}t7||| j| j| j| j| j | j!| j" S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rnrLrMrNr rOrrr'z7All examples must have the same number of tokens.z3Each example must contain at least one token.r. regressionsingle_label_classificationmulti_label_classificationro)#NotImplementedErrorrHrWreqr eos_token_idr#rMr#rOunique_consecutiver3numelrNrrwr problem_typerrLrQr[rr6rrrrrRrSr0rTrrU)rGr)rnrLrMrNr rOrmrrqrsreos_maskselectedsentence_representationrqrprvs r:rTz&MBartForSequenceClassification.forward&sHT  I  !:%d4>Kbdd '1dj  ' )/#9+'"7 ' '  ' '   << 899<<]=QRR  $X\\!__ 5 5 ; ; = = B E   !111- N1 !4Q!7 71 < A   #+-- 0B0B10E0Er=K]K]^`KaKa"b"bcdcdcdfhjkjkjkck"l))*ABB  YYv}--F{'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x//.#3")"?&9$5&-&G")"?&9    r<) NNNNNNNNN)rWrXrYr(rFrrrOr,r\listr-rrrrrrTr]r^s@r:r}r}se {      .2.259:>:>26:>*.!%l l #d*l  t+l !+d2 l !& 04 7 l e/047 l (4/l  %047l  4'l $;l +,l  0 0l l l ^l l l l l r<r}c@eZdZfdZee ddejdzdejdzdejdzdejdzde ej dzdejdzd ejdzd ej dzd ej dzd e dzd e e deezfdZxZS)MBartForQuestionAnsweringct|d|_|j|_t||_t j|j|j|_| dSrB) rErFrrBrrr hidden_size qa_outputsrrs r:rFz"MBartForQuestionAnswering.__init__sm     +'' )F$68IJJ r<Nr)rnrLrMrNstart_positions end_positionsr rOrrqrc ||d} |j|f|||||| | d| } | d} || }|dd\}}|d}|d}d}||t |dkr|d}t |dkr|d}|d}|d|}|d|}t|}|||}|||}||zd z }t|||| j | j | j | j | j| j| j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NFrrr'r.r,) ignore_indexrC) rp start_logits end_logitsrrRrSr0rTrrU)rrsplitr6r|rVrwrrrrrRrSr0rTrrU)rGr)rnrLrMrNrrr rOrrqrssequence_outputrqrr total_loss ignored_indexrv start_lossend_losss r:rTz!MBartForQuestionAnswering.forwardsP  &=+DI&0dj  ' )/#9+'"7 ' '  ' ' "!*11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM']CCCH!,@@Jx M::H$x/14J2%!#3")"?&9$5&-&G")"?&9    r<rX)rWrXrYrFrrrOr\r,rr-rrrrrrTr]r^s@r:rrsn     *..259:>:>371526:>!%W W <$&W  t+W !+d2 W !& 04 7 W e/047 W )D0W '$.W (4/W  %047W $;W +,W  4 4W W W ^W W W W W r<rc(eZdZdZfdZdZxZS)MBartDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ct|t||_|dSre)rErFr/rFrrs r:rFzMBartDecoderWrapper.__init__ s@    #F++  r<c|j|i|Sre)rF)rGargsrqs r:rTzMBartDecoderWrapper.forwardst|T,V,,,r<)rWrXrYrZrFrTr]r^s@r:rrsQ  -------r<rc(eZdZddiZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e dzd e j dzde j dzdedzdee j zdeedeezfdZxZS)MBartForCausalLMr\z!model.decoder.embed_tokens.weightcd|_d|_t|t ||_t j|j|j d|_ | dS)NTFr) rr8rErFrrrrrrr`rrs r:rFzMBartForCausalLM.__init__sp $)!    (00 y!3V5FUSSS  r<c$|jjjSrerrFrrs r:rHz%MBartForCausalLM.get_input_embeddings$sz!..r<c(||jj_dSrerrJs r:rKz%MBartForCausalLM.set_input_embeddings's*/ '''r<Nrr)rnrrrr rmrlogits_to_keeprqrc |jjd|||||||d| } | d} t| trt | dn| } || dd| ddf}d}|e||j}t}|| d|j j | d}t||| j | j| j| jS)aP labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25") >>> model = MBartForCausalLM.from_pretrained("facebook/mbart-large-cc25") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```)r)rnrrrr rrNr.)rprqrrrr0r)rrFrr[slicer`r#rMrrrrrrrrr0)rGr)rnrrrr rmrrrqrsr slice_indicesrqrprvs r:rTzMBartForCausalLM.forward*s)L>PTZ=O > )"7#9+' > >  > >   8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  YYv}--F'))H8FKKDK,BCCV[[QS__UUD0#3!/)$5     r<) NNNNNNNNr)rWrXrYrYrFrHrKrrrOr,r\r-r rr[rrrrrTr]r^s@r:rrs{=     ///000.2.2:>;?(,26*.!%-.A A #d*A  t+A  %047 A !& 1D 8 A  A (4/A  4'A $;A el*A +,A  2 2A A A ^A A A A A r<r)rrrr}rBr)Nri)RrZrcollections.abcrrOrtorch.nnrrrr r activationsr cache_utilsr r r generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!r"r# utils.genericr$utils.output_capturingr%r&configuration_mbartr( get_loggerrWrr\r[r;r{r>r`Modulerhrrrrrrrr/rBrr}rrr__all__rr<r:rsc $$$$$$ AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ:99999GFFFFFFF&&&&&&877777EEEEEEEE,,,,,, !!   H % %%,c*;;;;;bl;;;8 = = = = =r| = = =(!%% I% <% % < % L4' % T\ %%'(%%%%:r)r)r)r)r)RYr)r)r)j555552555pZZZZZ2ZZZ|bi0?4r@r@r@r@r@'r@r@r@jh h h h h 'h h h Vp p p p p %p p p f \D\D\D\D\D$8/\D\D  \D~ } } } } } %9} }  } @g g g g g 4g g g V - - - - -. - - - Y Y Y Y Y +_Y Y Y x   r<