tj)dZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'j+e,Z-Gddej.Z/Gddej.Z0Gddej.Z1Gddej.Z2Gddej.Z3Gddej.Z4Gddej.Z5Gdd eZ6Gd!d"ej.Z7e&Gd#d$e"Z8Gd%d&e8Z9e&Gd'd(e8Z:e&d)*Gd+d,e8eZ;e&Gd-d.e8Z<e&d/*Gd0d1e8Z=e&Gd2d3e8Z>e&Gd4d5e8Z?gd6Z@dS)7zPyTorch mT5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringloggingtorch_compilable_check) MT5Configc&eZdZdfd ZdZxZS) MT5LayerNormư>cttjt j||_||_dS)zd Construct a layernorm module in the MT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s b/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mt5/modeling_mt5.pyr%zMT5LayerNorm.__init__0sD l5:k#:#:;; #ch|tjddd}|tj||jzz}|jjtj tj fvr||jj}|j|zS)NT)keepdim) tor'float32powmeanrsqrtr*r)dtypefloat16bfloat16)r+ hidden_statesvariances r/forwardzMT5LayerNorm.forward8s !##EM2266q99>>r4>PP% Ht?T4T(U(UU  ;  ? ? ?),,T[->??M{]**r0)r")__name__ __module__ __qualname__r%r? __classcell__r.s@r/r!r!/sL$$$$$$ + + + + + + +r0r!c*eZdZdeffd ZdZxZS)MT5DenseActDenseconfigcJttj|j|jd|_tj|j|jd|_tj|j |_ t|j |_ dSNFbias)r$r%rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr+rGr.s r/r%zMT5DenseActDense.__init__Jsx )FNFKeDDD)FKeDDDz&"566 &-.r0c||}||}||}t|jjt jr]|j|jjjkrC|jjjt j kr$| |jjj}||}|SN) rOrUrS isinstancerPr)r'Tensorr:int8r5r+r=s r/r?zMT5DenseActDense.forwardQs .. //  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r0r@rArBrr%r?rCrDs@r/rFrFIsS/y//////       r0rFc*eZdZdeffd ZdZxZS)MT5DenseGatedActDenserGcttj|j|jd|_tj|j|jd|_tj|j|jd|_tj |j |_ t|j |_dSrI)r$r%rrLrMrNwi_0wi_1rPrQrRrSr rTrUrVs r/r%zMT5DenseGatedActDense.__init__as IfnfkFFF IfnfkFFF )FKeDDDz&"566 &-.r0c|||}||}||z}||}t |jjtjr]|j |jjj krC|jjj tj kr$| |jjj }||}|SrX) rUrarbrSrYrPr)r'rZr:r[r5)r+r= hidden_gelu hidden_linears r/r?zMT5DenseGatedActDense.forwardishhtyy7788  -00 #m3  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r0r]rDs@r/r_r_`sS/y//////r0r_c*eZdZdeffd ZdZxZS) MT5LayerFFrGc$t|jrt||_nt ||_t |j|j|_ tj |j |_ dS)Nr-)r$r% is_gated_actr_DenseReluDenserFr!rMlayer_norm_epsilon layer_normrrQrRrSrVs r/r%zMT5LayerFF.__init__sx    ;"7"?"?D  "26":":D &v~6;TUUUz&"566 r0c||}||}|||z}|SrX)rmrkrS)r+r=forwarded_statess r/r?zMT5LayerFF.forwardsF??=99../?@@% 5E(F(FF r0r]rDs@r/rgrg~sS7y777777r0rgcfeZdZ d dededzffd Zedd Zdd Z dd Z xZ S) MT5AttentionFNrG layer_idxc*t|j|_||_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |jz|_ ||_|/|jr(td|jjdt'j|j|j d|_t'j|j|j d|_t'j|j|j d|_t'j|j |jd|_|jr$t'j|j|j |_d|_dS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrJ)r$r% is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerMd_kvkey_value_proj_dim num_headsn_headsrRrS inner_dimrrlogger warning_oncer.r@rrLqkvo Embeddingrelative_attention_biasgradient_checkpointingr+rGrurrr.s r/r%zMT5Attention.__init__si  ++F(.4.S+/5/U,~ "(+' * (??"      ,4>+B,,,    4<eDDD4<eDDD4<eDDD4>4=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr2r) r5r'longabsmin zeros_likelogfloatmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r/_relative_position_bucketz&MT5Attention._relative_position_buckets>,  c A K !2Q!6 : :5: F F T T  % *; < <  !&+NO`>a>a!b!b b  1$ $y0&/ I'--//); < <h|i/00 1Y& ( "UZ.. &" &+Y &8RT_bcTc(d(d& & " EK2CE_```r0rc||jjj}tj|tj|dddf|z}tj|tj|dddf}||z }|||j |j|j }||} | gd d} | S)z%Compute binned relative position biasN)r:device)rrr)r2rrr) rr)rr'arangerrrtrvrwpermute unsqueeze) r+ query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess r/ compute_biaszMT5Attention.compute_biass >18?F < EJvVVVWXWXWXZ^W^_brr,zFSSSTXZ[Z[Z[T[\+.>>#'#A#A #.;= $B$ $  --.FGG **44Q77 r0c |jdd}g|d|jR} |||jnd} t | t jr| n| } |du} || |  dd} d} t |tr1|j |j} | r|j}n |j}n|}| r|n|}| r3|1| r/|j|jj}|j|jj}ng|jddd|jR}|| | dd}|| | dd}|E||||j\}}| r$t |trd|j |j<t j| | dd}||jd }|jsLt jd| jd|d|f|j|j }|jr|jrd|_n$||d||j| }|$|ddddddd|jd f}||z}|}||z }t>j !|"d #|}t>j $||j$|j }t j||}| dd%}|j&g|dR}|'|}||f}|r||fz}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr3rrr2FTr)rr:)rrdim)ptraining)(shaperyget_seq_lengthrrrYr'rZclonerview transposer is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdatematmulruzerosrr:rr requires_gradrr functionalsoftmaxrtype_asrS contiguousreshaper)r+r=maskkey_value_states position_biaspast_key_valuesoutput_attentionskwargs input_shape hidden_shaperis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapescoresr causal_maskposition_bias_masked attn_weights attn_outputoutputss r/r?zMT5Attention.forwards$)#2#. BBbB$*ABB M\Mh?99$.IIIno7ABRTYT`7a7aw+11333gw.T9vvm,,11,??II!QOO  o': ; ; 3(377GGJ! L'6'L$$'6'K$$#2 -?R))]  F/"=*"=-4T^DIJ/6t~FMLLP-crc2PBP8OPPH//44X>>HHANNJ66.1166x@@JJ1aPPL*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>lJ,@,@A,F,FGG  #)"-J3  %  *1-{1~zJSYS`hnht!!! .74=726M/ $ 1 1NJv}Wg!2!! "111aaa,Bj.>r.B,B#BC - ; ,&&},,V\\^^,DDLLVTT },,\T\TXTa,bb l<>> !++Aq11<<>> )k);;;;;; ff[))  .  0/Gr0FN)Trr)Nr)NNNNF) r@rArBrintr% staticmethodrrr?rCrDs@r/rqrqs%* $ , , ,: , , , , , ,D- - - \- ^([[[[[[[[r0rqc>eZdZddedzffd Z ddZxZS)MT5LayerSelfAttentionFNrrctt||||_t |j|j|_tj |j |_ dS)Nrurrri) r$r%rq SelfAttentionr!rMrlrmrrQrRrSrs r/r%zMT5LayerSelfAttention.__init__Usl ) 0KW`   'v~6;TUUUz&"566 r0c ||}|||||||} ||| dz}|f| ddz} | S)N)rrr use_cacherrr)rmrrS) r+r=attention_maskrrrrrnormed_hidden_statesattention_outputrs r/r?zMT5LayerSelfAttention.forward]s} $}==-- '+/ .  & 5Ea5H(I(II  "%5abb%99r0r)NNNFFr@rArBrr%r?rCrDs@r/rrTss77SSWZ777777r0rc<eZdZddedzffd Z ddZxZS)MT5LayerCrossAttentionNrrctt|d||_t |j|j|_tj |j |_ dS)NFrri) r$r%rqEncDecAttentionr!rMrlrmrrQrRrS)r+rGrrr.s r/r%zMT5LayerCrossAttention.__init__wsc +FPUajkkk&v~6;TUUUz&"566 r0Fc ||}|||||||} ||| dz} | f| ddz} | S)N)rrrrrrr)rmrrS) r+r=rrrrrrrr layer_outputrs r/r?zMT5LayerCrossAttention.forward}s| $}==// -'+/ 0  %t||4DQ4G'H'HH /$4QRR$88r0rX)NNNFrrDs@r/rrvso77#*777777r0rcFeZdZddedzffd Z ddZxZS) MT5BlockFNrrct|j|_tj|_|jt||||jr)|jt|||jt|dS)Nr)rr) r$r%rtr ModuleListlayerappendrrrgrs r/r%zMT5Block.__init__s  +]__   !&Famv w w w    ? S J  4VyQQQ R R R *V,,-----r0Tc  |jd|||||| } | d}| dd} |jtjkrtjtj|tj|jjdz tj|jj}tj || |}|j o|du}|r|jd|||||| }|d}|jtjkrtjtj|tj|jjdz tj|jj}tj || |}| |ddz} |jd|}|jtjkrtjtj|tj|jjdz tj|jj}tj || |}|f}|| zS)Nr)rrrrrri)rmax)rrrrrr3) rr:r'r;risinfanyfinforclamprt)r+r=rrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrrr return_dictrself_attention_outputsattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrs r/r?zMT5Block.forwardso"/A )'+/ " " " /q1 21226  %- / /+ M**..00 M/004t; M/004K "K K<[YYYM!_R1Fd1R  P&3djm!65; /"3 ''' #4A6M"em33#kK ..2244K 34484?K 3448 !& M |Q\ ] ] ] !24KABB4O O ' 2}55   %- / /+ M**..00 M/004t; M/004K "K K<[YYYM " ' ' r0r) NNNNNNFFTrrDs@r/rrs . .SSWZ . . . . . ."#&*J J J J J J J J r0rcLeZdZdZdeffd ZdejdejfdZxZ S)MT5ClassificationHeadz-Head for sentence-level classification tasks.rGcttj|j|j|_tj|j|_tj|j|j |_ dS)N)r) r$r%rrLrMdenserQclassifier_dropoutrS num_labelsout_projrVs r/r%zMT5ClassificationHead.__init__sc Yv~v~>> zF$=>>>  &.&2CDD r0r=returnc||}||}tj|}||}||}|SrX)rSrr'tanhrr\s r/r?zMT5ClassificationHead.forwards[ ]33  =11  =11  ]33  m44 r0) r@rArB__doc__rr%r'rZr?rCrDs@r/rrsw77EyEEEEEE U\elr0rceZdZUeed<dZdZdZdgZdgZ e dZ e j fdZdZxZS) MT5PreTrainedModelrG transformerTrrPcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r'tensorrr)r+r input_mask dummy_inputss r/rzMT5PreTrainedModel.dummy_inputss=L.. \*-- !*"&0  r0c\ t||jj}t |t rt j|j|dzd St |ttttfrt j |jjd|dzt|dr0|jjs$t j |jjd|dzt|drQt j |jjd||jjdzzt j|jjd Sd St |t,rVt|drDt j |jjd|dzt j|jjd Sd St |t0rt j |jjd||jjdzzt|jdr*|jjt j|jjt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |t6rt j |jjd||jjdzzt|jdr*|jjt j|jjt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |t>rVt j |j jd||jjdzzt|j dr*|j jt j|j jt j |j!jd||jjdzzt|j!dr*|j!jt j|j!jt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |tDr|jj}|jj#}|jj$}t j |j%jd|||zdzzt j |j&jd||dzzt j |j'jd||dzzt j |j(jd|||zdzz|j)r+t j |j*jd||dzzd Sd Sd S) zInitialize the weightsg?g)r8stdlm_head qa_outputsg classifierrKN)+r$ _init_weightsrGinitializer_factorrYr!init constant_r)MT5ModelMT5ForConditionalGenerationMT5EncoderModelMT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrMzeros_rKMT5ForTokenClassificationrrrrrFrOrPrNr_rarbrqrxrzrrrrrur)r+modulefactorrMryr{r.s r/rz MT5PreTrainedModel._init_weightss f%%%/ fl + +1 p N6=&3, 7 7 7 7 7   2OE\ ]  / p L-CVc\ J J J Jvy)) P$+2Q P V^2&3,OOOOv|,, 4 V.5CVPTP[PchlOlEmnnnn F-233333 4 4 9 : :% pv|,, 4 V.5CVc\RRRR F-233333 4 4 5 6 6! p L,3Ft{GZ_cFc%>%@%@#qrr'"$:&!  PQQ Q&&'8D'@,OOO  r0)r@rArBr__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr'no_gradrr3rCrDs@r/r r s %&*#!# !F XU]__5p5p5p5p_5pn!!!!!!!r0r c@eZdZfdZdZ ddZxZS)MT5Stackcttjjj|_j|_tjfdtj D|_ tjj |_tjj|_|d|_dS)Nc Vg|]%}tt|dk|&S)rr)rbool).0irGs r/ z%MT5Stack.__init__..ps4 w w wYZXf$qAv,,RS T T T w w wr0riF)r$r%rr vocab_sizerM embed_tokensrtrrange num_layersblockr!rlfinal_layer_normrQrRrS post_initrrVs `r/r%zMT5Stack.__init__is    L):FNKK +] w w w w^cdjdu^v^v w w w   !-V^AZ [ [ [z&"566  &+###r0c||_dSrX)rEr+new_embeddingss r/set_input_embeddingszMT5Stack.set_input_embeddingsys*r0Nc ||n |jj}||n |jj}| | n |jj} | | n |jj} |#|!|jrdnd} t d| d| d|1|} |d| d}n@||dd} n!|jrdnd} t d| d| d |j r%|j r|rt d d }|+|j t d | |}| \}}|d ur|jst d|d|jr]|rZ|X|jjr7tt!|jt!|j}nt!|j}n |jsd}|jjr5t#|j||t%|tr|jn|}nt)|j||}d}|jr|t)|j|||}| rdnd}|rdnd}|r |jrdnd}d}d}||}|jD]e}| r||fz}|||||||||||  }|d}|d}|jr| ||rdnd}|r||dfz}|jr ||dfz}f||}||}| r||fz}| st1d|||||fDSt3|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer3zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz#MT5Stack.forward..s4  =!===  r0)last_hidden_staterr= attentionscross_attentions)rGrroutput_hidden_statesrrtr/sizerrrr}r~rEis_encoder_decoderr r rrYrrrSrHrItupler)r+rrrrrRrrrrZrrerr_msg_prefixr batch_size seq_lengthencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrrr= layer_module layer_outputss r/r?zMT5Stack.forward|s"+!6IIDK+/?BZZNw>wwwww  "#..**K!r;r?;;II  &',,..ss3KK+/?BZZNu>uuXfuuuvv v  & "4= " "##p"   ( !_``` --i88M!, J   ? j !hT!h!h!hiii ? # G_4;1G&9$DK888,dk:Z:Z:Z''OO'3$+&F&F&FO ##O ; ! /{+-o/BCC!% D D$ NN7{+-N +/' ? 4@.G{+5&; /// +#7@BBD0:d&7VDOVrrRV (,% ]33  J V VL# I$58H$H!(L%/- /#"3'   M*!,M *!,M ]#8#D0=CT>[aaZ[0\-  V!/=3C2E!E?V+?=QRCSBU+U(--m<<  ]33   E 1]4D D    "#%"(       9+++%1     r0) NNNNNNNNNN)r@rArBr%rNr?rCrDs@r/r=r=hs,,,,, +++ "#![ [ [ [ [ [ [ [ r0r=c|eZdZUdZdZeed<dgZdddZdeffd Z dZ d Z e dd e jd zd e jd zd e jd zde jd zdeee jd zded zde jd zde jd zded zded zded zded zdee jezfdZxZS)raw Examples: ```python >>> from transformers import MT5Model, AutoTokenizer >>> model = MT5Model.from_pretrained("google/mt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, return_tensors="pt") >>> labels = tokenizer(text_target=summary, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```mt5rGFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightct|tj|j|j|_tj|}d|_ d|_ t||_ tj|}d|_ |j |_t||_|dSNFT)r$r%rrrDrMr$copydeepcopyrtrr=encodernum_decoder_layersrGdecoderrJr+rGencoder_configdecoder_configr.s r/r%zMT5Model.__init__6s    l6#4fnEE v..$)!#( // v..$(!$*$=!//  r0c|jSrXr$r+s r/get_input_embeddingszMT5Model.get_input_embeddingsH {r0c|||_|j||j|dSrXr$rrrNrtrLs r/rNzMT5Model.set_input_embeddingsL;$  )).999 )).99999r0Nrrrrencoder_outputsrrRdecoder_inputs_embedsrrrZrrc "| | n |jj} | | n |jj} |||||| | | }ne| rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}|||||||| | | |  }| s||zSt|j |j |j |j |j |j |j |j S) a\ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./mt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [MT5 Training](./mt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, MT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> model = MT5Model.from_pretrained("google/mt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for MT5Model. >>> # This is not needed for torch's MT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```NrrrRrrZrrrr2rWr=rX rrrRrrrrrrZr)rWrdecoder_hidden_statesdecoder_attentionsrYencoder_last_hidden_staterencoder_attentions)rGrrrrrYrlenrtrrWrr=rXrY)r+rrrrrrrRrrrrZrrr=decoder_outputss r/r?zMT5Model.forwardQsuF"+!6IIDKBF(,-159!%)-,0#'q q #d*q )D0q !+d2 q !& 04 7 q uU%6784? q q |d*q  %|d2q $;q  $;q #Tkq D[q  u !$6 6q q q ^q q q q q r0rz; MT5 Model with a `language modeling` head on top. ) custom_introceZdZUdZdZeed<dgZddddZdeffd Z dZ d Z e dd e jd zd e jd zd e jd zde jd zdeee jd zded zde jd zde jd zde jd zded zded zded zded zdee jezfdZde jfdZxZS)r a Examples: ```python >>> from transformers import MT5ForConditionalGeneration, AutoTokenizer >>> model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```rhrGrirj)rlrmzlm_head.weightct||j|_t j|j|j|_tj |}d|_ d|_ t||_ tj |}d|_ |j|_t||_t j|j|jd|_|dS)NFTrJ)r$r%rM model_dimrrrDr$rprqrtrr=rrrsrGrtrLrrJrus r/r%z$MT5ForConditionalGeneration.__init__s    l6#4fnEE v..$)!#( // v..$(!$*$=!// y1BOOO  r0c|jSrXryrzs r/r{z0MT5ForConditionalGeneration.get_input_embeddingsr|r0c|||_|j||j|dSrXr~rLs r/rNz0MT5ForConditionalGeneration.set_input_embeddingsrr0NrrrrrrrRrlabelsrrrZrrc | | n |jj} | | n |jj} |||||| | | }ne| rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}| |||| }|||||||| | | |  }|d}| |}d}| ptd }| |j } || d |d | d }| s|f|ddz|z}||f|zn|St|||j|j|j|j|j|j|j S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./mt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [MT5 Training](./mt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, MT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nrrrr2rrr, ignore_indexr3 losslogitsrrrrYrrr)rGrrrrrYrrr3rtrrr5rrr[rrr=rXrYrW)r+rrrrrrrRrrrrrZrrr=rsequence_output lm_logitsrloss_fctoutputs r/r?z#MT5ForConditionalGeneration.forwardsBR"+!6IIDK=A(,26:>*.!%)-,0#'J J #d*J )D0J !+d2 J !& 04 7 J uU\23d: J J (4/J  %047J  4'J $;J  $;J #TkJ D[J u !O 3!J J J ^J Z)EL))))))))r0r ceZdZUdZdZeed<ddiZdeffd ZdZ dZ e dd e j d zd e jd zd e jd zd ed zded zded zdee jezfdZxZS)r!a Examples: ```python >>> from transformers import MT5EncoderModel, AutoTokenizer >>> model = MT5EncoderModel.from_pretrained("google/mt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rhrGrlrjct|tj|j|j|_|}d|_d|_t||_ | dS)NF) r$r%rrrDrMr$rr\r=rrrJ)r+rGrvr.s r/r%zMT5EncoderModel.__init__so    l6#4fnEE #( ,1)//  r0c|jSrXryrzs r/r{z$MT5EncoderModel.get_input_embeddingsr|r0cH||_|j|dSrX)r$rrrNrLs r/rNz$MT5EncoderModel.set_input_embeddingss%$  )).99999r0NrrrRrrZrrc \||n |jj}|||||||}|S)aJ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./mt5#training). Example: ```python >>> from transformers import AutoTokenizer, MT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> model = MT5EncoderModel.from_pretrained("google/mt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr)rGrrr) r+rrrRrrZrrrs r/r?zMT5EncoderModel.forwardsJF&1% 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrrrRrrrrZrrrz7All examples must have the same number of tokens.z3Each example must contain at least one token.r3 regressionsingle_label_classificationmulti_label_classificationr)%rGrNotImplementedErrorr.r@r/r3req eos_token_idr5rrr'unique_consecutivesumnumelrrr problem_typerr:rrrsqueezerrrrrrrYrrr)r+rrrrrrRrrrrrZrrrreos_maskr__r,selectedsentence_representationrrrrs r/r?z$MT5ForSequenceClassification.forwardsb&1%Kbdd   $)>)F  U !% 1 1) < < "" )/#9+'"7/!5##   "!*<< 899<<_=STT  $X\\!__ 5 5 ; ; = = B E   &5%:" A{"8QQQ;/ N1  +q 0 A   #+-- B "L"LQQQPRTUTUTUX"V))*ABB  YYv}--F{'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x// FY,F)-)9TGf$$v E.#3")"?&9$5&-&G")"?&9    r0r)r@rArBrrr%rr'rrZlistrr@r]rr?rCrDs@r/rrs+s)s&y.2.259:>:>26:>*.!%)-,0#'F F #d*F  t+F !+d2 F !& 04 7 F e/047 F (4/F  %047F  4'F $;F  $;F #TkF D[F  0 0F F F ^F F F F F r0rceZdZdeffd Ze d dejdzdejdzdejdzdejdzdedzd edzd edzd e eje zfd Z xZ S)r(rGc6t||j|_t||_t j|j|_t j |j |j|_ | dSrX) r$r%rr!rrrQrrSrLr,rrJrVs r/r%z"MT5ForTokenClassification.__init__sz     +*622z&";<< )F$68IJJ r0NrrrRrrrZrrc ||n |jj}|||||||} | d} || } || } d} |Ft } | | d|j|d} |s| | ddf}| | f|zn|St| | | j | j S)a> input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./t5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrRrrZrrr3r2)rrr=rX) rGrrrSrrrrrr=rX)r+rrrRrrrZrrrr=rrrrs r/r?z!MT5ForTokenClassification.forwards 6&1%)F  U !% 1 1) < < !*!6IIDK=A371526:>!%)-,0#'I I #d*I )D0I !+d2 I !& 04 7 I uU\23d: I )D0I '$.I (4/I  %047I $;I  $;I #TkI D[I u !$G G!I I I ^I I I I I r0r")r!r r"rr(rr )Ar rprr'rtorch.nnrrrrQrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrconfiguration_mt5r get_loggerr@r}Moduler!rFr_rgrqrrrrr r=rr r!rr(r"__all__rSr0r/rso AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999.-----^^^^^^^^^^^^^^((((((  H % %+++++29+++4ry.BI<&29FBIDRY@W W W W W )W W W vBI$_!_!_!_!_!_!_!_!Fo o o o o !o o o di i i i i !i i i X G)G)G)G)G)"4oG)G)  G)TYYYYY(YYYx T T T T T #5T T  T nE E E E E 2E E E Ps s s s s 0s s s l   r0