\%"S)S*\=55r?\%"S+S,9"S-S.\=\55r@\%"S/S0\=55rA\%"S1S,9"S2S3\=55rB\%"S4S5\=55rC\%"S6S7\=55rD/S8QrEg)9zPyTorch UMT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilinglogging) UMT5Config) BlockMask)make_flex_block_causal_maskc2^\rSrSrSU4SjjrSrSrU=r$) UMT5LayerNorm=c>[TU]5 [R"[R "U55UlX lg)zU Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s ^/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/umt5/modeling_umt5.pyr)UMT5LayerNorm.__init__>s/ ll5::k#:; #cUR[R5RS5R SSS9nU[R "X R -5-nURR[R[R4;a%URURR5nURU-$)NT)keepdim) tor+float32powmeanrsqrtr.r-dtypefloat16bfloat16)r/ hidden_statesvariances r3forwardUMT5LayerNorm.forwardFs !##EMM266q9>>r4>P% H?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r5)r.r-)gư>)__name__ __module__ __qualname____firstlineno__r)rD__static_attributes__ __classcell__r2s@r3r%r%=s$ + +r5r%c6^\rSrSrS\4U4SjjrSrSrU=r$)UMT5DenseActDenseWconfigcX>[TU]5 [R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUl gNFbias)r(r)rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr/rPr2s r3r)UMT5DenseActDense.__init__Xsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r5cURU5nURU5nURU5n[URR [ R5(aURURR R:waaURR R[ R:wa/URURR R5nUR U5nU$N) rXr^r\ isinstancerYr-r+Tensorr?int8r:r/rBs r3rDUMT5DenseActDense.forward_s . /  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r5)r^r\rXrY rFrGrHrIr!r)rDrJrKrLs@r3rNrNWs/z/  r5rNc6^\rSrSrS\4U4SjjrSrSrU=r$)UMT5DenseGatedActDensenrPc>[TU]5 [R"URUR SS9Ul[R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUlgrR)r(r)rrUrVrWwi_0wi_1rYrZr[r\r r]r^r_s r3r)UMT5DenseGatedActDense.__init__os IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r5c8URURU55nURU5nX#-nURU5n[ UR R [R5(aURUR R R:waaUR R R[R:wa/URUR R R5nUR U5nU$rb) r^rmrnr\rcrYr-r+rdr?rer:)r/rB hidden_gelu hidden_linears r3rDUMT5DenseGatedActDense.forwardwshhtyy78  -0 #3  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r5)r^r\rmrnrYrhrLs@r3rjrjns/z/r5rjc6^\rSrSrS\4U4SjjrSrSrU=r$) UMT5LayerFFrPc>[TU]5 UR(a[U5UlO[ U5Ul[ URURS9Ul [R"UR5Ul g)Nr1)r(r) is_gated_actrjDenseReluDenserNr%rVlayer_norm_epsilon layer_normrrZr[r\r_s r3r)UMT5LayerFF.__init__s_    "8"@D "3F";D 'F[TU]5 URUlX lURUlUR UlUR UlURUlURUl URUl URUR-Ul X0lUc>UR(a-[R!SUR"R$S35 [&R("UR URSS9Ul[&R("UR URSS9Ul[&R("UR URSS9Ul[&R("URUR SS9UlUR(a0[&R2"URUR5Ul[75Ulg)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrS)r(r) is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerVd_kvkey_value_proj_dim num_headsn_headsr[r\ inner_dimrlogger warning_oncer2rFrrUqkvo Embeddingrelative_attention_biasset pruned_heads)r/rPrrr2s r3r)UMT5Attention.__init__se  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,, 4<<eD4<<eD4<<eD4>>4<FFq!QPQRr5cSnURnURnUR(dCUS-nX!S:R[R 5U-- n[R "U5nO,[R"U[R"U55*nUS-nX:n[R"UR5U- 5[R"XE- 5- nXsU- -nXWR[R 5-n[R"U[R"XS- 55nU[R"XaU5- nU$)aR Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr7r )rrrr:r+longabsmin zeros_likelogfloatmath full_likewhere) r/relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r3_relative_position_bucket'UMT5Attention._relative_position_buckets/*99 ;;  A K Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $0II/557)CDtxxP\PhGii y!89 %.ejj1I%I"%*YY &8RbcTc(d& " EKKE_``r5cUc URRRnUc,[R"U[R US9SS2S4nO USS2S4n[R"U[R US9SSS24nXe- nUR U5nURU5n U R/SQ5RS5n U $)z%Compute binned relative position biasN)r?device)r7rr r) rr-rr+arangerrr unsqueeze) r/ query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluess r3 compute_biasUMT5Attention.compute_biass >1188??F  !$||L SYZ[\^b[bc -ag6 ,,zFSTXZ[T[\+>#'#A#ABS#T --.FG *44Q7 r5rBencoder_hidden_statespast_key_valueattention_masklayer_head_maskrcXURSSupxUSLn URU5n U RUSURUR5R SS5n Ub[[ U[5(aFURRUR5n U (a URn OURn OUn U (aUOUn U (aQUbNW (aGU RURRnU RURRnOUR!U 5nUR#U 5nURUSURUR5R SS5nURUSURUR5R SS5nUbLU (dUOSnU R%XURSU05upU (aSURUR'[&R("XR SS55nUbXR+5-OUnURSnUR,(d9[&R."SURUU4UR0UR2S9nO.UR5UUUR0US 9nUSS2SS2U*S2SS24nUb#USS2SS2SS2SURS24nUU-nUR6(aS[&R8"URS5nS U[;UR65'USS2UR=54nOUnUU- n[>R@RCURE5SS 9RGU5n[>R@RIUURHURJS 9nUbUU-n[&R("UU5nUR SS5RM5nURXxS5nUROU5nUU4$) Nr7r8r rTr )rr?)rrrdim)ptraining)(shaperrrr transposercr is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysrrrupdater+matmulget_seq_lengthrzerosrr?rrr,listboolr functionalsoftmaxrtype_asr\r contiguousr)r/rBrrrrr batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuecurrent_states key_states value_statesscoresreal_seq_lengthr position_bias causal_maskmaskposition_bias_masked attn_weights attn_outputs r3rDUMT5Attention.forwards"/!4!4Ra!8 3$>vvm, #((RtG^G^_iijkmno   %*^EX*Y*Y'2266t~~FJ!&4&J&J#&4&I&I#"0 2D.- ."<,33DNNCHHJ.55dnnELLL/J66.1L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL)7It+>+E+Ednn?OQ_>`,( &@DN--dnn=l,@,@A,FGKYJd*'D'D'FFjt%%b) //!KKDLL*j9&--W]WcWcM!--FMMR`.M*!Q a*?@M  %(Aq2HJ4D4DR4H2H)HIK)K7M   ::m11!45D,-Dd''( )#0DIIK#@ #0 &&}},,V\\^,DLLVT }},,\T\\TXTaTa,b   &'/9Lll<> !++Aq1<<> !&&zrB ff[) L((r5)rVr\rrrrrrrrrrrrrr)FN)NNNNNNN)rFrGrHrI__doc__rintr)r+rdrrrtuplerDrJrKrLs@r3rrs"XVY]"":%,, - ^ $9=8<152615\)||\) ( 5\)!u||!45 \) !. \) "%,,/ \)!.\)\)r5rcL^\rSrSrSS\\4U4SjjjrSSjrSrU=r $)UMT5LayerSelfAttentionicrc>[TU]5 [USUS9Ul[ UR UR S9Ul[R"UR5Ul g)NTrrrx) r(r)r SelfAttentionr%rVr{r|rrZr[r\r/rPrr2s r3r)UMT5LayerSelfAttention.__init__dsN *6t_hi'F[TU]5 [USUS9Ul[ UR UR S9Ul[R"UR5Ul g)NFrrx) r(r)rEncDecAttentionr%rVr{r|rrZr[r\rs r3r) UMT5LayerCrossAttention.__init__sO ,VQVbkl'F[TU]5 URUl[R"5UlUR R [XS95 UR(a"UR R [XS95 UR R [U55 g)Nr) r(r)rr ModuleListlayerappendrrrurs r3r)UMT5Block.__init__sv  ++]]_  0MN ?? JJ  5fR S +f-.r5c URS"UUUUU S9upUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nSnUR=(a USLnU(aURS"UUUUUU S9upUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nURS"U5nUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nU4nU (aUX4- nU$)Nrri)rmaxr r r8) rr?r+r@finforrisinfanyclampr)r/rBrrencoder_attention_maskrcross_attn_layer_head_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrs r3rDUMT5Block.forwards,0::a= )+)) , (    %-- / M$7$78< >Gr5)rrrb) NNNNNNFFNrrLs@r3rrs?/(3-//"##';;r5rcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) UMT5ClassificationHeadiz-Head for sentence-level classification tasks.rPc(>[TU]5 [R"URUR5Ul[R "URS9Ul[R"URUR5Ul g)N)r) r(r)rrUrVdenserZclassifier_dropoutr\ num_labelsout_projr_s r3r)UMT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r5rBrcURU5nURU5n[R"U5nURU5nUR U5nU$rb)r\r(r+tanhr+rfs r3rDUMT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r5)r(r\r+) rFrGrHrIrr!r)r+rdrDrJrKrLs@r3r&r&s47EzE U\\ellr5r&cT\rSrSr%\\S'SrSrSrS/r S/r \ S5r Sr S rS rg ) UMT5PreTrainedModelirP transformerTrrYcz[R"[5n[R"[5nUUUS.nU$)N)decoder_input_ids input_idsdecoder_attention_mask)r+tensorrr)r/r5 input_mask dummy_inputss r3r9 UMT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r5cURRn[U[5(a)URR R US-5 g [U[[[[45(GaURRR RSUS-S9 [US5(aLURR(d1URRR RSUS-S9 [US5(avUR RR RSX RR"S--S9 UR R$R R'5 g g [U[(5(ar[US5(a`UR*RR RSUS-S9 UR*R$R R'5 g g [U[,5(GaQUR.RR RSX RR"S--S9 [UR.S5(aEUR.R$b.UR.R$R R'5 UR0RR RSX RR"S--S9 [UR0S5(aGUR0R$b/UR0R$R R'5 g g g [U[25(GaQUR4RR RSX RR"S--S9 [UR4S5(aEUR4R$b.UR4R$R R'5 UR6RR RSX RR8S--S9 [UR6S5(aGUR6R$b/UR6R$R R'5 g g g [U[:5(GaUR<RR RSX RR"S--S9 [UR<S5(aEUR<R$b.UR<R$R R'5 UR>RR RSX RR"S--S9 [UR>S5(aEUR>R$b.UR>R$R R'5 UR6RR RSX RR8S--S9 [UR6S5(aGUR6R$b/UR6R$R R'5 g g g [U[@5(GaZURR"nURRBnURRDnURFRR RSX#U-S--S9 URHRR RSX#S--S9 URJRR RSX#S--S9 URLRR RSX%U-S--S9 URN(a4URPRR RSX#S--S9 g g g ) zInitialize the weights?)r=stdlm_head qa_outputs classifierrTN))rPinitializer_factorrcr%r-datafill_ UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringsharednormal_hasattrtie_word_embeddingsr?r@rVrTzero_UMT5ForTokenClassificationrBr&r(r+rNrXrYrWrjrmrnrrrrrrrrr)r/modulefactorrVrrs r3 _init_weights!UMT5PreTrainedModel._init_weightss// fm , , MM   $ $Vc\ 2  , (     MM % % - -3FSL - Ivy))$++2Q2Q%%**22#2Nv|,,!!((--553F{{ObObgkNkDl5m!!&&++113- : ; ;v|,,!!((--553FSL5Q!!&&++113- 6 7 7 LL   $ $ , ,#6kkFYFY^bEb;c , dv||V,,1B1B1N !!&&,,. OO " " ' ' / /SfI\I\aeHe>f / gv//FOO4H4H4T$$))//15U/  1 2 2 II   ! ! ) )s;;CVCV[_B_8` ) avyy&))fiinn.H ##))+ II   ! ! ) )s;;CSCSX\B\8] ) ^vyy&))fiinn.H ##))+/I)  6 7 7 KK   # # + +&[[EXEX]aDa:b + cv{{F++ 0@0@0L   %%++- KK   # # + +&[[EXEX]aDa:b + cv{{F++ 0@0@0L   %%++- II   ! ! ) )s;;CSCSX\B\8] ) ^vyy&))fiinn.H ##))+/I)  . .kk))G!%!1!1 kk++G HHOO ( (cvL^B^cgAg7h ( i HHOO ( (cv$7O ( P HHOO ( (cv$7O ( P HHOO ( (cvL^B^cgAg7h ( i11..55::BBQWhl[lQmBn2/r5cURRnURRnUc [S5e[ U5(aE[ R "URSSS-U5n[ R"XASSS24/SS9nO=URUR5nUSSS24R5USSS24'X$S'Uc [S5eURUS :HU5 U$) Nzself.model.config.decoder_start_token_id has to be defined. In UMT5 it is usually set to the pad_token_id. See UMT5 docs for more information.r8)r .rr ).rz1self.model.config.pad_token_id has to be defined.) rPdecoder_start_token_id pad_token_id ValueErrorrr+fullrcat new_zerosclone masked_fill_)r/r5rVrWshifted_input_idss r3 _shift_right UMT5PreTrainedModel._shift_rightPs!%!C!C{{// ! )6  Y ' ' % 9??3B+?$+FH^ _  % +<SbS>Q*RXZ [  ) 3 3IOO D )238)<)B)B)D c12g &(>f %  PQ Q&&'8D'@,O  r5N)rFrGrHrIr!__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyr9rRr_rJrar5r3r1r1sI %&*#!$ !F @oD!r5r1c (^\rSrSrSU4SjjrSrSSjrSS\\RS4S\RS\RS \ S \ 4 S jjr \ S\RS \S \S\RS\RS\4 Sj5rSrU=r$) UMT5Stackilc >[TU]U5 X lURUl[R "[ UR5Vs/sH n[XS9PM sn5Ul [URURS9Ul [R"UR5UlSUlUR%5 gs snf)NrrxF)r(r) embed_tokensrrrrange num_layersrblockr%rVr{final_layer_normrZr[r\gradient_checkpointing post_init)r/rPrlir2s r3r)UMT5Stack.__init__ms  ( ++]]ERXRcRcLd#eLdqIf$BLd#ef -fnn&B[B[ \zz&"5"56 ',#  $fsC cXlgrb)rlr/new_embeddingss r3set_input_embeddingsUMT5Stack.set_input_embeddingsys*r5c U bU OURRn U bU OURRn U bU OURRn U bU OURRn Ub*Ub'UR (aSOSn[ SUSUS35eUb&UR5nURSUS5nO>UbUR5SSnO'UR (aSOSn[ SUSUS 35eUR(a/UR(aU (a[RS 5 S n Uc)URc [ S 5eURU5nUunnU S La UR (d[ SUS35eUR (aMU (aEUcBURR(a[[!5[!55nO[!5nOUR (dSnUbUR#5OSnU c#[$R&"UUU-UR(S9n Uc4[+5(d%UU-n[$R,"UUUR(S9nUR (a7UR/UUU [1U[5(a UR2OUU 5nO\UbWUSS2SSSS24nUR5UR6S9nSU- [$R8"UR65R:-nOSnUR (aOUbLUR5unnnUU4nUc[$R,"UUR(S9nUR=U5nOSnUR?X`RR@5nUR?XpRR@5nU (aSOSnU (aSOSnU (aUR (aSOSnURCU5n[EURF5H_unnUUn UUn!U (aUU4-nU"UUUUU U!UU U U S9 n"U"SnU (dM:UU"S4- nUR (dMVUU"S4- nMa URIU5nURCU5nU (aUU4-nU (d[KSUUUUU455$[MUUUUUS9$)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer8zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz$UMT5Stack.forward..s" As )last_hidden_statepast_key_valuesrB attentionscross_attentions)'rPrroutput_hidden_statesuse_return_dictrrXrrrqrrrrlis_encoder_decoderr r rr+rrrr,_update_causal_maskrcrr:r?rrinvert_attention_mask get_head_maskrnr\ enumeraterorprr)#r/r5rrrr} head_maskcross_attn_head_maskrrrr return_dictrerr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_lengthrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrBrs layer_modulerr layer_outputss# r3rDUMT5Stack.forward|s "+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii ??_4;;11&9,.,.&YO&2nO#OETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D*D4zAO"ZZ OML`L`aN ??22o/BCC 44$!K '(D$)9:K%..}/B/B.CK, MT]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr5rrr?rc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer?rr )diagonalr~r8r)rr+rrrYrtriurreshapeexpandr\rr: masked_fill) rrrr?rrkwargsrr mask_length padding_masks r3r?UMT5Stack._prepare_4d_causal_attention_mask_with_cache_positionjs}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r5)ror\rlrprqrrb) NNNNNNNNNNNNN)F)rFrGrHrIr)rxrDrr+rdr rr staticmethodrr?rrJrKrLs@r3rjrjls + "#!!g `#( BellK78BllB B  B  BH4 444{{ 4  4  44r5rjc&P^\rSrSr%SrSr\\S'SS/rU4Sjr Sr S r S r S r S rS r\S!S\\R&S\\R(S\\R&S\\R*S\\R(S\\R(S\\R,S\\\\R(S\\S\\R,S\\R,S\\S\\S\\S\\S\\R&S\\\R(\44"Sjj5rS rU=r$)"rFia? Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rPencoder.embed_tokens.weightdecoder.embed_tokens.weightc>[TU]U5 [R"URUR 5Ul[R"U5nSUl SUl SUl [X R 5Ul [R"U5nSUl SUl URUl[X0R 5UlUR#5 gNFT)r(r)rr vocab_sizerVrJcopydeepcopyrrtie_encoder_decoderrjencodernum_decoder_layersrndecoderrrr/rPencoder_configdecoder_configr2s r3r)UMT5Model.__init__s  ll6#4#4fnnE v.$)!#( -2* = v.$(!-2*$*$=$=! =  r5cUR$rbrJr/s r3get_input_embeddingsUMT5Model.get_input_embeddings {{r5c|XlURRU5 URRU5 grbrJrrxrrvs r3rxUMT5Model.set_input_embeddings+$  )).9 )).9r5cURR(aaURURRUR 5 URUR RUR 5 ggrbrPrM_tie_or_clone_weightsrrlrJrrs r3 _tie_weightsUMT5Model._tie_weightsP ;; * *  & &t||'@'@$++ N  & &t||'@'@$++ N +r5cUR$rbrrs r3 get_encoderUMT5Model.get_encoder ||r5cUR$rbrrs r3 get_decoderUMT5Model.get_decoderrr5cUR5H7up#URRURR U5 M9 g) Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr attention prune_headsr/heads_to_prunerheadss r3 _prune_headsUMT5Model._prune_headss< +002LE LL  u % / / ; ;E B3r5r5rr4r6rdecoder_head_maskrencoder_outputsrr}decoder_inputs_embedsrrrrrrc\U bU OURRn UbUOURRnUcURUUU UU UUS9nORU(aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnURUUU U UUUUU U UUUS9 nU(dUU-$[URURURURURURURURS9$) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr5rr}rrrrrr r7rrBr r5rr}rrrrrrrrrr)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rPrrrrcrlenrrrrrBrr)r/r5rr4r6rrrrrr}rrrrrrrBdecoder_outputss r3rDUMT5Model.forwards^b"+!6IDKKD $D>D 'tnD d^!D "!!1!12#D $ uU&&');; <%D D r5rFz< UMT5 Model with a `language modeling` head on top. ) custom_introc(^\rSrSrSrSr/SQrU4SjrSrSr Sr S r S r \ S S \\R S \\R"S \\R S\\R$S\\R"S\\R"S\\R&S\\\\R&S\\S\\R"S\\R"S\\R S\\S\\S\\S\\S\\R S\\\R"\44$Sjj5rS\R&4SjrSrU=r$)!rGiva Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```r)rrzlm_head.weightcL>[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl SUl [X R5Ul[R"U5nSUl SUl URUl[X0R5Ul[R$"URUR SS9UlUR)5 g)NFTrS)r(r)rV model_dimrrrrJrrrrrrjrrrnrrUr?rrrs r3r)%UMT5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( -2* = v.$(!-2*$*$=$=! = yy1B1BO  r5cUR$rbrrs r3r1UMT5ForConditionalGeneration.get_input_embeddingsrr5c|XlURRU5 URRU5 grbrrvs r3rx1UMT5ForConditionalGeneration.set_input_embeddingsrr5cURR(aaURURRUR 5 URUR RUR 5 ggrbrrs r3r)UMT5ForConditionalGeneration._tie_weightsrr5cUR$rbrrs r3r(UMT5ForConditionalGeneration.get_encoderrr5cUR$rbrrs r3r(UMT5ForConditionalGeneration.get_decoderrr5r5rr4r6rrrrrr}rlabelsrrrrrrcU bU OURRn UbUOURRnUcURUUU UUUUS9nORU(aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnU bUcU cURU 5nURUUU U UUUUU UUUUS9 nUSnURR(aUURS--nURU5nSnU b[[S S 9nU RUR5n U"URS UR!S 55U RS 55nU(dU4USS-U-nUbU4U-$U$[#UUUR$UR&UR(UR*UR,UR&UR(S 9 $) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrr r7rrrArU ignore_indexr8 losslogitsrrrrrrr)rPrrrrcrrr_rrMrr?rr:rrrrrrBrrr)r/r5rr4r6rrrrrr}rrrrrrrrBrsequence_output lm_logitsrloss_fctoutputs r3rD$UMT5ForConditionalGeneration.forwards.j"+!6IDKK_ $D>_ 'tn!_ "d^#_ $!!1!12%_ & uU&&'8 9'_ _ D)ELL))r5rGc8^\rSrSrSrSrS/rU4SjrSrSr Sr S r S r \ SS \\R S \\R"S \\R"S\\R"S\\S\\S\\S\\\R"\44Sjj5rSrU=r$)rHiea Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrc >[TU]U5 [R"URUR 5Ul[R"U5nSUl SUl [X R 5Ul UR5 gNF)r(r)rrrrVrJrrrrrjrrr)r/rPrr2s r3r)UMT5EncoderModel.__init__ysf  ll6#4#4fnnE v.#( ,1) =  r5cUR$rbrrs r3r%UMT5EncoderModel.get_input_embeddingsrr5cFXlURRU5 grb)rJrrxrvs r3rx%UMT5EncoderModel.set_input_embeddingss$  )).9r5cURR(a1URURRUR 5 ggrb)rPrMrrrlrJrs r3rUMT5EncoderModel._tie_weightss2 ;; * *  & &t||'@'@$++ N +r5cUR$rbrrs r3rUMT5EncoderModel.get_encoderrr5cUR5HDup#URRURSRR U5 MF g)rrN)rrrorrrrs r3rUMT5EncoderModel._prune_headssG +002LE LL  u % + +A . < < H H O3r5r5rrr}rrrrc fUbUOURRnURUUUUUUUS9nU$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rPrr) r/r5rrr}rrrrs r3rDUMT5EncoderModel.forwardsKF&1% -'tn-d^- uU&&'8 9--r5rHz UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c$^\rSrSrS/rSS/rS\4U4Sjjr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ \ RS\ \ RS\ \ RS\ \ RS\ \S\ \S\ \S\ \S\\\44 Sjj5rSrU=r$)UMT5ForSequenceClassificationiFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrrrPc>[TU]U5 [U5Ul[ U5UlUR 5 SUlgr+)r(r)rFr2r&classification_headrrmodel_parallelr_s r3r)&UMT5ForSequenceClassification.__init__s>  $V,#9&#A  #r5r5rr4r6rrrrr}rrrrrrrc4UbUOURRnU bSn Uc%U b"[SURR35eUc"U cUc [ S5eUR U5nURUUUUUUUUU U U U UUS9nUSnURURR5RUR5n[[R"URS555S:a [ S5eUR unnnUUSS24R#US U5SS2S SS24nUR%U5nSnU GbU RUR5n URR&cURR(S:XaS URlOyURR(S:aNU R*[R,:XdU R*[R.:XaS URlOS URlURR&S :XaT[15nURR(S:Xa&U"UR35U R355nOU"UU 5nOURR&S :XaG[55nU"UR#S URR(5U R#S 55nO-URR&S :Xa[75nU"UU 5nU(dU4USS-nUbU4U-$U$[9UUUR:UR<UR>UR@URBURDURFS 9 $)ak input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rr4r6rrrrr}rrrrrrr z7All examples must have the same number of tokens.r8 regressionsingle_label_classificationmulti_label_classificationr)$rPrNotImplementedErrorr2rFrXr_r2eq eos_token_idr:rrr+unique_consecutivesumrrr= problem_typer*r?rrrsqueezerrrrrrrrrr)r/r5rr4r6rrrrr}rrrrrrrr!eos_maskrrr0sentence_representationr rr#r$s r3rD%UMT5ForSequenceClassification.forwardsR|&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9/!5+'"7/!5##  "!*<< 8 89<<_=S=ST u'' Q8 9A =VW W%4%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r5)r=r>r2)NNNNNNNNNNNNNNN)rFrGrHrI"_keys_to_ignore_on_load_unexpectedrr!r)rrr+rrdrr rrrrrDrJrKrLs@r3r:r:s+s)s&79VW$z$15158<=A,0487;=A59=A-1$(,0/3&*!P E,,-P !.P $E$4$45 P !))9)9 : P ELL) P $ELL1P 'u||4P "$u'8'8"9:P   1 12P  ((9(9:P ))*P D>P $D>P 'tnP d^!P " u55 6#P P r5r:c@^\rSrSrS/rS/rS\4U4Sjjr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ S \ \ S \ \ S\ \\ R\44Sjj5rSrU=r$)rOi{r;z'transformer.encoder.embed_tokens.weightrPc0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 grb) r(r)r*rHr2rrZr)r\rUr0rBrrr_s r3r)#UMT5ForTokenClassification.__init__sj   +++F3zz&";";< ))F$6$68I8IJ r5r5rrr}rrrrrc UbUOURRnURUUUUUUUS9n U Sn URU 5n UR U 5n Sn Ub<[ 5n U "U R SUR5UR S55n U(dXSS4nU bU 4U-$U$[U U U RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrr}rrrrr8r7)rr rBr) rPrr2r\rBrrr*rrBr)r/r5rrr}rrrrrrBr rr#r$s r3rD"UMT5ForTokenClassification.forwards6&1%7 'tn7 d^7  uU\\"$99 :7 7 r5rOc&J^\rSrSrSS/rU4SjrSrSrSrSr S r \ SS \ \ RS \ \ RS \ \ RS \ \ R S\ \ RS\ \ RS\ \ R"S\ \\\ R"S\ \ RS\ \ RS\ \ RS\ \ RS\ \S\ \S\ \S\ \S\\\ R\44"Sjj5rSrU=r$)rIirrcp>[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl SUl [X R5Ul[R"U5nSUl SUl URUl[X0R5UlUR$Ul[R&"URUR$5UlUR+5 gr)r(r)rVrrrrrJrrrrrrjrrrnrr*rUr@rrrs r3r)!UMT5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#( -2* = v.$(!-2*$*$=$=! =  ++))FNNF4E4EF r5cUR$rbrrs r3r-UMT5ForQuestionAnswering.get_input_embeddingsrr5c|XlURRU5 URRU5 grbrrvs r3rx-UMT5ForQuestionAnswering.set_input_embeddingsrr5cURR(aaURURRUR 5 URUR RUR 5 ggrbrrs r3r%UMT5ForQuestionAnswering._tie_weightsrr5cUR$rbrrs r3r$UMT5ForQuestionAnswering.get_encoderrr5cUR$rbrrs r3r$UMT5ForQuestionAnswering.get_decoderrr5r5rr4r6rrrrstart_positions end_positionsr}rrrrrrcUbUOURRnU bU OURRn U bU bSn Uc"U cUc [S5eUR U5nU bU OURRn UbUOURRnUcUR UUU UUUUS9nORU(aK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nUSnURUUU SUUUUU UUUS 9 nUSnURU5nURSS S 9unnURS 5R5nURS 5R5nSnU bU b[U R55S:a*U RS 5RUR 5n [U R55S:a*U RS 5RUR 5n URS5nU R#SU5n U R#SU5n [%US 9nU"UU 5nU"UU 5nUU-S- nU(dUU4USS-U-nUbU4U-$U$['UUUUR(UR*UR,UR.UR0UR*UR,S 9 $)aY input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NFrArrr r7r) r5rr}rrrrrrrrrr8rr) r start_logits end_logitsrrrrrrr)rPrrrXr_rrcrrrr@splitrKrrr:rrrrrrBrrr)r/r5rr4r6rrrrrbrcr}rrrrrrBrr!r rerf total_loss ignored_indexr# start_lossend_lossr$s r3rD UMT5ForQuestionAnswering.forwards<x&1%)F  U !% 1 1) < !*!6IDKKZ $D>Z 'tn!Z "d^#Z $ uU&&')LL M%Z Z r5rI)rHrGrIr:rOrFr1)Frrrtypingrrr+rtorch.nnrrr activationsr cache_utilsr r r generationrmodeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrrconfiguration_umt5r!!torch.nn.attention.flex_attentionr"integrations.flex_attentionr# get_loggerrFrModuler%rNrjrurrrrr&r1rjrFrGrHr:rOrI__all__rar5r3r}s " AA!CC)>9.+ !!;J   H %+BII+4 .RYY<"))$B)BIIB)JRYY8bii