tjO!dZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+e(j,e-Z.Gddej/Z0Gddej/Z1Gddej/Z2Gddej/Z3Gddej/Z4Gddej/Z5Gddej/Z6Gdd eZ7Gd!d"ej/Z8e&Gd#d$e"Z9Gd%d&e9Z:e&Gd'd(e9Z;e&d)*Gd+d,e9eZe&Gd2d3e9Z?e&Gd4d5e9Z@gd6ZAdS)7zPyTorch UMT5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torchdynamo_compilingloggingtorch_compilable_check) UMT5Configc&eZdZdfd ZdZxZS) UMT5LayerNormư>cttjt j||_||_dS)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/umt5/modeling_umt5.pyr&zUMT5LayerNorm.__init__7sD l5:k#:#:;; #ch|tjddd}|tj||jzz}|jjtj tj fvr||jj}|j|zS)NT)keepdim) tor(float32powmeanrsqrtr+r*dtypefloat16bfloat16)r, hidden_statesvariances r0forwardzUMT5LayerNorm.forward?s !##EM2266q99>>r4>PP% Ht?T4T(U(UU  ;  ? ? ?),,T[->??M{]**r1)r#)__name__ __module__ __qualname__r&r@ __classcell__r/s@r0r"r"6sL$$$$$$ + + + + + + +r1r"c*eZdZdeffd ZdZxZS)UMT5DenseActDenseconfigcJttj|j|jd|_tj|j|jd|_tj|j |_ t|j |_ dSNFbias)r%r&rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr,rHr/s r0r&zUMT5DenseActDense.__init__Qsx )FNFKeDDD)FKeDDDz&"566 &-.r1c||}||}||}t|jjt jr]|j|jjjkrC|jjjt j kr$| |jjj}||}|SN) rPrVrT isinstancerQr*r(Tensorr;int8r6r,r>s r0r@zUMT5DenseActDense.forwardXs .. //  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r1rArBrCr r&r@rDrEs@r0rGrGPsS/z//////       r1rGc*eZdZdeffd ZdZxZS)UMT5DenseGatedActDenserHcttj|j|jd|_tj|j|jd|_tj|j|jd|_tj |j |_ t|j |_dSrJ)r%r&rrMrNrOwi_0wi_1rQrRrSrTr rUrVrWs r0r&zUMT5DenseGatedActDense.__init__hs IfnfkFFF IfnfkFFF )FKeDDDz&"566 &-.r1c|||}||}||z}||}t |jjtjr]|j |jjj krC|jjj tj kr$| |jjj }||}|SrY) rVrbrcrTrZrQr*r(r[r;r\r6)r,r> hidden_gelu hidden_linears r0r@zUMT5DenseGatedActDense.forwardpshhtyy7788  -00 #m3  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r1r^rEs@r0r`r`gsS/z//////r1r`c*eZdZdeffd ZdZxZS) UMT5LayerFFrHc$t|jrt||_nt ||_t |j|j|_ tj |j |_ dS)Nr.)r%r& is_gated_actr`DenseReluDenserGr"rNlayer_norm_epsilon layer_normrrRrSrTrWs r0r&zUMT5LayerFF.__init__sx    <"8"@"@D  "3F";";D 'Fforwarded_statess r0r@zUMT5LayerFF.forwardsF??=99../?@@% 5E(F(FF r1r^rEs@r0rhrhsS7z777777r1rhc eZdZdZddedzffd ZdejdejfdZd Z dd Z dd ejd ejdzde dzdejdzfdZ xZ S) UMT5Attentionz7 T5's attention using relative_attention_bias. FN layer_idxc t|j|_||_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |jz|_ ||_|/|jr(td|jjdt'j|j|j d|_t'j|j|j d|_t'j|j|j d|_t'j|j |jd|_|jr&t'j|j|j |_dSdS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrK)r%r& is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerNd_kvkey_value_proj_dim num_headsn_headsrSrT inner_dimrslogger warning_oncer/rArrMqkvo Embeddingrelative_attention_bias)r,rHrvrsr/s r0r&zUMT5Attention.__init__sk  ++F(.4.S+/5/U,~ "(+' * (??"      ,4>+B,,,    4<eDDD4<eDDD4<eDDD4>4>FFq!QPQRRr1c~d}|j}|j}|jsC|dz}||dktj|zz }t j|}n(t j|t j| }|dz}||k}t j | |z tj ||z z }|||z z}||tjz}t j|t j ||dz }|t j |||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr3r)rwrxrur6r(longabsmin zeros_likelogfloatmath full_likewhere) r,relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r0_relative_position_bucketz'UMT5Attention._relative_position_bucketsP*9 ;  c A K !2Q!6 : :5: F F T T  % *; < <  !&+NO`>a>a!b!b b  1$ $y0I/5577)CDDtxP\_hPhGiGii y!89 %.ej1I1I%I"%*Y &8RT_bcTc(d(d& & " EK2CE_```r1rc||jjj}tj|tj|dddf|z}tj|tj|dddf}||z }||}||} | gdd} | S)z%Compute binned relative position biasN)r;device)r3rrr) rr*rr(arangerrr unsqueeze) r, query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess r0 compute_biaszUMT5Attention.compute_biass >18?F < EJvVVVWXWXWXZ^W^_brr,zFSSSTXZ[Z[Z[T[\+.>>#'#A#ABS#T#T --.FGG **44Q77 r1r>encoder_hidden_statespast_key_valuesattention_maskc |jdd\}}|||jnd}t|tjr|n|}|du} ||} | |d|j |j  dd} d} |Ft|tr1|j |j} | r|j} n |j} n|} | r|n|} | r3|1| r/| j|jj}| j|jj}n|| }|| }||d|j |j  dd}||d|j |j  dd}|E| |||j\}}| r$t|trd|j |j<t j| | dd}|jd}|js+t jd|j ||f|j|j }n||||j| }|||z}|}||z }t:j| d !|}t:j"||j"|j# }t j||}| dd$}|||d}|%|}||fS) Nr3rr4rFTr)rr;)rrdim)ptraining)&shapeget_seq_lengthrsrZr(r[clonerrr|rz transposer is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdatematmulrvzerosrr;rr functionalsoftmaxrtype_asrTr contiguousr)r,r>rrrkwargs batch_size seq_lengthris_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_statesscoresr position_biasposition_bias_masked attn_weights attn_outputs r0r@zUMT5Attention.forwards"/!4RaR!8 JM\Mh?99$.IIIno7ABRTYT`7a7aw+11333gw3$>vvm,, #((RtG^__iijkmnoo   &:oGZ+[+[ &(377GGJ! L'6'L$$'6'K$$#2 2DW..-  F/"=*"=-4T^DIJ/6t~FMLL//J66.11L#RtG^__iijkmnooJ',,ZT\4KbccmmnoqrssL*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>lJ,@,@A,F,FGG%b) / !KDL*j9&-W]WcMM!--Jv}O_.M  %)N:M,&&},,V\\^^,DDLLVTT },,\T\TXTa,bb l<>> !++Aq11<<>> !&&z:rBB ff[)) L((r1)FN)NrNNN)rArBrC__doc__intr&r(r[rrrr r@rDrEs@r0rrrrskkSSWZkkkkkk6%, - - - ^    6:(,.2 N)N)|N) %|d2N) N)  t+ N)N)N)N)N)N)N)N)r1rrc8eZdZddedzffd Z ddZxZS)UMT5LayerSelfAttentionNrsctt|d||_t |j|j|_tj |j |_ dS)NTrvrsrj) r%r&rr SelfAttentionr"rNrmrnrrRrSrTr,rHrsr/s r0r&zUMT5LayerSelfAttention.__init__Jsb *6t_hiii'Frrrnormed_hidden_statesattention_outputoutputss r0r@zUMT5LayerSelfAttention.forwardPst $}==-- )+.   & 5Ea5H(I(II  "%5abb%99r1rY)NNrArBrCrr&r@rDrEs@r0rrIsi77#*777777 r1rc:eZdZddedzffd Z ddZxZS)UMT5LayerCrossAttentionNrsctt|d||_t |j|j|_tj |j |_ dS)NFrrj) r%r&rrEncDecAttentionr"rNrmrnrrRrSrTrs r0r&z UMT5LayerCrossAttention.__init__csc ,VQVbklll'Frrrrrr layer_outputrs r0r@zUMT5LayerCrossAttention.forwardisv $}==// "7)+ 0   %t||4DQ4G'H'HH /$4QRR$88r1rYrrrEs@r0rrbsl77#*777777# r1rc@eZdZddedzffd Z ddZxZS) UMT5BlockNrsct|j|_tj|_|jt|||jr)|jt|||jt|dS)Nrs) r%r&rur ModuleListlayerappendrrrhrs r0r&zUMT5Block.__init__~s  +]__  09MMMNNN ? T J  5f RRR S S S +f--.....r1Fc |jd|||\}} |jtjkrstj|jj} tjtj|| dz | } tj || | }d} |j o|du} | r|jd||||\}} |jtjkrstj|jj} tjtj|| dz | } tj || | }|jd|}|jtjkrstj|jj} tjtj|| dz | } tj || | }|f}|r|| | fz }|S)Nrri)rmaxrrr4) rr;r(r<finforrisinfanyclampru)r,r>rrencoder_attention_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrs r0r@zUMT5Block.forwards ,94:a= )+, , , ( (  %- / / M$788 >Gr1rY)NNNNFFrrEs@r0rr}su//#*//////"#55555555r1rcLeZdZdZdeffd ZdejdejfdZxZ S)UMT5ClassificationHeadz-Head for sentence-level classification tasks.rHcttj|j|j|_tj|j|_tj|j|j |_ dS)N)r) r%r&rrMrNdenserRclassifier_dropoutrT num_labelsout_projrWs r0r&zUMT5ClassificationHead.__init__sc Yv~v~>> zF$=>>>  &.&2CDD r1r>rc||}||}tj|}||}||}|SrY)rTrr(tanhrr]s r0r@zUMT5ClassificationHead.forwards[ ]33  =11  =11  ]33  m44 r1) rArBrCrr r&r(r[r@rDrEs@r0rrsw77EzEEEEEE U\elr1rceZdZUeed<dZdZdZdgZdgZ e dZ e j fdZdZxZS) UMT5PreTrainedModelrH transformerTrrQcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r(tensorrr)r,r input_mask dummy_inputss r0rz UMT5PreTrainedModel.dummy_inputss=L.. \*-- !*"&0  r1c\ t||jj}t |t rt j|j|dzd St |ttttfrt j |jjd|dzt|dr0|jjs$t j |jjd|dzt|drQt j |jjd||jjdzzt j|jjd Sd St |t,rVt|drDt j |jjd|dzt j|jjd Sd St |t0rt j |jjd||jjdzzt|jdr*|jjt j|jjt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |t6rt j |jjd||jjdzzt|jdr*|jjt j|jjt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |t>rVt j |j jd||jjdzzt|j dr*|j jt j|j jt j |j!jd||jjdzzt|j!dr*|j!jt j|j!jt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |tDr|jj}|jj#}|jj$}t j |j%jd|||zdzzt j |j&jd||dzzt j |j'jd||dzzt j |j(jd|||zdzz|j)r+t j |j*jd||dzzd Sd Sd S) zInitialize the weights?g)r9stdlm_head qa_outputs classifierrLN)+r% _init_weightsrHinitializer_factorrZr"init constant_r* UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrNzeros_rLUMT5ForTokenClassificationrrrrrGrPrQrOr`rbrcrrryr{rrrrrvr)r,modulefactorrNrzr|r/s r0rz!UMT5PreTrainedModel._init_weightss f%%%/ fm , ,= p N6=&3, 7 7 7 7 7  , (    ; p L-CVc\ J J J Jvy)) P$+2Q P V^2&3,OOOOv|,, 4 V.5CVPTP[PchlOlEmnnnn F-233333 4 4 : ; ;* pv|,, 4 V.5CVc\RRRR F-233333 4 4 6 7 7& p L,3Ft{GZ_cFc%>%@%@#qrr'"$:&!  PQQ Q&&'8D'@,OOO  r1)rArBrCr __annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr(no_gradrr3rDrEs@r0r r s %&*#!$ !F XU]__ApApApAp_ApF!!!!!!!r1r c@eZdZfdZdZ ddZxZS) UMT5Stackcttjjj|_j|_tjfdtj D|_ tjj |_tjj|_d|_|dS)Nc2g|]}t|S)r)r).0irHs r0 z&UMT5Stack.__init__..Gs&#e#e#eqIf$B$B$B#e#e#er1rjF)r%r&rr vocab_sizerN embed_tokensrurrange num_layersblockr"rmfinal_layer_normrRrSrTgradient_checkpointing post_initrWs `r0r&zUMT5Stack.__init__Cs    L):FNKK +]#e#e#e#eERXRcLdLd#e#e#eff -fn&B[ \ \ \z&"566 ',# r1c||_dSrY)rDr,new_embeddingss r0set_input_embeddingszUMT5Stack.set_input_embeddingsOs*r1Nc (||n |jj}||n |jj}| | n |jj} | | n |jj} |#|!|jrdnd} t d| d| d|1|} |d| d}n@||dd} n!|jrdnd} t d| d| d |j r%|j r|rt d d }|+|j t d | |}| \}}|d ur|jst d|d|jr]|rZ|X|jjr7tt!|jt!|j}nt!|j}n |jsd}||nd}|/t%s!||z}t'j|||j}|jrt-|j|||}nT|P|ddddddf}||j}d|z t'j|jjz}nd}|jr|t7|j|||}nd}| rdnd}|rdnd}|r |jrdnd}||}t;|jD]H\}}| r||fz}||||||||}|d}|r||dfz }|jr ||dfz }I||}||}| r||fz}| stAd|||||fDStC|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer4zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz$UMT5Stack.forward..s4  =!===  r1)last_hidden_staterr> attentionscross_attentions)"rHrroutput_hidden_states return_dictrur/rrrIrr~rrDis_encoder_decoderr r rrr(r)rrr6r;rrrrT enumeraterGrHtupler)r,rrrrrRrrrrYrZrerr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_length causal_maskencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsr>rA layer_module layer_outputss r0r@zUMT5Stack.forwardRs"+!6IIDK+/?BZZNw>wwwww  "#..**K!r;r?;;II  &',,..ss3KK+/?BZZNu>uuXfuuuvv v  & "4= " "##p"   ( !_``` --i88M!, J   ? j !hT!h!h!hiii ? # G_4;1G&9$DK888,dk:Z:Z:Z''OO'3$+&F&F&FO ##OETE`!?!?!A!A!Afg  !*B*D*D !4zAO"Z OML`aaaN ? ,{+- / KK  '(D$)9:K%..}/B.CCK, M>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rH shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightct|tj|j|j|_tj|}d|_ d|_ t||_ tj|}d|_ |j |_t||_|dSNFT)r%r&rrrCrNr$copydeepcopyrurr=encodernum_decoder_layersrFdecoderrJr,rHencoder_configdecoder_configr/s r0r&zUMT5Model.__init__s    l6#4fnEE v..$)!#(  00 v..$(!$*$=! 00  r1c|jSrYr$r,s r0get_input_embeddingszUMT5Model.get_input_embeddings {r1c|||_|j||j|dSrYr$rsrNrurLs r0rNzUMT5Model.set_input_embeddings;$  )).999 )).99999r1Nrrrrencoder_outputsrrRdecoder_inputs_embedsrrrYrZrc "| | n |jj} | | n |jj} |||||| | | }ne| rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}|||||||| | | |  }| s||zSt|j |j |j |j |j |j |j |j S) ah input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```NrrrRrrYrZrrr3rVr>rW rrrRrrrrrrYrZ)rVrdecoder_hidden_statesdecoder_attentionsrXencoder_last_hidden_staterencoder_attentions)rHrrZrsrZrlenrurrVrr>rWrX)r,rrrrrrrRrrrrYrZrr>decoder_outputss r0r@zUMT5Model.forwardsuD"+!6IIDKBF(,-159!%)-,0#'q q #d*q )D0q !+d2 q !& 04 7 q uU%6784? q q |d*q  %|d2q $;q  $;q #Tkq D[q  u !$6 6q q q ^q q q q q r1rz< UMT5 Model with a `language modeling` head on top. ) custom_introceZdZdZdZddddZfdZdZdZe dd e j dzd e j dzd e j dzd e j dzd eee jdzdedzde j dzde j dzde j dzdedzdedzdedzdedzdee j ezfdZde jfdZxZS)r a Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```rjrk)rmrnzlm_head.weightct||j|_t j|j|j|_tj |}d|_ d|_ t||_ tj |}d|_ |j|_t||_t j|j|jd|_|dS)NFTrK)r%r&rN model_dimrrrCr$rqrrrurr=rsrtrFrurMrrJrvs r0r&z%UMT5ForConditionalGeneration.__init__s    l6#4fnEE v..$)!#(  00 v..$(!$*$=! 00 y1BOOO  r1c|jSrYrzr{s r0r|z1UMT5ForConditionalGeneration.get_input_embeddingsr}r1c|||_|j||j|dSrYrrLs r0rNz1UMT5ForConditionalGeneration.set_input_embeddingsrr1NrrrrrrrRrlabelsrrrYrZrc | | n |jj} | | n |jj} |||||| | | }ne| rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}| |||| }|||||||| | | |  }|d}|jj r ||j dzz}| |}d}| ptd }| |j} ||d |d | d }| s|f|ddz|z}||f|zn|St#|||j|j|j|j|j|j|j S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrrr3rrrr, ignore_indexr4 losslogitsrrrrXrrr)rHrrZrsrZrrr3rur&rrrr6rrrrrr>rWrXrV)r,rrrrrrrRrrrrrYrZrr>rsequence_output lm_logitsrloss_fctoutputs r0r@z$UMT5ForConditionalGeneration.forwardscL"+!6IIDK=A(,26:>*.!%)-,0#'L L #d*L )D0L !+d2 L !& 04 7 L uU\23d: L L (4/L  %047L  4'L $;L  $;L #TkL D[L u !O 3!L L L ^L ^)EL))))))))r1r ceZdZdZdZddiZfdZdZdZe dd e j dzd e j dzd e j dzd e dzd e dzde dzdee j ezfdZxZS)r!a Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rjrmrkc&t|tj|j|j|_tj|}d|_ d|_ t||_ | dS)NF)r%r&rrrCrNr$rqrrrr[r=rsrJ)r,rHrwr/s r0r&zUMT5EncoderModel.__init__zsy    l6#4fnEE v..#( ,1) 00  r1c|jSrYrzr{s r0r|z%UMT5EncoderModel.get_input_embeddingsr}r1cH||_|j|dSrY)r$rsrNrLs r0rNz%UMT5EncoderModel.set_input_embeddingss%$  )).99999r1NrrrRrrYrZrc \||n |jj}|||||||}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr)rHrZrs) r,rrrRrrYrZrrs r0r@zUMT5EncoderModel.forwardsJF&1% 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrrrRrrrrYrZrrz7All examples must have the same number of tokens.z3Each example must contain at least one token.r4 regressionsingle_label_classificationmulti_label_classificationr)%rHrZNotImplementedErrorr/rAr/r3r eq eos_token_idr6rrr(unique_consecutivesumnumelrrr problem_typerr;rrrsqueezerrrrrrrXrrr)r,rrrrrrRrrrrrYrZrrreos_maskr_r-selectedsentence_representationrrrrs r0r@z%UMT5ForSequenceClassification.forwards`&1%Kbdd   $)>)F  U !% 1 1) < < "" )/#9+'"7/!5##   "!*<< 899<<_=STT  $X\\!__ 5 5 ; ; = = B E   &5%:" A{"8QQQ;/ N1  +q 0 A   #+-- B "L"LQQQPRTUTUTUX"V))*ABB  YYv}--F{'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x// FY,F)-)9TGf$$v E.#3")"?&9$5&-&G")"?&9    r1r)rArBrC"_keys_to_ignore_on_load_unexpectedr r&rr(rr[listrrr]rr@rDrEs@r0rrs+s)s&z.2.259:>:>26:>*.!%)-,0#'F F #d*F  t+F !+d2 F !& 04 7 F e/047 F (4/F  %047F  4'F $;F  $;F #TkF D[F  0 0F F F ^F F F F F r1rceZdZdgZdeffd Ze ddejdzdejdzdejdzdejdzd e dzd e dzd e dzd e eje zfd Z xZ S)r(rrHc6t||j|_t||_t j|j|_t j |j |j|_ | dSrY) r%r&rr!r rrRrrTrMr-rrJrWs r0r&z#UMT5ForTokenClassification.__init__asz     ++F33z&";<< )F$68IJJ r1NrrrRrrrYrZrc ||n |jj}|||||||} | d} || } || } d} |Ft } | | d|j|d} |s| | ddf}| | f|zn|St| | | j | j S)aB input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrRrrYrZrr4r3)rrr>rW) rHrZr rTrrrrrr>rW)r,rrrRrrrYrZrrr>rrrrs r0r@z"UMT5ForTokenClassification.forwardls 6&1%rWrXrV)r,rrrrrrrrRrrrrYrZrr>rrrrr total_loss ignored_indexr start_lossend_lossrs r0r@z UMT5ForQuestionAnswering.forwards\&1%)F  U !% 1 1) < < !*!6IIDK=A371526:>!%)-,0#'I I #d*I )D0I !+d2 I !& 04 7 I uU\23d: I )D0I '$.I (4/I  %047I $;I  $;I #TkI D[I u !$G G!I I I ^I I I I I r1r")r!r r"rr(rr )Brrqrr(rtorch.nnrrrrQrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrconfiguration_umt5r get_loggerrAr~Moduler"rGr`rhrrrrrrr r=rr r!rr(r"__all__rSr1r0rs AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999.-----+*****  H % %+++++BI+++4 .RY<")$o)o)o)o)o)BIo)o)o)dRY2bi6@@@@@*@@@HRY$k!k!k!k!k!/k!k!k!\e e e e e #e e e Pf f f f f #f f f R H)H)H)H)H)#6H)H)  H)VXXXXX*XXXv S S S S S $7S S  S lG G G G G !4G G G Tp p p p p 2p p p f   r1