tjdZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'j+e,Z-Gddej.Z/Gddej.Z0Gddej.Z1Gddej.Z2Gddej.Z3Gddej.Z4Gddej.Z5Gdd eZ6Gd!d"ej.Z7e&Gd#d$e"Z8Gd%d&e8Z9e&Gd'd(e8Z:e&d)*Gd+d,e8eZ;e&Gd-d.e8Z<e&d/*Gd0d1e8Z=e&Gd2d3e8Z>e&Gd4d5e8Z?gd6Z@dS)7zPyTorch T5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringloggingtorch_compilable_check)T5Configc&eZdZdfd ZdZxZS) T5LayerNormư>cttjt j||_||_dS)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s `/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.pyr%zT5LayerNorm.__init__/sD l5:k#:#:;; #ch|tjddd}|tj||jzz}|jjtj tj fvr||jj}|j|zS)NT)keepdim) tor'float32powmeanrsqrtr*r)dtypefloat16bfloat16)r+ hidden_statesvariances r/forwardzT5LayerNorm.forward7s !##EM2266q99>>r4>PP% Ht?T4T(U(UU  ;  ? ? ?),,T[->??M{]**r0)r")__name__ __module__ __qualname__r%r? __classcell__r.s@r/r!r!.sL$$$$$$ + + + + + + +r0r!c*eZdZdeffd ZdZxZS)T5DenseActDenseconfigcJttj|j|jd|_tj|j|jd|_tj|j |_ t|j |_ dSNFbias)r$r%rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr+rGr.s r/r%zT5DenseActDense.__init__Hsx )FNFKeDDD)FKeDDDz&"566 &-.r0c||}||}||}t|jjt jr]|j|jjjkrC|jjjt j kr$| |jjj}||}|SN) rOrUrS isinstancerPr)r'Tensorr:int8r5r+r=s r/r?zT5DenseActDense.forwardOs .. //  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r0r@rArBrr%r?rCrDs@r/rFrFGsS/x//////       r0rFc*eZdZdeffd ZdZxZS)T5DenseGatedActDenserGcttj|j|jd|_tj|j|jd|_tj|j|jd|_tj |j |_ t|j |_dSrI)r$r%rrLrMrNwi_0wi_1rPrQrRrSr rTrUrVs r/r%zT5DenseGatedActDense.__init__^s IfnfkFFF IfnfkFFF )FKeDDDz&"566 &-.r0c|||}||}||z}||}t |jjtjr]|j |jjj krC|jjj tj kr$| |jjj }||}|SrX) rUrarbrSrYrPr)r'rZr:r[r5)r+r= hidden_gelu hidden_linears r/r?zT5DenseGatedActDense.forwardfshhtyy7788  -00 #m3  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r0r]rDs@r/r_r_]sS/x//////r0r_c*eZdZdeffd ZdZxZS) T5LayerFFrGc$t|jrt||_nt ||_t |j|j|_ tj |j |_ dS)Nr-)r$r% is_gated_actr_DenseReluDenserFr!rMlayer_norm_epsilon layer_normrrQrRrSrVs r/r%zT5LayerFF.__init__{sx    :"6v">">D  "1&"9"9D %fn&:STTTz&"566 r0c||}||}|||z}|SrX)rmrkrS)r+r=forwarded_statess r/r?zT5LayerFF.forwardsF??=99../?@@% 5E(F(FF r0r]rDs@r/rgrgzsS7x777777r0rgcfeZdZ d dededzffd Zedd Zdd Z dd Z xZ S) T5AttentionFNrG layer_idxc*t|j|_||_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |jz|_ ||_|/|jr(td|jjdt'j|j|j d|_t'j|j|j d|_t'j|j|j d|_t'j|j |jd|_|jr$t'j|j|j |_d|_dS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrJ)r$r% is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerMd_kvkey_value_proj_dim num_headsn_headsrRrS inner_dimrrlogger warning_oncer.r@rrLqkvo Embeddingrelative_attention_biasgradient_checkpointingr+rGrurrr.s r/r%zT5Attention.__init__si  ++F(.4.S+/5/U,~ "(+' * (??"      ,4>+B,,,    4<eDDD4<eDDD4<eDDD4>4=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr2r) r5r'longabsmin zeros_likelogfloatmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r/_relative_position_bucketz%T5Attention._relative_position_buckets>,  c A K !2Q!6 : :5: F F T T  % *; < <  !&+NO`>a>a!b!b b  1$ $y0&/ I'--//); < <h|i/00 1Y& ( "UZ.. &" &+Y &8RT_bcTc(d(d& & " EK2CE_```r0rc||jjj}tj|tj|dddf|z}tj|tj|dddf}||z }|||j |j|j }||} | gd d} | S)z%Compute binned relative position biasN)r:device)rrr)r2rrr) rr)rr'arangerrrtrvrwpermute unsqueeze) r+ query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess r/ compute_biaszT5Attention.compute_biass >18?F < EJvVVVWXWXWXZ^W^_brr,zFSSSTXZ[Z[Z[T[\+.>>#'#A#A #.;= $B$ $  --.FGG **44Q77 r0c |jdd}g|d|jR} |||jnd} t | t jr| n| } |du} || |  dd} d} t |tr1|j |j} | r|j}n |j}n|}| r|n|}| r3|1| r/|j|jj}|j|jj}ng|jddd|jR}|| | dd}|| | dd}|E||||j\}}| r$t |trd|j |j<t j| | dd}||jd }|jsLt jd| jd|d|f|j|j }|jr|jrd|_n$||d||j| }|$|ddddddd|jd f}||z}|}||z }t>j !|"d #|}t>j $||j$|j }t j||}| dd%}|j&g|dR}|'|}||f}|r||fz}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr3rrr2FTr)rr:)rrdim)ptraining)(shaperyget_seq_lengthrrrYr'rZclonerview transposer is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdatematmulruzerosrr:rr requires_gradrr functionalsoftmaxrtype_asrS contiguousreshaper)r+r=maskkey_value_states position_biaspast_key_valuesoutput_attentionskwargs input_shape hidden_shaperis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapescoresr causal_maskposition_bias_masked attn_weights attn_outputoutputss r/r?zT5Attention.forwards$)#2#. BBbB$*ABB M\Mh?99$.IIIno7ABRTYT`7a7aw+11333gw.T9vvm,,11,??II!QOO  o': ; ; 3(377GGJ! L'6'L$$'6'K$$#2 -?R))]  F/"=*"=-4T^DIJ/6t~FMLLP-crc2PBP8OPPH//44X>>HHANNJ66.1166x@@JJ1aPPL*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>lJ,@,@A,F,FGG  #)"-J3  %  *1-{1~zJSYS`hnht!!! .74=726M/ $ 1 1NJv}Wg!2!! "111aaa,Bj.>r.B,B#BC - ; ,&&},,V\\^^,DDLLVTT },,\T\TXTa,bb l<>> !++Aq11<<>> )k);;;;;; ff[))  .  0/Gr0FN)Trr)Nr)NNNNF) r@rArBrintr% staticmethodrrr?rCrDs@r/rqrqs%* $ , , ,: , , , , , ,D- - - \- ^([[[[[[[[r0rqc>eZdZddedzffd Z ddZxZS)T5LayerSelfAttentionFNrrctt||||_t |j|j|_tj |j |_ dS)Nrurrri) r$r%rq SelfAttentionr!rMrlrmrrQrRrSrs r/r%zT5LayerSelfAttention.__init__Osl ( 0KW`   &fn&:STTTz&"566 r0c ||}|||||||} ||| dz}|f| ddz} | S)N)rrr use_cacherrr)rmrrS) r+r=attention_maskrrrrrnormed_hidden_statesattention_outputrs r/r?zT5LayerSelfAttention.forwardWs} $}==-- '+/ .  & 5Ea5H(I(II  "%5abb%99r0r)NNNFFr@rArBrr%r?rCrDs@r/rrNss77SSWZ777777r0rc<eZdZddedzffd Z ddZxZS)T5LayerCrossAttentionNrrctt|d||_t |j|j|_tj |j |_ dS)NFrri) r$r%rqEncDecAttentionr!rMrlrmrrQrRrS)r+rGrrr.s r/r%zT5LayerCrossAttention.__init__psb *6u`ijjj%fn&:STTTz&"566 r0Fc ||}|||||||} ||| dz} | f| ddz} | S)N)rrrrrrr)rmrrS) r+r=rrrrrrrr layer_outputrs r/r?zT5LayerCrossAttention.forwardvs| $}==// -'+/ 0  %t||4DQ4G'H'HH /$4QRR$88r0rX)NNNFrrDs@r/rroso77#*777777r0rcFeZdZddedzffd Z ddZxZS) T5BlockFNrrct|j|_tj|_|jt||||jr)|jt|||jt|dS)Nr)rr) r$r%rtr ModuleListlayerappendrrrgrs r/r%zT5Block.__init__s  +]__   E`lu v v v    ? R J  3FiPPP Q Q Q )F++,,,,,r0Tc  |jd|||||| } | d}| dd} |jtjkrtjtj|tj|jjdz tj|jj}tj || |}|j o|du}|r|jd|||||| }|d}|jtjkrtjtj|tj|jjdz tj|jj}tj || |}| |ddz} |jd|}|jtjkrtjtj|tj|jjdz tj|jj}tj || |}|f}|| zS)Nr)rrrrrri)rmax)rrrrrr3) rr:r'r;risinfanyfinforclamprt)r+r=rrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrrr return_dictrself_attention_outputsattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrs r/r?zT5Block.forwardso"/A )'+/ " " " /q1 21226  %- / /+ M**..00 M/004t; M/004K "K K<[YYYM!_R1Fd1R  P&3djm!65; /"3 ''' #4A6M"em33#kK ..2244K 34484?K 3448 !& M |Q\ ] ] ] !24KABB4O O ' 2}55   %- / /+ M**..00 M/004t; M/004K "K K<[YYYM " ' ' r0r) NNNNNNFFTrrDs@r/rrs - -SSWZ - - - - - -"#&*J J J J J J J J r0rcLeZdZdZdeffd ZdejdejfdZxZ S)T5ClassificationHeadz-Head for sentence-level classification tasks.rGcttj|j|j|_tj|j|_tj|j|j |_ dS)N)r) r$r%rrLrMdenserQclassifier_dropoutrS num_labelsout_projrVs r/r%zT5ClassificationHead.__init__sc Yv~v~>> zF$=>>>  &.&2CDD r0r=returnc||}||}tj|}||}||}|SrX)rSrr'tanhrr\s r/r?zT5ClassificationHead.forwards[ ]33  =11  =11  ]33  m44 r0) r@rArB__doc__rr%r'rZr?rCrDs@r/rrsw77ExEEEEEE U\elr0rceZdZUeed<dZdZdZdgZdgZ e dZ e j fdZdZxZS) T5PreTrainedModelrG transformerTrrPcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r'tensorrr)r+r input_mask dummy_inputss r/rzT5PreTrainedModel.dummy_inputss=L.. \*-- !*"&0  r0c\ t||jj}t |t rt j|j|dzd St |ttttfrt j |jjd|dzt|dr0|jjs$t j |jjd|dzt|drQt j |jjd||jjdzzt j|jjd Sd St |t,rVt|drDt j |jjd|dzt j|jjd Sd St |t0rt j |jjd||jjdzzt|jdr*|jjt j|jjt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |t6rt j |jjd||jjdzzt|jdr*|jjt j|jjt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |t>rVt j |j jd||jjdzzt|j dr*|j jt j|j jt j |j!jd||jjdzzt|j!dr*|j!jt j|j!jt j |jjd||jjdzzt|jdr,|jj"t j|jjd Sd Sd St |tDr|jj}|jj#}|jj$}t j |j%jd|||zdzzt j |j&jd||dzzt j |j'jd||dzzt j |j(jd|||zdzz|j)r+t j |j*jd||dzzd Sd Sd S) zInitialize the weightsg?g)r8stdlm_head qa_outputs classifierrKN)+r$ _init_weightsrGinitializer_factorrYr!init constant_r)T5ModelT5ForConditionalGenerationT5EncoderModelT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrMzeros_rKT5ForTokenClassificationrrrrrFrOrPrNr_rarbrqrxrzrrrrrur)r+modulefactorrMryr{r.s r/rzT5PreTrainedModel._init_weightss f%%%/ fk * *1 p N6=&3, 7 7 7 7 7   0.BX Y  / p L-CVc\ J J J Jvy)) P$+2Q P V^2&3,OOOOv|,, 4 V.5CVPTP[PchlOlEmnnnn F-233333 4 4 8 9 9% pv|,, 4 V.5CVc\RRRR F-233333 4 4 4 5 5! p L,3Ft{GZ_cFc%>%@%@#qrr'"$:&!  PQQ Q&&'8D'@,OOO  r0)r@rArBr__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr'no_gradrr4rCrDs@r/r r s %&*#!" !F XU]__5p5p5p5p_5pn!!!!!!!r0r c@eZdZfdZdZ ddZxZS)T5Stackcttjjj|_j|_tjfdtj D|_ tjj |_tjj|_|d|_dS)Nc Vg|]%}tt|dk|&S)rr)rbool).0irGs r/ z$T5Stack.__init__..es4 v v vXYWVa1fQR S S S v v vr0riF)r$r%rr vocab_sizerM embed_tokensrtrrange num_layersblockr!rlfinal_layer_normrQrRrS post_initrrVs `r/r%zT5Stack.__init__^s    L):FNKK +] v v v v]bcict]u]u v v v   !,FN@Y Z Z Zz&"566  &+###r0c||_dSrX)rFr+new_embeddingss r/set_input_embeddingszT5Stack.set_input_embeddingsns*r0Nc ||n |jj}||n |jj}| | n |jj} | | n |jj} |#|!|jrdnd} t d| d| d|1|} |d| d}n@||dd} n!|jrdnd} t d| d| d |j r%|j r|rt d d }|+|j t d | |}| \}}|d ur|jst d|d|jr]|rZ|X|jjr7tt!|jt!|j}nt!|j}n |jsd}|jjr5t#|j||t%|tr|jn|}nt)|j||}d}|jr|t)|j|||}| rdnd}|rdnd}|r |jrdnd}d}d}||}|jD]e}| r||fz}|||||||||||  }|d}|d}|jr| ||rdnd}|r||dfz}|jr ||dfz}f||}||}| r||fz}| st1d|||||fDSt3|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer3zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz"T5Stack.forward..s4  =!===  r0)last_hidden_staterr= attentionscross_attentions)rGrroutput_hidden_statesrrtr0sizerrrr}r~rFis_encoder_decoderr r rrYrrrSrIrJtupler)r+rrrrrSrrrr[rrerr_msg_prefixr batch_size seq_lengthencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrrr= layer_module layer_outputss r/r?zT5Stack.forwardqs"+!6IIDK+/?BZZNw>wwwww  "#..**K!r;r?;;II  &',,..ss3KK+/?BZZNu>uuXfuuuvv v  & "4= " "##p"   ( !_``` --i88M!, J   ? j !hT!h!h!hiii ? # G_4;1G&9$DK888,dk:Z:Z:Z''OO'3$+&F&F&FO ##O ; ! /{+-o/BCC!% D D$ NN7{+-N +/' ? 4@.G{+5&; /// +#7@BBD0:d&7VDOVrrRV (,% ]33  J V VL# I$58H$H!(L%/- /#"3'   M*!,M *!,M ]#8#D0=CT>[aaZ[0\-  V!/=3C2E!E?V+?=QRCSBU+U(--m<<  ]33   E 1]4D D    "#%"(       9+++%1     r0) NNNNNNNNNN)r@rArBr%rOr?rCrDs@r/r>r>]s,,,,, +++ "#![ [ [ [ [ [ [ [ r0r>cheZdZdgZdddZdeffd ZdZdZe dd e j dzd e j dzd e j dzd e j dzd eee j dzdedzde jdzde jdzdedzdedzdedzdedzdee j ezfdZxZS)r Fdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightrGct|tj|j|j|_tj|}d|_ d|_ t||_ tj|}d|_ |j |_t||_|dSNFT)r$r%rrrErMr%copydeepcopyrtrr>encodernum_decoder_layersrHdecoderrKr+rGencoder_configdecoder_configr.s r/r%zT5Model.__init__s    l6#4fnEE v..$)!#( ~.. v..$(!$*$=!~..  r0c|jSrXr%r+s r/get_input_embeddingszT5Model.get_input_embeddings* {r0c|||_|j||j|dSrXr%rrrOrtrMs r/rOzT5Model.set_input_embeddings-;$  )).999 )).99999r0Nrrrrencoder_outputsrrSdecoder_inputs_embedsrrr[rrc "| | n |jj} | | n |jj} |||||| | | }ne| rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}|||||||| | | |  }| s||zSt|j |j |j |j |j |j |j |j S) aV input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, T5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5Model.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for T5Model. >>> # This is not needed for torch's T5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```NrrrSrr[rrrr2rXr=rY rrrSrrrrrr[r)rXrdecoder_hidden_statesdecoder_attentionsrZencoder_last_hidden_staterencoder_attentions)rGrrrrrYrlenrtrrXrr=rYrZ)r+rrrrrrrSrrrr[rrr=decoder_outputss r/r?zT5Model.forward2suD"+!6IIDKBF(,-159!%)-,0#'q q #d*q )D0q !+d2 q !& 04 7 q uU%6784? q q |d*q  %|d2q $;q  $;q #Tkq D[q  u !$6 6q q q ^q q q q q r0r z: T5 Model with a `language modeling` head on top. ) custom_introceZdZdgZddddZdeffd ZdZdZe dd e j dzd e j dzd e j dzd e j dzd eee jdzdedzde j dzde j dzde j dzdedzdedzdedzdedzdee j ezfdZde jfdZxZS)r!rirj)zlm_head.weightrlrmrGct||j|_t j|j|j|_tj |}d|_ d|_ t||_ tj |}d|_ |j|_t||_t j|j|jd|_|dS)NFTrJ)r$r%rM model_dimrrrEr%rprqrtrr>rrrsrHrtrLrrKrus r/r%z#T5ForConditionalGeneration.__init__s    l6#4fnEE v..$)!#( ~.. v..$(!$*$=!~.. y1BOOO  r0c|jSrXryrzs r/r{z/T5ForConditionalGeneration.get_input_embeddingsr|r0c|||_|j||j|dSrXr~rMs r/rOz/T5ForConditionalGeneration.set_input_embeddingsrr0NrrrrrrrSrlabelsrrr[rrc | | n |jj} | | n |jj} |||||| | | }ne| rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}| |||| }|||||||| | | |  }|d}|jj r ||j dzz}| |}d}| ptd }| |j} ||d |d | d }| s|f|ddz|z}||f|zn|St#|||j|j|j|j|j|j|j S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, T5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nrrrr2rrrr- ignore_indexr3 losslogitsrrrrZrrr)rGrrrrrYrrr4rtscale_decoder_outputsrrrr5rrr\rrr=rYrZrX)r+rrrrrrrSrrrrr[rrr=rsequence_output lm_logitsrloss_fctoutputs r/r?z"T5ForConditionalGeneration.forwardsaR"+!6IIDK=A(,26:>*.!%)-,0#'M M #d*M )D0M !+d2 M !& 04 7 M uU\23d: M M (4/M  %047M  4'M $;M  $;M #TkM D[M u !O 3!M M M ^M ^)EL))))))))r0r!ceZdZddiZdgZdeffd ZdZdZe dd e j dzd e j dzd e j dzd e dzd e dzde dzdee j ezfdZxZS)r"rlrjrtrGct|tj|j|j|_|}d|_d|_t||_ | dS)NF) r$r%rrrErMr%rr]r>rrrK)r+rGrvr.s r/r%zT5EncoderModel.__init__lso    l6#4fnEE #( ,1)~..  r0c|jSrXryrzs r/r{z#T5EncoderModel.get_input_embeddingsxr|r0cH||_|j|dSrX)r%rrrOrMs r/rOz#T5EncoderModel.set_input_embeddings{s%$  )).99999r0NrrrSrr[rrc \||n |jj}|||||||}|S)aI input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). Example: ```python >>> from transformers import AutoTokenizer, T5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5EncoderModel.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr)rGrrr) r+rrrSrr[rrrs r/r?zT5EncoderModel.forwardsJD&1% 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrrrSrrrr[rrrz7All examples must have the same number of tokens.z3Each example must contain at least one token.r3 regressionsingle_label_classificationmulti_label_classificationr)%rGrNotImplementedErrorr.r@r0r4req eos_token_idr5rrr'unique_consecutivesumnumelrrr problem_typerr:rrrsqueezerrrrrrrZrrr)r+rrrrrrSrrrrr[rrrreos_maskr`_r,selectedsentence_representationrrrrs r/r?z#T5ForSequenceClassification.forwards`&1%Kbdd   $)>)F  U !% 1 1) < < "" )/#9+'"7/!5##   "!*<< 899<<_=STT  $X\\!__ 5 5 ; ; = = B E   &5%:" A{"8QQQ;/ N1  +q 0 A   #+-- B "L"LQQQPRTUTUTUX"V))*ABB  YYv}--F{'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x// FY,F)-)9TGf$$v E.#3")"?&9$5&-&G")"?&9    r0r)r@rArBrrr%rr'rrZlistrrAr^rr?rCrDs@r/rrs+s)s&x.2.259:>:>26:>*.!%)-,0#'F F #d*F  t+F !+d2 F !& 04 7 F e/047 F (4/F  %047F  4'F $;F  $;F #TkF D[F  0 0F F F ^F F F F F r0rceZdZdeffd Ze d dejdzdejdzdejdzdejdzdedzd edzd edzd e eje zfd Z xZ S)r)rGc6t||j|_t||_t j|j|_t j |j |j|_ | dSrX) r$r%rr"rrrQrrSrLr,rrKrVs r/r%z!T5ForTokenClassification.__init__Lsz     +)&11z&";<< )F$68IJJ r0NrrrSrrr[rrc ||n |jj}|||||||} | d} || } || } d} |Ft } | | d|j|d} |s| | ddf}| | f|zn|St| | | j | j S)a< input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrSrr[rrr3r2)rrr=rY) rGrrrSrrrrrr=rY)r+rrrSrrr[rrrr=rrrrs r/r?z T5ForTokenClassification.forwardWs 4&1%rrrsrHrtrrLr,rrKrus r/r%zT5ForQuestionAnswering.__init__s    l6#4fnEE v..$)!#( ~.. v..$(!$*$=!~..  +)F$68IJJ r0c|jSrXryrzs r/r{z+T5ForQuestionAnswering.get_input_embeddingsr|r0c|||_|j||j|dSrXr~rMs r/rOz+T5ForQuestionAnswering.set_input_embeddingsrr0Nrrrrrstart_positions end_positionsrSrrrr[rrc .| | n |jj} | | n |jj} ||d} |(| &|td||}| | n |jj} | | n |jj} |||||| | | }ne| rct |tsNt|dt|dkr|dndt|dkr|dnd}|d}| ||| d||| | | | }|d}| |}| dd \}}| d  }| d  }d}||t|dkr-| d |j}t|dkr-| d |j}|d}|d|}|d|}t%| }|||}|||}||zdz }| s||f|ddz|z}||f|zn|St'||||j|j|j|j|j|j|j S)az input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrrrrr2rrr3rr) r start_logits end_logitsrrrrZrrr)rGrrr0r4rrrYrrrtrsplitrrr\r5rrrrrr=rYrZrX)r+rrrrrrrrSrrrr[rrr=rrrrr total_loss ignored_indexr start_lossend_lossrs r/r?zT5ForQuestionAnswering.forwards\&1%)F  U !% 1 1) < < !*!6IIDK=A371526:>!%)-,0#'I I #d*I )D0I !+d2 I !& 04 7 I uU\23d: I )D0I '$.I (4/I  %047I $;I  $;I #TkI D[I u !$G G!I I I ^I I I I I r0r#)r"r!r r r#rr))Ar rprr'rtorch.nnrrrrRrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrconfiguration_t5r get_loggerr@r}Moduler!rFr_rgrqrrrrr r>r r!r"rr)r#__all__rTr0r/rsn AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999.-----^^^^^^^^^^^^^^&&&&&&  H % %+++++")+++2bi,29: $")D29BBI>W W W W W (W W W t29$_!_!_!_!_!_!_!_!Do o o o o o o o dT T T T T T T T n x)x)x)x)x)!2Ox)x)  x)vDDDDD&DDDN R R R R R "3R R  R jC C C C C 0C C C Lo o o o o .o o o d   r0