tj ldZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'e$j(e)Z*dej+de,de,fdZ-Gddej.Z/Gddej0Z1GddeZ2GddeZ3Gddej0Z4Gdd ej0Z5e#Gd!d"e!Z6Gd#d$e6Z7Gd%d&e6Z8e#Gd'd(e6Z9e#d)*Gd+d,e6eZ:e#d-*Gd.d/e6Z;e#Gd0d1e6Z<Gd2d3e6Z=Gd4d5e6eZ>gd6Z?dS)7zPyTorch MVP model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)PreTrainedModel)auto_docstringloggingtorch_compilable_check) MvpConfig input_ids pad_token_iddecoder_start_token_idc||j}|ddddf|ddddf<||dddf<|td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrr shifted_input_idss b/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mvp/modeling_mvp.pyshift_tokens_rightr*.s"++IO<<(CRC06688aaae4aaadLMMM""#4#(HPUPZcgcncufS"oo L(11!44Lww|dk9:::r+)rN) __name__ __module__ __qualname____doc__intr5r=TensorrC __classcell__r7s@r)r-r-?sFsF3FFFFFFmq ; ; ;?B ;V[VbeiVi ; ; ; ; ; ; ; ; ; ;r+r-ceZdZdZ ddedededzd edzd edzd edzf fd Z dd ej dej dzde dzdej dzdej dzdede ej ej dze ej dzffdZ xZ S) MvpAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias layer_idxct||_||_||_||z|_|j|z|jkrt d|jd|d|jdz|_||_||_ tj ||||_ tj ||||_ tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rU)r4r5rQrRrShead_dimr&scalingrTrVrLineark_projv_projq_projout_proj)r6rQrRrSrTrUrVr7s r)r5zMvpAttention.__init__]s  "" !Y. MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r+ hidden_stateskey_value_statespast_key_valuesattention_mask attn_promptoutput_attentionsreturnc |du}|\} } } |||jz} d} |Ht|tr1|j|j} |r|j}n |j }n|}|r|n|}|r3|1| r/|j |jj }|j |jj }n| |}||}|| d|j|jdd}|| d|j|jdd}|E||||j\}}|r$t|trd|j|j<|t)j|d| ddd|gd}t)j|d| ddd|gd}|`t)j| d| |dd|j}t)j||gd}| |jzd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t)j| |dd}|| |jz| |fkr2t9d | |jz| |fd |||| d| |fkr+t9d | d| |fd ||| |j| ||z}|| |jz| |}t:j|d}|r=|| |j| |}|| |jz| |}nd}t:j ||j |j! }t)j||}|| |jz| |jfkr5t9d | |j| |jfd ||| |j| |j}|dd}|| | |j"}|#|}||fS)z#Input shape: Batch x Time x ChannelNFr"rr2Trdimz$Attention weights should be of size z , but is z!Attention mask should be of size ptrainingz `attn_output` should be of size )$sizer^rZ isinstancer is_updatedgetrVcross_attention_cacheself_attention_cachelayerskeysvaluesr\r]viewrRrY transposeupdater=catrAzerostor<reshapebmmr&r functionalsoftmaxrSrlrQr_)r6r`rarbrcrdrekwargsis_cross_attentionrDtgt_len_ query_statesrocurr_past_key_valuescurrent_states key_states value_states prompt_mask proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r)rCzMvpAttention.forwardzs.T9',,..Wa{{=11DL@   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66L#b$.$-PPZZ[\^_``J',,S"dndmTT^^_`bcddL*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>  "KN$9$9#r2r$J$JJ#W]^___J 9k!n&;&;CR&L&Ll%[abcccL)#k#q';q>;N;Nq;Q;QRRUUVdVkll !&K+Hr!S!S!SDN*B > #((gt~t}UU__`acdee +|+Z8 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4?"(";9T^V-CDD9V3T^DD " T^ < <r+Fr`rcself_attn_promptreNrfc|}|||||\}}tj||j|j}||z}||}|}|||}tj||j|j}| |}tj||j|j}||z}| |}|j tj kr_tj|s9tj|j jdz }tj|| |}||fS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, encoder_attention_heads, pro_len, head_dim)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r`rcrdrerji)minmax)rrr~rSrlrrrrrrr;r=float16isfiniteallfinforclamp)r6r`rcrreresidualr clamp_values r)rCzMvpEncoderLayer.forwardsr$!&*nn')(/ '5' ' # |  --mt|VZVc-dd  =0 11-@@  **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0 --m<<  %- / /}8U8U8Y8Y8[8[ /+m&9::>EK!K K<[YYYMl**r+F) rFrGrHrr5r= FloatTensorrrrCrLrMs@r)rrs=y======**/ )+)+()+))+ + )+  $; )+ u %"3d":: ; )+)+)+)+)+)+)+)+r+rceZdZddeffd Z ddejdejdzdejdzd ejdzd ejdzd ejdzd edzd edzdedzde ej e ej ej fdzffdZ xZ S)MvpDecoderLayerNrct|j|_t |j|j|jd||_|j|_t|j |_ |j |_ tj|j|_t |j|j|jd||_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rQrRrSrTrV)rSrTrV)r4r5rrQrOdecoder_attention_headsrrrSr rrrrrr encoder_attnencoder_attn_layer_normr[decoder_ffn_dimrrr)r6rrVr7s r)r5zMvpDecoderLayer.__init__+s %n4,    ~ #F$>?"(";$&L$@$@!( N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r+FTr`rcencoder_hidden_statesencoder_attention_maskrcross_attn_promptrbre use_cacherfc  |} ||||||\}} tj||j|j}| |z}||}d} |f|} |||||||\}} tj||j|j}| |z}||}|} || |}tj||j |j}| |}tj||j|j}| |z}| |}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. cross_attn_prompt (`torch.FloatTensor`): prompt of cross attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r`rbrcrdrerjN)r`rarcrdrbre) rrr~rSrlrrrrrrrr)r6r`rcrrrrrbrerrrself_attn_weightscross_attn_weightsoutputss r)rCzMvpDecoderLayer.forwardGs>!,0>>'+)(/ ,:, , ( ( --mt|VZVc-dd  =0 11-@@ " ,$H040A0A+!65- /"3 1B11 -M-M11-4 >Gr+N)NNNNNNFT) rFrGrHrr5r=rKr rrrrCrLrMs@r)rr*sE==y======>/3596:0415(,).!%LL|L t+L %|d2 L !& t 3 L  ,- L!<$.LL $;L$;L u %(95;L(L"MPT"TT ULLLLLLLLr+rcXeZdZdZdedededeffd Zdejdejfd Z xZ S) MvpClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcttj|||_tj||_tj|||_dS)Nrk)r4r5rr[denseDropoutrSr_)r6rrrrr7s r)r5zMvpClassificationHead.__init__sY Yy)44 zN333  )[99 r+r`rfc||}||}tj|}||}||}|Sr)rSrr=tanhr_)r6r`s r)rCzMvpClassificationHead.forwards[ ]33  =11  =11  ]33  m44 r+) rFrGrHrIrJrr5r=rKrCrLrMs@r)rrs77 : : : :  : : : : : :U\elr+rcReZdZdZfdZdejdeejfdZxZ S) MvpPromptz)Layer-wise prompt for encoder or decoder.c t|j|_||_||_|j|z|_tj|j |_ tj |j|j|_ tj tj |j|jtjtj |j|dz|jz|_dS)Nrr2)r4r5 prompt_length num_layersrRrrYrrrS Embeddingprompt_embedding Sequentialr[prompt_mid_dimGELU prompt_trans)r6rrrRr7s r)r5zMvpPrompt.__init__s #1$")3 zFN333 " V-A6> R RM Ifnf&; < < GII If+Z!^fn-L M M  r+ prompt_idsrfc2|||}||j|jdz|j|j}||}|gd d}|S)Nr2)rr2rr) rrrvrrrRrYrSpermutesplit)r6rprompts r)rCzMvpPrompt.forwards""4#8#8#D#DEET/11DdnVZVcddf%% --33A66 r+) rFrGrHrIr5r=rKrrCrLrMs@r)rrsi33      %,53Fr+rcHeZdZUeed<dZdZfdZedZ xZ S)MvpPreTrainedModelrmodelTct|t|trt j|jdSdSr)r4 _init_weightsrnMvpForConditionalGenerationinitzeros_final_logits_bias)r6moduler7s r)rz MvpPreTrainedModel._init_weightssQ f%%% f9 : : 2 K0 1 1 1 1 1 2 2r+c|jj}tjgddddd|gg|j}|||d}|S)N)r r2r r2r<)rcr)rrr=tensorr<ne)r6 pad_tokenr dummy_inputss r)rzMvpPreTrainedModel.dummy_inputssaK, L"2"2"2Q2q)4L!MVZVabbb 'll955"  r+) rFrGrHr__annotations__base_model_prefixsupports_gradient_checkpointingrpropertyrrLrMs@r)rrsm &*#22222 Xr+rceZdZdZddedejdzdedzffd Z dde j dzd e j dzd e j dzd edzd edzd edzde ezfdZxZS) MvpEncodera Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MvpEncoderLayer`]. Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFr embed_tokens use_promptctj|_j|_j}j|_j|_ j rtj |nd|_ tjj||j|_t%j||_tjfdt+jD|_tj||_||_|r,j|_t9jj|_d|_| dS)N?c.g|]}tS)r).0rrs r) z'MvpEncoder.__init__..s!$c$c$c_V%<%<$c$c$cr+F)!r4r5rSencoder_layerdrop layerdroprr padding_idxmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerr vocab_sizerr-embed_positions ModuleListrangeencoder_layersrsrlayernorm_embeddingrrrrrgradient_checkpointing post_init)r6rrrrQr7s ` r)r5zMvpEncoder.__init__sB    ~ 1N !.$*$B!393IR49Y///sL):ItGWXX<  *   m$c$c$c$ceFLaFbFb$c$c$cdd #%< #:#: $  !'!5D $-%.%%D ! ',# r+rrc inputs_embedsreoutput_hidden_states return_dictrfc ||n |jj}||n |jj}||n |jj}||t d|&|}|j} |d| d}n=|,|dd} |dddddf}nt d||||j z}| |} || z} | | } tj | |j|j} |jrFt#j|j|j} || } |t/|j| |}|rdnd}|rdnd}t1|jD]p\}}|r|| fz}d}|jr!t#jg}||jkrd }|rd }n&|| ||jr| |nd| }|d } |r ||d fz}q|r|| fz}|st9d| ||fDSt;| ||S)a8 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embedsrj)rrrcrFT)NN)rrerrc3K|]}||V dSrrrvs r) z%MvpEncoder.forward..{s(eeqWXWdWdWdWdWdeer+last_hidden_stater` attentions)rrerrr&r$rvrmrr r rrr~rSrlrr=r>rr{r<rr enumeratersrandrrr)r6rrcrrerrrinput input_shape embed_posr`rrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputss r)rCzMvpEncoder.forward sP2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "E+K!r;r?;;II  &',,..ss3K!!!!QQQ(+EETUU U   --i884;KKM((// % 1 00??  --mt|VZVc-dd  ? Ad&899<>N fee]NN$Seeeee e+>Vd    r+NF)NNNNNN)rFrGrHrIrrrrr5r= LongTensorrKrrrrCrLrMs@r)rrsy t8K`dgk`kF.2.226)-,0#'t t #d*t  t+t (4/ t  $; t #Tk t D[t   t t t t t t t t r+rceZdZdZddededzffd Z ddejdzdej dzd ej dzd ejdzd e dzd ej dzd edzdedzdedzdedzde e zfdZxZS) MvpDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MvpDecoderLayer`] Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt FrrNctj|_j|_j|_j|_j rtj j nd|_ tjjj |j|_t%jj |_tjfdt+jD|_tjj |_||_|rLj|_t9jj|_t9jj|_d|_ |!dS)Nrc2g|]}t|S))rV)r)rirs r)rz'MvpDecoder.__init__..s&$p$p$pa_Vq%I%I%I$p$p$pr+F)"r4r5rSdecoder_layerdroprrrrmax_target_positionsrrrrr rrr rr-r r r decoder_layersrsrrrrrrrrrr)r6rrr7s ` r)r5zMvpDecoder.__init__si    ~ 1!.$*$B!8>8NW49V^444TWL):FNDL\]]<  * N  m$p$p$p$pSXY_YnSoSo$p$p$pqq #%<#?#? $  !'!5D $-%.%%D ! &/%.&&D " ',# r+rrcrrrbrrrerrrfc ||n |jj}| | n |jj} ||n |jj}| | n |jj} ||t d|&|} |j} |d| d}n=|,|dd} |dddddf} nt d|| ||j z}|j r%|j r|rtdd}|r[|Y| |jjr6t!t#|jt#|jnt#|j}||nd}t'|j||| }||t)|j||| }|| |}||z}||}t.j||j|j }|jr[t7j|j|j}| |}|!|}| rd nd}|rd nd}|r|d nd}tE|j#D]\}}| r||fz }|j r t7j$g}||j%kr4||||||jr||nd|jr||nd||| }|d}|r||dfz }| ||dfz }| r||fz }| stMd|||||fDStO|||||S)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer"zEYou have to specify either decoder_input_ids or decoder_inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rr)rrrcrb)rrrcrrjr)rrrrbrerrr2c3K|]}||V dSrrrs r)rz%MvpDecoder.forward..Ws0= ===r+)rrbr`rcross_attentions)(rrerrrr&r$rvrmrr rrllogger warning_onceis_encoder_decoderr r get_seq_lengthrrr rrr~rSrr=r>rr{r<rrrrsrrrr)r6rrcrrrbrrrerrrrr r8 positionsr`rrrall_hidden_statesall_self_attnsall_cross_attentionsr$ decoder_layerr'r(s r)rCzMvpDecoder.forwards@2C1N--TXT_Tq$8$D $+Jj "+!6IIDKstt t  "E#/K!r;r?;;II  &',,..ss3K!!!!QQQ(+EEdee e   --i884;KKM  & "4= " "##p"  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  FUE`!?!?!A!A!Afg+;')+     ! ,1G1S%>{+5&; &&& "((0FGG % 1 00??  --mt|VZVc-dd  ? Cd&899<:>(,26!%)-,0#'u u #d*u  t+u  %047 u !& 04 7 u  u (4/u $;u  $;u #Tku D[u  : :u u u u u u u u r+r,cLeZdZdgZdddZdeffd ZdZdZdZ e dd e j d zd e j d zd e j d zd e j d zdee jd zded zde jd zde jd zded zded zded zded zdeezfdZxZS)MvpModelrz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightrcNt||j|j}}|j|_t j||j||_t||j|_ t||j|_ | dSr)r4r5rr rrrrsharedrencoderr,decoderr)r6rrr r7s r)r5zMvpModel.__init__ms    "("5v7HZ  +l:v~{KK !&&*;<< !&&*;<<  r+c|jSr)rBr6s r)get_input_embeddingszMvpModel.get_input_embeddingszs {r+cX||_|j|j_|j|j_dSr)rBrCrrDr6values r)set_input_embeddingszMvpModel.set_input_embeddings}s' $(K !$(K !!!r+c |js Jd|d|jjd|jjd|jjddS)NzHIf you want to use lightweight tuning, make sure that `use_prompt=True`.FT)rrequires_grad_rCrrDrrFs r)set_lightweight_tuningzMvpModel.set_lightweight_tunings}jj jjj E""" %44T::: %44T::: &55d;;;;;r+Nrrcdecoder_input_idsdecoder_attention_maskencoder_outputsrbrdecoder_inputs_embedsrrerrrfc |8|6|tdt||jj|jj}| | n |jj} | | n |jj} | | n |jj} | | n |jj} || |||| | | }ne| rct|tsNt|dt|dkr|dndt|dkr|dnd}| |||d|||| | | |  }| s||zSt|j|j|j|j|j|j|j|j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)rrcrrerrrrr2r rrcrrrbrrrerr)rrbdecoder_hidden_statesdecoder_attentionsr5encoder_last_hidden_staterencoder_attentions)r&r*rrr rerrrrCrnrlenrDrrrbr`rr5)r6rrcrOrPrQrbrrRrrerrrdecoder_outputss r)rCzMvpModel.forwardsT  $)>)F  U !34;3T[5W!! 2C1N--TXT_Tq$8$D $+Jj "+!6IIDK&'6'6 y      000 <<<.2.259:>:>(,26:>!%)-,0#'g g #d*g  t+g !+d2 g !& 04 7 g e/047 g g (4/g  %047g $;g  $;g #Tkg D[g  # #g g g ^g g g g g r+r@ze The MVP Model with a language modeling head. Can be used for various text generation tasks. ) custom_introceZdZddiZdeffd Z ddededzd ed ej ffd Z ded dfd Z d Z e ddejdzdejdzdejdzdejdzdeejdzdedzdejdzdejdzdejdzdedzdedzdedzdedzd eezfdZdejfdZxZS)rlm_head.weightzmodel.shared.weightrclt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)NrrFrX)r4r5r@rregister_bufferr=rzrBr.rr[rlm_headrrs r)r5z$MvpForConditionalGeneration.__init__s    f%%  0%+q$*BSBb>c2d2deeey1B1QX]^^^  r+NTnew_num_tokenspad_to_multiple_of mean_resizingrfcxt|||}|||Sr)r4resize_token_embeddings_resize_final_logits_bias)r6rerfrgnew_embeddingsr7s r)riz3MvpForConditionalGeneration.resize_token_embeddingss<88I[]jkk &&~666r+c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nr"rrrhr)rr$r=rzr<ryrc)r6reold_num_tokensnew_bias extra_biass r)rjz5MvpForConditionalGeneration._resize_final_logits_biass/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r+cl|j|jddSr)rrNrdrMrFs r)rNz2MvpForConditionalGeneration.set_lightweight_tuning2 ))+++ ##E*****r+rrcrOrPrQrbrrRlabelsrrerrc | | n |jj} | G| rtdd} |'|%t | |jj|jj}|||||||||| | | |  }||d|j z}d}| Kt}|| d|jj | d}| s|f|ddz}||f|zn|St|||j|j|j|j|j|j|j S) a\ decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example of summarization: Fine-tuning a model ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.", ... return_tensors="pt", ... ) >>> labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"] >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... generated_ids = model.generate(**inputs) >>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) rcrOrQrPrbrrRrrerrrr"r losslogitsrbrUrVr5rWrrX)rrr6warningr*rr rrdrrrvr rrbrUrVr5rWrrX)r6rrcrOrPrQrbrrRrsrrerrrr lm_logitsmasked_lm_lossloss_fctoutputs r)rCz#MvpForConditionalGeneration.forwardsR&1%:>(,26:>*.!%)-,0#'v v #d*v  t+v !+d2 v !& 04 7 v e/047 v v (4/v  %047v  4'v $;v  $;v #Tkv D[v  !v v v ^v phELhhhhhhhhr+rz Mvp model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c:eZdZdeffd ZdZe ddejdzdej dzdejdzdejdzd e ej dzd ej dzd ej dzd ejdzd e dzde dzde dzde dzde ezfdZxZS)MvpForSequenceClassificationrc tj|fi|t||_t |j|j|j|j|_| dSr) r4r5r@rrr num_labelsclassifier_dropoutclassification_headr)r6rrr7s r)r5z%MvpForSequenceClassification.__init__sq**6***f%% #8 N N    % $ $   r+cl|j|jddSr))rrNrrMrFs r)rNz3MvpForSequenceClassification.set_lightweight_tunings3 ))+++  //66666r+NrrcrOrPrQrrRrsrrerrrfc ~| | n |jj} |d} ||td|jj||||||||| | | |  }|d}||jj|j }ttj | ddkd||ddf}t|jd|jdzdkd||dd |d ddd ddf}||}d}|n|jjp|jjdkr d |j_nS|jjdkr7|jtjks|jtjkr d |j_n d |j_|jjd kr\t1}|jjdkr1|||}n|||}n|jjd krLt5}||d |jj|d }n*|jjd krt7}|||}| s|f|ddz}||f|zn|St9|||j|j|j|j |j!|j"|j# S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: Fine-tuning a model on `num_labels` classes ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForSequenceClassification >>> num_labels = 2 # for example, this is a binary classification task >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForSequenceClassification.from_pretrained("RUCAIBox/mvp", num_labels=num_labels) >>> inputs = tokenizer("Classify: Hello, my dog is cute", return_tensors="pt") >>> labels = torch.tensor(1) # the real label for inputs >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax() ``` NFz8Passing input embeddings is currently not supported for rcrOrPrQrrRrrerrrrz7All examples must have the same number of tokens.z3Each example must contain at least one token.r" regressionsingle_label_classificationmulti_label_classificationru)$rrNotImplementedErrorr7rFreq eos_token_idr{r<rr=unique_consecutivesumnumelr$rvrmr problem_typerr;r?rJrsqueezerrrrbrUrVr5rWrrX)r6rrcrOrPrQrrRrsrrerrrrr`eos_maskselectedsentence_representationrwrvr{r|s r)rCz$MvpForSequenceClassification.forwardsvJ&1%Kbdd ** )/#9+'"7/!5#     << 899<<]=QRR  $X\\!__ 5 5 ; ; = = B E   !111- N1 !4Q!7 71 < A   #+-- 0B0B10E0Er=K]K]^`KaKa"b"bcdcdcdfhjkjkjkck"l))*ABB  {'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x// FY,F)-)9TGf$$v E.#3")"?&9$5&-&G")"?&9    r+r[)rFrGrHrr5rNrr=r*rKr^rrrrrCrLrMs@r)rrs y      777.2.259:>:>26:>*.!%)-,0#'N N #d*N  t+N !+d2 N !& 04 7 N e/047 N (4/N  %047N  4'N $;N  $;N #TkN D[N  0 0N N N ^N N N N N r+rcJeZdZfdZdZe ddejdzdejdzdejdzdejdzde ej dzd ejdzd ejdzd ej dzd ej dzd e dzde dzde dzde dzde e zfdZxZS)MvpForQuestionAnsweringct|d|_|j|_t||_t j|j|j|_| dSr1) r4r5rr@rrr[ hidden_size qa_outputsrrs r)r5z MvpForQuestionAnswering.__init__Esm     +f%% )F$68IJJ r+cl|j|jddSr))rrNrrMrFs r)rNz.MvpForQuestionAnswering.set_lightweight_tuningQs2 ))+++ &&u-----r+NrrcrOrPrQstart_positions end_positionsrrRrrerrrfc | | n |jj} ||d} |||||||| | | | |  }|d}||}|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zd z }| s||f|ddz}||f|zn|St||||j |j |j|j|j|j|j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. Example: Fine-tuning a model for extrative question answering, and our model also supports generative question answering using `BartForConditionalGeneration` ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForQuestionAnswering >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForQuestionAnswering.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Answer the following question: Who was Jim Henson? [SEP] Jim Henson was a nice puppet", ... return_tensors="pt", ... ) >>> target_start_index = torch.tensor([18]) >>> target_end_index = torch.tensor([19]) >>> loss = model(**inputs, start_positions=target_start_index, end_positions=target_end_index).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1] >>> predict_answer = tokenizer.decode(predict_answer_tokens) ``` NFrrrr"rh) ignore_indexr2) rv start_logits end_logitsrbrUrVr5rWrrX)rrrrrr contiguousrYrmrrrrbrUrVr5rWrrX)r6rrcrOrPrQrrrrRrrerrrrsequence_outputrwrr total_loss ignored_indexr{ start_lossend_lossr|s r)rCzMvpForQuestionAnswering.forwardUsRV&1%:>371526:>!%)-,0#'F F <$&F  t+F !+d2 F !& 04 7 F e/047 F )D0F '$.F (4/F  %047F $;F  $;F #TkF D[F 4 4!F F F ^F F F F F r+rc(eZdZdZfdZdZxZS)MvpDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ct|t||_|dSr)r4r5r,rDrrs r)r5zMvpDecoderWrapper.__init__s@    !&))  r+c|j|i|Sr)rD)r6argsrs r)rCzMvpDecoderWrapper.forwardst|T,V,,,r+)rFrGrHrIr5rCrLrMs@r)rrsQ  -------r+rc2eZdZddiZfdZdZdZdZe dd e j dzd e j dzd e j dzd e j dzd e dzde j dzde j dzdedzdedzdedzdedzdee j zdeezfdZxZS)MvpForCausalLMraz!model.decoder.embed_tokens.weightcd|_d|_t|t ||_t j|j|j d|_ | dS)NTFrX) rTr8r4r5rrrr[rr rdrrs r)r5zMvpForCausalLM.__init__sp $)!    &v.. y!3V5FUSSS  r+c$|jjjSrrrDrrFs r)rGz#MvpForCausalLM.get_input_embeddingssz!..r+c(||jj_dSrrrIs r)rKz#MvpForCausalLM.set_input_embeddingss*/ '''r+cl|j|jddSr)rqrFs r)rNz%MvpForCausalLM.set_lightweight_tuningrrr+Nrrrcrrrbrrsrrerrlogits_to_keeprfc r| | n |jj} | | n |jj} | | n |jj} |j|||||||| | |  }|d}t | trt| dn| }| |dd|ddf}d}|Kt}|| d|jj | d}| s|f|ddz}||f|zn|St|||j|j|j|jS)ap labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MvpForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForCausalLM.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 50267] ```NrTrr"r)rvrwrbr`rr5)rrerrrrDrnrJslicerdrrvr rrbr`rr5)r6rrcrrrbrrsrrerrrrrr` slice_indicesrwrvr{r|s r)rCzMvpForCausalLM.forwardsN2C1N--TXT_Tq$8$D $+Jj &1%SV8W8Wk~ot444]k mAAA}aaa,?@AA  '))H8FKKDK,BCCV[[QS__UUD DY,F'+'7D7V##V C0#3!/)$5     r+) NNNNNNNNNNNr)rFrGrHr]r5rGrKrNrr=r*rKrr rrJrrrCrLrMs@r)rrs*,OP     ///000+++.2.2:>;?(,26*.!%)-,0#'-.O O #d*O  t+O  %047 O !& 1D 8 O  O (4/O  4'O $;O  $;O #TkO D[O el*O  2 2O O O ^O O O O O r+r)rrrrr@r)@rIrr=rtorch.nnrrrrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrconfiguration_mvpr get_loggerrFr6rKrJr*rr-ModulerOrrrrrrr,r@rrrrr__all__rr+r)rs AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999.-----DDDDDDDDDD((((((  H % % %, c [^    ";;;;;BL;;;6P2P2P2P2P229P2P2P2f:+:+:+:+:+0:+:+:+ziiiii0iiiZBI0 2*` ` ` ` ` #` ` ` Fa a a a a #a a a HL L L L L !L L L ^ \h\h\h\h\h"4o\h\h  \h~ a a a a a #5a a  a HX X X X X 0X X X x - - - - -* - - -h h h h h 'h h h V   r+