tj ddlZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2ddl3m4Z4e,j5e6Z7Gddej8Z9e)Gdde$Z:Gddej8Z; dCdej<dej=dej=dej=d ej=dzd!e>dzd"e>d#e&e(fd$Z?Gd%d&ej<Z@Gd'd(eZAGd)d*e:ZBGd+d,eZCGd-d.e:ZDd/ej=d0eEfd1ZFe)Gd2d3e:ZGe)d45Gd6d7e:eZHGd8d9ej<ZIe)d:5Gd;de:ZKe)d?5Gd@dAe:eZLgdBZMdS)DN)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSeq2SeqSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilingloggingtorch_compilable_check)merge_with_config_defaults)OutputRecordercapture_outputs) PLBartConfigc VeZdZdZd dededededzffd Zd ejffd Z xZ S) PLBartScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?num_embeddings embedding_dim padding_idx embed_scaleNc\t|||||_dSN)super__init__r-)selfr*r+r,r- __class__s h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/plbart/modeling_plbart.pyr1z"PLBartScaledWordEmbedding.__init__Bs-  DDD& input_idscVt||jzSr/)r0forwardr-)r2r6r3s r4r8z!PLBartScaledWordEmbedding.forwardFs!wwy))D,<<(HPUPZcgcncufS"oo L(11!44Lww|dk9:::r5)rN) r:r;r<r=r>r1r@rAr8rBrCs@r4rZrZZsFsF3FFFFFFmq ; ; ;?B ;V[VbeiVi ; ; ; ; ; ; ; ; ; ;r5rZrQquerykeyvalueattention_maskscalingdropoutkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nrcr\rdimptrainingr%) sizer@matmul transposer functionalsoftmaxrrrz contiguous) rQrmrnrorprqrrrs attn_weights attn_outputs r4eager_attention_forwardrus**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r5ceZdZdZ ddededed ed ed ed edzd edzffd Z dde j de j dzde dzde j dzde e dee j e j dzff dZxZS)PLBartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrlFTN embed_dim num_headsrr is_decoderbias is_causalrF layer_idxc zt||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ ||_ |/|j r(t d|jjdt!j||||_t!j||||_t!j||||_t!j||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).ruzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)r0r1rrrrhead_dimrF ValueErrorrqrrrlogger warning_oncer3r:rLineark_projv_projq_projout_proj) r2rrrrrrrrFrr3s r4r1zPLBartAttention.__init__sY "" !Y.  MI %$. 8 83dn33%.333 }d* $""      ,4>+B,,,    i 94@@@ i 94@@@ i 94@@@  )YTBBB r5 hidden_stateskey_value_statespast_key_valuesrprsreturnc |du}|jdd}g|d|jR}|||dd} d} |Ht |t r1|j|j } |r|j } n |j } n|} |r|n|} |r3|1| r/| j |j j } | j |j j}n|| } || }g| jddd|jR}| |dd} ||dd}|E| | ||j \} }|r$t |t rd|j|j <t%j|jjt,}||| | ||f|jsdn|j|jd|\}}|jg|dR}||}||fS) z#Input shape: Batch x Time x ChannelNrcr%r\FTrl)rrrq)rdrrviewr}rLr is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacerF_attn_implementationrrzrrrqreshaperr)r2rrrrprsis_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacerrs r4r8zPLBartAttention.forwards.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66LF-crc2FBF FFH#22<(?(M K ,.E) ) %8$7      % $}>CC$,L % %  % % ! \*k);;;;;;FFHH mmK00 L((r5)rlFTFNNNNN)r:r;r<r=r>r?boolr&r1r@rAr rrtupler8rBrCs@r4rrs]GG  &* $%C%C%C%C %C  %C  %C%Ct#%C:%C%C%C%C%C%CT15(,.2 H)H)|H) ,-H) H)  t+ H)-.H) u|U\D00 1H)H)H)H)H)H)H)H)r5rcreZdZd dededzffd Zdejdejdee dej fd Z xZ S) rINrFrct|j|_t |j|j|j|||_tj |j|_ |j |_ t|j |_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)rrrrrFr)r0r1d_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrrr activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normr2rFrr3s r4r1zPLBartEncoderLayer.__init__s (n4,    %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r5rrprsrc |}|j|fd|i|\}}tj||j|j}||z}||}|}|||}tj||j|j}| |}tj||j|j}||z}| |}|j tj kr_tj|s9tj|j jdz }tj|| |}|S)Nrprxi)minmax)rrr~rrrzrrrrrrrar@float16isfiniteallfinforclamp)r2rrprsresidual_ clamp_values r4r8zPLBartEncoderLayer.forwardsv !)4>   )    q  --mt|VZVc-dd  =0 11-@@  **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0 --m<<  %- / /}8U8U8Y8Y8[8[ /+m&9::>EK!K K<[YYYMr5r/) r:r;r<r&r>r1r@ FloatTensorrrrAr8rBrCs@r4rIrIs==|=d ======&()+,   r5rIceZdZdZeedZdeffd Ze e e d de j dzde jdzde jdzd eed ef d ZxZS) PLBartEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`PLBartEncoderLayer`]. Args: config: PLBartConfig embed_tokens (nn.Embedding): output embedding )r attentionsrFcBtj|_j|_j}j|_j|_ j rtj |nd}tj||j||_t!j||_t%jfdt)jD|_t%j||_d|_|dS)Nr)r-c2g|]}t|S)r)rI.0irFs r4 z*PLBartEncoder.__init__..[($s$s$sQR%7!%L%L%L$s$s$sr5F)r0r1rrencoder_layerdrop layerdropr pad_token_idr,max_position_embeddingsmax_source_positionsscale_embeddingmathsqrtr( vocab_size embed_tokensrZembed_positionsr ModuleListrangeencoder_layersrrlayernorm_embeddinggradient_checkpointing post_init)r2rFrr-r3s ` r4r1zPLBartEncoder.__init__Hs    ~ 1N !.$*$B!.4.DMdi ***# 5  y$*:     @  *   m$s$s$s$sV[\b\qVrVr$s$s$stt #%< #:#: &+# r5Nr6rp inputs_embedsrsrc R|du|duz rtd|||}||dddddf}||j}||z}||}t j||j|j }t|j ||}t|j D];\}}d} |j r!tjg} | |jkrd} | s |||fi|}?"(";$&L$@$@!+ N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r5Trrpencoder_hidden_statesencoder_attention_maskr use_cachersrc |}|j|f||d|\}} tj||j|j}||z}||}|]|}|j|f|||d|\}} tj||j|j}||z}||}|}|| |}tj||j |j}| |}tj||j|j}||z}| |}|S)N)rrprx)rrpr) rrr~rrrzrrrrrrrr) r2rrprrrrrsrrs r4r8zPLBartDecoderLayer.forwards!*4>  +)      q  --mt|VZVc-dd  =0 11-@@  ! ,$H0t0 !65 /        M1M11-4r1r@rAr rrrr8rBrCs@r4rHrHs==|=d ======D/3596:(,!%//|/ t+/ %|d2 / !& t 3 /  /$;/+,/ ////////r5rHc8eZdZdZeeeddeedddZdeffd Z e e e dd e jd zd e jd zd e jd zd e jd zded zde jd zded zdeedefdZxZS) PLBartDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`PLBartDecoderLayer`] Args: config: PLBartConfig embed_tokens (nn.Embedding): output embedding r%r)index layer_namer)rrcross_attentionsrFc\tj|_j|_j|_j|_j rtj j nd}tjj |j||_t!jj |_t%jfdt)jD|_t%jj |_d|_|dS)Nr)rc2g|]}t|Sr)rHrs r4rz*PLBartDecoder.__init__..rr5F)r0r1rrdecoder_layerdroprrr,rmax_target_positionsrrrrr(rrrZrrrrdecoder_layersrrrrr)r2rFr-r3s ` r4r1zPLBartDecoder.__init__s    ~ 1!.$*$B!393IRdi///s 5  v~t/?[    @  * N  m$s$s$s$sV[\b\qVrVr$s$s$stt #%<#?#? &+# r5Nr6rprrrrrrsrc t|du|duz rtd|||}|r[|Y| |jjr6t t |jt |jnt |j}|dd\} } ||nd} tj | |j | z} |/ts!| | z} tj | | |j }t|tr|jn|}t|j|||}t!|j|||}||| | }||j }||z}||}t(j||j|j }t1|jD];\}}|jr tjg}||jkr,||||f|||d |}?? !O 0 0   ,;')+     ";;'1"7 " " " ((4JYe(ff LL!566 % 1 00??  --mt|VZVc-dd "+DK"8"8   C} &+jnn#&77)M%(> /# MM9++    r5)NNNNNNN)r:r;r<r=rHr#rrr&r1r"r$rr@rrArr rrrrr8rBrCs@r4rrsy,$n_A+VVV*N?!P^___ |0 .2.2:>:>(,26!%R R #d*R  t+R  %047 R !& 04 7 R  R (4/R $;R +,R  3R R R ^_ R R R R R r5rr6rc|}|td||dk|||ddz d}|d|}|ddddf|ddddf<||dddf<|S)z Shift input ids one token to the right, and wrap the last non pad token (the token) Note that PLBart does not have a single `decoder_start_token_id` in contrast to other Bart-like models. Nz1self.model.config.pad_token_id has to be defined.ir%rvrcr)cloner masked_fill_nesumrigathersqueeze)r6rprev_output_tokens index_of_eosdecoder_start_tokenss r4shift_tokens_rightr$cs #**LMMM##$6$$> MMM&)),77;;;BBQFQQRTUUL-44Q EEMMOO 2111crc6 : @ @ B Bqqq!""u3qqq!t r5cbeZdZdddZdeffd ZdZdZee e dde j dzd e j dzd e j dzd e j dzd ee jdzd edzde jdzde jdzdedzdeedee j ezfdZxZS) PLBartModelz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightrFc\t||j|j}}|jrt j|jnd}t||j|||_ t||_ t||_ |dS)Nr)r)r0r1rrrrrrr(sharedrencoderrdecoderr)r2rFr,rr-r3s r4r1zPLBartModel.__init__~s    "("5v7HZ 393IRdi///s / FNKepqqq $V,, $V,,  r5c|jSr/)r(r2s r4get_input_embeddingsz PLBartModel.get_input_embeddingss {r5cX||_|j|j_|j|j_dSr/)r(r)rr*r2ros r4set_input_embeddingsz PLBartModel.set_input_embeddingss' $(K !$(K !!!r5Nr6rpdecoder_input_idsdecoder_attention_maskencoder_outputsrrdecoder_inputs_embedsrrsrc ||t||jj}||jd |||d| }nct |t sNt |dt |dkr|dndt |dkr|dnd}|jd |||d|||| d| } t| j | j | j | j | j |j |j |j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)r6rprrr%r\)rrrr6rprrrrr)rrdecoder_hidden_statesdecoder_attentionsr encoder_last_hidden_staterencoder_attentions)r$rFrr)rLrlenr*rrrrrr ) r2r6rpr1r2r3rrr4rrsdecoder_outputss r4r8zPLBartModel.forwardsYP  $)>)F 29dk>V W W   "/;t|0#-+00 00OO O_== -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO '$,  '1"1!"4#1+/      "-?+;"1"?.9,=&5&G"1"?.9    r5 NNNNNNNNN)r:r;r<_tied_weights_keysr&r1r-r0r"r$rr@rrAlistrr rrrrrr8rBrCs@r4r&r&ws(7'6 |      000  .226596::>(,26:>!%J J #d*J (4/J !+d2 J !& t 3 J e/047 J J (4/J  %047J $;J +,J  u| 1 1J J J ^_ J J J J J r5r&zv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introceZdZdZdgZddiZdeffd Z dd ed edzd e d e j ffd Z d ed dfdZ eee ddejdzdejdzdejdzdejdzdeejdzdedzdejdzdejdzdejdzde dzdeed eejezfdZdejfdZxZS)rMrGrPlm_head.weightzmodel.shared.weightrFclt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)NrPr%Fr)r0r1r&rGregister_bufferr@zerosr(r*rrrlm_headrr2rFr3s r4r1z'PLBartForConditionalGeneration.__init__s     ((  0%+q$*BSBb>c2d2deeey1B1QX]^^^  r5NTnew_num_tokenspad_to_multiple_of mean_resizingrct|||}||jjd|S)Nr)r0resize_token_embeddings_resize_final_logits_biasrgrd)r2rIrJrKnew_embeddingsr3s r4rMz6PLBartForConditionalGeneration.resize_token_embeddingssG88I[]jkk &&~'<'B1'EFFFr5c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nrcr%rrvrP)rPrdr@rFrbcatrE)r2rIold_num_tokensnew_bias extra_biass r4rNz8PLBartForConditionalGeneration._resize_final_logits_biass/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r5r6rpr1r2r3rrr4labelsrrsc | ||t| |jj}|j|f|||||||| d| } || j} | |j| jz} d}| Kt}|| d|jj | d}t|| | j | j| j| j| j| j| j S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N)rpr1r3r2rrr4rrc losslogitsrr7r8r r9rr:)r$rFrrGrGrrPrrbrrrrrr7r8r r9rr:)r2r6rpr1r2r3rrr4rUrrsoutputs lm_logitsmasked_lm_lossloss_fcts r4r8z&PLBartForConditionalGeneration.forwards3@   (-B-J$6vt{?W$X$X!&0dj  ' )/+#9+'"7 ' '  ' ' LL!:;;  6 9 9):J K KK   '))H%XinnR9O&P&PRXR]R]^`RaRabbN#3")"?&9$5&-&G")"?&9    r5c6t||jjSr/)r$rFr)r2rUs r4%prepare_decoder_input_ids_from_labelszDPLBartForConditionalGeneration.prepare_decoder_input_ids_from_labelsjs!&$+*BCCCr5)NT) NNNNNNNNNN)r:r;r<rS_keys_to_ignore_on_load_missingr?r&r1r>rr EmbeddingrMrNr"r$rr@rrAr@rr rrrrr8r_rBrCs@r4rMrMs:  ':&;#/|ae!7:TzY] <<<<<< .226596::>(,26:>&*!%_ _ #d*_ (4/_ !+d2 _ !& t 3 _ e/047 _ _ (4/_  %047_  t#_ $;_ +,_  u|  ._ _ _ ^_ _ BDELDDDDDDDDr5rMcXeZdZdZdedededeffd Zdejdejfd Z xZ S) PLBartClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcttj|||_tj||_tj|||_dS)N)ry)r0r1rrdenseDropoutrrr)r2rdrerfrgr3s r4r1z!PLBartClassificationHead.__init__qsY Yy)44 zN333  )[99 r5rrc||}||}tj|}||}||}|Sr/)rrrir@tanhr)r2rs r4r8z PLBartClassificationHead.forward}s[ ]33  =11  =11  ]33  m44 r5r9rCs@r4rcrcns77 : : : :  : : : : : :U\elr5rcz PLBart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c0eZdZdeffd Zee ddejdzdej dzdejdzdejdzde ej dzd ej dzd ej dzd ejdzd e dzd e edeezfdZxZS)PLBartForSequenceClassificationrFc tj|fi|t||_t |j|j|j|j|_| dSr/) r0r1r&rGrcr num_labelsclassifier_dropoutclassification_headr)r2rFrsr3s r4r1z(PLBartForSequenceClassification.__init__sq**6*** (( #; N N    % $ $   r5Nr6rpr1r2r3rr4rUrrsrc J|d} ||td|jj|j|f||||||| d| } | d} ||jj| j} ttj | d dkd| | ddf}t|jd| jdzdkd|| dd | d ddd ddf}||}d}|||j}|jjp|jjdkr d |j_nS|jjdkr7|jtjks|jtjkr d |j_n d |j_|jjd kr\t/}|jjdkr1|||}n|||}n|jjd krLt3}||d |jj|d }n*|jjd krt5}|||}t7||| j| j| j| j| j | j!| j" S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for )rpr1r2r3rr4rrr%z7All examples must have the same number of tokens.z3Each example must contain at least one token.rc regressionsingle_label_classificationmulti_label_classificationrW)#NotImplementedErrorr3r:rGeqrF eos_token_idrrbr!r@unique_consecutivernumelrdrr{rr problem_typerprarfr>rr rrrrr7r8r r9rr:)r2r6rpr1r2r3rr4rUrrsrZreos_maskselectedsentence_representationrYrXr]s r4r8z'PLBartForSequenceClassification.forwardsHP  I  !:%d4>Kbdd '1dj  ' )/#9+'"7 ' '  ' '   << 899<<]=QRR  $X\\!__ 5 5 ; ; = = B E   !111- N1 !4Q!7 71 < A   #+-- 0B0B10E0Er=K]K]^`KaKa"b"bcdcdcdfhjkjkjkck"l))*ABB  YYv}--F{'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x//.#3")"?&9$5&-&G")"?&9    r5r>)r:r;r<r&r1rrr@rrAr@rrrrrrr8rBrCs@r4rnrnse |      .2.259:>:>26:>*.!%k k #d*k  t+k !+d2 k !& 04 7 k e/047 k (4/k  %047k  4'k $;k +,k  0 0k k k ^k k k k k r5rnc(eZdZdZfdZdZxZS)PLBartDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ct|t||_|dSr/)r0r1rr*rrHs r4r1zPLBartDecoderWrapper.__init__s@    $V,,  r5c|j|i|Sr/)r*)r2argsrss r4r8zPLBartDecoderWrapper.forwardst|T,V,,,r5)r:r;r<r=r1r8rBrCs@r4rr sQ  -------r5rzw PLBART decoder with a language modeling head on top (linear layer with weights tied to the input embeddings). c(eZdZddiZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e dzd e j dzde j dzdedzdee j zdeedeezfdZxZS)PLBartForCausalLMrCz!model.decoder.embed_tokens.weightcd|_d|_t|t ||_t j|j|j d|_ | dS)NTFr) rrr0r1rrGrr hidden_sizerrGrrHs r4r1zPLBartForCausalLM.__init__#sp $)!    )&11 y!3V5FUSSS  r5c$|jjjSr/rGr*rr,s r4r-z&PLBartForCausalLM.get_input_embeddings.sz!..r5c(||jj_dSr/rr/s r4r0z&PLBartForCausalLM.set_input_embeddings1s*/ '''r5Nrr6rprrrrrUrlogits_to_keeprsrc |jjd|||||||d| } | d} t| trt | dn| } || dd| ddf}d}|e||j}t}|| d|j j | d}t||| j | j| j| jS)aF labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```r6rNrc)rXrYrrrr r;)rGr*rLr>slicerGrrbrrrFrrrrrr )r2r6rprrrrrUrrrsrZr slice_indicesrYrXr]s r4r8zPLBartForCausalLM.forward4s)L>PTZ=O > )"7#9+' > >  > >   8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  YYv}--F'))H8FKKDK,BCCV[[QS__UUD0#3!/)$5     r5) NNNNNNNNr)r:r;r<r?r1r-r0rrr@rrArr rr>rrrrr8rBrCs@r4rrs} =     ///000.2.2:>;?(,26*.!%-.A A #d*A  t+A  %047 A !& 1D 8 A  A (4/A  4'A $;A el*A +,A  2 2A A A ^A A A A A r5r)rrMrnr&rE)Nrl)Nrcollections.abcrr@rtorch.nnrrrr rN activationsr cache_utilsr r r generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrrr r! utils.genericr"utils.output_capturingr#r$configuration_plbartr& get_loggerr:rrar(rErZModulerAr?rrrIrrHrr>r$r&rMrcrnrr__all__r;r5r4rs* $$$$$$ AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJBBBBBB999999GFFFFFFF&&&&&&877777EEEEEEEE......  H % % = = = = = = = = 2 2 2 2 2O 2 2 2;;;;;r|;;;B!%% I% <% % < % L4' % T\ %%'(%%%%8r)r)r)r)r)bir)r)r)j000003000fV V V V V )V V V rOOOOO3OOOd| | | | | )| | | ~%,c(g g g g g 'g g g T DDDDDDDDDD%:ODDDD  DDNry0 { { { { { &;{ {  { | - - - - -0 - - - Y Y Y Y Y -Y Y  Y x   r5