tjC8dZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZmZmZdd lmZdd lmZdd lmZmZmZdd lmZddlmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(Gdde"Z)eGddeZ*Gdde Z+GddeZ,eGdde*Z-ed Gd!d"e*e Z.Gd#d$eZ/Gd%d&e$Z0Gd'd(e!Z1gd)Z2dS)*zPyTorch PLBART model.N)nn)CrossEntropyLoss)initialization)Cache)GenerationMixin)BaseModelOutputSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)merge_with_config_defaults)capture_outputs)BartClassificationHead BartDecoder BartEncoderBartForCausalLMBartScaledWordEmbedding)'BigBirdPegasusForSequenceClassification)shift_tokens_right) PLBartConfigceZdZdS)PLBartScaledWordEmbeddingN__name__ __module__ __qualname__g/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/plbart/modular_plbart.pyrr/Dr$rcFeZdZUeed<dZdZddgZdZdZ dZ fdZ xZ S)PLBartPreTrainedModelconfigmodelTPLBartDecoderLayerPLBartEncoderLayerct|t|trt j|jdSdSN)super _init_weights isinstancePLBartForConditionalGenerationinitzeros_final_logits_bias)selfmodule __class__s r%r0z#PLBartPreTrainedModel._init_weights=sQ f%%% f< = = 2 K0 1 1 1 1 1 2 2r$) r r!r"r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnr0 __classcell__r8s@r%r(r(3so &*#-/CDN222222222r$r(ceZdZdS) PLBartEncoderNrr#r$r%rCrCCr&r$rCceZdZdS) PLBartDecoderNrr#r$r%rErEGr&r$rEcbeZdZdddZdeffd ZdZdZee e dde j dzd e j dzd e j dzd e j dzd ee jdzd edzde jdzde jdzdedzdeedee j ezfdZxZS) PLBartModelz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightr)c\t||j|j}}|jrt j|jnd}t||j|||_ t||_ t||_ |dS)Ng?) embed_scale)r/__init__ pad_token_id vocab_sizescale_embeddingmathsqrtd_modelrsharedrCencoderrEdecoder post_init)r6r) padding_idxrLrIr8s r%rJzPLBartModel.__init__Rs    "("5v7HZ 393IRdi///s / FNKepqqq $V,, $V,,  r$c|jSr.)rQ)r6s r%get_input_embeddingsz PLBartModel.get_input_embeddings^s {r$cX||_|j|j_|j|j_dSr.)rQrR embed_tokensrS)r6values r%set_input_embeddingsz PLBartModel.set_input_embeddingsas' $(K !$(K !!!r$N input_idsattention_maskdecoder_input_idsdecoder_attention_maskencoder_outputspast_key_values inputs_embedsdecoder_inputs_embeds use_cachekwargsreturnc ||t||jj}||jd |||d| }nct |t sNt |dt |dkr|dndt |dkr|dnd}|jd |||d|||| d| } t| j | j | j | j | j |j |j |j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)r\r]rbrrr)last_hidden_state hidden_states attentions)r\r]encoder_hidden_statesencoder_attention_maskrarbrd)rhradecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_staterkencoder_attentionsr#)rr)rKrRr1r lenrSr rhrarirjro) r6r\r]r^r_r`rarbrcrdredecoder_outputss r%forwardzPLBartModel.forwardfsYP  $)>)F 29dk>V W W   "/;t|0#-+00 00OO O_== -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO '$,  '1"1!"4#1+/      "-?+;"1"?.9,=&5&G"1"?.9    r$) NNNNNNNNN)r r!r"_tied_weights_keysrrJrWr[rrrtorch LongTensorTensorlist FloatTensorrboolr rtupler rtr@rAs@r%rGrGKs(7'6 |      000  .226596::>(,26:>!%J J #d*J (4/J !+d2 J !& t 3 J e/047 J J (4/J  %047J $;J +,J  u| 1 1J J J ^_ J J J J J r$rGzv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introceZdZdZdgZddiZdeffd Z dd ed edzd e d e j ffd Z d ed dfdZ eee ddejdzdejdzdejdzdejdzdeejdzdedzdejdzdejdzdejdzde dzdeed eejezfdZdejfdZxZS)r2r*r5zlm_head.weightzmodel.shared.weightr)clt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)Nr5rF)bias)r/rJrGr*register_bufferrvzerosrQnum_embeddingsrLinearrPlm_headrT)r6r)r8s r%rJz'PLBartForConditionalGeneration.__init__s     ((  0%+q$*BSBb>c2d2deeey1B1QX]^^^  r$NTnew_num_tokenspad_to_multiple_of mean_resizingrfct|||}||jjd|S)Nr)r/resize_token_embeddings_resize_final_logits_biasweightshape)r6rrrnew_embeddingsr8s r%rz6PLBartForConditionalGeneration.resize_token_embeddingssG88I[]jkk &&~'<'B1'EFFFr$c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nr)device)dimr5)r5rrvrrcatr)r6rold_num_tokensnew_bias extra_biass r%rz8PLBartForConditionalGeneration._resize_final_logits_biass/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r$r\r]r^r_r`rarbrclabelsrdrec | ||t| |jj}|j|f|||||||| d| } || j} | |j| jz} d}| Kt}|| d|jj | d}t|| | j | j| j| j| j| j| j S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N)r]r^r`r_rarbrcrdr) losslogitsrarmrnrorprkrq)rr)rKr*rrhr5torrviewrLr rarmrnrorprkrq)r6r\r]r^r_r`rarbrcrrdreoutputs lm_logitsmasked_lm_lossloss_fcts r%rtz&PLBartForConditionalGeneration.forwards3@   (-B-J$6vt{?W$X$X!&0dj  ' )/+#9+'"7 ' '  ' ' LL!:;;  6 9 9):J K KK   '))H%XinnR9O&P&PRXR]R]^`RaRabbN#3")"?&9$5&-&G")"?&9    r$c6t||jjSr.)rr)rK)r6rs r%%prepare_decoder_input_ids_from_labelszDPLBartForConditionalGeneration.prepare_decoder_input_ids_from_labels>s!&$+*BCCCr$)NT) NNNNNNNNNN)r r!r"r:_keys_to_ignore_on_load_missingrurrJintr{r Embeddingrrrrrrvrwrxryrzrr rr|r rtrr@rAs@r%r2r2s:  ':&;#/|ae!7:TzY] <<<<<< .226596::>(,26:>&*!%_ _ #d*_ (4/_ !+d2 _ !& t 3 _ e/047 _ _ (4/_  %047_  t#_ $;_ +,_  u|  ._ _ _ ^_ _ BDELDDDDDDDDr$r2ceZdZdS)PLBartClassificationHeadNrr#r$r%rrBr&r$rceZdZfdZxZS)PLBartForSequenceClassificationc :tjdi|dS)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr#r/rt super_kwargsr8s r%rtz'PLBartForSequenceClassification.forwardGs'4 '','''''r$)r r!r"rtr@rAs@r%rrFs8(((((((((r$rc>eZdZeefdZxZS)PLBartForCausalLMc :tjdi|dS)aF labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nr#rrs r%rtzPLBartForCausalLM.forwardes'2 '','''''r$)r r!r"rrrtr@rAs@r%rrdsM((((^(((((r$r)rr2rrGr()3__doc__rNrvrtorch.nnrrr3 cache_utilsr generationrmodeling_outputsr r r modeling_utilsr processing_utilsr utilsrrr utils.genericrutils.output_capturingrbart.modeling_bartrrrrr(bigbird_pegasus.modeling_bigbird_pegasusrmbart.modeling_mbartrconfiguration_plbartrrr(rCrErGr2rrr__all__r#r$r%rs %%%%%%&&&&&& )))))) .-----&&&&&&IIIIIIIIII777777555555_^^^^^555555......     7    2 2 2 2 2O 2 2 2     K        K   g g g g g 'g g g T DDDDDDDDDD%:ODDDD  DDN     5   (((((&M(((<((((((((:   r$