tjDdZddlZddlZddlmZddlZddlmZddlmZm Z m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4m5Z5ddl6m7Z7e/j8e9Z:dej;deZ?Gddej>Z@ dKdejAd ej;d!ej;d"ej;d#ej;dzd$eBdzd%eBd&e)e+fd'ZCGd(d)ejAZDGd*d+eZEGd,d-eZFGd.d/ejAZGe,Gd0d1e'ZHGd2d3eHZIGd4d5eHZJGd6d7eHZKGd8d9eHZLe,Gd:d;eHZMe,d<=Gd>d?eHeZNe,d@=GdAdBeHZOe,GdCdDeHZPGdEdFeHZQe,dG=GdHdIeHeZRgdJZSdS)LzPyTorch BART model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilingloggingtorch_compilable_check)merge_with_config_defaults)OutputRecordercapture_outputs) BartConfig input_ids pad_token_iddecoder_start_token_idc||j}|ddddf|ddddf<||dddf<|td||dk||S)z1 Shift input ids one token to the right. Nr&rz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r(r)r*shifted_input_idss d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/bart/modeling_bart.pyshift_tokens_rightr4:s"++IO<<(CRC06688aaae4aaadLMMM""#4#forwardr=)r@r(rBrCbszseq_lenrAs r3rMz&BartLearnedPositionalEmbedding.forwardUs  $?2A2.LC <&(>(HPUPZcgcncufS"oo L(11!44Lww|dk9:::r5)rN) __name__ __module__ __qualname____doc__intr?rGTensorrM __classcell__rAs@r3r7r7JsFsF3FFFFFFmq ; ; ;?B ;V[VbeiVi ; ; ; ; ; ; ; ; ; ;r5r7c VeZdZdZd dededededzffd Zd ejffd Z xZ S) BartScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?r8r9 padding_idx embed_scaleNc\t|||||_dSN)r>r?r\)r@r8r9r[r\rAs r3r?z BartScaledWordEmbedding.__init__js-  DDD&r5r(cVt||jzSr^)r>rMr\)r@r(rAs r3rMzBartScaledWordEmbedding.forwardns!wwy))D,<<>L=((6?([[L,|U33K''1--88::K  $$r5ceZdZdZ ddededed ed ed ed edzd edzffd Z dde j de j dzde dzde j dzde e dee j e j dzff dZxZS) BartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrbFTN embed_dim num_headsri is_decoderbias is_causalconfig layer_idxc zt||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ ||_ |/|j r(t d|jjdt!j||||_t!j||||_t!j||||_t!j||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rlzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)r>r?r}r~rihead_dimrr0rhrrrlogger warning_oncerArPrLineark_projv_projq_projout_proj) r@r}r~rirrrrrrAs r3r?zBartAttention.__init__sY "" !Y.  MI %$. 8 83dn33%.333 }d* $""      ,4>+B,,,    i 94@@@ i 94@@@ i 94@@@  )YTBBB r5 hidden_stateskey_value_statespast_key_valuesrgrjreturnc |du}|jdd}g|d|jR}|||dd} d} |Ht |t r1|j|j } |r|j } n |j } n|} |r|n|} |r3|1| r/| j |j j } | j |j j}n|| } || }g| jddd|jR}| |dd} ||dd}|E| | ||j \} }|r$t |t rd|j|j <t%j|jjt,}||| | ||f|jsdn|j|jd|\}}|jg|dR}||}||fS) z#Input shape: Batch x Time x ChannelNr,r&r<FTrb)rirh)r.rrviewrt isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationrzrqrirhreshaperwr)r@rrrrgrjis_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfaceryrxs r3rMzBartAttention.forwards.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66LF-crc2FBF FFH#22<(?(M K ,.E) ) %8$7      % $}>CC$,L % %  % % ! \*k);;;;;;FFHH mmK00 L((r5)rbFTFNNNNN)rPrQrRrSrTraboolr'r?rGrUr rrtuplerMrVrWs@r3r|r|s]GG  $( $%C%C%C%C %C  %C  %C%CT!%C:%C%C%C%C%C%CT15(,.2 H)H)|H) ,-H) H)  t+ H)-.H) u|U\D00 1H)H)H)H)H)H)H)H)r5r|creZdZd dededzffd Zdejdejdee dej fd Z xZ S) BartEncoderLayerNrrct|j|_t |j|j|j|||_tj |j|_ |j |_ t|j |_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)r}r~rirr)r>r?d_modelr}r|encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrir activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normr@rrrAs r3r?zBartEncoderLayer.__init__s &n4,    %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r5rrgrjrc |}|j|fd|i|\}}tj||j|j}||z}||}|}|||}tj||j|j}| |}tj||j|j}||z}| |}|j tj kr_tj|s9tj|j jdz }tj|| |}|S)Nrgroi)minmax)rrrurirqrrrrrrrErGfloat16isfiniteallfinforclamp)r@rrgrjresidual_ clamp_values r3rMzBartEncoderLayer.forwardsv !)4>   )    q  --mt|VZVc-dd  =0 11-@@  **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0 --m<<  %- / /}8U8U8Y8Y8[8[ /+m&9::>EK!K K<[YYYMr5r^) rPrQrRr'rTr?rG FloatTensorrrrUrMrVrWs@r3rrs==z=cDj======&()+,   r5rceZdZddededzffd Z ddejdejdzdejdzd ejdzd edzd e dzd e e d ejfdZ xZ S)BartDecoderLayerNrrc t|j|_t |j|j|jdd|||_|j|_t|j |_ |j |_ tj|j|_t |j|j|jd|||_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)r}r~rirrrr)rirrr)r>r?rr}r|decoder_attention_headsrrrir rrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrrrs r3r?zBartDecoderLayer.__init__8s  &n4,   ~ #F$>?"(";$&L$@$@!) N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r5Trrgencoder_hidden_statesencoder_attention_maskr use_cacherjrc |}|j|f||d|\}} tj||j|j}||z}||}|]|}|j|f|||d|\}} tj||j|j}||z}||}|}|| |}tj||j |j}| |}tj||j|j}||z}| |}|S)N)rrgro)rrgr) rrrurirqrrrrrrrr) r@rrgrrrrrjrrs r3rMzBartDecoderLayer.forwardWs!*4>  +)      q  --mt|VZVc-dd  =0 11-@@  ! ,$H0t0 !65 /        M1M11-4r?rrdenseDropoutrir)r@rrrrrAs r3r?zBartClassificationHead.__init__sY Yy)44 zN333  )[99 r5rrc||}||}tj|}||}||}|Sr^)rirrGtanhr)r@rs r3rMzBartClassificationHead.forwards[ ]33  =11  =11  ]33  m44 r5r`rWs@r3rrs77 : : : :  : : : : : :U\elr5rcneZdZUeed<dZdZddgZddgZdgZ dZ dZ dZ dZ fd Zed ZxZS) BartPreTrainedModelrmodelTzencoder.versionzdecoder.versionrrrct|t|trt j|jdSdSr^)r> _init_weightsrBartForConditionalGenerationinitzeros_final_logits_bias)r@rcrAs r3rz!BartPreTrainedModel._init_weightssQ f%%% f: ; ; 2 K0 1 1 1 1 1 2 2r5c|jj}tjgddddd|gg|j}|||d}|S)N)r r<r r<rF)rgr()rr)rGtensorrFne)r@ pad_tokenr( dummy_inputss r3rz BartPreTrainedModel.dummy_inputssaK, L"2"2"2Q2q)4L!MVZVabbb 'll955"  r5)rPrQrRr'__annotations__base_model_prefixsupports_gradient_checkpointing"_keys_to_ignore_on_load_unexpected_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrpropertyrrVrWs@r3rrs &*#*;=N)O&,.AB#4"5N!22222 Xr5rceZdZdZdS)PretrainedBartModelc:tjdtdSNz_The class `PretrainedBartModel` has been depreciated, please use `BartPreTrainedModel` instead.warningswarn FutureWarningr@s r3__init_subclass__z%PretrainedBartModel.__init_subclass__& m      r5NrPrQrRrr5r3rr#     r5rceZdZdZdS)BartPretrainedModelc:tjdtdSrrrs r3rz%BartPretrainedModel.__init_subclass__r r5Nr r r5r3rrr r5rceZdZdZeedZdeffd Ze e e d de j dzde jdzde jdzd eed ef d ZxZS) BartEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`BartEncoderLayer`]. Args: config: BartConfig embed_tokens (nn.Embedding): output embedding )r attentionsrcBtj|_j|_j}j|_j|_ j rtj |nd}tj||j||_t!j||_t%jfdt)jD|_t%j||_d|_|dS)NrZr\c2g|]}t|S)r)r.0irs r3 z(BartEncoder.__init__..'$q$q$qq%5f%J%J%J$q$q$qr5F)r>r?riencoder_layerdrop layerdroprr)r[max_position_embeddingsmax_source_positionsscale_embeddingmathsqrtrY vocab_size embed_tokensr7embed_positionsr ModuleListrangeencoder_layersrrlayernorm_embeddinggradient_checkpointing post_init)r@rr}r\rAs ` r3r?zBartEncoder.__init__s    ~ 1N !.$*$B!.4.DMdi ***# 3  y$*:     >  *   m$q$q$q$qTYZ`ZoTpTp$q$q$qrr #%< #:#: &+# r5Nr(rg inputs_embedsrjrc R|du|duz rtd|||}||dddddf}||j}||z}||}t j||j|j }t|j ||}t|j D];\}}d} |j r!tjg} | |jkrd} | s |||fi|}.Grr5F)r>r?ridecoder_layerdroprr)r[rmax_target_positionsr r!r"rrYr#r$r7r%rr&r'decoder_layersrrr)r*r+)r@rr\rAs ` r3r?zBartDecoder.__init__7s    ~ 1!.$*$B!393IRdi///s 3  v~t/?[    >  * N  m$q$q$q$qTYZ`ZoTpTp$q$q$qrr #%<#?#? &+# r5Nr(rgrrrr,rrjrc t|du|duz rtd|||}|r[|Y| |jjr6t t |jt |jnt |j}|dd\} } ||nd} tj | |j | z} |/ts!| | z} tj | | |j }t|tr|jn|}t|j|||}t!|j|||}||| | }||j }||z}||}t(j||j|j }t1|jD];\}}|jr tjg}||jkr,||||f|||d |}?? !O 0 0   ,;')+     ";;'1"7 " " " ((4JYe(ff LL!566 % 1 00??  --mt|VZVc-dd "+DK"8"8   C} &+jnn#&77)M%(> /# MM9++    r5)NNNNNNN)rPrQrRrSrr$r|r7r'r?r#r%rrGr8rUrr rrrrrMrVrWs@r3r:r:(sx*$n]! TTT*N=n]]] z0 .2.2:>:>(,26!%R R #d*R  t+R  %047 R !& 04 7 R  R (4/R $;R +,R  3R R R ^_ R R R R R r5r:c<eZdZdddZdeffd ZdZdZee dde j dzd e j dzd e j dzd e j dzd e e jdzd edzde jdzde jdzdedzdeedeezfdZxZS) BartModelz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrc\t||j|j}}|jrt j|jnd}t||j|||_ t||_ t||_ |dS)NrZr)r>r?r)r#r r!r"rrYsharedrencoderr:decoderr+)r@rr[r#r\rAs r3r?zBartModel.__init__s    "("5v7HZ 393IRdi///s -j&.+cnooo "6** "6**  r5c|jSr^)rPrs r3get_input_embeddingszBartModel.get_input_embeddingss {r5cX||_|j|j_|j|j_dSr^)rPrQr$rRr@rfs r3set_input_embeddingszBartModel.set_input_embeddingss' $(K !$(K !!!r5Nr(rgdecoder_input_idsdecoder_attention_maskencoder_outputsrr,decoder_inputs_embedsrrjrc |8|6|tdt||jj|jj}||jd |||d| }nct |tsNt|dt|dkr|dndt|dkr|dnd}|j d |||d|||| d| } t| j | j | j | j| j|j |j |j S)  decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)r(rgr,rr&r<)r.rrr(rgrrrr,r)r.rdecoder_hidden_statesdecoder_attentionsr=encoder_last_hidden_staterencoder_attentionsr )r0r4rr)r*rQrrlenrRrr.rrrr=) r@r(rgrXrYrZrr,r[rrjdecoder_outputss r3rMzBartModel.forwardsP  $)>)F  U !34;3T[5W!!   "/;t|0#-+00 00OO O_== -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO FRT\ F '1"1!"4#1+/ F F  F F "-?+;"1"?.9,=&5&G"1"?.9    r5 NNNNNNNNN)rPrQrR_tied_weights_keysr'r?rTrWrrrGr8rUlistrr rrrrrrMrVrWs@r3rNrNs(7'6 z      000 .2.259:>:>(,26:>!%T T #d*T  t+T !+d2 T !& 04 7 T e/047 T T (4/T  %047T $;T +,T  # #T T T ^T T T T T r5rNzV The BART Model with a language modeling head. Can be used for summarization. ) custom_introceZdZdZddiZdgZdeffd Z dd ed edzd e d e j ffd Z d ed dfdZ ee ddejdzdejdzdejdzdejdzdeejdzdedzdejdzdejdzdejdzde dzdeed eezfdZdejfdZxZS)rrlm_head.weightzmodel.shared.weightrrclt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)Nrr&Fr)r>r?rNrregister_bufferrGzerosrPr8rrrlm_headr+r@rrAs r3r?z%BartForConditionalGeneration.__init__(s    v&&  0%+q$*BSBb>c2d2deeey1B1QX]^^^  r5NTnew_num_tokenspad_to_multiple_of mean_resizingrct|||}||jjd|S)Nr)r>resize_token_embeddings_resize_final_logits_biasrJr.)r@rprqrrnew_embeddingsrAs r3rtz4BartForConditionalGeneration.resize_token_embeddings1sG88I[]jkk &&~'<'B1'EFFFr5c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nr,r&rrmr)rr.rGrmrFcatrl)r@rpold_num_tokensnew_bias extra_biass r3ruz6BartForConditionalGeneration._resize_final_logits_bias8s/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r5r(rgrXrYrZrr,r[labelsrrjc | G| rtdd} |'|%t| |jj|jj}|j|f|||||||| d| } || d} | |j | j z} d}| e| | j } t}|| d|jj | d}t|| | j| j| j| j| j| j| j S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example summarization: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn") >>> ARTICLE_TO_SUMMARIZE = ( ... "PG&E stated it scheduled the blackouts in response to forecasts for high winds " ... "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were " ... "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow." ... ) >>> inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="pt") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"], num_beams=2, min_length=0, max_length=20) >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'PG&E scheduled the blackouts in response to forecasts for high winds amid dry conditions' ``` Mask filling example: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-base") >>> TXT = "My friends are but they eat too many carbs." >>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"] >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['not', 'good', 'healthy', 'great', 'very'] ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rgrXrZrYrr,r[rrr, losslogitsrr_r`r=rarrb)rwarningr4rr)r*rrnrr/rFrrr#rrr_r`r=rarrb)r@r(rgrXrYrZrr,r[r|rrjoutputs lm_logitsmasked_lm_lossloss_fcts r3rMz$BartForConditionalGeneration.forwardAswh   mklllI (-B-J$6DK4dk6X%%!'1dj  ' )/+#9+'"7 ' '  ' ' LL,,  6 9 9):J K KK   YYy/00F'))H%XinnR9O&P&PRXR]R]^`RaRabbN#3")"?&9$5&-&G")"?&9    r5cLt||jj|jjSr^)r4rr)r*)r@r|s r3%prepare_decoder_input_ids_from_labelszBBartForConditionalGeneration.prepare_decoder_input_ids_from_labelss!&$+*BDKDfgggr5)NT NNNNNNNNNN)rPrQrRrrf_keys_to_ignore_on_load_missingr'r?rTrr EmbeddingrtrurrrGr8rUrgrr rrrrrMrrVrWs@r3rrs,  /(;&;#zae!7:TzY] <<<<<<.2.259:>:>(,26:>*.!%{ { #d*{  t+{ !+d2 { !& 04 7 { e/047 { { (4/{  %047{  4'{ $;{ +,{   { { { ^{ zhELhhhhhhhhr5rz Bart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c0eZdZdeffd Zee ddejdzdej dzdejdzdejdzde ej dzd ej dzd ej dzd ejdzd e dzd e edeezfdZxZS)BartForSequenceClassificationrc tj|fi|t||_t |j|j|j|j|_| dSr^) r>r?rNrrr num_labelsclassifier_dropoutclassification_headr+)r@rrjrAs r3r?z&BartForSequenceClassification.__init__sq**6***v&& #9 N N    % $ $   r5Nr(rgrXrYrZr,r[r|rrjrc J|d} ||td|jj|j|f||||||| d| } | d} ||jj| j} ttj | d dkd| | ddf}t|jd| jdzdkd|| dd | d ddd ddf}||}d}|||j}|jjp|jjdkr d |j_nS|jjdkr7|jtjks|jtjkr d |j_n d |j_|jjd kr\t/}|jjdkr1|||}n|||}n|jjd krLt3}||d |jj|d }n*|jjd krt5}|||}t7||| j| j| j| j| j | j!| j" S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rgrXrYrZr,r[rrr&z7All examples must have the same number of tokens.z3Each example must contain at least one token.r, regressionsingle_label_classificationmulti_label_classificationr~)#NotImplementedErrorrArPreqr eos_token_idr/rFr"rGunique_consecutivesumnumelr.rrrr problem_typerrErIrTrsqueezerrrrr_r`r=rarrb)r@r(rgrXrYrZr,r[r|rrjrreos_maskselectedsentence_representationrrrs r3rMz%BartForSequenceClassification.forwardsHR  I  !:%d4>Kbdd '1dj  ' )/#9+'"7 ' '  ' '   << 899<<]=QRR  $X\\!__ 5 5 ; ; = = B E   !111- N1 !4Q!7 71 < A   #+-- 0B0B10E0Er=K]K]^`KaKa"b"bcdcdcdfhjkjkjkck"l))*ABB  YYv}--F{'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x//.#3")"?&9$5&-&G")"?&9    r5re)rPrQrRr'r?rrrGr8rUrgrrrrrrrMrVrWs@r3rrse z      .2.259:>:>26:>*.!%l l #d*l  t+l !+d2 l !& 04 7 l e/047 l (4/l  %047l  4'l $;l +,l  0 0l l l ^l l l l l r5rc@eZdZfdZee ddejdzdejdzdejdzdejdzde ej dzdejdzd ejdzd ej dzd ej dzd e dzd e e deezfdZxZS)BartForQuestionAnsweringct|d|_|j|_t||_t j|j|j|_| dSr;) r>r?rrNrrr hidden_size qa_outputsr+ros r3r?z!BartForQuestionAnswering.__init__Ksm     +v&& )F$68IJJ r5Nr(rgrXrYrZstart_positions end_positionsr,r[rrjrc ||d} |j|f|||||| | d| } | d} || }|dd\}}|d}|d}d}||t |dkr|d}t |dkr|d}|d}|d|}|d|}t|}|||}|||}||zd z }t|||| j | j | j | j | j| j| j S) r]NFrrr&r,rm) ignore_indexr<) r start_logits end_logitsrr_r`r=rarrb)rrsplitrrwrcrrrrrrr_r`r=rarrb)r@r(rgrXrYrZrrr,r[rrjrsequence_outputrrr total_loss ignored_indexr start_lossend_losss r3rMz BartForQuestionAnswering.forwardWsN  &=+DI&0dj  ' )/#9+'"7 ' '  ' ' "!*11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM']CCCH!,@@Jx M::H$x/14J2%!#3")"?&9$5&-&G")"?&9    r5r)rPrQrRr?rrrGrUr8rgrrrrrrrMrVrWs@r3rrIsn     *..259:>:>371526:>!%W W <$&W  t+W !+d2 W !& 04 7 W e/047 W )D0W '$.W (4/W  %047W $;W +,W  4 4W W W ^W W W W W r5rc(eZdZdZfdZdZxZS)BartDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ct|t||_|dSr^)r>r?r:rRr+ros r3r?zBartDecoderWrapper.__init__s@    "6**  r5c|j|i|Sr^)rR)r@argsrjs r3rMzBartDecoderWrapper.forwardst|T,V,,,r5)rPrQrRrSr?rMrVrWs@r3rrsQ  -------r5rzu BART decoder with a language modeling head on top (linear layer with weights tied to the input embeddings). c(eZdZddiZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e dzd e j dzde j dzdedzdee j zdeedeezfdZxZS)BartForCausalLMrjz!model.decoder.embed_tokens.weightcd|_d|_t|t ||_t j|j|j d|_ | dS)NTFr) rrDr>r?rrrrrr#rnr+ros r3r?zBartForCausalLM.__init__sp $)!    '// y!3V5FUSSS  r5c$|jjjSr^rrRr$rs r3rTz$BartForCausalLM.get_input_embeddingssz!..r5c(||jj_dSr^rrVs r3rWz$BartForCausalLM.set_input_embeddingss*/ '''r5Nrr(rgrrrr,r|rlogits_to_keeprjrc |jjd|||||||d| } | d} t| trt | dn| } || dd| ddf}d}|e||j}t}|| d|j j | d}t||| j | j| j| jS)a@ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForCausalLM.from_pretrained("facebook/bart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```r^rNr,)rrrrrr=r )rrRrrTslicernr/rFrrrr#rrrrr=)r@r(rgrrrr,r|rrrjrr slice_indicesrrrs r3rMzBartForCausalLM.forwards)L>PTZ=O > )"7#9+' > >  > >   8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  YYv}--F'))H8FKKDK,BCCV[[QS__UUD0#3!/)$5     r5) NNNNNNNNr)rPrQrRrfr?rTrWrrrGr8rUrr rrTrrrrrMrVrWs@r3rrs} =     ///000.2.2:>;?(,26*.!%-.A A #d*A  t+A  %047 A !& 1D 8 A  A (4/A  4'A $;A el*A +,A  2 2A A A ^A A A A A r5r)rrrrrNrrr)Nrb)TrSr!rcollections.abcrrGrtorch.nnrrrr r activationsr cache_utilsr r r generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!r" utils.genericr#utils.output_capturingr$r%configuration_bartr' get_loggerrPrrUrTr4rr7rYModulerarzr|rrrrrrrr:rNrrrrr__all__r r5r3rs $$$$$$ AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJBBBBBB999999GFFFFFFF&&&&&&877777EEEEEEEE******  H % % %, c [^     ;;;;;R\;;;6 = = = = =bl = = =(!%% I% <% % < % L4' % T\ %%'(%%%%8r)r)r)r)r)BIr)r)r)j000001000fOOOOO1OOOdRY0/:     -        -   V V V V V %V V V r| | | | | %| | | ~q q q q q #q q q h `h`h`h`h`h#6`h`h  `hF | | | | | $7| |  | ~f f f f f 2f f f R - - - - -, - - - Y Y Y Y Y )?Y Y  Y x   r5