tjxdZddlZddlZddlmZddlZddlZddlmZddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2ddl3m4Z4e-j5e6Z7dej8de9de9fdZ:Gddej;Z< d=dej=dej8dej8d ej8d!ej8dzd"e>dzd#e>d$e'e)fd%Z?Gd&d'ej=Z@Gd(d)eZAGd*d+eZBe*Gd,d-e%ZCGd.d/eCZDGd0d1eCZEe*Gd2d3eCZFe*d45Gd6d7eCeZGGd8d9eCZHGd:d;eCeZIgd<ZJdS)>zPyTorch PEGASUS model.N)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging)merge_with_config_defaults)OutputRecordercapture_outputs) PegasusConfig input_ids pad_token_iddecoder_start_token_idc||j}|ddddf|ddddf<||dddf<|td||dk||S)z1 Shift input ids one token to the right. Nr!rz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r#r$r%shifted_input_idss j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/pegasus/modeling_pegasus.pyshift_tokens_rightr/9s"++IO<<(CRC06688aaae4aaadLMMM""#4#zQPegasusSinusoidalPositionalEmbedding.create_weight...Ws7 L L LAcBHUAaL3$6777 L L Lr0)range)rCrFrEs @r.rGzFPegasusSinusoidalPositionalEmbedding.create_weight..Ws9 e e eQT L L L L Ls L L L e e er0F)dtype requires_gradr@rr!N) weightr)rAarrayrHtorchemptyrI FloatTensorsincos)r;n_pos position_encoutsentinelrEs @r. create_weightz2PegasusSinusoidalPositionalEmbedding.create_weightPs [& sx e e e eX]^cXdXd e e e  k%DK,=USSS"Qw!||3!88#(a".rvl111add76K/L/LMMAAAqzM!-bf\!!!QTT'5J.K.KLLAAAxyyL r0rinput_ids_shapepast_key_values_length position_idsc|<|dd\}}tj|||ztj|jj}t |S)z3`input_ids_shape` is expected to be [bsz x seqlen].Nr@)rIdevice)rMarangelongrKr[r9forward)r;rWrXrYbszseq_lenr<s r.r^z,PegasusSinusoidalPositionalEmbedding.forward_se  *2A2.LC <&(>(HPUPZcgcncuLww|,,,r0N)rN) __name__ __module__ __qualname____doc__intr:rVrMno_gradSizeTensorr^ __classcell__r<s@r.r2r2JsNNEEcE#ECRVJEbfEEEEEE   U]__pt - -$z -CF -Z_ZfimZm -  - - - - -_ - - - - -r0r2modulequerykeyvalueattention_maskscalingdropoutkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nr'r@rrEptrainingr!) sizerMmatmul transposer functionalsoftmaxrsrz contiguous) rmrnrorprqrrrsrt attn_weights attn_outputs r.eager_attention_forwardrms**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r0ceZdZdZ ddededed ed ed ed edzd edzffd Z dde j de j dzde dzde j dzde e dee j e j dzff dZxZS)PegasusAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrlFTN embed_dim num_headsrs is_decoderbias is_causalconfig layer_idxc zt||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ ||_ |/|j r(t d|jjdt!j||||_t!j||||_t!j||||_t!j||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rvzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)r9r:rrrshead_dimrr+rrrrrlogger warning_oncer<rbrLineark_projv_projq_projout_proj) r;rrrsrrrrrr<s r.r:zPegasusAttention.__init__sY "" !Y.  MI %$. 8 83dn33%.333 }d* $""      ,4>+B,,,    i 94@@@ i 94@@@ i 94@@@  )YTBBB r0 hidden_stateskey_value_statespast_key_valuesrqrtr6c |du}|jdd}g|d|jR}|||dd} d} |Ht |t r1|j|j } |r|j } n |j } n|} |r|n|} |r3|1| r/| j |j j } | j |j j}n|| } || }g| jddd|jR}| |dd} ||dd}|E| | ||j \} }|r$t |t rd|j|j <t%j|jjt,}||| | ||f|jsdn|j|jd|\}}|jg|dR}||}||fS) z#Input shape: Batch x Time x ChannelNr'r!r@FTrl)rsrr)r)rrviewr} isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationrrzrsrrreshaperr)r;rrrrqrtis_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacerrs r.r^zPegasusAttention.forwards.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66LF-crc2FBF FFH#22<(?(M K ,.E) ) %8$7      % $}>CC$,L % %  % % ! \*k);;;;;;FFHH mmK00 L((r0)rlFTFNNNNN)rbrcrdrerffloatboolr"r:rMrir rrtupler^rjrks@r.rrs]GG  '+ $%C%C%C%C %C  %C  %C%C$%C:%C%C%C%C%C%CT15(,.2 H)H)|H) ,-H) H)  t+ H)-.H) u|U\D00 1H)H)H)H)H)H)H)H)r0rcfeZdZdeffd ZdejdejdeedejfdZ xZ S)PegasusEncoderLayerrct|j|_t |j|j|j||_tj |j|_ |j |_ t|j |_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)rrrsr)r9r:d_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrsractivation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normr;rr<s r.r:zPegasusEncoderLayer.__init__s )n4,     %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r0rrqrtr6c |}||}|jd||d|\}}tj||j|j}||z}|}||}|||}tj||j |j}| |}tj||j|j}||z}|j tj kr9tj|j jdz }tj|| |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rrqrxi)minmax)rrrr~rsrzrrrrrrIrMfloat16finforclamp)r;rrqrtresidual_ clamp_values r.r^zPegasusEncoderLayer.forwards[!11-@@ )4> ')     q  --mt|VZVc-dd  =0  --m<< **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0  %- / /+m&9::>EK!K K<[YYYMr0) rbrcrdr"r:rMrirrr^rjrks@r.rrs=}======$"|" "+, "  """"""""r0rceZdZddededzffd Z ddejdejdzdejdzd ejdzd edzd e dzd e e d ejfdZ xZ S)PegasusDecoderLayerNrrc t|j|_t |j|j|jdd|||_|j|_t|j |_ |j |_ tj|j|_t |j|j|jd|||_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rrrsrrrr)rsrrr)r9r:rrrdecoder_attention_headsrrrsrrrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrr)r;rrr<s r.r:zPegasusDecoderLayer.__init__:s  )n4,   ~ #F$>?"(";$&L$@$@!, N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r0Trrqencoder_hidden_statesencoder_attention_maskr use_cachertr6c |}||}|jd|||d|\}} tj||j|j}||z}|]|}||}|jd||||d|\}} tj||j|j}||z}|}||}| | |}tj||j |j}| |}tj||j|j}||z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )rrrqrxN)rrrqrr) rrrr~rsrzrrrrrrr) r;rrqrrrrrtrrs r.r^zPegasusDecoderLayer.forwardYs*!11-@@ *4> '+)      q  --mt|VZVc-dd  =0  ! ,$H 88GGM0t0 +!65 /        M1M11-4.s"$g$g$gQ%8%@%@$g$g$gr0F)r9r:rsencoder_layerdrop layerdroprr$r5max_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scaler Embedding vocab_size embed_tokensr2embed_positions ModuleListrHencoder_layersrr layer_normgradient_checkpointing post_init)r;rrr<s ` r.r:zPegasusEncoder.__init__s     ~ 1N !.$*$B!393IR49Y///sL):ItGWXXC  *      m$g$g$g$g%PVPeJfJf$g$g$ghh ,v~66&+# r0new_num_position_embeddingscbtd|d||j_t |jj|jj|j|_tj |jj |j |j |j dS Resizes position embeddings matrix of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embeddings. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. z(Setting `config.max_position_embeddings=z`...Nrinforrr2rr5rrrrKrVtor[r;rs r.resize_position_embeddingsz)PegasusEncoder.resize_position_embeddings  `?Z```aaa.I +C K / K      4'.0D0R0R0T0TUUU  ,,,,,r0r6c|jSz8 Returns the position embeddings matrix rr;s r.get_position_embeddingsz&PegasusEncoder.get_position_embeddings ##r0Nrtc 2|du|duz rtd||||jz}|jdd}||}||z}t j||j|j}t|j ||}t|j D];\}} d} |jr!tjg} | |jkrd} | s | ||fi|}<||}t#|S)Nz:You must specify exactly one of input_ids or inputs_embedsr'rx)r inputs_embedsrqFT)last_hidden_state)r+rrr)rrr~rsrzr r enumeraterrMrandrrr) r;r#rqrrtr embed_posridx encoder_layerto_dropdropout_probabilitys r.r^zPegasusEncoder.forwardsf  -t"; < [YZZ Z   --i884;KKM#)#2#. ((55 % 1  --mt|VZVc-dd 2;')    #,DK"8"8   CG} #&+jnn#&77"G  - !"!!!!  66 +    r0r)rbrcrdrerr_can_record_outputsr"r:rfr rrrrr rrrrr^rjrks@r.rrs-& }2-c----0$$$$$   - - +, -  - - - ^_ - - - - - r0rceZdZdZeeeddeedddZdeffd Z d e fd Z d e j fd Zeee ddeed efdZxZS)PegasusDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`PegasusDecoderLayer`] Args: config: PegasusConfig embed_tokens (nn.Embedding): output embedding r!r)index layer_namer)rrcross_attentionsrcxtj|_j|_j|_j|_j rtj j nd|_ tjjj |j|_t%jj |j|_tjfdt+jD|_tjj |_d|_|dS)Nrc2g|]}t|S))r)r)rCirs r.rGz+PegasusDecoder.__init__..@s($t$t$tRS%81%M%M%M$t$t$tr0F)r9r:rsdecoder_layerdroprr$r5rmax_target_positionsrrrrrrrrrr2rrrHdecoder_layersrrrrrrs `r.r:zPegasusDecoder.__init__1s    ~ 1!.$*$B!8>8NW49V^444TWL):FNDL\]]C  * N     m$t$t$t$tW\]c]rWsWs$t$t$tuu ,v~66&+# r0rcbtd|d||j_t |jj|jj|j|_tj |jj |j |j |j dSrrr s r.r z)PegasusDecoder.resize_position_embeddingsGr r0r6c|jSrrrs r.rz&PegasusDecoder.get_position_embeddings_rr0Nrtc X|du|duz rtd|||}||jz}|r[|Y| |jjr6t t |jt |jnt |j}|dd\} } ||nd} tj | |j | z} |/ts!| | z} tj | | |j }t|t r|jn|}t!|j|||}t#|j|||}|| | f| | }||z}t&j||j|j }t/|jD];\}}|jr tjg}||jkr,||||f|||d |}<||}t9|| S) NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time)rr'rr[)rrrqr)rrrqr)rYrx)rrr)rr)r+rrris_encoder_decoderr r r{get_seq_lengthrMr\r[ronesrrrr rrr~rsrzrrrrrr)r;r#rqrrrrrrt batch_size seq_lengthrXrYmask_seq_lengthself_attn_cache causal_mask positionsrr decoder_layerrs r.r^zPegasusDecoder.forwardes  -t"; < ustt t   --i88M&(88   0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  "/!3!3!5!5crc!: JETE`!?!?!A!A!Afg|J}7KLLLOee  !*B*D*D !4zAO"Z OML`aaaN/+>?? !O 0 0   );')+    ";;'1"7 " " " ((*j)ACYht(uu % 1  --mt|VZVc-dd "+DK"8"8   C} &+jnn#&77)M%(> /# MM 66 8++    r0)NNNNNNN)rbrcrdrerrrrr"r:rfr rrrrr rrrrr^rjrks@r.r r "s9-$n%5Q;WWW*N+;1Q_``` },-c----0$$$$$  "#S S +,S  3S S S ^_ S S S S S r0r cjeZdZdddZdeffd ZdZdZdefdZ d e e j fd Z ee dd ejd zd ejd zdejd zdejd zde ejd zded zdejd zdejd zded zdeed e ezfdZxZS) PegasusModelz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrct||j|j}}t j||j||_t||_ t||_ | dSra) r9r:r$rrrrsharedrencoderr decoderr)r;rr5rr<s r.r:zPegasusModel.__init__sw    "("5v7HZ l:v~{KK %f-- %f--  r0c|jSra)r;rs r.get_input_embeddingsz!PegasusModel.get_input_embeddingss {r0cX||_|j|j_|j|j_dSra)r;r<rr=r;rps r.set_input_embeddingsz!PegasusModel.set_input_embeddingss' $(K !$(K !!!r0rc||j_|j||j|dSrN)rrr<r r=r s r.r z'PegasusModel.resize_position_embeddingssC/J + //0KLLL //0KLLLLLr0r6cf|j|jfSr)r<rr=rs r.rz$PegasusModel.get_position_embeddingss+ 4466 8\8\8^8^__r0Nr#rqdecoder_input_idsdecoder_attention_maskencoder_outputsrrdecoder_inputs_embedsrrtc ||jd |||d| }nct|tsNt|dt|dkr|dndt|dkr|dnd}|jd |||d|||| d| } t | j| j| j| j | j |j|j|j S) aE decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pegasus uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, PegasusModel >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-large") >>> model = PegasusModel.from_pretrained("google/pegasus-large") >>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt") >>> decoder_inputs = tokenizer("Studies show that", return_tensors="pt") >>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_inputs.input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 1024] ```N)r#rqrrr!r@)rrrr#rqrrrrr)rrdecoder_hidden_statesdecoder_attentionsr#encoder_last_hidden_staterencoder_attentionsr) r<rrlenr=rrrrrr#) r;r#rqrFrGrHrrrIrrtdecoder_outputss r.r^zPegasusModel.forwards8`  "*dl#-+ OOO_== -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO'$,  '1"1!"4#1+/      "-?+;"1"?.9,=&5&G"1"?.9    r0) NNNNNNNNN)rbrcrd_tied_weights_keysr"r:r?rBrfr rrrrrrrMrirOr rrrrr^rjrks@r.r9r9s(7'6 }      000 McMMMM"`r|)<```` *..2156:;?(,-159!%R R <$&R  t+R !<$. R !& t 3 R u01D8 R R |d*R  %|d2R $;R +,R  # #R R R ^R R R R R r0r9zY The PEGASUS Model with a language modeling head. Can be used for summarization. ) custom_introceZdZdZdgZddiZdeffd Z dd ed edzd e d e j ffd Z d ed dfdZ defdZd ee j fdZee d dejdzdejdzdejdzdejdzdeejdzdedzdejdzdejdzdejdzde dzdeed eezfdZdejfdZxZS)!rrrlm_head.weightzmodel.shared.weightrclt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)Nrr!Fr)r9r:r9rregister_bufferrMzerosr;num_embeddingsrrrlm_headrrs r.r:z(PegasusForConditionalGeneration.__init__Ss    !&))  0%+q$*BSBb>c2d2deeey1B1QX]^^^  r0NTnew_num_tokenspad_to_multiple_of mean_resizingr6ct|||}||jjd|S)Nr)r9resize_token_embeddings_resize_final_logits_biasrKr))r;r[r\r]new_embeddingsr<s r.r_z7PegasusForConditionalGeneration.resize_token_embeddings\sG88I[]jkk &&~'<'B1'EFFFr0c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nr'r!r-rwr)rr)rMrXr[catrW)r;r[old_num_tokensnew_bias extra_biass r.r`z9PegasusForConditionalGeneration._resize_final_logits_biascs/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r0rc||j_|jj||jj|dSrD)rrrr<r r=r s r.r z:PegasusForConditionalGeneration.resize_position_embeddingslsI/J + 556QRRR 556QRRRRRr0cz|jj|jjfSr)rr<rr=rs r.rz7PegasusForConditionalGeneration.get_position_embeddings}s1 "::<P>h>h>j>jkkr0r#rqrFrGrHrrrIlabelsrrtc | G| rtdd} |'|%t| |jj|jj}|j|f|||||||| d| } || j|j z} d}| Kt}|| d|jj | d}t|| | j| j| j| j| j| j| j S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pegasus uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Summarization: ```python >>> from transformers import AutoTokenizer, PegasusForConditionalGeneration >>> model = PegasusForConditionalGeneration.from_pretrained("google/pegasus-xsum") >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-xsum") >>> ARTICLE_TO_SUMMARIZE = ( ... "PG&E stated it scheduled the blackouts in response to forecasts for high winds " ... "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were " ... "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow." ... ) >>> inputs = tokenizer(ARTICLE_TO_SUMMARIZE, max_length=1024, return_tensors="pt") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"]) >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "California's largest electricity provider has turned off power to hundreds of thousands of customers." ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rqrFrHrGrrrIrr') losslogitsrrLrMr#rNrrO)rwarningr/rr$r%rrZrrrrrrrrLrMr#rNrrO)r;r#rqrFrGrHrrrIrirrtoutputs lm_logitsmasked_lm_lossloss_fcts r.r^z'PegasusForConditionalGeneration.forwardsLr   mklllI (-B-J$6DK4dk6X%%!'1dj  ' )/+#9+'"7 ' '  ' ' LL!:;;d>TT   '))H%XinnR9O&P&PRXR]R]^`RaRabbN#3")"?&9$5&-&G")"?&9    r0cLt||jj|jjSra)r/rr$r%)r;ris r.%prepare_decoder_input_ids_from_labelszEPegasusForConditionalGeneration.prepare_decoder_input_ids_from_labelss!&$+*BDKDfgggr0)NT) NNNNNNNNNN)rbrcrdr_keys_to_ignore_on_load_missingrRr"r:rfrrrr_r`r rrrrrMrirOr rrrr^rsrjrks@r.rrGsk  ':&;#/}ae!7:TzY] <<<<<<ScSSSS"lr|)<llll *..2156:;?(,-159&*!%] ] <$&]  t+] !<$. ] !& t 3 ] u01D8 ] ] |d*]  %|d2]  t#] $;] +,]   ] ] ] ^] ~hELhhhhhhhhr0rc(eZdZdZfdZdZxZS)PegasusDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ct|t||_|dSra)r9r:r r=rrs r.r:zPegasusDecoderWrapper.__init__s@    %f--  r0c|j|i|Sra)r=)r;argsrts r.r^zPegasusDecoderWrapper.forwardst|T,V,,,r0)rbrcrdrer:r^rjrks@r.rvrvsQ  -------r0rvcJeZdZddiZfdZdZdZdejfdZ de fd Z e e dd ejd zd ejd zdejd zdejd zded zdejd zdejd zded zde ejzdeedeezfdZxZS)PegasusForCausalLMrUz!model.decoder.embed_tokens.weightc*tj|}d|_d|_t |t ||_tj |j |j d|_ | dS)NTFr)copydeepcopyrr.r9r:rvrrr hidden_sizerrZrrs r.r:zPegasusForCausalLM.__init__sv&& $)!    *622 y!3V5FUSSS  r0c$|jjjSrarr=rrs r.r?z'PegasusForCausalLM.get_input_embeddings sz!..r0c(||jj_dSrarrAs r.rBz'PegasusForCausalLM.set_input_embeddings s*/ '''r0r6c>|jjSr)rr=rrs r.rz*PegasusForCausalLM.get_position_embeddingssz!99;;;r0rc\||j_|jj|dSrD)rrrr=r r s r.r z-PegasusForCausalLM.resize_position_embeddingss//J + 556QRRRRRr0Nrr#rqrrrrrirlogits_to_keeprtc |jjd|||||||d| } | d} t| trt | dn| } || dd| ddf}d}|e||j}t}|| d|j j | d}t||| j | j| j| jS)aJ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PegasusForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-large") >>> model = PegasusForCausalLM.from_pretrained("google/pegasus-large") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```rKrNr')rkrlrrrr#r)rr=rrfslicerZr r[rrrrrrrrr#)r;r#rqrrrrrirrrtrnr slice_indicesrlrkrqs r.r^zPegasusForCausalLM.forward%s)N>PTZ=O > )"7#9+' > >  > >   8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  YYv}--F'))H8FKKDK,BCCV[[QS__UUD0#3!/)$5     r0) NNNNNNNNr)rbrcrdrRr:r?rBrrrrfr rrrM LongTensorrirOr rrrrrr^rjrks@r.r{r{s=     ///000<<<<< ScSSSS .2.2:>;?(,26*.!%-.A A #d*A  t+A  %047 A !& 1D 8 A  A (4/A  4'A $;A el*A +,A  2 2A A A ^A A A A A r0r{)r{rr9r)Nrl)Krer}rcollections.abcrnumpyrArMrtorch.nnrrr activationsr cache_utilsr r r generationr masking_utilsr rmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrutils.output_capturingrr configuration_pegasusr" get_loggerrbrrirfr/rr2Modulerrrrrrrr r9rrvr{__all__rr0r.rs- $$$$$$ %%%%%%&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJBBBBBB999999GFFFFFFF&&&&&&877777EEEEEEEE000000  H % % %, c [^    "-----2<---R!%% I% <% % < % L4' % T\ %%'(%%%%:r)r)r)r)r)ryr)r)r)l555554555rZZZZZ4ZZZz22222_222$v v v v v +v v v rY Y Y Y Y +Y Y Y xE E E E E )E E E P YhYhYhYhYh&