\-5r;/S?Qr10!Q& hh-u/?/?@V/WX\\^all m "'+A!B!&+A5CSCSTjCk!lq Ixx.:Huyy)?)E)E)G))STW[W_W_ X    ""L99\5??<+HKZ[O+\]aacL/%++hHbHbHdfr2ss r#cxURS5RSURS5S5nX2RS5- n[R"US- U4SS9RS5nURSURS5S5nXBRS5- n[ XUSS9n[ XUSS9nXV4$)ze This function computes both main and predict relative position buckets. For more detail, see paper. rrF)rD) unsqueezerepeatsizerr0rJ)rArB position_idsmain_stream_relative_positions$predicting_stream_relative_positionsmain_relative_position_buckets!predict_relative_position_bucketss r"#compute_all_stream_relative_bucketsrU\s &2%;%;A%>%E%EaIZIZ[]I^`a%b"%CF\F\]_F`%`",199lQ6F 5U[]+^+h+hij+k(+O+V+VWXZfZkZklnZoqr+s(+ORhRhikRl+l(&>#ATY&")A#GZ_)% * LLr#zF Base class for sequence-to-sequence language models outputs. ) custom_introc:\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'Sr\\ \R\ S 'Sr\\ \R\ S 'Sr\\ \R\ S 'Sr\\ \R\ S 'Sr\\R\ S 'Sr\\ \R\ S'Sr\\ \R\ S'\S5rSrg)ProphetNetSeq2SeqLMOutputsa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, decoder_sequence_length, config.vocab_size)`): Prediction scores of the main stream language modeling head (scores for each vocabulary token before SoftMax). logits_ngram (`torch.FloatTensor` of shape `(batch_size, ngram * decoder_sequence_length, config.vocab_size)`): Prediction scores of the predict stream language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`list[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_attn_heads, decoder_sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_ngram_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. decoder_ngram_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length)`. Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. Nlosslogits logits_ngrampast_key_valuesdecoder_hidden_statesdecoder_ngram_hidden_statesdecoder_attentionsdecoder_ngram_attentionscross_attentionsencoder_last_hidden_stateencoder_hidden_statesencoder_attentionscP[R"S[5 UR$Nzi`decoder_cross_attentions` is deprecated and will be removed soon. Please use `cross_attentions` instead.warningswarn FutureWarningrbselfs r"decoder_cross_attentions2ProphetNetSeq2SeqLMOutput.decoder_cross_attentions$    $$$r#)__name__ __module__ __qualname____firstlineno____doc__rZrr FloatTensor__annotations__r[r\r]tupler^r_r`rarbrcrdrepropertyrn__static_attributes__rqr#r"rXrXssG>)-D(5$$ %,*.FHU&& '.04L(5,,-4:>OXeE$5$567>@D8E%*;*;$<=DFJ%0A0A*B!CJ=Au'8'8!9:ACGhuU->->'?@G;?huU%6%678?=Ax(9(9:A@D8E%*;*;$<=D=Au'8'8!9:A %%r#rXz Base class for model encoder's outputs that also contains : pre-computed hidden states that can speed up sequential decoding. c\rSrSr%Sr\R \S'Sr\ \R \S'Sr \ \ \R \S'Sr \ \ \R \S'Sr \ \ \R \S'Sr\ \ \R \S 'Sr\ \ \R \S 'Sr\ \ \R \S 'Sr\ \R \S 'Sr\ \ \R \S 'Sr\ \ \R \S'\S5rSrg)ProphetNetSeq2SeqModelOutputa last_hidden_state (`torch.FloatTensor` of shape `(batch_size, decoder_sequence_length, hidden_size)`): Sequence of main stream hidden-states at the output of the last layer of the decoder of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. last_hidden_state_ngram (`torch.FloatTensor` of shape `(batch_size,ngram * decoder_sequence_length, config.vocab_size)`, *optional*): Sequence of predict stream hidden-states at the output of the last layer of the decoder of the model. past_key_values (`list[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_attn_heads, decoder_sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_ngram_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. decoder_ngram_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length)`. Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the weighted average in the encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. last_hidden_stateNlast_hidden_state_ngramr]r^r_r`rarbrcrdrecP[R"S[5 UR$rgrhrls r"rn5ProphetNetSeq2SeqModelOutput.decoder_cross_attentionsrpr#rq)rrrsrtrurvrrwrxrrr]ryr^r_r`rarbrcrdrerzrnr{rqr#r"r}r}s*<(((;?Xe&7&78?:>OXeE$5$567>@D8E%*;*;$<=DFJ%0A0A*B!CJ=Au'8'8!9:ACGhuU->->'?@G;?huU%6%678?=Ax(9(9:A@D8E%*;*;$<=D=Au'8'8!9:A %%r#r}zs Base class for model's outputs that may also contain a past key/values (to speed up sequential decoding). ct\rSrSr%Sr\R \S'Sr\ \R \S'Sr \ \ \R \S'Sr \ \ \R \S'Sr \ \ \R \S'Sr\ \ \R \S 'Sr\ \ \R \S 'Sr\ \ \R \S 'S rg) ProphetNetDecoderModelOutputa last_hidden_state (`torch.FloatTensor` of shape `(batch_size, decoder_sequence_length, hidden_size)`): Sequence of main stream hidden-states at the output of the last layer of the decoder of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. last_hidden_state_ngram (`torch.FloatTensor` of shape `(batch_size, ngram * decoder_sequence_length, config.vocab_size)`): Sequence of predict stream hidden-states at the output of the last layer of the decoder of the model. past_key_values (`list[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_attn_heads, decoder_sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states_ngram (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. ngram_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length)`. Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the weighted average in the rNrr] hidden_stateshidden_states_ngram attentionsngram_attentionsrbrq)rrrsrtrurvrrwrxrrr]ryrrrrrbr{rqr#r"rrs8(((;?Xe&7&78?:>OXeE$5$567>8B%(9(9":;B59Ju00129;?huU%6%678?;?huU%6%678?r#rc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'Sr\\ \R\ S 'Sr\\ \R\ S 'Sr\\ \R\ S 'Sr\\ \R\ S 'S rg)ProphetNetDecoderLMOutputiav ngram_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, decoder_sequence_length, config.vocab_size)`): Prediction scores of the main stream language modeling head (scores for each vocabulary token before SoftMax). logits_ngram (`torch.FloatTensor` of shape `(batch_size, ngram * decoder_sequence_length, config.vocab_size)`): Prediction scores of the predict stream language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`list[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_attn_heads, decoder_sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states_ngram (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. ngram_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length)`. Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the weighted average in the NrZr[r\r]rrrrrbrq)rrrsrtrurvrZrrrwrxr[r\r]ryrrrrrbr{rqr#r"rrs!F)-D(5$$ %,*.FHU&& '.04L(5,,-4:>OXeE$5$567>8B%(9(9":;B59Ju00129;?huU%6%678?;?huU%6%678?r#rc4\rSrSr%\\S'SrSrSrSr Sr g) ProphetNetPreTrainedModeliNconfig prophetnetTcF[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 ggg)N)meanstd) isinstancerLinearweightdatanormal_rinit_stdbiaszero_ Embedding padding_idx)rmmodules r" _init_weights'ProphetNetPreTrainedModel._init_weightsTs fbii ( ( MM   & &CT[[5I5I & J{{&   &&('  - - MM   & &CT[[5I5I & J!!- ""6#5#56<<>..r#cURRnURRnUcS5eURUR5nUSSS24R 5USSS24'X$S'UcS5eUR US:HU5 [R"US:5R5(dS 5eU$) Nzself.model.config.decoder_start_token_id has to be defined. In ProphetNet it is usually set to the pad_token_id. See ProphetNet docs for more information.rLr).rz1self.model.config.pad_token_id has to be defined.rz8Verify that `shifted_input_ids` has only positive values) rdecoder_start_token_id pad_token_id new_zerosshaper, masked_fill_rallitem)rm input_idsrrshifted_input_idss r" _shift_right&ProphetNetPreTrainedModel._shift_right^s!%!C!C{{// %1  F 1 &// @%.sCRCx%8%>%>%@#qr'"$:&!'\)\\'&&'8D'@,Oyy*a/05577s9ss7  r#rqN) rrrsrtrurrxbase_model_prefixsupports_gradient_checkpointingrrr{rqr#r"rrNs $&*#?!r#rcT^\rSrSrSrS\SS4U4SjjrS U4SjjrU4SjrS r U=r $) ProphetNetPositionalEmbeddingsiuz This module learns positional embeddings up to a fixed maximum size. Padding ids are ignored by either offsetting based on padding_idx or by setting padding_idx to None and ensuring that the appropriate position ids are passed to the forward function. rreturnNc>URUl[TU] URURUR 5 gN)max_position_embeddings max_lengthsuper__init__ hidden_sizerrmr __class__s r"r'ProphetNetPositionalEmbeddings.__init__|s3 88 779K9KVM`M`ar#cD>UbURbS5eUcUbjUR5S:waVUR5nUSU-n[R"S[RUS9[ URU-5-nOUc$[R"U[RUS9n[R "USS9RU5U-R 5UR-nURSURS- 5n[TU]-U5U4$)NzCIf position_ids is pre-computed then padding_idx should not be set.rr)rrrr%r) rget_seq_lengthrr(longr:cumsumtype_asclamprrforward) rm inputs_shaper%attention_maskr]rPprev_num_input_ids num_input_idsrs r"r&ProphetNetPositionalEmbeddings.forwards$$*:*:*B Q C  */M/M/OST/T&5%C%C%E" ,Q2D D $zz& 6R((=89  ")%*ZZ EJJW]%^NLLQ7??OR``$&4++ , ,11!T__q5HI w|,l::r#c">[TU]U5$r)rr)rmrPrs r"_forward'ProphetNetPositionalEmbeddings._forwardsw|,,r#)r)NNN) rrrsrtrurvrrrrr{ __classcell__rs@r"rrus. b/bDb;8--r#rc^\rSrSrSrSS\S\S\\4U4SjjjrSS\\ S\\ S \\ S \\ S \\ S \\ RS \ \ \\ 44SjjrSrU=r$)ProphetNetAttentioniz=Multi-headed attention from 'Attention Is All You Need' paperrnum_attn_heads layer_idxc>[TU]5 URnURUlURUlX lXB-UlX0lUR U-U:XdS5e[R"XD5Ul [R"XD5Ul [R"XD5Ul [R"XD5Ul g)Nzw`config.hidden_size` must be divisible by `config.num_encoder_attention_heads` and `config.num_decoder_attention_heads`)rrrattention_dropoutdropoutrhead_dimrrrkey_proj value_proj query_projout_proj)rmrrrrrs r"rProphetNetAttention.__init__s (( !'!9!9~~ ,#5 "}}~-<  4 <  +; ))K=))K= +; r#key_value_statesrlayer_head_maskpast_key_valueoutput_attentionscache_positionrc UR5upn USLn [UR55UU U /:XdSXU 4SUR535eURU5URS-- n Ub][ U[ 5(aFUR RUR5n U (a URnOURnOUnU (aUOUnU (aQUbNW (aGWRURRnURURRnOURU5nURU5nUR!USUR"UR5R%SS5nUR!USUR"UR5R%SS5nUbNU (dUOSnWR'UUURSU05unnU (aSUR UR'U R!XUR"UR5R%SS5n URS5n[(R*"S U UR%SS 55nXR"U U4nUR5U:wa[-S USUR535eUbUR/5S :XaSnXR"SU4nUb3UR5U:wa[-S USUR535eUbUU-nU(aUnOSn[0R2R5USS9nUbUR5UR"4:Xd&SUR"4SUR535eUR!SSSS5UR!XR"U U5-nUR!SSSS5U-n[0R2R7UUR8UR:S9n[(R*"S UU5nXR"XR4nUR5U:wa[-SUSUR535eUR%SS5R=XU 5nUR?U5n[0R2R7UUR6UR:S9nUU4$)Nz Size of hidden states should be , but is ?rLrr'rTzbsij,bsjk->bsikr z#Attention weights should have size rz Attention mask should have size r/Head mask for a single layer should be of size ptrainingz `attn_output` should have shape , but is of shape ) rOlistrrrr is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrviewr transposeupdatereinsum ValueErrorrrrrrrrreshaper)rmrrrrrrr batch_sizetgt_lenris_cross_attention query_statesrcurr_past_key_valuecurrent_states key_states value_statessrc_len attn_weightsexpected_shapeattn_weights_reshaped attn_probs attn_outputs r"rProphetNetAttention.forwards,9+=+=+?( [.T9M&&()   .   p.j;.N-OyYfYkYkYmXn o  p }59KL  %.*=>>+66::4>>J %*8*N*N'*8*M*M'&4#-?)] ."<,33DNNCHHJ.55dnnELLL~6J??>:L#R9L9Ldmm\ffghjklJ',,ZT=P=PRVR_R_`jjklnopL)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=#((d>Q>QSWS`S`akklmopq //!$||$5|ZEYEYZ[]^E_` $&9&97GL    . 0B>BRR[\h\m\m\o[pqr r  %.*<*<*>!*C!N$&9&91gF  %.*=*=*?>*Q??OyYgYlYlYnXopq q  %'.8L $0 !$( !}},,\r,B  &"'')d.A.A-CC A4CVCVBXAYZ#((*+- C+//2q!<|?P?P//'@L %4$8$8B1$EH]$] !]]** $$]]+ ll#4j,O $&9&97MMR     /??OOabmbrbrbtauvw w!++Aq199*{[ mmK0 mm++K4<j2!.j2 vx'' (j2j2r#rc>^\rSrSrSrS\S\4U4SjjrSrSr U=r $)ProphetNetFeedForwardi%ze This is the residual two feed-forward layer block based on the original Transformer implementation. rffn_dimc,>[TU]5 [URUl[ R "URU5Ul[ R "X!R5Ul URUl URUl gr) rrr activation_function activation_fnrrr intermediateoutputactivation_dropoutr)rmrr rs r"rProphetNetFeedForward.__init__*si #F$>$>?IIf&8&8'Bii););< "(";";~~ r#c4URU5nURU5n[RR XR UR S9nURU5n[RR XRUR S9nU$)Nr)r r rrrrrr)rmrs r"rProphetNetFeedForward.forward2s))-8 **=9  --m?V?Vaeanan-o  M2  --m||VZVcVc-d r#)rr rr r) rrrsrtrurvrr:rrr{rrs@r"rr%s&&/&#&r#rcz^\rSrSrS S\4U4SjjjrSrSrS S\\ \ 4Sjjr Sr S r S rU=r$) ProphetNetNgramSelfAttentioni<rc>[TU]5 URUlURUlURUlUR UlURUlURUlURUR -Ul URUl X l URUR -UR:XdS5e[R"URUR5Ul[R"URUR5Ul[R"URUR5Ul[R"URUR5Ul[R"URURUR -5UlSUlg)Nz6config.hidden_size must be divisible by num_attn_headsF)rrrrArelative_max_distancenum_decoder_attention_headsrrrrr2rrrrrrrrelative_pos_embeddingsr!rmrrrs r"r%ProphetNetNgramSelfAttention.__init__=sf !--!--%+%A%A"$@@~~ !'!9!9**d.A.AA \\ "}}t222f6H6HH D H &"4"4f6H6HI ))F$6$68J8JK))F$6$68J8JK &"4"4f6H6HI (*yy1C1CTEUEUX\XkXkEk'l$ r#cURX2URUR5RSS5R 5$Nrr')rrrr contiguous)rmtensorseq_lenrs r"_shape#ProphetNetNgramSelfAttention._shape[s7{{:0C0CT]]S]]^_abcnnppr#cSUlg)NT)r!rls r"prepare_for_onnx_export_5ProphetNetNgramSelfAttention.prepare_for_onnx_export_^s r#rc  UR5upn [UR55XU /:XdSXU 4SUR35eURU5n UR U5nUR U5nXR S-- n URXU 5n URUSU 5nURUSU 5nXRSUR 4nU R"U6n UR"U6nUR"U6nURSUR-SS9nU RSUR-SS9nURSUR-SS9nURSUR-SS9nUSUSSnnUSUSSnnUSUSSnnUSUSSnnUbG[U[5(a URnOUnURUUUR S U 05unnU SUR--n["R$"S UUR'SS 55nUR)UUX5n UU -nUbUU-n[+USUR,S 9R/U5n!UbwUR5UR4:Xd&S UR4SUR535eUR1SSSS5U!R1XRSU5-n![2R4R7U!UR8UR:S9n!["R$"S U!U5n"U"R'SS5RU SUU 5n"UR=U"5n"["R>"US5R1XRURUUR 5n#["R>"UV$s/sHn$["R@"UU$/S5PM sn$S5n%["R>"USS9n&["R@"UV's/sH+n'["R@"UU'/S5RCS5PM- sn'S5n(["R$"SU#U%45n)UREU&U)X5n*U)U*-n)Ub5URGSSSS S5nURIU)RJ5nU)U-n)[+U)SUR,S 9R/U)5n+Ub]UR5UR4:Xd&S UR4SUR535eUR1SSSSS5U+-n+[2R4R7U+UR8UR:S9n+["R$"SU+U(R'SS545n,U,R'SS 5n,U,RXRUU 5n,UR=U,5n,["R@"U"U,/S5R1U SU 5n-U!R1XRUS5n![2R4R7U-UR6UR:S9n-U-U!U+4$s sn$fs sn'f)Nz#`hidden_states` should be of shape rrrLrrr'rrzbntc,bncs->bntsr )rr!rrrzbnhtc,bnhsc->bnhtszbnhts,bnhsc->bnhtc)&rOrrrrrrr rrchunkr2rr rrrrrr get_main_relative_pos_embeddingsrr!rrrrrrrrstackr0rM#get_predict_relative_pos_embeddingspermutetor).rmrrrrextended_predict_attention_maskrSrTrPrrngram_sequence_lengthrrrr proj_shapehidden_states_listquery_states_listkey_states_listvalue_states_listmain_hidden_stateshidden_states_predict_listmain_query_statespredict_query_states_listmain_key_statespredict_key_states_listmain_value_statespredict_value_states_listrr1main_attn_weightsmain_relative_pos_embeddingsmain_attn_probsmain_attn_outputpredict_query_stateskeypredict_key_statespredict_hidden_statesv_ppredict_value_statespredict_attn_weightspredict_relative_pos_embeddingspredict_attn_probspredict_attn_outputrs. r"r$ProphetNetNgramSelfAttention.forwardas :G9K9K9M6 ;M&&()jQ\-]] 1*U`2`1ab##$ & ] }5 ]]=1 }5 $}}c'9: {{< S [[R< {{<Z@  "5"5r4==I #++Z8 ''4 #++Z8 +00TZZQ0G(..q4::~1.E$**1tzz>q*A(..q4::~1.E9KA9NPbcdcePf67H7KM^_`_aMb43B13EWXWYGZ07H7KM^_`_aMb4  %.*=>>&4&I&I#&4#1D1K1K!2DNNEUWeDf2 .O. 0A NC"LL):PRW. * "99-H/44  $ $Ra (D,<,Q>Q+R R 0771aC/77 8J8J2A8NQV8VW)G)N)NqRVReRegh)i&)G)L)L .44R8* &*H)L)L)N&/77# , 4"."4"4R"8 ?1%)PRW1 - &//15 !99-H044    $(8(8$:M:M'N N 0771aAF/77D[TU]5 [XR5Ul[ UR 5Ul[XR5Ul [ UR 5Ul gr) rrrnum_encoder_attention_heads self_attnrrself_attn_layer_normrencoder_ffn_dim feed_forwardfeed_forward_layer_normrs r"rProphetNetEncoderLayer.__init__s[ ,V5W5WX$-f.@.@$A!2&:P:PQ'01C1C'D$r#rcURUUUUS9upVURXQ-5nURU5nURXq-5nU4nU(aX4- nU$)N)rrrr)rYrZr\r]) rmrrrrattention_outputrfeed_forward_outputoutputss r"rProphetNetEncoderLayer.forwards~*.')+/ *8* & 112B2RS #// >445H5XY  "   &Gr#)r\r]rYrZF) rrrsrtrurvrrrrr{rrs@r"rVrV~s0E/E#(   r#rVc^\rSrSrSrS S\4U4SjjjrS S\\S\\S\\ R4Sjjr S r U=r $) ProphetNetDecoderLayeriz Decoder block for Prophetnet rcb>[TU]5 [XS9Ul[ UR 5UlUR(a3[XRUS9Ul [ UR 5Ul [XR5Ul[ UR 5Ulg)Nr)rrrrYrrrZadd_cross_attentionrr cross_attncross_attn_layer_normrdecoder_ffn_dimr\r]rs r"rProphetNetDecoderLayer.__init__s 5fR$-f.@.@$A!  % %1&:\:\hqrDO)263E3E)FD &2&:P:PQ'01C1C'D$r# use_cacherrc .URUU UUUUU U S9unnnURX-5nSnUb+URUUUUU U S9unnURUU-5nUR U5nUR UU-5nU4nU (aUUUU4- nU$)N)rrrrr-rSrTrP)rrrrrr)rYrZrjrkr\r])rmrrrdencoder_attn_maskrcross_attn_layer_head_maskr-rSrTrPrrnrrngram_attention_outputself_attn_weightsself_attn_weights_ngramcross_attn_weightsr`rarbs r"rProphetNetDecoderLayer.forwards$NR^^'))+,K+I.O%N\ N J 13J11-2XY ! ,37??+!60 :-"3 4C4 0 0!667G-7WXM#// >445H=5XY  "  )+BDVW WGr#)rjrkr\r]rYrZr) NNNNNNNNNNTFN)rrrsrtrurvrrrrrrrr{rrs@r"rfrfs E/ E E$"#'(,'+*.$(,1154D>4$D>4!.44r#rfz= The standalone encoder part of the ProphetNetModel. c"^\rSrSrSS\S\R 4U4SjjjrSrSr \ SS\ \ RS\ \ RS \ \ RS \ \ RS \ \S \ \S \ \S\\\44Sjj5rSrU=r$)ProphetNetEncoderirword_embeddingsc>[TU]U5 UbUO3[R"URUR UR S9Ul[U5Ul [UR 5Ul [R"[UR5Vs/sHn[U5PM sn5UlSUlUR%5 gs snf) word_embeddings (`torch.nn.Embeddings` of shape `(config.vocab_size, config.hidden_size)`, *optional*): The word embedding parameters. This can be used to initialize [`ProphetNetEncoder`] with pre-defined word embeddings instead of randomly initialized word embeddings. NrF)rrrr vocab_sizerrryrposition_embeddingsrembeddings_layer_norm ModuleListr-num_encoder_layersrVrgradient_checkpointing post_init)rmrry_rs r"rProphetNetEncoder.__init__s  * f//1C1CQWQdQde  $B&#I %.v/A/A%B"mmUSYSlSlMm$nMm%;F%CMm$no &+#  %os CcUR$rryrls r"get_input_embeddings&ProphetNetEncoder.get_input_embeddings###r#cXlgrrrmvalues r"set_input_embeddings&ProphetNetEncoder.set_input_embeddings$r#rr head_mask inputs_embedsroutput_hidden_states return_dictrcUbUOURRnUbUOURRnUbUOURRnUcUc [ S5eUbUb [ S5eUbUcUR U5nUb}SUSS2SSSS24R SURRSS5- [R"UR5R-nURUR5nOSnURURSSUR5upXI-n UR!U 5n ["R$R'XRR&UR(S9n U(aSOSn U(aSOSn Ub\UR+5S [-UR.5:Xd2S [-UR.5S UR+5S S 35e[1UR.5H8upU(aX4-n U"U UUbX>OSUS 9nUS n U(dM/U US4-n M: U(aX4-n U(d[3SXU 455$[5XU S9$)a Example: ```python >>> from transformers import AutoTokenizer, ProphetNetEncoder >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/prophetnet-large-uncased") >>> model = ProphetNetEncoder.from_pretrained("patrickvonplaten/prophetnet-large-uncased-standalone") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```Nz3Either input_ids or inputs_embeds has to be passed.z2Make sure to only pass input_ids or inputs_embeds.?rr'rrqrz&The head_mask should be specified for  layers, but it is for .)rrrc3,# UH ocMUv M g7frrq.0vs r" ,ProphetNetEncoder.forward..hsl$Zq$Z )rrr)rrruse_return_dictrryrNrXrr)rr*r,r~rr%rrrrrrOlenr enumerateryr)rmrrrrrrrextended_attention_maskr~rPrrdall_attentionsidx encoder_layer layer_outputss r"rProphetNetEncoder.forwards42C1N-TXT_T_TqTq$8$D $++JjJj &1%&@&@ATAT&U #&* #,0,D,D]EXEXY[Z[E\^k^r^r,s)%; 22=A  --m{{?R?R]a]j]j-k &:0d  >>#A&3t{{+;< 8T[[9I8JJabkbpbpbrstbuavvwx <#,DKK"8 C#(=@P(P%)63<3Hd"3 M*!,M  !/=3C2E!E#9 $9$Zll l+]k  r#)rrrr~ryr)NNNNNNN)rrrsrtrurrrrrrrrrrrrryrrr{rrs@r"rxrxs /",,,$%-115,004,0/3&*T ELL)T !.T ELL) T   - T $D> T 'tnT d^T  uo% &T T r#rxz= The standalone decoder part of the ProphetNetModel. c ^\rSrSrSS\S\\R4U4SjjjrSr Sr \ SS\\ RS\\ RS \\ RS \\ RS \\ RS \\ RS \\\\ RS\\ RS\\S\\S\\S\\S\\ RS\\\44Sjj5rSrSrSrSrU=r$)ProphetNetDecoderinrryc >[TU]U5 URUlURUlURUlUR UlUR UlUbUO3[R"URURURS9Ul [U5Ul[R"URURS5Ul[R""[%UR&5Vs/sH n[)XS9PM sn5Ul[-UR5UlSUlUR35 gs snf)r{Nr|rhF)rrr2rArrrmax_target_positionsrrr}rrryrr~ngram_embeddingsrr-num_decoder_layersrfrrrrr)rmrryirs r"rProphetNetDecoder.__init__ts  \\ !--%+%A%A"~~ $*$B$B!* f//1C1CQWQdQde  $B&#I " TZZ9K9KT RmmBGHaHaBb cBbQ #F 8Bb c  &/v/A/A%B"&+#  ds EcUR$rrrls r"r&ProphetNetDecoder.get_input_embeddingsrr#cXlgrrrs r"r&ProphetNetDecoder.set_input_embeddingsrr#rrrdencoder_attention_maskrcross_attn_head_maskr]rrnrrrrrc U bU OURRn U bU OURRn U bU OURRn U bU OURRn UcUc [ S5eUbUb [ S5eUbUcUR U5nURSSupUR(a/UR(aU (a[RS5 Sn SnU (aB[U[5(d-[RS5 Sn[R"U5nUbUR!5OS nUR#X4UR$US 9unnUS :waS unnOUR'U5unnUR"R)US -5nUU-nUR*R,nUS :wa_UR/S 5S :XdS 5e[1UR25Vs/sHnUUS - U-R5US S 5PM! nnSnSnOQ[1UR25Vs/sHnUUS - U-PM nnUR7UU5nUR9UU5nUb}SUSS2SSSS24R5S URR:S S 5- [<R>"UR@5RB-nUREUR@5nOSn[<RF"U/U-S 5nURH(aURIU5n[JRLROUURNURS9nU (aSOSnU (aURR2S :aSOSnU (aSOSn U (aSOSn!U (aURRP(aSOSn"[SXV/SS/5Hjun#n$U#cM U#R/5S [UURV5:XaM7SU$S[UURV5SUR/5S S35e [YURV5Hun%n&U (a8UUSS2SU244- nURR2S :aUUSS2US244- nU&"UUUUUbUU%OSUbUU%OSUUUUUU U U S9n'U'S nU (dMvU U'S 4- n U!U'S4- n!URRP(dMU"U'S4- n"M U (a8UUSS2SU244- nURR2S :aUUSS2US244- nU(aUR[5nUSS2SU24n(URR2S :a USS2US24OSn)U (d[]SU(U)UUUU U!U"455$[_U(U)UUUU U!U"S9$s snfs snf)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, ProphetNetDecoder >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/prophetnet-large-uncased") >>> model = ProphetNetDecoder.from_pretrained("microsoft/prophetnet-large-uncased", add_cross_attention=False) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzGEither `decoder_input_ids` or `decoder_inputs_embeds` has to be passed.zFMake sure to only pass `decoder_input_ids` or `decoder_inputs_embeds`.r'zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.Tr)r%r])NNrzOAt the moment `use_cache` is only supported for `decoder_input_ids` of length 1rrrqrrzThe `z` should be specified for rr) rprrqr-rSrTrPrrnrrr c3.# UH nUcMUv M g7frrqrs r"r,ProphetNetDecoder.forward..Ts"  A s )rrr]rrrrrb)0rrnrrrrryrrrlogger warning_oncerr r from_legacy_cacherr~r%!compute_buffered_relative_bucketsrrrrOr-r2rNprepare_attention_maskprepare_predict_attention_maskrrr)rr*r,r0rrrrriziprrrto_legacy_cacheryr)*rmrrrdrrrr]rrnrrrrrr1return_legacy_cachepast_key_values_lengthmain_stream_pos_embedrPrSrTpredicting_stream_pos_embedrrr2ngram_hidden_statesrr-extended_encoder_attention_maskall_main_stream_hidden_statesall_ngram_stream_hidden_statesall_main_stream_attnsall_ngram_stream_attnsall_cross_attns attn_mask mask_namer decoder_layerrrrs* r"rProphetNetDecoder.forwardsJ"+!6IDKK&>&G&G WXHX&Y#&(== 0077 "Q & %%a(A- a - #4::.#.E"%!),/JJRRS]_`bcd. #'+ #.2 +Z__c_i_iYj#YjPU!%!),/JJYj #'+&A&A-Q_&` #.2.Q.QR_ao.p + " -,QdA-=>EEaIpIprsuvww DJJ'++/, +/N.P.PQ^QdQd.e +.2 + =/4G"GK  % % 66}EM --mt||VZVcVc-d /C%/C HYHY\]H]cg&&7T'8d 1dkk6U6U"[_%((IKYoKp$q Iy$ ~~'*s4;;/?@I;&@T[[AQ@RS!(+,A/@%r #,DKK"8 C#--CSOCS@S2T1VV-;;$$q(2}QHXEX7Y6[[2)'%"A3<3H3dI]Ii,@,Eos0O/M2S).#"3-M"*!,M  %-*:)<<%&=+;*==&;;222# a(8'::O=#9@ )mA?O?OHL HYHY\]H]-?3C0C"Dcg &+#12)*#    ,/$;+7 >,3,  O##s %W  W%c URup#[R"SUR5R UR 5R SS5n[URURU5upEUSS2SU2SU24R USS5n[R"USS2SU2SU24USS2SU2URURU-24/S5R USS5nXE4$r) rrrNrr,r%rNrUrArr0)rmrPrr1main_relative_bucketspredict_relative_bucketss r"r3ProphetNetDecoder.compute_buffered_relative_bucketsms&2&8&8# ||At'@'@ADD\EXEXY``abdef :]   d88,; 7 !6a9I/9IK[OK[6[ \ c cdnpqst u#(99(,<_,<>N>N)NO('')B)BTE^E^apEp)pp  $  &Q " !%>>r#cHURSSup4[R"XD4[R"UR5R URUR S9n[R"US5nUSU2SU24SSSS2SS24RX0RR4UR-5nUb@SUSS2SSSS24- [R"UR5R -nXg-nOUnURUR5$)Nr'rrr) rrfullr)rr*r%triuexpandrrr,)rmrrr seq_length causal_maskextended_causal_maskrs r"r(ProphetNetDecoder.prepare_attention_masks!.!4!4Ra!8 jj  $ KK ++ , 0 0%% ''  jja0 *;J;  +CDT4QRTUEUV]] @@ AKDUDU U   %'*^AtT1>r#c URSSup4[URURURUR 5n[ R"USS2SU2SU24USS2SU2URURU-24/SS9nUSSSS2SS2SS24RX0RR4UR-5nUbSUSS2SSSSS24- [ R"UR 5R-nURX0RRURXD45n[ R"U[ R"U5/SS9nXg-nOUnURUR 5$)Nr'rLrr)rr6rr2r%rrr0rrrr)r*r9r,) rmrrrrpredict_causal_maskextended_predict_causal_maskrr-s r"r0ProphetNetDecoder.prepare_predict_attention_masks!.!4!4Ra!8 3  % %tzz=3G3GI\I\ $ii#A{ {KZK$?@#{ {D$=$=@Y@Y\f@f$ff   (;4q!Q;N'O'V'V @@ ADWD]D] ]( $  %'*^AtT4QR$%-1158<9=,07;@D04$(,0/3&*15Q ELL)Q !.Q  ( 5 Q !) 6 Q ELL) Q 'u||4Q "%ell(;"<=Q   -Q D>Q $D>Q 'tnQ d^Q !.Q  u22 3Q Q f?,?0!G!Gr#rc&$^\rSrSrSS/rS\4U4SjjrSrSrSr S r S r \ SS \ \RS \ \RS \ \RS\ \R S\ \RS\ \RS\ \RS\ \S\ \\\RS\ \RS\ \RS\ \S\ \S\ \S\ \S\ \RS\\\44"Sjj5rSrU=r$)ProphetNetModeliencoder.word_embeddings.weightdecoder.word_embeddings.weightrc>[TU]U5 [R"URUR UR S9Ul[R"U5nSUl SUl [X R5Ul [R"U5nSUlSUl [X0R5UlUR#5 g)Nr|FT)rrrrr}rrrycopydeepcopyrntie_encoder_decoderrxencoder is_decoderrdecoderr)rmrencoder_configdecoder_configrs r"rProphetNetModel.__init__s  !||F,=,=v?Q?Q_e_r_rsv.#( -2*(9M9MN v.$(!-2*(9M9MN  r#cUR$rrrls r"r$ProphetNetModel.get_input_embeddingsrr#c|XlURURlURURlgr)ryrrrs r"r$ProphetNetModel.set_input_embeddingss,$'+';'; $'+';'; $r#cURR(aaURURRUR5 URUR RUR5 ggr)rtie_word_embeddings_tie_or_clone_weightsrryrrls r" _tie_weightsProphetNetModel._tie_weightssT ;; * *  & &t||'C'CTEYEY Z  & &t||'C'CTEYEY Z +r#cUR$r)rrls r" get_encoderProphetNetModel.get_encoder ||r#cUR$rrrls r" get_decoderProphetNetModel.get_decoderrr#rrdecoder_input_idsdecoder_attention_maskrdecoder_head_maskrencoder_outputsr]rdecoder_inputs_embedsrnrrrrrc`U bU OURRn U bU OURRn UbUOURRnUbUOURRnUcUR UUUU U UUS9nUR UUUSUUUU U U UU UUS9 nU(dUU-$[URURURURURURURURURURURS9 $)aW decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) ProphetNet uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, ProphetNetModel >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/prophetnet-large-uncased") >>> model = ProphetNetModel.from_pretrained("microsoft/prophetnet-large-uncased") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state # main stream hidden states >>> last_hidden_states_ngram = outputs.last_hidden_state_ngram # predict hidden states ```)rrrrrrrr) rrrdrrrr]rrrrnrr) rrr]r^r_r`rarbrcrdre)rrnrrrrrr}rrr]rrrrrb)rmrrrrrrrrr]rrrnrrrrdecoder_outputss r"rProphetNetModel.forwardsWt"+!6IDKKj $D>j 'tnj d^!j "!.#j $ u22 3%j j r#rzh The ProphetNet Model with a language modeling head. Can be used for sequence generation tasks. c(~^\rSrSr/SQrS\4U4SjjrSrSr\ SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \\\ RS\ \ RS\ \ RS\ \ RS\ \S\ \S\ \S\ \S\ \ RS\\\44$Sjj5rS SjrS\ R4SjrSrSrSrU=r$)!"ProphetNetForConditionalGenerationiU)rrlm_head.weightrc>[TU]U5 [U5UlURUlUR Ul[R"URURSS9Ul UR5 g)NFr) rrrrrrdisable_ngram_lossrrrr}lm_headrrs r"r+ProphetNetForConditionalGeneration.__init__]sd  )&1!.."(";";yy!3!3V5F5FUS  r#cURR(a1URURRUR 5 ggr)rrrrryrrls r"r/ProphetNetForConditionalGeneration._tie_weightshs2 ;; * *  & &t'F'F U +r#c.URR$r)rryrls r"r7ProphetNetForConditionalGeneration.get_input_embeddingsls...r#rrrrrrrrr]rrlabelsrnrrrrrczUbUOURRnU bUcU cURU 5nURUUUUUUUUU U U U UUUUS9nUb UROU RSSunnUSR UURR US5nURU5nUSS2S4nURR S:a USS2SS24OSnUR5(dUR5nSnU bURUU 5nU(d+[SUU455nUb U4U-USS-$UUSS-$[UUUURURURUR UR"UR$UR&UR(UR*S9 $) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) ProphetNet uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, ProphetNetForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/prophetnet-large-uncased") >>> model = ProphetNetForConditionalGeneration.from_pretrained("microsoft/prophetnet-large-uncased") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> logits_next_token = outputs.logits # logits to predict next token as usual >>> logits_ngram_next_tokens = outputs.logits_ngram # logits to predict 2nd, 3rd, ... next tokens ```N)rrrrrrrrr]rrrnrrrrr'rrLrc3,# UH ocMUv M g7frrqrs r"r=ProphetNetForConditionalGeneration.forward..R*@Qqq*@r) rZr[r\r]r^r_r`rarbrcrdre)rrrrrrr2r is_contiguousr _compute_lossryrXr]r^r_r`rarbrcrdre)rmrrrrrrrrr]rrrrnrrrrrbrr1predicting_streamspredict_logitsr[r\rZ all_logitss r"r*ProphetNetForConditionalGeneration.forwardos~&1%y$D>y 'tn!y"d^#y$!.%y& u// 0'yyv8)ELL)'''r#r zt The standalone decoder part of the ProphetNetModel with a lm head on top. The model can be used for causal c ^\rSrSr/SQrS\4U4SjjrSrSrSr Sr S r \ SS \ \RS \ \RS \ \RS \ \RS\ \RS\ \RS\ \\\RS\ \RS\ \RS\ \S\ \S\ \S\ \S\\\44Sjj5rSSjrSSjrSrU=r$)ProphetNetForCausalLMi)z!prophetnet.word_embeddings.weightz)prophetnet.decoder.word_embeddings.weightr rcN>[R"U5nSUlSUl[TU]U5 [ U5UlURUl URUl [R"URURSS9UlUR!5 g)NTFr)rrris_encoder_decoderrrProphetNetDecoderWrapperrrrrrrrr}rrrs r"rProphetNetForCausalLM.__init__sv& $)!  26:!.."(";";yy!3!3V5F5FUS  r#cBURRR$rrrryrls r"r*ProphetNetForCausalLM.get_input_embeddings-s&&666r#c8XRRlgrrHrs r"r*ProphetNetForCausalLM.set_input_embeddings0s27/r#cURR(a;URURRR UR 5 ggr)rrrrrryrrls r"r"ProphetNetForCausalLM._tie_weights3s; ;; * *  & &t'>'>'N'NPTP\P\ ] +r#c$XRlgrr=)rmrs r" set_decoder!ProphetNetForCausalLM.set_decoder7s ")r#c.URR$rr=rls r"r!ProphetNetForCausalLM.get_decoder:r;r#rrrdrrrr]rrrnrrrrcU bU OURRn URRUUUUUUUUU U U U S9 nUb UROURSSunnUSR XRR US5nURU5nUSS2S4nURR S:a USS2SS24OSnSnU bURUU 5nU (d+[SUU455nUb U4U-USS-$UUSS-$[UUUURURURURURUR S9 $) a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, ProphetNetForCausalLM >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/prophetnet-large-uncased") >>> model = ProphetNetForCausalLM.from_pretrained("microsoft/prophetnet-large-uncased") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # Model can also be used with EncoderDecoder framework >>> from transformers import BertTokenizer, EncoderDecoderModel, AutoTokenizer >>> import torch >>> tokenizer_enc = BertTokenizer.from_pretrained("google-bert/bert-large-uncased") >>> tokenizer_dec = AutoTokenizer.from_pretrained("microsoft/prophetnet-large-uncased") >>> model = EncoderDecoderModel.from_encoder_decoder_pretrained( ... "google-bert/bert-large-uncased", "microsoft/prophetnet-large-uncased" ... ) >>> ARTICLE = ( ... "the us state department said wednesday it had received no " ... "formal word from bolivia that it was expelling the us ambassador there " ... "but said the charges made against him are `` baseless ." ... ) >>> input_ids = tokenizer_enc(ARTICLE, return_tensors="pt").input_ids >>> labels = tokenizer_dec( ... "us rejects charges against its ambassador in bolivia", return_tensors="pt" ... ).input_ids >>> outputs = model(input_ids=input_ids, decoder_input_ids=labels[:, :-1], labels=labels[:, 1:]) >>> loss = outputs.loss ```N) rrrdrrrr]rrnrrrr'rrLrc3,# UH ocMUv M g7frrqrs r"r0ProphetNetForCausalLM.forward..rr) rZr[r\r]rrrrrb)rrrrrrr2rrryrr]rrrrrb)rmrrrdrrrr]rrrnrrrrbrr1rrr[r\rZr s r"rProphetNetForCausalLM.forward=sB&1%l$D>l'tnld^l u// 0ll\>   r#rBc@^\rSrSrSrS\4U4SjjrSrSrSr U=r $)rEizx This is a wrapper class, so that [`ProphetNetForCausalLM`] can correctly be loaded from pretrained prophetnet classes. rc>[TU]U5 [R"URUR UR S9Ul[XRS9Ul UR5 g)Nr|r) rrrrr}rrryrrrrs r"r!ProphetNetDecoderWrapper.__init__sV  !||F,=,=v?Q?Q_e_r_rs(AUAUV  r#clURURURR55 gr)rryrrrls r"r%ProphetNetDecoderWrapper._tie_weightss% ""4#7#79Z9Z9\]r#c&UR"U0UD6$rr)rmargsr[s r"r ProphetNetDecoderWrapper.forwards||T,V,,r#)rry) rrrsrtrurvrrrrr{rrs@r"rErEs% /^--r#rE)rrxrBr rrrd)=rvrr>ri dataclassesrtypingrrrtorch.utils.checkpointrrtorch.nnr activationsr cache_utilsr r generationr modeling_layersrmodeling_outputsrmodeling_utilsrutilsrrrconfiguration_prophetnetr get_loggerrrrrr6rJrUrXr}rrrrrModulerrrrVrfrxrrr rBrE__all__rqr#r"rusY !" !5)9/-996   H %Q7" 6M.  4% 4%  4%n  2%;2%  2%j  $@;$@  $@N  ,@ ,@  ,@^#!#!#!L(-R\\(-VB2"))B2JBII./299/D (7(VH7HV r 1r  r j IG1IG  IGX R /R R j t')BOt'  t'n J 5J  J Z-8-, r#