tjdZddlZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l mZmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZddlmZmZmZddl m!Z!ej"e#Z$dej%de&de&fdZ'dFdej%dej(de&dzfdZ)Gddej*Z+Gddej,Z-Gddej,Z.Gd d!ej,Z/Gd"d#eZ0Gd$d%eZ1Gd&d'ej,Z2eGd(d)eZ3ed*+eGd,d-eZ4ed.+eGd/d0eZ5ed1+eGd2d3eZ6ed4+eGd5d6eZ7ed7+eGd8d9eZ8Gd:d;e3Z9Gd<d=e3Z:eGd>d?e3Z;ed@+GdAdBe3eZdS)GzPyTorch LED model.N) dataclass)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions)PreTrainedModel) ModelOutputauto_docstringlogging) LEDConfig input_ids pad_token_iddecoder_start_token_idc||j}|ddddf|ddddf<||dddf<|td||dk||S)z1 Shift input ids one token to the right. Nrrz&config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrrshifted_input_idss b/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/led/modeling_led.pyshift_tokens_rightr#&s"++IO<<(CRC06688aaae4aaadABBB""#4#z&LEDLearnedPositionalEmbedding.__init__Ms# 77777r$rinput_ids_shapepast_key_values_lengthc|dd\}}tj|||ztj|jj}t |S)z3`input_ids_shape` is expected to be [bsz x seqlen].N)r&device)r.arangelongweightrEr=forward)r?rArBr1seq_len positionsr@s r"rIz%LEDLearnedPositionalEmbedding.forwardPs[&rr* WL "$:W$DEJ_c_j_q   wwy)))r$)r) __name__ __module__ __qualname____doc__intr>r.SizerI __classcell__r@s@r"r8r8Hs8s83888888**uz*3**********r$r8ceZdZfdZ ddZedZedZeddefdZ ed e j fd Z d e j d e j d e fdZde j de j d e fdZedZdZdZdZxZS)LEDEncoderSelfAttentionct|j|jzdkr t d|jd|jd|j|_t |j|jz |_|j|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj |j|j|_tj |j|j|_tj |j|j|_|j|_||_|j|j}|dzdksJd|jd||dksJd|jd||dz|_||_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()rDz`attention_window` for layer z has to be an even value. Given z has to be positive. Given )r=r> hidden_sizenum_attention_headsr num_headsrPhead_dim embed_dimrLinearquerykeyvalue query_global key_global value_globalattention_probs_dropout_probdropoutlayer_idattention_windowone_sided_attn_window_sizeconfig)r?rirfrgr@s r"r>z LEDEncoderSelfAttention.__init__[s    : :a ? ?8F$688 4888  3F.1KKLL +Yv14>BB 9V/@@Yv14>BB If&8$.II)F$6GGIf&8$.II:   !24=A!#q((( mDM m m[k m m)(( !### hDM h hVf h h$##+;a*?' r$NFc D |dd}||}||}||} |\} } } | |jksJd|jd| |t j|jz}| | | |j |jdd}| | | |j |jdd}| |||j } |dkddddddf}| ||tj|jj}| ||||j }| |z } t)| | | |j |j dzdzgks;Jd| d | d |j d |j dzdzd |  |rN||\}}}}||||||| }tj|| fd } ~t0j| d tj}tj||ddddddfd}| | }~ t0j||j|j}| | | |j |jdd} |r|| ||||}n||| |j }|| | |j |jfks Jd|dd | | | !}|rq|"||||||\}}||ddd|df}| tG|dd ||ddd <d||<|ddf}|r||fz }|r|r||fzn|S)a [`LEDEncoderSelfAttention`] expects *len(hidden_states)* to be multiple of *attention_window*. Padding to *attention_window* happens in [`LEDEncoderModel.forward`] to avoid redoing the padding on each layer. The *attention_mask* is changed in [`LEDEncoderModel.forward`] from 0, 1, 2 to: - -10000: no attention - 0: local attention - +10000: global attention rrz&hidden_states should have embed_dim = z , but has N)r)rDz$local_attn_probs should be of size (z, z), but is of size ) query_vectors key_vectorsmax_num_global_attn_indicesis_index_global_attn_nonzero"is_local_index_global_attn_nonzero%is_local_index_no_global_attn_nonzerordimrrr&ptraining) value_vectors attn_probsrmrnrozUnexpected size) hidden_statesrmrornrpis_index_masked)$ transposer^r_r`r)r\mathsqrtr[viewrZ _sliding_chunks_query_key_matmulrhtype_asr,r.r/r&r0new_oneslist_get_global_attn_indices"_concat_with_global_key_attn_probscatr functionalsoftmaxfloat32rerw(_compute_attn_output_with_global_indices'_sliding_chunks_matmul_attn_probs_valuereshape contiguous'_compute_global_attn_output_from_hiddenlen)r?rzattention_maskr{is_index_global_attnis_global_attnoutput_attentionsrkrlrxrJ batch_sizer\ attn_scores#remove_from_windowed_attention_mask float_mask diagonal_maskrmrnrorpglobal_key_attn_scoresry attn_outputglobal_attn_outputglobal_attn_probsnonzero_global_attn_outputoutputss r"rIzLEDEncoderSelfAttention.forward~s&&//155  =11 hh}--  =11 )6););)=)=&YDN*** ZT^ Z Zy Z Z+** 4=111 %**7JPTP]^^hhijlmnn !&&w DNDMZZddefhijj ;; ;(G  0>/BAAAqqq$PTDT.U+9@@OO[[ /]=P1Q1Q1U  ==   Z__%6%6  7 7TEd  }$ K$$&&''   N  +a /! 3 ,     `: ` ` ` `DN ` `/!3a7 ` `KVK[K[K]K] ` `      '--.BCC  +,25&*%L%L+',G-I3U6[ &M&& " )%;[$IrRRRK']** Ru}+  &z?111aaatCS3TVYZZ '' 44  ]**:PTP]*^^ %**7JPTP]^^hhijlmnn   GG+%,G-I3U HKKFFM4+JK!!j'4>4=%YYYY[lYYY!++Aq1199':yYYddff   9484`4`+,G3U-I6[ / 5a55 1  1*<215qqq:\]^:__* & ?Y>]>]6q9::B??K4TTrT: ; 89J3 4((A..0  %  } $G2@dEVdw+---]ddr$ctj||}|jg|dd|d|dR}|S)z)pads rows and then flips rows and columnsNr)rrpadrr))hidden_states_paddedpaddings r" _pad_and_transpose_last_two_dimsz8LEDEncoderSelfAttention._pad_and_transpose_last_two_dimss "}00 '   938 ! & & ( (" - /C/H/H/L/L NbNgNghjNkNk   $#r$c2|\}}}}tj|d|dzf}|||d}|ddddd| f}||||||z}|ddddddddf}|S)aY shift every row 1 step right, converting columns into diagonals. Example: ```python chunked_hidden_states: [ 0.4983, 2.6918, -0.0071, 1.0492, -1.8348, 0.7672, 0.2986, 0.0285, -0.7584, 0.4206, -0.0405, 0.1599, 2.0514, -1.1600, 0.5372, 0.2629, ] window_overlap = num_rows = 4 ``` (pad & diagonalize) => [ 0.4983, 2.6918, -0.0071, 1.0492, 0.0000, 0.0000, 0.0000 0.0000, -1.8348, 0.7672, 0.2986, 0.0285, 0.0000, 0.0000 0.0000, 0.0000, -0.7584, 0.4206, -0.0405, 0.1599, 0.0000 0.0000, 0.0000, 0.0000, 2.0514, -1.1600, 0.5372, 0.2629 ] rrrN)r)rrrr)chunked_hidden_statestotal_num_heads num_chunkswindow_overlap hidden_dims r"_pad_and_diagonalizez,LEDEncoderSelfAttention._pad_and_diagonalize)sBCXB\B\B^B^?^Z " 1 1 !A~'9#:! ! !6 : : Z! ! !6 AAqqq"N?" "! !6 : : Z*9T! ! !6aaaAAAssl C$$r$ onnx_exportc@|s||dtj|d|dzd|dz|d}t |}|ddzdz |d<t |}|ddz|d<|||S|dtj|d|ddz |dz|dg}tj||j}t|dD],}|dd||z||zd|zzddf|dd|ddddf<-|S) zBconvert into overlapping chunks. Chunk size = 2w, overlap size = wrrrDtrunc rounding_moder)striderEN) rr)r.divrr as_stridedemptyrErange)rzrr chunk_size chunk_strideoverlapping_chunkschunks r"_chunkzLEDEncoderSelfAttention._chunkZs R)..""1%% -,,Q//.12DU\]]]"""1%% Mm002233J&qMA-1JqM 4 4 6 677L*1o2LO ++L+QQ Q   q ! ! Im((++^7 S S SVW W Q    q ! !  #[M5>)EN,BQEW,WWYZYZYZZ2 qqq%AAA~ . ."!r$returnc>|||dzdg}|ddddddf}|d}|ddd|ddd|dzf}||}t j|td | ||ddd|ddd|dzf<|dd| ddd|dz df}||}t j|td | ||dd| ddd|dz df<dS)Nrr)dims)rrinf) rtrilflipr*r)r. full_likefloatwherer-) input_tensoraffected_seq_lenbeginning_mask_2dbeginning_mask ending_maskbeginning_input ending_inputs r"_mask_invalid_locationsz/LEDEncoderSelfAttention._mask_invalid_locationss(112BDTWXDXYY^^``eelmkneoo*4D!!!+;<$))v)66 &qqq*;+;*;QQQ@VBRUVBV@V'VW'../C/C/E/EFFHM eEll]I I %##%% 7 7 QQQ)))111.D0@10D.DDE$AAA(8'8'9'9111@PST@T>U>W>W$WX !(():):)<)<== LQO 5<<-M M %   ""L 1 1 QQQ))**AAA1AA1E/F/H/HHIIIr$r^r_rc |\}}}}||dzzdksJd|dzd|||ksJtj||ddz }|dd||z||}|dd||z||}|||t |jdd }|||t |jdd }tjd ||f} | | d } | ||z|dz||dzdzf} | d d d d d |d |dzf| d d d dd d |d f<| d d d|d d |dzf| d d dd d |d f<| d d d d |dz d|dzd f| d d dd d d d |f<| d d dd |dz d|z d f| d d dd|d|f<| |||d|zdzdd} | | || S)a Matrix multiplication of query and key tensors using with a sliding window attention pattern. This implementation splits the input into overlapping chunks of size 2w (e.g. 512 for pretrained LEDEncoder) with an overlap of size window_overlap rDrz&Sequence length should be multiple of z. Given rrrrFzbcxd,bcyd->bcxy)rrrr)rNr) r)r.rr|rrgetattrrieinsumrrrr) r?r^r_rrrJrZr[ chunks_count!diagonal_chunked_attention_scoresdiagonal_attention_scoress r"rz8LEDEncoderSelfAttention._sliding_chunks_query_key_matmulsJ 49::<<0 GY.1,-222 Z^a5G Z ZQX Z Z322zz||sxxzz))))y.PPPSTT 1%%--j9.DgxXXmmAq!!))*y*@'8TT E>74; W\3]3]^^kk#~wt{MSX/Y/YZZ -2L9JUTWL,Y,Y)-1,Q,Q -|-R- - )%F$O$O ) #\A%5~~XYGY\]G] ^% % ! Ab AAqqq/>/#7^a%7#7 7A !!!!SbS!!!^__"<=@a AAr>??$8nq&8$8 8@ !!!!RNOO";<@a AAqqqNQ&'",nq.@.B.B B@ !!!!QRRO^O";<Op AAq&NQ&&N(:(<(< <O !!!!Q.(8!N:J"JK %>$B$B  7A,>,B% % )Aq// " $$%>OOO((r$ryr`c|\}}}}||dzzdksJ|dd|ddksJ|dd|zdzksJtj||ddz }|dd||ztj||d|d|zdz} |dd||z||}t j|dd||fd } ||z|dzd|z|f} | } | d|| dz| d| df} | | | } | | } tj d | | f}| ||||ddS) z Same as _sliding_chunks_query_key_matmul but for attn_probs and value tensors. Returned tensor will be of the same shape as `attn_probs` rDrNrrrrrr`rzbcwd,bcdh->bcwh) r)r.rr|rrrrrrrrr)r?ryr`rrrJrZr[rchunked_attn_probs padded_valuechunked_value_sizechunked_value_stride chunked_valuecontexts r"rz?LEDEncoderSelfAttention._sliding_chunks_matmul_attn_probs_values)49::<<0 GY.1,-2222  !$ RaR(88888q!!Q%7!%;;;;;y.PPPSTT (11!Q77??  " Ig~W E E E    "   1%%--j9.DgxXX}((A~~0V^`(aa )94lQ6FNHZ\de+2244  # 1!4 4  #  #   %//5GPd/ee !667IJJ,03E}2UVV||J 7HEEOOPQSTUUUr$cx|d}|}|d}t j||j|dk}|d}|dkd}||||fS)z&A&A&C&C#(<'C'CT'C'R'R$&+\ '0D0K& & & # - -" - 5 5&6" .H-O-OY]-O-^-^*2Lq1P0Y0Ycg0Y0h0h- ' ( . 1   r$cj|jd}||||j|j}||||<t jd||f} | dd} t j| jj | |d|dddddf<| dd} | S)Nrzblhd,bshd->blhsrr) rrrZr[r.rr|r/r&r0) r?rlrkrmrnrorprkey_vectors_only_globalattn_probs_from_global_keys r"rz:LEDEncoderSelfAttention._concat_with_global_key_attn_probss!&q) #."7"7 3T^T]# # GRRnFo BC&+\2CmUlEm%n%n"&@%I%I!Q%O%O" K28 9 9 = # 1! 46[\]6^`a`a`acdcdcd d &@%I%I!Q%O%O"))r$cN|jd}|dd|}||||j|j}||||<t j|dd|dddd} |d|| d|z  } | | ||j } | | zS)NrrrrD) rnarrowrrZr[r.matmulr|rr)rrrh) r?rxryrmrnrorattn_probs_only_globalvalue_vectors_only_globalattn_output_only_globalattn_probs_without_globalattn_output_without_globals r"rz@LEDEncoderSelfAttention._compute_attn_output_with_global_indices<s6 %a( ",!2!22q:U!V!V$1$;$; 3T^T]% % !IVVrHs!"DE #(, " , ,Q 2 2 8 8 : : ? ?4rI staticmethodrrr-rr.TensorrrPrrrrrrrRrSs@r"rUrUZs!!!!!L!^e^e^e^e@$$\$.%.%\.%`&"&"4&"&"&"\&"P 25< 2 2 2\ 2@)el@)@)gj@)@)@)@)D*V,*V/4|*VMP*V*V*V*VX  \ 8***<"D"D"DH]5]5]5]5]5]5]5r$rUceZdZfdZ d dejdejdzdejdzdejdzdedzd ed eejejdzeejdzffd ZxZ S) LEDEncoderAttentionctt|||_t j|j|j|_dS)N)rf)r=r>rUlongformer_self_attnrr]d_modeloutputr?rirfr@s r"r>zLEDEncoderAttention.__init__sI $;FX$V$V$V!i?? r$NFrzrr{rrrrc|||||||}||d}|f|ddz} | S)#Input shape: Batch x Time x Channelrzrr{rrrrrN)rr) r?rzrr{rrr self_outputsrrs r"rIzLEDEncoderAttention.forwardsa00')+!5)/ 1  kk,q/22 .<#33r$r) rLrMrNr>r.rr-tuplerIrRrSs@r"rrs@@@@@/3/348&*"'| t+,  $lT1  t   u|U\D0% 2E2LL Mr$rceZdZdZ ddedededzd edzd edzd edzf fd Z dd ej dej dzde dzdej dzdede ej ej dze dzff dZ xZ S)LEDDecoderAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrtFTNr\rZre is_decoderbias layer_idxct||_||_||_||z|_|j|z|jkrt d|jd|d|jdz|_||_||_ tj ||||_ tj ||||_ tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩r )r=r>r\rZrer[rscalingr r rr]k_projv_projq_projout_proj)r?r\rZrer r r r@s r"r>zLEDDecoderAttention.__init__s  "" !Y. =9 $ 6 6"dn""""" }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r$rzkey_value_statespast_key_valuesrrrc |du}|\}}} |||jz} d} |Ht|tr1|j|j} |r|j} n |j } n|} |r|n|} |r3|1| r/| j |jj }| j |jj }n| | }|| }||d|j|jdd}||d|j|jdd}|E| |||j\}}|r$t|trd|j|j<||jzd|jf}| |||j|jdd} | j|} |j|}|j|}|d}t+j| |dd}|||jz||fkr2t/d||jz||fd||||d||fkr+t/d |d||fd||||j|||z}|||jz||}t0j|d }|r=|||j||}|||jz||}nd}t0j||j|j }t+j||}|||jz||jfkr5t/d ||j||jfd||||j||jdd||| }||}|||fS) rNFrrrDTz$Attention weights should be of size rz!Attention mask should be of size rqruz `attn_output` should be of size )r)rr isinstancer is_updatedgetr cross_attention_cacheself_attention_cachelayerskeysvaluesrrrrZr[r|updaterr.rrrrrrerwr)r?rzrrrris_cross_attentionr1r'r\ query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shaper2 attn_weightsattn_weights_reshapedryrs r"rIzLEDDecoderAttention.forwards.T9"/"4"4"6"6Wi{{=11DL@   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66L#b$.$-PPZZ[\^_``J',,S"dndmTT^^_`bcddL*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>DN*B > #((gt~t}UU__`acdee +|+Z8 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4r.rr rrIrRrSs@r"r r sYGG !$"' !%CCCC C 4K C Tk C$;CCCCCC>15(,.2"' eCeC|eC ,-eC eC  t+ eC  eC u|U\D0%$,> ?eCeCeCeCeCeCeCeCr$r cVeZdZdedeffd Z d dejdejfdZxZ S) LEDEncoderLayerrirfct|j|_t |||_t j|j|_|j |_ t|j |_ |j |_ t j|j|j|_t j|j|j|_t j|j|_dSr<)r=r>rr\r self_attnr LayerNormself_attn_layer_normreractivation_function activation_fnactivation_dropoutr]encoder_ffn_dimfc1fc2final_layer_normrs r"r>zLEDEncoderLayer.__init__hs ,VX>>$&L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r$NFrzrc>|}|||||||}|d}tj||j|j}||z}||}|}|||}tj||j|j}| |}tj||j|j}||z}| |}|j tj kr_tj|s9tj|j jdz } tj|| | }|f|ddzS)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape *(batch, seq_len, embed_dim)* attention_mask (`torch.FloatTensor`): attention mask of size *(batch, 1, tgt_len, src_len)* where padding elements are indicated by very large negative values. rrrui)r0rrN)r-rrrerwr/r1r4r2r5r6r&r.float16isfiniteallr/rclamp) r?rzrr{rrrresidual attn_outputs clamp_values r"rIzLEDEncoderLayer.forwardts!~~')+!5)/ &  %Q  --mt|VZVc-dd  =0 11-@@  **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0 --m<<  %- / /}8U8U8Y8Y8[8[ /+m&9::>EK!K K<[YYYM,qrr"222r$r)) rLrMrNrrPr>r.rrIrRrSs@r"r+r+gs =y =C = = = = = = !(3(3|(3 (3(3(3(3(3(3(3(3r$r+ceZdZddeffd Z ddejdejdzdejdzd ejdzd edzd edzd edzfd Z xZ S)LEDDecoderLayerNrict|j|_t |j|j|jd||_|j|_t|j |_ |j |_ tj|j|_t |j|j|jd||_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)r\rZrer r )rer r )r=r>rr\r decoder_attention_headsattention_dropoutr-rerr0r1r2rr.r/ encoder_attnencoder_attn_layer_normr]decoder_ffn_dimr4r5r6)r?rir r@s r"r>zLEDDecoderLayer.__init__s ,n4,    ~ #F$>?"(";$&L$@$@!/ N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r$FTrzrencoder_hidden_statesencoder_attention_maskrr use_cachec 2|} |||||\}} } tj||j|j}| |z}||}d} d} |f|} ||||||\}} } tj||j|j}| |z}||}|} || |}tj||j |j}| |}tj||j|j}| |z}| |}|f}|r|| | fz }|r||fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape *(batch, seq_len, embed_dim)* attention_mask (`torch.FloatTensor`): attention mask of size *(batch, 1, tgt_len, src_len)* where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape *(batch, seq_len, embed_dim)* encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size *(batch, 1, tgt_len, src_len)* where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`): Whether the base model outputs attentions. This requires the attentions tensor to be reshaped in this function. )rzrrrruN)rzrrrr) r-rrrerwr/rDrEr1r4r2r5r6)r?rzrrGrHrrrIkwargsr<self_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightsrs r"rIzLEDDecoderLayer.forwards0!?Cnn'+)/ ?M? ? ; (*;  --mt|VZVc-dd  =0 11-@@ (,$! ,$HNRN_N_+!65 /"3 O`OO KM-/KM11-4 >G  * ) )Gr$r<)NNNNFT) rLrMrNrr>r.rr r-rIrRrSs@r"r@r@s==y======>/3596:(,).!%GG|G t+G %|d2 G !& t 3 G  G $;G$;GGGGGGGGr$r@cJeZdZdZdedededeffd ZdejfdZ xZ S) LEDClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcttj|||_tj||_tj|||_dS)N)rv)r=r>rr]denseDropoutrer)r?rRrSrTrUr@s r"r>zLEDClassificationHead.__init__ sY Yy)44 zN333  )[99 r$rzc||}||}tj|}||}||}|Sr<)rerWr.tanhr)r?rzs r"rIzLEDClassificationHead.forwards[ ]33  =11  =11  ]33  m44 r$) rLrMrNrOrPrr>r.rrIrRrSs@r"rQrQs77 : : : :  : : : : : :U\r$rQcHeZdZUeed<dZdZedZfdZ xZ S)LEDPreTrainedModelriledTc|jj}tjgddddd|gg|j}|||d}|S)N)r rDr rDr)rr)rirr.tensorrEne)r? pad_tokenr dummy_inputss r"rgzLEDPreTrainedModel.dummy_inputs$saK, L"2"2"2Q2q)4L!MVZVabbb 'll955"  r$ct|t|trt j|jdSdSr<)r= _init_weightsrLEDForConditionalGenerationinitzeros_final_logits_bias)r?moduler@s r"riz LEDPreTrainedModel._init_weights.sQ f%%% f9 : : 2 K0 1 1 1 1 1 2 2r$) rLrMrNr__annotations__base_model_prefixsupports_gradient_checkpointingpropertyrgrirRrSs@r"r\r\sk &*# X222222222r$r\zi Base class for LEDEncoder's outputs, with potential hidden states, local and global attentions. ) custom_introceZdZUdZejed<dZeejdfdzed<dZ eejdfdzed<dZ eejdfdzed<dS)LEDEncoderBaseModelOutputaI attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x + attention_window + 1)`, where `x` is the number of tokens with global attention mask. Local attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token in the sequence to every token with global attention (first `x` values) and to every token in the attention window (remaining `attention_window + 1` values). Note that the first `x` values refer to tokens with fixed positions in the text, but the remaining `attention_window + 1` values refer to tokens with relative positions: the attention weight of a token to itself is located at index `x + attention_window / 2` and the `attention_window / 2` preceding (succeeding) values are the attention weights to the `attention_window / 2` preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the first `x` attention weights. If a token has global attention, the attention weights to all other tokens in `attentions` is set to 0, the values should be accessed from `global_attentions`. global_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x)`, where `x` is the number of tokens with global attention mask. Global attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token with global attention to every token in the sequence. last_hidden_stateN.rz attentionsglobal_attentions) rLrMrNrOr. FloatTensorrorzrrwrxr$r"ruru4s2((((:>M5*C/047>>>7;Je',-4;;;>BuU.34t;BBBBBr$ruz Base class for model encoder's outputs that also contains : pre-computed hidden states that can speed up sequential decoding. cxeZdZUdZdZejdzed<dZe dzed<dZ e ejdfdzed<dZ e ejdfdzed<dZ e ejdfdzed<dZejdzed <dZe ejdfdzed <dZe ejdfdzed <dZe ejdfdzed <dS) LEDSeq2SeqModelOutputal last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the decoder of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. encoder_global_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x)`, where `x` is the number of tokens with global attention mask. Global attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token with global attention to every token in the sequence. Nrvr.decoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_staterGencoder_attentionsencoder_global_attentions)rLrMrNrOrvr.ryrorr r}rr~rrrGrrrzr$r"r|r|[s6(37u(4/666$(OUT\(((BF5!2C!784?FFF?Ce/45<CCC=AeE-s23d:AAA:>u047>>>BF5!2C!784?FFF?Ce/45<CCCFJuU%6%;u047>>>BF5!2C!784?FFF?Ce/45<CCCFJuU%6%;u047>>>BF5!2C!784?FFF?Ce/45<CCCFJuU%6%;z'LEDEncoder.__init__..s#$f$f$fA_VQ%?%?$f$f$fr$F)r=r>reencoder_layerdrop layerdroprr padding_idxmax_encoder_position_embeddingsmax_source_positionsrrgrPrnum_hidden_layersrr Embedding vocab_size embed_tokensr8embed_positions ModuleListrencoder_layersrr.layernorm_embeddinggradient_checkpointing post_init)r?rir\r@s ` r"r>zLEDEncoder.__init__s    ~ 1N !.$*$J! f-s 3 3 &*a// !TUUU&!++ !OPPP'-'>&?&BZ&ZF # #6*++v/GGG a & 8aaBEfF]B^B^aa L):ItGWXX<  %   m$f$f$f$fvOdIeIe$f$f$fgg #%< #:#: &+# r$rglobal_attention_maskc&| ||dzz}n|dz}|S)Nrrz)r?rrs r"_merge_to_attention_maskz#LEDEncoder._merge_to_attention_mask!s.  %+/Dq/HINN3Q6Nr$r inputs_embedsrct|jjtr |jjnt |jj}|dzdkrt d|||jn|j}|dd\}}|||zz |z} | dkrtd|d|| zd||$tj |d| f|}|[| || f|jj tj } || } tj|| gd }tj |d| fd }| |||fS) zbA helper function to pad tokens and mask to work with implementation of Longformer self-attention.rDrz2`attention_window` should be an even value. Given Nz(Input ids are automatically padded from z to z0 to be a multiple of `config.attention_window`: r)r&rrqF)rrirgrPrrrlogger warning_oncerrrnew_fullrr.rGrr) r?rrrrrg input_shaperrJ padding_leninput_ids_paddinginputs_embeds_paddings r"_pad_to_window_sizezLEDEncoder._pad_to_window_size-s$+6<< 3DK ( (T[122  a 1 $ $dRbddee e)2)>iooMDW )"1"o G''4D*DDHXX ??   A7AAR]H]AA.>AA   $M--i![9IQ]-^^ ($1$:$:-K,*%;%%! )-(9(9:K(L(L% % =:O*PVX Y Y Y ]..K 0/NI~}DDr$Nc ||n |jj}||n |jj}||n |jj}||t d||t d|||}|@t j|dd|j t j }|| ||}| ||||jj \}}}|1|} |d| d}n||dd} |#t||jddddddf}|dk} |dk} | } || } || z}||}t,j||j|j}|rd nd}|rd nd}|r| rd nd}t5|jD]\}}|r||fz}t jg}|jr||jkrd }n|||| | | | }|d}|rB||d d d fz}| r ||d d dfz}|r||fz}dkrI|ddd f}|rt?fd|D}|rt?fd|D}|st?d||||fDStA||||S)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) global_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to decide the attention given on each token, local attention or global attention for the encoder. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the [Longformer paper](https://huggingface.co/papers/2004.05150) for more details. Mask values selected in `[0, 1]`: - 0 for local attention (a sliding window attention), - 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them). inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsr)rEr&)rrrrrrurz)NNN)rr{rrrrrDrc36K|]}|ddd fVdSr<rzrstaters r" z%LEDEncoder.forward..s7&[&[5uQQQ + -='>&[&[&[&[&[&[r$c3BK|]}|ddddd ddfVdSr<rzrs r"rz%LEDEncoder.forward..sC&a&auQQQ=[L=!!!-C'D&a&a&a&a&a&ar$c3K|]}||V dSr<rzrvs r"rz%LEDEncoder.forward..s1eferererererr$rvrzrwrx)!riroutput_hidden_states return_dictrrr.onesr)rErGrrrrr6r&flattenanyitemrrrrrerw enumeraterrandrr|rru)r?rrrrrrrrKrr{rr embed_posrzencoder_statesall_attentionsall_global_attentionsidx encoder_layerdropout_probability layer_outputsrs @r"rIzLEDEncoder.forwardXsbf2C1N--TXT_Tq$8$D $+Jj &1%cdd d  =#8TUU U   --i88M  !"Z (:(:(<(K`aaNAE@X@X)'1 AYA A = Y   #..**K!r;r?;;II  &',,..ss3K  %@Q^QdeefgfgfgijlmopopopfpqN)1,-1-5577;;==BBDD((55 % 1 00??  --mt|VZVc-dd 3=0:d'8 V^ VRV"+DK"8"8 h h C# C!/=2B!B"'*R.. } 1"5"F"F 2 - !#1$3)=#1&7 !!! !.a 0   h!/=3C3M3MaQR3S3S2U!U!h,A]STEUE_E_`acdEeEeDg,g)  ?+}.>>N ??)!!!]{l]*:;M# \!&&[&[&[&[N&[&[&[![![  b!&&a&a&a&aR`&a&a&a!a!a )>>K`a )+(%3     r$)NNNNNNN) rLrMrNrOrr>r.rrrPrrIrRrSs@r"rrs"y""""""H u| \a\h    )E<)E )E| )E  )E)E)E)EZ"!^ ^ ^ ^ ^ ^ ^ ^ r$rcFeZdZdZdeffd Z ddZxZS) LEDDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`LEDDecoderLayer`] Args: config: LEDConfig embed_tokens (nn.Embedding): output embedding rictj|_j|_j|_j|_tj j j |j|_ t|jj |_tjfdt#jD|_tjj |_d|_|dS)Nc2g|]}t|S))r )r@rs r"rz'LEDDecoder.__init__..s&$p$p$pa_Vq%I%I%I$p$p$pr$F)r=r>redecoder_layerdroprrrmax_decoder_position_embeddingsmax_target_positionsrrrrrr8rrrdecoder_layersrr.rrrr?rir@s `r"r>zLEDDecoder.__init__s    ~ 1!.$*$J!L):FNDL\]]<  % N  m$p$p$p$pSXY_YnSoSo$p$p$pqq #%<#?#? &+# r$Nc ^| | n |jj} | | n |jj} ||n |jj}| | n |jj} ||t d|1|} |d| d}n.||dd} nt d|||}|j r%|j r|rt dd}|r8|6tt|jt|j}||nd}d}| dd krt!|j||| }t#|j||| }|| |}||z}||}t(j||j|j }| rd nd}| rd nd}| rd nd}t/|jD]h\}}| r||fz }|j r t3jg}||jkr4||||||| |}|d}| r||d fz }||dfz }i| r||fz }| st9d|||||fDSt;|||||S)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) global_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to decide the attention given on each token, local attention or global attention. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the [Longformer paper](https://huggingface.co/papers/2004.05150) for more details. Mask values selected in `[0, 1]`: - 0 for local attention (a sliding window attention), - 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them). encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timerzEYou have to specify either decoder_input_ids or decoder_inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rirr)rirrr)rirrrGrurz)rHrrrIrDc3K|]}||V dSr<rzrs r"rz%LEDDecoder.forward..s0= ===r$)rvrrzrwr)rirrrIrrr)rrrrwrrr r get_seq_lengthrr rrrrrerrr.rrrr)r?rrrrGrHrrrIrrrrKrrBcombined_attention_maskrKrzall_hidden_statesall_self_attnsall_cross_attentionsr decoder_layerrrs r"rIzLEDDecoder.forwardsV2C1N--TXT_Tq$8$D $+Jj "+!6IIDKstt t  "#..**K!r;r?;;II  &',,..ss3KKdee e   --i88M  & "4= " "##p"  v01,dk2R2R2RT`hlhsTtTtTtuuOETE`!?!?!A!A!Afg"& r?Q  &8{+- / ''' #";;'1"7 " " " ((6LMM % 1 00??  --mt|VZVc-dd #7@BBD0:d%6@rrD"+DK"8"8 < < C# 6!m%55!} &+jnn#&77)M'%'= /"3#M*!,M  <=#3"55$q)9(;;$  2 -!1 1  ':K^]qr  9+++%1     r$) NNNNNNNNNNN)rLrMrNrOrr>rIrRrSs@r"rrsy,""#!q q q q q q q q r$rcxeZdZdddZdeffd ZdZdZe dde j dzd e j dzd e j dzd e j dzd e e e j dzd e j dzdedzde j dzde j dzdedzdedzdedzdedzde e j ezfdZxZS)LEDModelz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightrict||j|j}}t j||j||_t||_ t||_ | dSr<) r=r>rrrrrsharedrencoderrdecoderr)r?rirrr@s r"r>zLEDModel.__init__sw    "("5v7HZ l:v~{KK !&)) !&))  r$c|jSr<)r)r?s r"get_input_embeddingszLEDModel.get_input_embeddingss {r$cX||_|j|j_|j|j_dSr<)rrrr)r?r`s r"set_input_embeddingszLEDModel.set_input_embeddingss' $(K !$(K !!!r$Nrrdecoder_input_idsdecoder_attention_maskencoder_outputsrrrdecoder_inputs_embedsrIrrrrc | | n |jj} | | n |jj} | | n |jj} | | n |jj} |'| %t ||jj|jj}||||||| | | }n| rt|tsjt|dt|dkr|dndt|dkr|dndt|dkr|dnd}| |||d||| | | | |  }| s||zSt|j|j|j|j|j|j|j|j|j S) K decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`LedTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) LED uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_led._prepare_decoder_inputs`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. global_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to decide the attention given on each token, local attention or global attention for the encoder. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the [Longformer paper](https://huggingface.co/papers/2004.05150) for more details. Mask values selected in `[0, 1]`: - 0 for local attention (a sliding window attention), - 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them). N)rrrrrrrrrrDrr) rrrGrHrrrIrrr) rvrr}r~rrrGrr)rirrrIrr#rrrrrurrr|rvrrzrwrrx)r?rrrrrrrrrrIrrrrKdecoder_outputss r"rIzLEDModel.forwards^2C1N--TXT_Tq$8$D $+Jj "+!6IIDK)F 24;3T[5W!!   ""ll#-&;+"3%9'+OO O=V!W!W 7"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--t8;O8L8Lq8P8P/!"4"4VZ O,,'1"1!"4#1+//!5#'    5"_4 4$-?+;"1"?.9,=&5&G"1"?.9&5&G    r$) NNNNNNNNNNNNN)rLrMrN_tied_weights_keysrr>rrrr. LongTensorrrryr r-r|rIrRrSs@r"rrs(7'6 y      000 .2.259:>BF:>(,26:>!%)-,0#'k k #d*k  t+k !+d2 k !& 04 7 k uU%6784? k  %047k k (4/k  %047k $;k  $;k #Tkk D[k u| 4 4!k k k ^k k k k k r$rzU The LED Model with a language modeling head. Can be used for summarization. c!eZdZdZdgZddiZdeffd Z dd ed edzd e d e j ffd Z d ed dfdZ e d dejdzdejdzdejdzdejdzdeeejdzdejdzdedzdejdzdejdzdejdzde dzde dzde dzde dzd eejezfdZdejfdZxZS)!rjr]rmzlm_head.weightzled.shared.weightriclt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)NrmrFr)r=r>rr]register_bufferr.zerosrr9rr]rlm_headrrs r"r>z$LEDForConditionalGeneration.__init__`s    F## 0%+q$(/B`>a2b2bcccy1OV[\\\  r$NTnew_num_tokenspad_to_multiple_of mean_resizingrct|||}||jjd|S)Nr)r=resize_token_embeddings_resize_final_logits_biasrHr)r?rrrnew_embeddingsr@s r"rz3LEDForConditionalGeneration.resize_token_embeddingsisG88I[]jkk &&~'<'B1'EFFFr$c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nrrrrqrm)rmrr.rrErr)r?rold_num_tokensnew_bias extra_biass r"rz5LEDForConditionalGeneration._resize_final_logits_biasps/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r$rrrrrrrrrlabelsrIrrrc ||n |jj}| G| rtdd} |'| %t | |jj|jj}|||||||||| | | | | }||d|j z}d}| Kt}|| d|jj | d}|s|f|ddz}||f|zn|St|||j|j|j|j|j|j|j|j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`LedTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) LED uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_led._prepare_decoder_inputs`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. global_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to decide the attention given on each token, local attention or global attention for the encoder. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the [Longformer paper](https://huggingface.co/papers/2004.05150) for more details. Mask values selected in `[0, 1]`: - 0 for local attention (a sliding window attention), - 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Summarization: ```python >>> import torch >>> from transformers import AutoTokenizer, LEDForConditionalGeneration >>> model = LEDForConditionalGeneration.from_pretrained("allenai/led-large-16384-arxiv") >>> tokenizer = AutoTokenizer.from_pretrained("allenai/led-large-16384-arxiv") >>> ARTICLE_TO_SUMMARIZE = '''Transformers (Vaswani et al., 2017) have achieved state-of-the-art ... results in a wide range of natural language tasks including generative language modeling ... (Dai et al., 2019; Radford et al., 2019) and discriminative ... language understanding (Devlin et al., 2019). ... This success is partly due to the self-attention component which enables the network to capture contextual ... information from the entire sequence. While powerful, the memory and computational requirements of ... self-attention grow quadratically with sequence length, making it infeasible (or very expensive) to ... process long sequences. To address this limitation, we present Longformer, a modified Transformer ... architecture with a self-attention operation that scales linearly with the sequence length, making it ... versatile for processing long documents (Fig 1). This is an advantage for natural language tasks such as ... long document classification, question answering (QA), and coreference resolution, where existing approaches ... partition or shorten the long context into smaller sequences that fall within the typical 512 token limit ... of BERT-style pretrained models. Such partitioning could potentially result in loss of important ... cross-partition information, and to mitigate this problem, existing methods often rely on complex ... architectures to address such interactions. On the other hand, our proposed Longformer is able to build ... contextual representations of the entire context using multiple layers of attention, reducing the need for ... task-specific architectures.''' >>> inputs = tokenizer.encode(ARTICLE_TO_SUMMARIZE, return_tensors="pt") >>> # Global attention on the first token (cf. Beltagy et al. 2020) >>> global_attention_mask = torch.zeros_like(inputs) >>> global_attention_mask[:, 0] = 1 >>> # Generate Summary >>> summary_ids = model.generate(inputs, global_attention_mask=global_attention_mask, num_beams=3, max_length=32) >>> print(tokenizer.decode(summary_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)) ``` Example Conditional generation : ```python >>> from transformers import AutoTokenizer, LEDForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384") >>> TXT = "My friends are but they eat too many carbs." >>> model = LEDForConditionalGeneration.from_pretrained("allenai/led-base-16384") >>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"] >>> prediction = model.generate(input_ids)[0] >>> print(tokenizer.decode(prediction, skip_special_tokens=True)) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) rrrrrrrrrIrrrrrr) rrrr}r~rrrGrr)rirrwarningr#rrr]rrmrrrrrr}r~rrrGrr)r?rrrrrrrrrrrIrrrrKr lm_logitsmasked_lm_lossloss_fctrs r"rIz#LEDForConditionalGeneration.forwardysN&1%rPr-rrrrrr.rrrryr rrIrrRrSs@r"rjrjTsg ':&;#-yae!7:TzY] <<<<<<.2.259:>BF:>(,26:>*.!%)-,0#'V V #d*V  t+V !+d2 V !& 04 7 V uU%6784? V  %047V V (4/V  %047V  4'V $;V  $;V #TkV D[V " u| 1 1#V V V ^V phELhhhhhhhhr$rjc!|eZdZfdZe ddejdzdejdzdejdzdejdzdeeej dzdej dzd ejdzd ejdzd ej dzd ej dzd e dzde dzde dzde dzdeeje zfdZ xZ S)LEDForQuestionAnsweringct|d|_|j|_t||_t j|j|j|_| dS)NrD) r=r> num_labelsrr]rr]rX qa_outputsrrs r"r>z LEDForQuestionAnswering.__init__sm     +F##)F$68IJJ r$Nrrrrrrstart_positions end_positionsrrrIrrrrc ||n |jj}||d} |||||||| | | | | | }|d}||}|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zd z }|s||f|ddz}||f|zn|St||||j |j |j|j|j|j|j|j S) rNF) rrrrrrrrIrrrrrrrq) ignore_indexrD) rrrrr}r~rrrGrr)rirr]rsplitsqueezerrr)r;rrrr}r~rrrGrr)r?rrrrrrrrrrrIrrrrKrsequence_outputrrr total_loss ignored_indexr start_lossend_lossrs r"rIzLEDForQuestionAnswering.forward$s[`&1%rr.rrrryr-rrIrRrSs@r"r r s     .2.259:>BF:>371526:>!%)-,0#'m m #d*m  t+m !+d2 m !& 04 7 m uU%6784? m  %047m )D0m '$.m (4/m  %047m $;m  $;m #Tkm D[m " u| E E#m m m ^m m m m m r$r )rjr rr\r<)?rOr} dataclassesrr.rtorch.nnrrrk activationsr cache_utilsr r r generationr masking_utilsr rmodeling_layersrmodeling_outputsrmodeling_utilsrutilsrrrconfiguration_ledr get_loggerrLrrrPr#r&r6rr8ModulerUrr r+r@rQr\rur|rrrrrrrjr __all__rzr$r"r*s !!!!!! %%%%%%&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999IIIIII------9999999999((((((  H % % %, c [^     ##el#5;#Y\_cYc####$*****BL***$c 5c 5c 5c 5c 5bic 5c 5c 5L")@DCDCDCDCDC")DCDCDCN53535353530535353pddddd0dddNBI022222222*  CCCCC CC  C@  KKKKKKKK  K@  KKKKKKK   K@  KKKKKKK   K@  KKKKK[KK   K>C C C C C #C C C LN N N N N #N N N bF F F F F !F F F R zhzhzhzhzh"4ozhzh  zhz{ { { { { 0{ { { |   r$