tj-c.dZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z ddl mZdd lmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%ej&e'Z(Gddej)Z*Gddej+Z,Gddej+Z-GddeZ.eGddeZ/eGdde/Z0ed Gd!d"e/eZ1gd#Z2dS)$zPyTorch XGLM model.N)nn)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)merge_with_config_defaults)OutputRecordercapture_outputs) XGLMConfigc VeZdZdZd dededededzffd Zd ejffd Z xZ S) XGLMScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?num_embeddings embedding_dim padding_idx embed_scaleNc\t|||||_dSN)super__init__r )selfrrrr __class__s d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/xglm/modeling_xglm.pyr$z XGLMScaledWordEmbedding.__init__-s-  DDD& input_idscVt||jzSr")r#forwardr )r%r)r&s r'r+zXGLMScaledWordEmbedding.forward1s!wwy))D,<<r@rAregister_buffer)r%rrr emb_weightss r'r<z.XGLMSinusoidalPositionalEmbedding.make_weights@sl(( TT 4 # # _%..t|/A$,J].^^K Y FFFFFr(c|dz}tjd|dz z }tjtj|tj| z}tj|tjd|dz}tjtj |tj |gd |d}|dzdkr+tj|tj |dgd}| d||ddf<| tjS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". r:i'r)r@rdimN)mathlogr2exparangeint64r1 unsqueezecatsincosviewzerosrEget_default_dtype)rrrhalf_dimembs r'rCz/XGLMSinusoidalPositionalEmbedding.get_embeddingHs?!A%huooA.i XU[AAAGGIISDPQQl>===CCEEOOPQRRUXUbUbcdUeUeei338a@@@EEnVXYY 1  ! !)S%+na"@"@AqIIIC  ""#C QQQ vve-//000r(r position_idspast_key_values_lengthc|\}}||jz}d|z|z}||jdkr!|||j|j|jd|d|||jjd S)Nr:rrK) sizer;r>r<rr index_selectrUshapedetach)r%rZr[bszseq_lenmax_poss r'r+z)XGLMSinusoidalPositionalEmbedding.forward]s#((** W#dk1 g+ 66 T\&&q)) ) )   gt'94;K L L L|((L,=,=b,A,ABBGGWVZVbVhikVlmmttvvvr(r")Nr)r,r-r.r/r0r$r< staticmethodrCr2no_gradr3r+r4r5s@r'r7r75s6NNSScS#SCRVJSSSSSSGG3GsGQTW[Q[GGGG11c1#1CRVJ111\1(U]__wwEL4$7wX[www_wwwwwr(r7c eZdZdZ ddedededzd edzd edzd edzf fd Z dd ej dej dzde dzdej dzde e de ej ej dze ej dzff dZxZS) XGLMAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias layer_idxct||_||_||_||z|_|j|z|jkrt d|jd|d|jdz|_||_||_ tj ||||_ tj ||||_ tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rm)r#r$rirjrkhead_dim ValueErrorscalingrlrnrLineark_projv_projq_projout_proj)r%rirjrkrlrmrnr&s r'r$zXGLMAttention.__init__ls  "" !Y. MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r( hidden_stateskey_value_statespast_key_valuesattention_maskkwargsreturnc n |du}|\}}} |r |jdn|} |||jz} d} |Ht |t r1|j|j} |r|j } n |j } n|} |r|n|}|r3|1| r/| j |jj }| j |jj }n||}||}||| d|jdd}||| d|jdd}|E| |||j\}}|r$t |t rd|j|j<||jzd|jf}| |||j|jdd} | j|} |j|}|j|}|d} t-j| |dd}|||jz|| fkr2t1d||jz|| fd||||d|| fkr+t1d |d|| fd||||j|| |z}t-j|t-jt-j|jj|j }|||jz|| }|jt,jkrJt@j!"|dt,j# $t,j}n!t@j!"|d }|||j|| }|||jz|| }t@j!%||j%|j& }t-j||}|||jz||jfkr5t1d||j||jfd||||j||j}|dd}||||j'}|(|}||fS)z#Input shape: Batch x Time x ChannelNrFrKr:Tz$Attention weights should be of size z , but is z!Attention mask should be of size )rA)rJr@rIptrainingz `attn_output` should be of size ))r]r_rwrs isinstancer is_updatedgetrncross_attention_cacheself_attention_cachelayerskeysvaluesrurvrUrq transposeupdaterjreshaper2bmmrrmaxtensorfinfor@minrAfloat16r functionalsoftmaxfloat32rErkrrirx)r%ryrzr{r|r}is_cross_attentionratgt_len_src_len query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shape attn_weightsattn_weights_reshaped attn_probs attn_outputs r'r+zXGLMAttention.forwardsi.T9',,..Wa/AN"(++w{{=11DL@   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66L#gr4=IISSTUWXYYJ',,S'2t}MMWWXY[\]]L*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>DN*B > #((gt~t}UU__`acdee +|+Z8 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL 9el5;|7I+J+J+NWcWjkkkL(,,S4>-A7GTTL   . .=002U]0[[^^_d_lmmLL=0020FFL !- 1 1#t~wPW X X,11#2FQXYY ]**<4?"(";  % H -. 00# !!!D ,.<+G+GD ($&L$@$@!9T^V^<<9V^T^<< " T^ < <r(Tryr|encoder_hidden_statesencoder_attention_maskr{ use_cacher}r~c |}||}|j|f||d|\}} tj||j|j}||z}|]|}||}|j|f|||d|\}} tj||j|j}||z}|}||}| | |}tj||j |j}| |}tj||j|j}||z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )r{r|rN)rzr|r{) rrrrrkrrrrrrrr) r%ryr|rrr{rr}residualrs r'r+zXGLMDecoderLayer.forwards*!11-@@ *4>  +)      q  --mt|VZVc-dd  =0  ! ,$H 88GGM0t0 !65 /        M1M11-4)r%modulerGr&s r'rz!XGLMPreTrainedModel._init_weights\s f%%% f? @ @ 4 ..$v}4f6JFL^K Jv~{ 3 3 3 3 3  4 4r() r,r-r.r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr4r5s@r'rrUs[ &*#+,444444444r(rcfeZdZeeeddeedddZdeffd Ze e e dd e j dzd e j dzd e j dzd e j dzd e j dzdedzde j dzdedzdeedee j ezfdZxZS) XGLMModelrr)index layer_namer)ry attentionscross_attentionsrchtj|_j|_j|_j|_jrtj j nd}tj j |j||_tjj j|_t#jfdt'jD|_t#jj |_d|_|dS)Nr)r c2g|]}t|S))rn)r).0irs r' z&XGLMModel.__init__..~s'$m$m$mq%5f%J%J%J$m$m$mr(F)r#r$rk layerdrop pad_token_idrmax_position_embeddingsmax_target_positionsscale_embeddingrLsqrtrr vocab_size embed_tokensr7embed_positionsr ModuleListrange num_layersrr layer_normgradient_checkpointing post_init)r%rr r&s ` r'r$zXGLMModel.__init__ms    ~ )!.$*$B!393IRdi///s 3  v~t/?[    A  * N     m$m$m$m$mTYZ`ZkTlTl$m$m$mnn ,v~66&+# r(Nr)r|rZrrr{ inputs_embedsrr}r~c  |du|duz rtd|||}|r[|Y| |jjr6t t |jt |jnt |j}||nd} t|j|||}|Stj | |j d| ztj ||j n|j }| d}||t|j|||}|||| |j z} t"j| t)|j|j } t-|jD];\} } |jr tjg}||jkr,| | ||f|||d | } <|| } t7| | S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embeds)rr)rrr|r{rr?)rrr|rr)rr{r)last_hidden_stater{)rrrris_encoder_decoderr rget_seq_lengthr r2rOr_longrArQr rrErrrkr1r enumeraterrandrrr)r%r)r|rZrrr{rrr}r[ryidx decoder_layerdropout_probabilitys r'r+zXGLMModel.forwards8  -t"; < [YZZ Z   --i88M  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  FUE`!?!?!A!A!Afg+;')+       <&#A&)??j+4+@y''mFZ L (11!44L ! ,1G1S%>{+5&; &&& "&(<(<\Ka(b(b(e(e  ) )   --muT\?R?R]a]j-kk "+DK"8"8   C} &+jnn#&77)M%(> /# MM 66 8++    r()NNNNNNNN)r,r-r.rrrg_can_record_outputsrr$rrrr2r3rrrrrrr+r4r5s@r'rres*$n]! TTT*N=n]]] z0 *..2,0596:(,-1!%] ] <$&]  t+] lT) ] %|d2 ] !& t 3 ] ] |d*] $;] +,]  u| H H] ] ] ^_ ] ] ] ] ] r(rz The XGLM Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc\eZdZdZddiZfdZeee dde j dzde j dzd e j dzd e j dzd e j dzd e dzd e j dzde j dzde dzde e j zdeedee j ezfdZxZS)XGLMForCausalLMrzlm_head.weightzmodel.embed_tokens.weightct|t||_t j|j|jd|_| dS)NFrp) r#r$rrrrt hidden_sizerlm_headr)r%rr&s r'r$zXGLMForCausalLM.__init__s`    v&& y!3V5FUSSS  r(Nrr)r|rZrrr{rlabelsrlogits_to_keepr}r~c |jd|||||||| d| } | j} t| trt | dn| }|| dd|ddf}d}|-||||jj|jj }t||| j | j | j | jS)ai encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )r)r|rZrrr{rrN)rr)losslogitsr{ryrr)rrrr0slicer loss_functionrrrrr{ryrr)r%r)r|rZrrr{rrrrr}outputsry slice_indicesrrs r'r+zXGLMForCausalLM.forwards F>HTZ > )%"7#9+' > >  > >  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %%;1![5 &D1#3!/)$5     r() NNNNNNNNNr)r,r-r.r_tied_weights_keysr$rrrr2r3rrr0rrrrr+r4r5s@r'rrs *,GH *..2,0596:(,-1&*!%-.A A <$&A  t+A lT) A %|d2 A !& t 3 A A |d*A  t#A $;A el*A +,A  u| @ @A A A ^_ A A A A A r(r)rrr)3r/rLr2rrr activationsr cache_utilsrrr generationr masking_utilsr r modeling_layersr modeling_outputsrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrrconfiguration_xglmr get_loggerr,logger EmbeddingrModuler7rgrrrr__all__rr(r'rs &&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999llllllll------&&&&&&@@@@@@@@@@777777EEEEEEEE******  H % % = = = = =bl = = =1w1w1w1w1w 1w1w1whL2L2L2L2L2BIL2L2L2^ZZZZZ1ZZZz 4 4 4 4 4/ 4 4 4     #   D P P P P P )?P P  P f B A Ar(