tj\ddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0ddl1m2Z2e*j3e4Z5edGddej6Z7Gddej6Z8Gddej6Z9Gddej6Z:Gd d!ej6Z;Gd"d#ej6Zd'ej?d(e@d)ej?fd*ZA dGd,ej6d-ej?d.ej?d/ej?d0ej?dzd1eBd2eBd3e%e'fd4ZCGd5d6ej6ZDGd7d8eZEe(Gd9d:e#ZFe(Gd;dej?eHej?zdzd?e@dzd0ej?dzd)ej?e@zfd@ZIGdAdBeFeZJGdCdDeeFZKgdEZLdS)I)Callable)OptionalN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hub)create_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) JetMoeConfigRMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) JetMoeRMSNormư>epsreturnNcttjt j||_||_dS)z< JetMoeRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer' __class__s h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/jetmoe/modeling_jetmoe.pyr+zJetMoeRMSNorm.__init__2sD l5:k#:#:;; # hidden_statesc|j}|tj}|ddd}|tj||jzz}|j||zS)NT)keepdim) dtypetor-float32powmeanrsqrtr0r/)r1r6 input_dtypevariances r4forwardzJetMoeRMSNorm.forward:s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r5cHt|jjd|jS)Nz, eps=)tupler/shaper0)r1s r4 extra_reprzJetMoeRMSNorm.extra_reprAs& )**II$2GIIIr5)r&) __name__ __module__ __qualname__floatr+r-TensorrCrG __classcell__r3s@r4r%r%0s$$$$$$$$$$;U\;el;;;;JJJJJJJr5r%c eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)JetMoeRotaryEmbeddinginv_freqNconfigct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrQF) persistentoriginal_inv_freq) r*r+max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrRrope_parametersrTcompute_default_rope_parametersrattention_scalingregister_bufferclone)r1rRdevice rope_init_fnrQr3s r4r+zJetMoeRotaryEmbedding.__init__Hs "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr5r`z torch.deviceseq_lenr(z torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr8r;)r`r;) r[getattrr2num_attention_headsr-arangeint64r<rK)rRr`rbbasedimattention_factorrQs r4r\z5JetMoeRotaryEmbedding.compute_default_rope_parametersXs&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r5cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) Nrr9r!mpscpuF) device_typeenabledr8rlrf)rQrKexpandrFr<r` isinstancetypestrr transposer-catcosr]sinr;) r1x position_idsinv_freq_expandedposition_ids_expandedrqfreqsembrzr{s r4rCzJetMoeRotaryEmbedding.forwardvs!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*NNNN)rHrIrJr-rL__annotations__r"r+ staticmethodrintrErKr\no_gradrrCrMrNs@r4rPrPEslVV|VVVVVV &*+/"**t#*(*t* ~u$ % ***\*:U]__ < <_ < < < <  >(=a(@AADYZ[D\\ ++M;GG'+aaag*>> S6\4?;>CW`n`uvvvq+~FF #((fdoFF #di/ r5)rHrIrJ__doc__r"r+rCrMrNs@r4rrs] |      r5rc:eZdZdZdeffd ZdZdZdZxZ S) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. rRct|j|_|j|_|j|jz|_|j|_ tj tj |j|_t|j|j|j|_t|j|j|j|_t%|j|j|j |_dS)Nr)r*r+rrr2r kv_channelsnum_key_value_headsrrr-rr,rrrrrrrrs r4r+zJetMoeMoA.__init__'s !3 ,!-0JJ/ H&&u{4?'C'CDD 1$2BDOUYUeff243CTEUW[Wfgg&(*    r5c|\}}}|d|}||\}}}}} ||||f} ||} || |} t j||z|jz|jf| j| j } | d|| }| |||jd}|| | fS)z Map inputs to attention experts according to routing decision and compute query projection inside each experts. r9rr) rrrrr-rrr2r;r`rr)r1rrrrrrrrr topo_inforrrrs r4mapz JetMoeMoA.map9s !, 0 0 2 2VX!))"h77 UYU`U`alUmUmRk; ]); [Q $K0 **=+FF 6\DJ &(8 9AU^l^s   q*>OO #((fdj"EE ]I55r5c|\}}}}|d|}|\}}} } ||} || | } | | dddfz} tj||z|jf| j| j} | d|| }| |||j}||j z}|S)zu Compute output projection inside each attention experts and merge the outputs of different experts. r9Nrr) rrrr-rrr;r`rrr)r1rrrrkr2rrrrrrrrs r4reducezJetMoeMoA.reducePs'2&6&6&8&8#VQ !))"k:: FOCk; $$89 ++M;GG(+aaag*>> S6\4?;>CW`n`uvvvq+~FF #((fdoFF #di/ r5c td)Nz-This module doesn't support call and forward.)NotImplementedError)r1rs r4rCzJetMoeMoA.forwardfs!"QRRRr5) rHrIrJrr"r+rrrCrMrNs@r4rrs |      $666.,SSSSSSSr5rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr9r8rs)rFr-ry)r|x1x2s r4 rotate_halfrjs] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r5rotary_pos_embc||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qrrzr{ unsqueeze_dimq_embedk_embeds r4apply_rotary_pos_embrqsc& -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r5r6n_repr(c|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rFrtr)r6rbatchrslenres r4 repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr5modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nr8rr9)rlr;)ptrainingr!)rnum_key_value_groupsr-matmulrxrrrr=r<r;rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r4eager_attention_forwardr s3 ;<#>#D#DQB#D#O#O L#((66@@AFF __\22<E+, r5r&ceZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedeed geeed d Zejfd ZxZS) JetMoePreTrainedModelrRmodelFr&rTr8)indexr!)rr6 attentionsc4t|t|tr(t j|jd|jjdSt|ttzrt j |j dSdS)zInitialize the weights.r)r?stdN) r* _init_weightsrurinitnormal_r/rRinitializer_rangerrzeros_r)r1rr3s r4r9z#JetMoePreTrainedModel._init_weightsCs f%%% f3 4 4 % LSdk6S T T T T T T  I 5 6 6 % K $ $ $ $ $ % %r5)rHrIrJr"rbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr rr&_can_record_outputsr-rr9rMrNs@r4r2r21s &+#-.#4"5N""&(.BBBNNScklDmDmDmn+$n_A>>> U]__%%%%_%%%%%r5r2ceZdZdeffd Zeee d dej dzdej dzdej dzde dzdej dzd e dzd eed efd ZxZS) JetMoeModelrRctj|_j|_t jjj|j|_t j fdtj D|_ tjj|_t!|_d|_j|_|dS)Nc0g|]}t|Sr.)r&).0rrRs r4 z(JetMoeModel.__init__..Vs$ d d dy  2 2 d d dr5)r'rRF)r*r+ pad_token_id padding_idx vocab_sizer Embeddingr2 embed_tokens ModuleListrnum_hidden_layerslayersr% rms_norm_epsnormrP rotary_embgradient_checkpointingr post_initrs `r4r+zJetMoeModel.__init__Os    !. +L):F>>&+#$*$?! r5N input_idsrr}r inputs_embedsr,rr(c F|du|duz rtd|r|t|j}|||}|V||nd}t j|jd|j|z}| d}t|j||||} |} | | |} |j d|jj D]} | | f| | |||d|} || } t| |S) Nz:You must specify exactly one of input_ids or inputs_embedsrNrr!)r`)rRr]rrr})rrrr,r})last_hidden_stater) ValueErrorr rRrSget_seq_lengthr-rirFr`rrrYrVrUrXr) r1r\rr}rr]r,rpast_seen_tokens causal_maskr6r decoder_layers r4rCzJetMoeModel.forward`s  -t"; < [YZZ Z  ?0*$+>>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    & #oom\JJ![)H4;+H)HI  M)M$7* /#) MM -00 %++    r5)NNNNNN)rHrIrJr"r+rr rr-r$rLr FloatTensorr0rrrrCrMrNs@r4rIrIMs |" .2.204(,26!%5 5 #d*5  t+5 &- 5  5 (4/ 5 $;5 +,5  5 5 5 ^_ 5 5 5 5 5 r5rIr8 gate_logitsrc|t|tsdSt|tr/|djtjfd|Dd}tjj|d}tj||d\}}tjj ||}|@tj | d} tj |d} n.|j \} } |j d| | zz} |dddddddf | | | ||fd||}tj| |zdtj|dz } |ddddddf | | | |fd|}tj||zdtj|dz } tj| | dz}||zS)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrc:g|]}|Sr.)r<)rL layer_gatecompute_devices r4rMz,load_balancing_loss_func..s&-j-j-jPZjmmN.K.K-j-j-jr5rsr9)rurEr`r-ryrrrrone_hotr?rKrFrtrr<rr)rfrrrconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthrUexpert_attention_mask router_per_expert_attention_mask overall_lossrjs @r4load_balancing_loss_funcrws:*[%"@"@q+u%%s$Q.#(9-j-j-j-j^i-j-j-jpq#r#r#r h)112JPR1SSO*_eDDDA(%--.> LLK!J{'8'8':':BBB"'O!C!C!C&4&:# O4:1=*B^_ 4AAAtT1 2 V& OUKX Y Y WR , , R   "Ik&7&7&9&9SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4% ;1 D  F/% ( H !*X[[-E-EEE(#3!/)!/    r5) NNNNNNNrF)rHrIrJ_tied_weights_keysr+rrr-r$rLr rer0rrrCrMrNs@r4ryrys-*,GH     .2.204(,26*.!%-.,19 9 #d*9  t+9 &- 9  9 (4/ 9  4'9 $;9 el*9 #Tk9  #9 9 9 ^9 9 9 9 9 r5ryceZdZdS)JetMoeForSequenceClassificationN)rHrIrJr.r5r4rr;sr5r)ryrIr2r)r!)r)Nr8N)Mcollections.abcrtypingrr-rtorch.nnrrrr: activationsr cache_utilsr r generationr integrationsr r masking_utilsrmodeling_layersrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrr configuration_jetmoer" get_loggerrHrModuler%rPrrrrrrrLrrrKr r r&r2rIrErwryr__all__r.r5r4rs*%$$$$$ $$$$$$&&&&&&!!!!!!........))))))QQQQQQQQ//////[[[[[[[[QQQQQQQQKKKKKKKKFFFFFFFF&&&&&&RRRRRRRRRRRRGGGGGGGGEEEEEEEE......  H % %Y''JJJJJBIJJ('J(><><><><><><>