tj]dZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l mZmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZddlmZddlmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3e"j4e5Z6Gdde-Z7Gdde.Z8Gddej9Z:Gddej9Z;Gd d!ej9Z<Gd"d#ej9Z=Gd$d%ej9Z>Gd&d'e)Z?e Gd(d)e,Z@e Gd*d+e+ZAGd,d-e@eZBGd.d/ee@ZCgd0ZDdS)1zPyTorch JetMoe model.)CallableN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask) GenericForSequenceClassification)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs)LlamaDecoderLayer) MixtralModelMixtralPreTrainedModelMixtralRMSNormMixtralRotaryEmbeddingapply_rotary_pos_embeager_attention_forwardload_balancing_loss_func) JetMoeConfigceZdZdS) JetMoeRMSNormN__name__ __module__ __qualname__g/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/jetmoe/modular_jetmoe.pyr&r&4Dr,r&ceZdZdS)JetMoeRotaryEmbeddingNr'r+r,r-r0r08r.r,r0c6eZdZdedededdffd ZdZxZS)JetMoeParallelExperts num_experts input_size output_sizereturnNcttjt j||||_||_||_||_ dS)a Initialize the JetMoeParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) super__init__r Parametertorchemptyweightr3r4r5)selfr3r4r5 __class__s r-r9zJetMoeParallelExperts.__init__=sW" l5;{K#T#TUU &$&r,c||d}g}t|jD];}|t j|||j|inputs expert_size input_list output_listiresultss r-forwardzJetMoeParallelExperts.forwardTs\\+1\55  t'(( H HA   qx 1 t{1~FF G G G G)KQ///r,r(r)r*intr9rO __classcell__r?s@r-r2r2<sh'C'S's't''''''.r,r2c2eZdZdededeffd ZdZxZS)JetMoeTopKGatingr4r3top_kct||_||_||_t j||d|_dS)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. FbiasN)r8r9r3r4rVrLinearlayer)r>r4r3rVr?s r-r9zJetMoeTopKGating.__init__jsM &$ Yz;UCCC r,c||}||jd\}}t j|d|}t j|d|j g|j |j }| d|d}| d}|}|} | d\} } | |jd} |}|| } | | | ||fS)Nr#rArdtypedevicetrunc) rounding_mode)r[floattopkrVr;softmaxtype_aszerossizer3r^r_scatterlongsumtolistflattensortdiv)r> hidden_stateslogits top_k_logits top_k_indices top_k_gatesrfgatesrJ top_k_experts_index_sorted_experts batch_index batch_gatess r-rOzJetMoeTopKGating.forward~sSM**0022&,kk$*!k&D&D# mmLa888@@OO    a $"2 3;;LU`Ug    a22jjll&&q)) "((** &--// "/"4"4Q"7"7 *..tz.QQ "))++ !"67 #[+{FRRr,rPrSs@r-rUrUisqD3DSDDDDDDD(SSSSSSSr,rUc.eZdZdZdeffd ZdZxZS) JetMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. configct|j|_|j|_t |j|_tj tj |j|_ t|j|j|jdz|_t|j|j|j|_t#|j|j|j|_dS)Nrr4r3rV)r8r9 hidden_sizer4intermediate_sizeractivation_function activationr;rr:r<rYr2num_local_experts input_linear output_linearrUnum_experts_per_tokrouterr>r|r?s r-r9zJetMoeMoE.__init__s  ,!3 !;<H&&u{4?'C'CDD 1&2JDO]a]mpq]qrr263KTM]_c_noo&0,    r,cd|\}}}|d|}||\}}}}} ||} || |} | dd} || d| dz} || |} | |dddfz} tj||z|j f| j | j }| d|| }| |||j }||jz}|S)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. rrArr#Nr])rgreshaperrchunkrrr;rfr4r^r_ index_addviewrY)r> layer_inputbszlengthemb_sizervrxryrJ router_logits expert_inputsrochunked_hidden_statesexpert_outputsrf layer_outputs r-rOzJetMoeMoE.forwards<!, 0 0 2 2VX!))"h77 BF++kBZBZ?; [-#K0 ))-EE - 3 3A2 3 > >(=a(@AADYZ[D\\ ++M;GG'+aaag*>> S6\4?;>CW`n`uvvvq+~FF #((fdoFF #di/ r,)r(r)r*__doc__r$r9rOrRrSs@r-r{r{s] |      r,r{c:eZdZdZdeffd ZdZdZdZxZ S) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. r|ct|j|_|j|_|j|jz|_|j|_ tj tj |j|_t|j|j|j|_t|j|j|j|_t%|j|j|j |_dS)Nr~)r8r9rr3rr4 kv_channelsnum_key_value_headsrrVr;rr:r<rYr2rrrUrrs r-r9zJetMoeMoA.__init__s !3 ,!-0JJ/ H&&u{4?'C'CDD 1$2BDOUYUeff243CTEUW[Wfgg&(*    r,c|\}}}|d|}||\}}}}} ||||f} ||} || |} t j||z|jz|jf| j| j } | d|| }| |||jd}|| | fS)z Map inputs to attention experts according to routing decision and compute query projection inside each experts. rr]r) rgrrrr;rfrVrr^r_rr)r>rrrrrwrxryrJr topo_inforrrfrs r-mapz JetMoeMoA.maps !, 0 0 2 2VX!))"h77 UYU`U`alUmUmRk; ]); [Q $K0 **=+FF 6\DJ &(8 9AU^l^s   q*>OO #((fdj"EE ]I55r,c|\}}}}|d|}|\}}} } ||} || | } | | dddfz} tj||z|jf| j| j} | d|| }| |||j}||j z}|S)zu Compute output projection inside each attention experts and merge the outputs of different experts. rNr]r) rgrrr;rfr4r^r_rrrY)r>rrrrkrrwrxryrJrrrfrs r-reducezJetMoeMoA.reduces'2&6&6&8&8#VQ !))"k:: FOCk; $$89 ++M;GG(+aaag*>> S6\4?;>CW`n`uvvvq+~FF #((fdoFF #di/ r,c td)Nz-This module doesn't support call and forward.)NotImplementedError)r>rs r-rOzJetMoeMoA.forwards!"QRRRr,) r(r)r*rr$r9rrrOrRrSs@r-rrs |      $666.,SSSSSSSr,rceZdZdZd dededzffd Z d dejdejdzdej dzd e dzd e ejejdze ejdzff d Z xZ S)JetMoeAttentionzH Multi-headed attention from 'Attention Is All You Need' paper. Nr| layer_idxc"t||_||_d|_|(t d|jjdd|_ |j |_ |j |_ |j |jz|_|j|_|j|_|j |_|jdz|_t)||_t,j|j|jdzd |_dS) z Initialize the JetMoeAttention module. Args: config: Configuration object with model hyperparameters. layer_idx: Index of the layer in the model. TNzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r#grFrX)r8r9r|r is_causallogger warning_oncer?r(num_key_value_groupsrrVattention_dropoutrrkv_projection_sizenum_attention_heads num_headshead_dimscalingrexpertsr;rrZrkv_projr>r|rr?s r-r9zJetMoeAttention.__init__%s  "     ,!8,,,    %&!/ !'!9"("4v7Q"Q#)#= 3* }d*  (( xv'94;RUV;V]bcc r,roattention_maskposition_embeddingspast_key_valuesr6c |jdd}g|d|jR}|j|\}} } ||dd\} } ||dd}| |dd} | |dd} |\} }t|| | |\}} || | | |j \} } tj |j jt}| d|jdd} | d|jdd} |||| | |f|jsdn|j|jd|\}}|jg||jdR}|j|| }|jg|dR}||| fS)NrrrAr#)dropoutr)shaperrrrrr transposer updaterr get_interfacer|_attn_implementationr!repeatrVtrainingrrr)r>rorrrkwargs input_shape hidden_shape query_statesrr key_states value_statescossinattention_interface attn_output attn_weightss r-rOzJetMoeAttention.forwardFs2$)#2#. 88b8$-88 151A1A-1P1P. mY#'<< #>#>#D#DQB#D#O#O L#((66@@AFF __\22<rorrrrrrresidualrvs r-rOzJetMoeDecoderLayer.forwards!,,];; 1d1 ')%+ 3      q!!=0 !55mDD //  =0 r,r)NNNFN)r(r)r*r$rQr9r;rrr boolrrrrOrRrSs@r-rrxs|d /304(,!&HL| t+&-    $; #5<#=>E+, r,rceZdZUeedeedgeeeddZee d<dZ dZ d gZ d gZ d Zd ZdZejd Zd S)JetMoePreTrainedModelr)indexr#)rro attentionsr|modelFrrTctj||t|tr(t j|jd|jjdSt|ttzrt j |j dSdS)zInitialize the weights.r)meanstdN) r _init_weights isinstancer2initnormal_r=r|initializer_rangerr{zeros_rY)r>modules r-rz#JetMoePreTrainedModel._init_weightss %dF333 f3 4 4 % LSdk6S T T T T T T  I 5 6 6 % K $ $ $ $ $ % %r,N)r(r)r*rrrUr_can_record_outputsr$__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphr;no_gradrr+r,r-rrs).BBBNNScklDmDmDmn+$n_A>>> &+#-.#4"5N"U]__%%_%%%r,rceZdZdeffd Zeee d dej dzdej dzdej dzde dzdej dzd e dzd eed efd ZxZS) JetMoeModelr|ctj|_j|_t jjj|j|_t j fdtj D|_ j |_ tjj|_dS)Nc0g|]}t|Sr+)r).0rr|s r- z(JetMoeModel.__init__..s$ d d dy  2 2 d d dr,)eps)r8r9 pad_token_id padding_idx vocab_sizer Embeddingr embed_tokens ModuleListrDnum_hidden_layerslayersrr& rms_norm_epsnormrs `r-r9zJetMoeModel.__init__s    !. +L):Fr rrrr rrpast_seen_tokens causal_maskror decoder_layers r-rOzJetMoeModel.forwards  -t"; < [YZZ Z  ?0*$+>>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    & #oom\JJ![)H4;+H)HI  M)M$7* /#) MM -00 %++    r,)NNNNNN)r(r)r*r$r9rrrr;rrr FloatTensorrrrrrOrRrSs@r-rrs O| O O O O O O .2.204(,26!%5 5 #d*5  t+5 &- 5  5 (4/ 5 $;5 +,5  5 5 5 ^_ 5 5 5 5 5 r,rceZdZddiZfdZee ddejdzdej dzd ejdzd e dzd ej dzd ejdzd e dzde ej zde dzdefdZxZS)JetMoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightc^t|t||_|j|_|j|_t j|j|jd|_ |j |_ |j |_ |j |_ |dS)NFrX)r8r9rrr aux_loss_coefrrZrlm_headtie_word_embeddingsrr3r post_initrs r-r9zJetMoeForCausalLM.__init__s     ((  +#1y!3V5FUSSS #)#= !3#)#=  r,NrFr rrrr labelsrlogits_to_keepoutput_router_logitsr6c |jd||||||| d| } | j} t|trt | dn|} || dd| ddf}d}||j||fd|jji| }d}| rHt| j |j |j |}|%||j ||jzz }t!|||| j| j| j| j S)N)r rrrr rr!r)lossaux_lossrprrorrr+)rrrrQslicer loss_functionr|rr"rr3rrtor_rrror)r>r rrrr rrr r!routputsro slice_indicesrpr#r$s r-rOzJetMoeForCausalLM.forwardsm+5$* + )%+'!5 + +  + +  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4% ;1 D  F/% ( H !*X[[-E-EEE(#3!/)!/    r,) NNNNNNNrF)r(r)r*_tied_weights_keysr9rrr;rrr rrrQrrOrRrSs@r-rrs-*,GH     .2.204(,26*.!%-.,19 9 #d*9  t+9 &- 9  9 (4/ 9  4'9 $;9 el*9 #Tk9  #9 9 9 ^9 9 9 9 9 r,rceZdZdS)JetMoeForSequenceClassificationNr'r+r,r-r,r,Rsr,r,)rrrr,)Ercollections.abcrr;rtorch.nnrrFrr activationsr cache_utilsr r generationr masking_utilsr modeling_layersr modeling_outputsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrrllama.modeling_llamarmixtral.modeling_mixtralrrrrr r!r"configuration_jetmoer$ get_loggerr(rr&r0Moduler2rUr{rrrrrrr,__all__r+r,r-rAsR$$$$$$ $$$$$$&&&&&&!!!!!!........))))))//////RQQQQQQQFFFFFFFF&&&&&&RRRRRRRRRRRR777777EEEEEEEE444444/.....  H % %     N        2   *****BI***Z.S.S.S.S.Sry.S.S.Sb77777 777tISISISISIS ISISISXU8U8U8U8U8biU8U8U8p&&&&&*&&&R%%%%%2%%%2E E E E E ,E E E PK K K K K -K K K \dcccc&FH]ccc k j jr,