tjCdZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z dd l mZmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZmZmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ej1e2Z3Gdde)Z4Gdde%Z5Gdde.Z6Gddej7Z8Gdd e-Z9Gd!d"ej7Z:Gd#d$e'Z;Gd%d&eZ<Gd'd(eZ=eGd)d*e=Z>Gd+d,e(e=e Z?gd-Z@dS).zPyTorch AFMoE model.)CallableN)nn)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs) GptOssRMSNorm)LlamaAttentionLlamaForCausalLMLlamaRotaryEmbeddingapply_rotary_pos_embeager_attention_forward)Qwen2MoeExperts Qwen2MoeMLP) AfmoeConfigceZdZdS)AfmoeRotaryEmbeddingN__name__ __module__ __qualname__e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/afmoe/modular_afmoe.pyr%r%/Dr+r%ceZdZdS) AfmoeRMSNormNr&r*r+r,r/r/3r-r+r/ceZdZdS)AfmoeMLPNr&r*r+r,r1r17r-r+r1cFeZdZdZfdZdejdejfdZxZS)AfmoeTokenChoiceRouterz Token-choice top-K router for MoE routing. This router assigns each token to the top-K experts based on sigmoid scores, matching the released checkpoints. ct||_|j|_|j|_|j|_tj|j |jd|_ dSNFbias) super__init__confignum_experts_per_toktop_k num_experts route_scalerLinear hidden_sizegateselfr: __class__s r,r9zAfmoeTokenChoiceRouter.__init__Bsb  / !-!-If0&2D5QQQ r+ hidden_states expert_biasc|j\}}}|d|}||tj}t j|}t j||z|jd\}}| d|}| dddz} || z }||j z}|||fS)Nr")kdim)rJindexT)rJkeepdimg#B ;) shapeviewrAtotorchfloat32sigmoidtopkr<gathersumr>) rCrErF_ hidden_dim router_logitsscoresselected_experts top_scores denominators r,forwardzAfmoeTokenChoiceRouter.forwardJs(.1j%**2z::  -0033EMBB }--#j+)=QRSSS ]]q0@]AA  nnTn::UB +- $"22 j*:::r+) r'r(r)__doc__r9rPTensorr] __classcell__rDs@r,r3r3;sm RRRRR ;U\ ; ; ; ; ; ; ; ; ;r+r3ceZdZdS) AfmoeExpertsNr&r*r+r,rcrcYr-r+rcc(eZdZdZfdZdZxZS)AfmoeSparseMoeBlockz Mixture of Experts (MoE) module for AFMoE. This module implements a sparse MoE layer with both shared experts (always active) and routed experts (activated based on token-choice routing). cNt||_t||_t ||j|jz|_t||_ tj tj|jd|_dS)NF) requires_grad)r8r9r:r3routerr1moe_intermediate_sizenum_shared_expertsshared_expertsrcexpertsr ParameterrPzerosr=rFrBs r,r9zAfmoeSparseMoeBlock.__init__es  ,V44 &vv/KfNg/ghh#F++ < F4F(G(GW\]]]r+c8|j\}}}|d|}|||j\}}}|||||} |||||||} | | zS)NrH)rMrNrhrFrkrl) rCrE batch_sizeseq_lenrWhidden_states_flatrVr[rZ shared_output routed_outputs r,r]zAfmoeSparseMoeBlock.forwardms*7*=' GZ*//J??+/++mTEU*V*V':'++,>??DDZQXZdee  %79I:VV[[   },,r+)r'r(r)r^r9r]r`ras@r,rere]sV^^^^^ - - - - - - -r+receZdZdZdedeffd Z d dejde ejejfdejdzd e dzd e e d e ejejff d Z xZS)AfmoeAttentionaJ Multi-headed attention module with optional sliding window and gating. This attention mechanism supports both full attention and sliding window attention, and includes Q/K normalization and gating of the output. It inherits from [`LlamaAttention`] to minimize the amount of custom logic we need to maintain. r: layer_idxc~t|||j|dk|_|jr|jnd|_t |j|j|_t |j|j|_ tj |j |j |jzd|_dS)Nsliding_attentionepsFr6)r8r9 layer_typesis_local_attentionsliding_windowr/head_dim rms_norm_epsq_normk_normrr?r@num_attention_heads gate_projrCr:rwrDs r,r9zAfmoeAttention.__init__s +++#)"4Y"?CV"V7;7NXf33TX"4=f6IJJJ "4=f6IJJJ 6#5v7QTXTa7ahmnnnr+NrEposition_embeddingsattention_maskpast_key_valuekwargsreturnc |jdd}g|d|jR}|||}|||} |||} ||} ||dd}| | dd} | dd} |j r|\} } t|| | | \}} || | | |j \} } tj|jjt$}|||| | f||jsdn|j|j|jd|\}}|jg|dR}|t1j| z}||}||fS)NrHr"r)rdropoutscalingr~)rMrq_projrNk_projv_projrr transposerr}rupdaterwr get_interfacer:_attn_implementationrtrainingattention_dropoutrr~ contiguousrPrRo_proj)rCrErrrr input_shape hidden_shape query_states key_states value_states gate_statescossinattention_interfaceoutput attn_weights attn_outputs r,r]zAfmoeAttention.forwards$)#2#. 88b8$-88 {{=1166|DD [[//44\BB {{=1166|DD nn]33 {{<00::1a@@ [[,,66q!<< #--a33  " `*HC';L*VY[^'_'_ $L*  %'5'<'.) t+ .)  .) +, .) u|U\) *.).).).).).).).)r+rvceZdZdZdedeffd Z ddejdejdzdej dzd e dzd e dzd e ejejfdzd e ed ejfdZxZS)AfmoeDecoderLayerz AFMoE decoder layer with dual normalization. This layer applies self-attention followed by either a dense MLP or MoE block, with dual normalization (pre and post) around each component. r:rwct|j|_||_t |||_t |j|j|_t |j|j|_ t |j|j|_ t |j|j|_ ||j k|_ |j rt||_dSt!||_dS)N)r:rwrz)r8r9r@rwrv self_attnr/rinput_layernormpost_attention_layernormpre_mlp_layernormpost_mlp_layernormnum_dense_layers moe_enabledremlpr1rs r,r9zAfmoeDecoderLayer.__init__s !-"'vKKK ,F,>FDWXXX(4V5GVM`(a(a(a%".f.@fFY!Z!Z!Z".v/AvGZ"["["[%(??   (*622DHHH''DHHHr+NrEr position_idsr use_cacherrrc "|}||}|jd||||||d|\}} ||}||z}|}||}||}||}||z}|S)N)rErrrrrr*)rrrrrr) rCrErrrrrrresidualrVs r,r]zAfmoeDecoderLayer.forwards!,,];; )4> ')%) 3      q55mDD  =0 !..}== // // >>  =0 r+)NNNNN)r'r(r)r^r#rr9rPr_ LongTensorrboolrrr FloatTensorr]r`ras@r,rrs({(s((((((2/304'+!%HL!!|! t+!&- !  ! $; !#5<#=>E!+,!  !!!!!!!!r+rceZdZUdZeed<dZdgZdgZe e de e dZ gd Zd Zd Zd Zd Zd Zd Zfd ZxZS) AfmoePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r:modelrpast_key_valuesr)rK)rXrE attentions)rrrrrrnormrFTct||jj}t |t r:t j|jd|t j|j d|dSt |tr t j |j j dSt |trt j |jdSdS)zInitialize the weightsr)meanstdN)r8 _init_weightsr:initializer_range isinstancercinitnormal_ gate_up_proj down_projr3zeros_rAweightrerF)rCmodulerrDs r,rz"AfmoePreTrainedModel._init_weights#s f%%%k+ fl + + , L,3C @ @ @ @ L) = = = = = =  6 7 7 , K * + + + + +  3 4 4 , K* + + + + + , ,r+)r'r(r)r^r#__annotations__base_model_prefix_no_split_modules_skip_keys_device_placementrr3rrv_can_record_outputs_keep_in_fp32_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_can_compile_fullgraph_supports_attention_backendsupports_gradient_checkpointingrr`ras@r,rrs ,-#4"5'(>aHHH*$   N!"&&*# , , , , , , , , ,r+rceZdZdZdeffd Zeee dde j dzde j dzde j dzde j dzd e dzd edzd eed eezfd ZxZS) AfmoeModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`AfmoeDecoderLayer`] Args: config: AfmoeConfig r:ctj|_j|_t jjj|j|_t j fdtj D|_ tjj|_t!|_d|_|dS)Nc0g|]}t|Sr*)r).0rwr:s r, z'AfmoeModel.__init__..@s$ c c ci vy 1 1 c c cr+rzr:F)r8r9 pad_token_id padding_idx vocab_sizer Embeddingr@ embed_tokens ModuleListrangenum_hidden_layerslayersr/rrr% rotary_embgradient_checkpointing post_initrBs `r,r9zAfmoeModel.__init__9s    !. +L):F>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L?-FF +!."0#2 K#5"C"C{"C"C%F%U%U%U%U##  &  ; " K)T[-Dc-IJM"oom\JJ )$+ 6 6   A})M24;3J13MN).#$7 MM -00 %+/8BOOd    r+)NNNNNN)r'r(r)r^r#r9rrrrPrr_rrrrrrrr]r`ras@r,rr0s {      .2.22604(,!%< < #d*<  t+< (4/ < &- <  < $;< +,<  ' '< < < _ ^< < < < < r+rceZdZddiZddiZddgdgfiZdZee dd e j dzd e j dzd e j dzd e dzde j dzde j dzdedzdedzdee j zdeedefdZdS)AfmoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputrElogitsct||t||_|j|_t j|j|jd|_| dSr5) rr9rrrrr?r@rr)rCr:s r,r9zAfmoeForCausalLM.__init__se%%dF333''  +y!3V5FUSSS  r+Nrrrrrrlabelsroutput_router_logitslogits_to_keeprrc r||n |jj}|jd|||||||d| } | j} t | t rt | dn| } || dd| ddf}d}||j|||j fi| }t||| j | j | j | jS)N)rrrrrrr)lossrrrErrXr*)r:rrrrrslicer loss_functionrr rrErrX)rCrrrrrrrrrroutputsrE slice_indicesrrs r,r]zAfmoeForCausalLM.forwards %9$D $+Jj +5$* + )%+'!5 + +  + +  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%ffdoPPPPD(#3!/)!/     r+) NNNNNNNNr)r'r(r)_tied_weights_keys_tp_plan_pp_planr9rrrPrr_rrrrrrr r]r*r+r,rrsG*,GH23H_-z:;H.2.204(,26*.!%,0-.+ + #d*+  t++ &- +  + (4/ +  4'+ $;+ #Tk+ el*+ +,+  #+ + + ^+ + + r+r)rrr)Ar^collections.abcrrPrrr cache_utilsrr generationr masking_utilsr r modeling_layersr modeling_outputsr rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrrgpt_oss.modeling_gpt_ossrllama.modeling_llamarrrrrqwen2_moe.modeling_qwen2_moer r!configuration_afmoer# get_loggerr'loggerr%r/r1Moduler3rcrervrrrr__all__r*r+r,r$s$$$$$$ &&&&&&........))))))RRRRRRRR999999QQQQQQQQFFFFFFFF&&&&&&RRRRRRRRRRRR777777EEEEEEEE444444HGGGGGGG,,,,,,  H % %     /        =        {   ;;;;;RY;;;<     ?   -----")--->B)B)B)B)B)^B)B)B)J?????2???D*,*,*,*,*,?*,*,*,ZV V V V V %V V V r9 9 9 9 9 ')=9 9 9 x   r+