tjhu<ddlmZddlmZddlZddlmcmZddlmZddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZmZmZdd lmZdd lmZdd lmZmZddlm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1edGddej2Z3Gddej2Z4Gddej2Z5eGddej2Z6Gdd ej2Z7d!Z8ed"dAd#Z9d$ej:d%e;d&ej:fd'Z< dBd)ej2d*ej:d+ej:d,ej:d-ej:dzd.e=d/e=d0e&e(fd1Z>ee9Gd2d3ej2Z?Gd4d5eZ@e)Gd6d7e$ZAe)Gd8d9eAZB dCd;ej:eCej:zdzdd?eAeZEgd@ZFdS)D)Callable)OptionalN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_experts_implementationuse_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstring)can_return_tuplemaybe_autocastmerge_with_config_defaults)capture_outputs)GraniteMoeConfigRMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) GraniteMoeRMSNormư>epsreturnNcttjt j||_||_dS)z@ GraniteMoeRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer% __class__s p/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/granitemoe/modeling_granitemoe.pyr)zGraniteMoeRMSNorm.__init__5sD l5:k#:#:;; # hidden_statesc|j}|tj}|ddd}|tj||jzz}|j||zS)NT)keepdim) dtypetor+float32powmeanrsqrtr.r-)r/r4 input_dtypevariances r2forwardzGraniteMoeRMSNorm.forward=s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r3cHt|jjd|jS)Nz, eps=)tupler-shaper.)r/s r2 extra_reprzGraniteMoeRMSNorm.extra_reprDs& )**II$2GIIIr3)r$) __name__ __module__ __qualname__floatr)r+TensorrArE __classcell__r1s@r2r#r#3s$$$$$$$$$$;U\;el;;;;JJJJJJJr3r#c eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)GraniteMoeRotaryEmbeddinginv_freqNconfigct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrOF) persistentoriginal_inv_freq) r(r)max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrPrope_parametersrRcompute_default_rope_parametersrattention_scalingregister_bufferclone)r/rPdevice rope_init_fnrOr1s r2r)z"GraniteMoeRotaryEmbedding.__init__Ks "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr3r^z torch.deviceseq_lenr&z torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr6r9)r^r9) rYgetattrr0num_attention_headsr+arangeint64r:rI)rPr^r`basedimattention_factorrOs r2rZz9GraniteMoeRotaryEmbedding.compute_default_rope_parameters[s&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r3cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) Nrr7rmpscpuF) device_typeenabledr6rjrd)rOrIexpandrDr:r^ isinstancetypestrr transposer+catcosr[sinr9) r/x position_idsinv_freq_expandedposition_ids_expandedrofreqsembrxrys r2rAz!GraniteMoeRotaryEmbedding.forwardys!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*NNNN)rFrGrHr+rJ__annotations__r r) staticmethodrintrCrIrZno_gradrrArKrLs@r2rNrNHslVV/VVVVVV *.+/"** 4'*(*t* ~u$ % ***\*:U]__ < <_ < < < < B22B69B6 rFrGrHrr r)r+rJrArKrLs@r2rrs<<0/000000#|#\#| #  ########r3rcLeZdZdZdeffd ZdejdejfdZxZ S) GraniteMoeMoEzISparsely-gated mixture-of-experts block: router decides, experts compute.rPct|j|_t ||_t ||_dSr)r(r)r0 input_sizerrouterrexpertsrs r2r)zGraniteMoeMoE.__init__sE  ,*622 (00 r3 layer_inputr&c|\}}}|d|}||\}}}||||} | |||jS)Nr7)sizereshaperrviewr) r/rbszlengthemb_sizer4rr_ layer_outputs r2rAzGraniteMoeMoE.forwardsv + 0 0 2 2VX#++B99 (, M(B(B% ]A||M; NN   fdo>>>r3rrLs@r2rrsqSS1/111111 ?5<?EL????????r3rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr7r6rq)rDr+rw)rzx1x2s r2 rotate_halfrs] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r3rotary_pos_embc||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrxry unsqueeze_dimq_embedk_embeds r2apply_rotary_pos_embrsc& -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r3r4n_repr&c|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rDrrr)r4rbatchnum_key_value_headsslenrcs r2 repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr3modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nr6rr7)rjr9)ptrainingr)rnum_key_value_groupsr+matmulrvrrrr;r:r9rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r2eager_attention_forwardrs3 ;<  " F4F&Jd4dee $*$>&B\$\!2 !'!9i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    r3Nr4position_embeddingsrpast_key_valuesrr&c "|jdd}g|d|jR}|||dd}|||dd} |||dd} |\} } t|| | | \}} ||| | |j \} } tj |j j t} | ||| | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)Nr7rr6r)rr)rDrcrrrvrrrupdaterr get_interfacerP_attn_implementationrrrrrrr)r/r4rrrr input_shape hidden_shape query_statesrrrxryattention_interfacerrs r2rAzGraniteMoeAttention.forward9s$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r3r)rFrGrHrr rr)r+rJrCr rrrArKrLs@r2rrsGG / C      4IM.2(, &)&)|&)#5<#=>E&) t+ &)  &) +, &) u|U\) *&)&)&)&)&)&)&)&)r3rc eZdZdedeffd Z d dejdejdzdedzde ejejfdzd ejf d Z xZ S) GraniteMoeDecoderLayerrPrcLt|j|_t|||_t |j|j|_t |j|j|_t||_ |j |_ dS)N)rPrr%) r(r)r0r self_attnr# rms_norm_epsinput_layernormpost_attention_layernormrblock_sparse_moeresidual_multiplierrs r2r)zGraniteMoeDecoderLayer.__init__cs !-,FiPPP01CI\]]](9&:LRXRe(f(f(f% -f 5 5#)#=   r3Nr4rrrr&c |}||}|jd||||d|\}}|||jzz}|}||}||}|||jzz}|S)N)r4rrr)rrrrr)r/r4rrrrresidualrs r2rAzGraniteMoeDecoderLayer.forwardls!,,];; )4> ')+ 3       q!=43K#KK  55mDD --m<<  =43K#KK r3r) rFrGrHr rr)r+rJr rCrArKrLs@r2rrbs>/>C>>>>>>/3(,HL | t+  #5<#=>E  r3rceZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZejfdZxZS)GraniteMoePreTrainedModelrPmodelTrr)r4 attentionsct|t|trNt j|jd|jjt j|j d|jjdSt|tr(t j|j d|jjdSdS)Nr)r=std) r( _init_weightsrsrinitnormal_rrPinitializer_rangerrr-)r/rr1s r2r z'GraniteMoePreTrainedModel._init_weightss f%%% f/ 0 0 U L,3DKz,GraniteMoeModel.__init__..s$ h h h9 #FI 6 6 h h hr3rrPF)r(r) pad_token_id padding_idx vocab_sizer Embeddingr0 embed_tokens ModuleListrangenum_hidden_layerslayersr#rnormrN rotary_embgradient_checkpointingembedding_multiplier post_initrs `r2r)zGraniteMoeModel.__init__s    !. +L):FQRRR 36BBB&+#$*$?! r3N input_idsrr{r inputs_embeds use_cacherr&c Z|du|duz rtd|r|t|j}|||}|V||nd}t j|jd|j|z}| d}t|j||||} ||j z}|} | | |} |j d|jjD]} | | f| | |||d|} || } t!| |S) Nz:You must specify exactly one of input_ids or inputs_embedsr4rr)r^)rPrDrrr{)rrr{rrE)last_hidden_stater) ValueErrorr rPr9get_seq_lengthr+rgrDr^rrrAr?r=r<r>r) r/rCrr{rrDrErpast_seen_tokens causal_maskr4r decoder_layers r2rAzGraniteMoeModel.forwards  -t"; < [YZZ Z  ?0*$+>>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    &(AA % #oom\JJ![)H4;+H)HI  M)M$7*) /# MM -00 %++    r3)NNNNNN)rFrGrHr r)rrrr+ LongTensorrJr FloatTensorboolrrrrArKrLs@r2r/r/s/" .2.204(,26!%5 5 #d*5  t+5 &- 5  5 (4/ 5 $;5 +,5  5 5 5 ^_ 5 5 5 5 5 r3r/r6 gate_logitsrc|t|tsdSt|tr/|djtjfd|Dd}tjj|d}tj||d\}}tjj ||}|@tj | d} tj |d} n.|j \} } |j d| | zz} |dddddddf | | | ||fd||}tj| |zdtj|dz } |ddddddf | | | |fd|}tj||zdtj|dz } tj| | dz}||zS)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrc:g|]}|Sr)r:)r2 layer_gatecompute_devices r2r3z,load_balancing_loss_func..s&-j-j-jPZjmmN.K.K-j-j-jr3rqr7)rsrCr^r+rwrrrrrr=rIrDrrrr:rr)rPrrrconcatenated_gate_logitsrouting_weightsrselected_expertsrtokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthr<expert_attention_mask router_per_expert_attention_mask overall_lossrTs @r2load_balancing_loss_funcr_s:*[%"@"@q+u%%s$Q.#(9-j-j-j-j^i-j-j-jpq#r#r#r h)112JPR1SSO*_eDDDA(%--.> LLK!J{'8'8':':BBB"'O!C!C!C&4&:# O4:1=*B^_ 4AAAtT1 2 V& OUKX Y Y WR , , R   "Ik&7&7&9&9>> from transformers import AutoTokenizer, GraniteMoeForCausalLM >>> model = GraniteMoeForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rCrr{rrDr7)lossaux_lossrdrr4rrr)rPrirrGrsrslicerbrg loss_functionr7r_rrrrfr:r^rrr4r)r/rCrr{rrDrhrirjroutputsr4 slice_indicesrdrlrms r2rAzGraniteMoeForCausalLM.forwardSsJ%9$D $+Jj $* )%+'       1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA$+44  %4% ;1 D  M/% ( H !1HKK 4L4LLL(#3!/)!/    r3)NNNNNNNr)rFrGrH_tied_weights_keys_tp_plan_pp_planr r)rrr+rMrJr rNrOrrCrrArKrLs@r2rara@s`*,GH23H_-z:;H /      .2.204(,26*.,0-.Q Q #d*Q  t+Q &- Q  Q (4/ Q  4'Q #TkQ el*Q  * *Q Q Q ^Q Q Q Q Q r3ra)rar/r)r)r)Nr6N)Gcollections.abcrtypingrr+torch.nn.functionalrrrrr! activationsr cache_utilsr r generationr integrationsr r rr masking_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrr utils.genericrrrutils.output_capturingrconfiguration_granitemoer Moduler#rNrrrrrrJrrrIrrrrr/rCr_ra__all__rr3r2rs,%$$$$$ &&&&&&!!!!!!........)))))) 0/////999999QQQQQQQQKKKKKKKKFFFFFFFF&&&&&&77777777YYYYYYYYYY555555666666Y''JJJJJ JJ('J(><><><><>< ><><>