tjA@ddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z dd lmZdd lmZmZmZdd lmZmZdd lmZmZdd lmZmZddlmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0Gddej1Z2dZ3eddCdZ4dej5de6dej5fdZ7 dDd ej1d!ej5d"ej5d#ej5d$ej5dzd%e8d&e8d'e$e&fd(Z9ee4Gd)d*ej1Z:Gd+d,ej;j<Z=eGd-d.ej1Z>dEd0Z?Gd1d2ej@ZAGd3d4ej1ZBGd5d6eZCe'Gd7d8e"ZDe'Gd9d:eDZE dFd;ej5eFej5zdzdd?eDeZHGd@dAeeDZIgdBZJdS)G)Callable)OptionalN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_experts_implementationuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask!create_sliding_window_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) PhimoeConfigc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejedd ZxZS)PhimoeRotaryEmbeddinginv_freqNconfigct|j|_|j|_||_|jjd|_|j|_ |jdkrt|j|_ | |j|\}|_ | d|d| d| ddS)N rope_typedefaultr%F) persistentoriginal_inv_freq)super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr&rope_parametersr(compute_default_rope_parameters rope_init_fnrattention_scalingregister_bufferclone)selfr&devicer% __class__s h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/phimoe/modeling_phimoe.pyr-zPhimoeRotaryEmbedding.__init__0s "("@$*$B! 4[A&*&J >Y & & 3DN CD +/+<+*ll_illl )L))A- ; &{ 3y @ @W @T[89[\\\ +M::[0@  '+&7&7 QXw&W&W##&,n""&$T111d]399;;BB > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C % %&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))f$C''))f$C  % % % % % % % % % % % % % % % vvagqw//s=BH  HHNNNN)NN)r[ __module__ __qualname__rGTensor__annotations__r"r- staticmethodrinttuplerKr2no_gradrrn __classcell__r9s@r:r$r$-slVV|VVVVVV &*+/"**t#*(*t* ~u$ % ***\*:U]__000_00000r;r$c|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..NrTrBrY)r^rGrc)rfx1x2s r: rotate_halfr~|s] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r;rotary_pos_embc||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer~)qkrdre unsqueeze_dimq_embedk_embeds r:apply_rotary_pos_embrsc& -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r; hidden_statesn_repr=c|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)r^r]reshape)rrbatchnum_key_value_headsslenr@s r: repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr;modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)NrBrrT)rMrC)ptrainingr!)rnum_key_value_groupsrGmatmulrbr functionalsoftmaxfloat32rJrCrr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r:eager_attention_forwardrs3 ;<&B\$\!}d* !'!9i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    r;Nrposition_embeddingsrpast_key_valuesrr=c "|jdd}g|d|jR}|||dd}|||dd} |||dd} |\} } t|| | | \}} ||| | |j \} } tj |j j t} | ||| | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)NrTr!rBr)rr)r^r@rviewrbrrrupdaterr get_interfacer&_attn_implementationrrrrrrr)r7rrrrr input_shape hidden_shape query_statesrrrdreattention_interfacerrs r:rnzPhimoeAttention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r;rp)r[rqrr__doc__r"rvr-rGrsrwr rrrnryrzs@r:rrsGG |       4IM.2(, &)&)|&)#5<#=>E&) t+ &)  &) +, &) u|U\) *&)&)&)&)&)&)&)&)r;rc eZdZedejdejdejdejdejf dZedejfdZd S) PhimoeMultiplierscores multiplierselected_experts masked_gates mask_for_onec:||||||zS)a Forward pass for the custom autograd function. Args: ctx: Context object to save information for backward computation. scores (torch.Tensor): Input scores tensor. multiplier (torch.Tensor): Multiplier tensor. selected_experts (torch.Tensor): Tensor of selected experts. masked_gates (torch.Tensor): Masked gates tensor. mask_for_one (torch.Tensor): Mask for one tensor. Returns: torch.Tensor: Result of the forward pass. )save_for_backward)ctxrrrrrs r:rnzPhimoeMultiplier.forwards(. j*:LIIIL((r;grad_at_outputc|j\}}}||z}||dz}|d|||ddddfS)aB Backward pass for the custom autograd function. Args: ctx: Context object with saved tensors from the forward pass. grad_at_output (torch.Tensor): Gradient at the output. Returns: tuple[torch.Tensor, None, None, None, None]: Gradients for the inputs. rT)rMindexsrcN) saved_tensorsmul scatter_add_)rrrrrgrad_at_scores_expandeds r:backwardzPhimoeMultiplier.backward!sx695F2 $l'*4".1C1CB1G1G"G,," -    $       r;N)r[rqrrrurGrsrnrr;r:rrs) )L) , ) l ) l )))\)2     \   r;rcheZdZdZdeffd ZdejdejdejdejfdZxZ S) PhimoeExpertsz2Collection of expert weights stored as 3D tensors.r&ct|j|_|j|_|j|_tj tj |jd|jz|j|_ tj tj |j|j|j|_ t|j|_dS)NrB)r,r-num_local_experts num_expertsrE hidden_dimintermediate_sizeintermediate_dimr ParameterrGempty gate_up_proj down_projr hidden_actact_fnr7r&r9s r:r-zPhimoeExperts.__init__Hs !3 , & 8LT5Eq4K`G`bfbq)r)rssek$2BDOUYUj&k&kllV./ r;r top_k_index top_k_weightsr=ctj|}tj5tjj||j}|ddd}tj| dd }dddn #1swxYwY|D]}|d}||jkrtj ||\}} || } tj | |j |dd\} } || | z} tj | |j|} | || |dfz} |d| | |j|S)N) num_classesrBr!r)rTrYrT)rG zeros_likerxrrone_hotrpermutegreatersumnonzerowherelinearrchunkrr index_add_rJrC)r7rrrfinal_hidden_states expert_mask expert_hit expert_idx top_k_pos token_idx current_stategateupcurrent_hidden_statess r:rnzPhimoeExperts.forwardQs $.}== ]__ S S(-55ktO_5``K%--aA66K{8'D'DaHHPPRRJ S S S S S S S S S S S S S S S % n nJ#AJT---#(;{:/F#G#G Iy))4M}++M4;LZ;XYY__`agi_jjHD"$(KK$5$5$: !$&M$8$89NPTP^_iPj$k$k !$9M)U^`dJdB22B69B6) r[rqrrrr"r-rGrsrnryrzs@r:rrDs<<0|000000#|#\#| #  ########r;rrBcBtj5|dd\}}||}||z |z d|zk}dddn #1swxYwY||t d}|ru|tj|tj z d d  d}n|}tj |d }| d| } |r|dd\} }tj||ktj| d k} tjd | d|} t$|| ||| } n| } tj|d|t d} tj5| dd\}}||}||z |z d|zk}dddn #1swxYwY| |t d}|ru|tj|tj z d d  d}n|}tj |d }| d| }|r|dd\} }tj||ktj| d k}tjd |d|}t$|||||}n|}tj| |fd } tj||fd }| |fS)ud Sparse mixer function to select top-k experts and compute multipliers. Based on the paper: https://huggingface.co/papers/2409.12136 We first replace the TopK(·) function as random sampling of discrete variables in model training. Then, following Liu et al. (2023a) and Liu et al. (2023b), we apply Heun's third order method to approximate the expert routing gradient and construct a modified back-propagation to give a mathematically sound gradient estimation for expert routing. Args: scores (torch.Tensor): Input scores tensor. jitter_eps (float): Jitter epsilon for numerical stability. training (bool): Flag indicating if the model is in training mode. top_k (int): Number of top experts to select. Returns: tuple[torch.Tensor, torch.Tensor]: Multiplier and selected experts tensors. rTT)rMkeepdim)minrBNz-inf) memory_formatrYr!)rMrg?gioT?gK=U?)alpha)rGrxr\absclamp masked_fillrK empty_likelegacy_contiguous_format exponential_logrrgather logical_or rand_likeaddtype_asrapplyscatteruniform_concat)r jitter_epsrtop_kmask_logits_thresholdmax_indfactorrr multiplier_o max_scoresrr masked_scoresmasked_gates_top2selected_experts_top2multiplier_top2_omask_for_one_top2multiplier_top2s r: sparsemixerr ls$ __)/D)I)I&w##(=#>>"7&"@F!JqS]~ ^ _______________%%&;U6]]KKL #">"7&"@F!JqS]~ ^ _______________&112GvWW (""#4EDbccc SRS[[  Yr]] !( &7R@@@)00R?T0UU,/33D3II G!, !W , OJ ' ' 0 0 2 2T 9   "If.?vNNNVVWhii*00   !     ,z?;DDDJ|%57L$MSUVVV  s%AA22A69A6AI11I58I5cfeZdZdeffd Zdejdeejejfffd ZxZ S)PhimoeTopKRouterr&ct|j|jd|j|_|j|_|j|_|j|_dS)NFr) r,r-rErrouter_jitter_noiseinput_jitter_noisenum_experts_per_tokrrrs r:r-zPhimoeTopKRouter.__init__sY +V-EERRR#)#= "(";/ !3r;rr=c4|jrF|jdkr;|tj|d|jz d|jzz}t |}t||j|j|j \}}|||fS)NrrA)rrr) rr%rGrrr,rnr r$r)r7r router_logitsrouting_weightsrr9s r:rnzPhimoeTopKRouter.forwards = T4q88 U-m<<EEd--sT5L/L M 66 ,7 d&>^b^h- - - ))o/???r;) r[rqrrr"r-rGrsrwrnryrzs@r:r"r"s4|444444 @U\ @eEL%,FDWlpqqq(*  F$7D) ) ) %%%r;Nrrrrgrrr=c |}||}|jd|||||d|\}}||z}|}||}||}||z}|S)N)rrrrgrr)r>r:r?r;) r7rrrrgrrresidualr3s r:rnzPhimoeDecoderLayer.forward.s!,,];; )4> ' 3)%+       q!=0  55mDD //  =0 r;)NNNN)r[rqrrr"rvr-rGrsrw LongTensorr rrrnryrzs@r:r5r5s  |         "IM.204(, |#5<#=>E t+  &-   +, r;r5ceZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedeedZejfd ZxZS) PhimoePreTrainedModelr&modelTr5rr)r)r(r attentionscft||jj}t |t r:t j|jd|t j|j d|dSt |trt j|j d|dSdS)Nr)meanstd) r, _init_weightsr&initializer_ranger_rinitnormal_rrr"weight)r7rrIr9s r:rJz#PhimoePreTrainedModel._init_weights\s f%%%k+ fm , , ; L,3C @ @ @ @ L) = = = = = =  0 1 1 ; LSc : : : : : : ; ;r;)r[rqrrr"rtbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr"r5r_can_record_outputsrGrxrJryrzs@r:rDrDIs &*#-.#4"5N!"&'(8BBB+% U]__;;;;_;;;;;r;rDceZdZdeffd Zeee d dej dzdej dzdej dzde dzdej dzd e dzd eed efd ZxZS) PhimoeModelr&ctj|_j|_t jjj|j|_t j fdtj D|_ t j jjd|_t!|_d|_|dS)Nc0g|]}t|Sr)r5).0rr&s r: z(PhimoeModel.__init__..ps$ d d dy  2 2 d d dr;Tr7r&F)r,r- pad_token_id padding_idx vocab_sizer EmbeddingrE embed_tokens ModuleListrangenum_hidden_layerslayersr<r=normr$ rotary_embgradient_checkpointing post_initrs `r:r-zPhimoeModel.__init__is    !. +L):F>>&+# r;N input_idsrrgr inputs_embeds use_cacherr=c t|du|duz rtd|r|t|j}|||}|V||nd}t j|jd|j|z}| d}|jj tnt} | |j||||} |} | | |} |jd|jjD]} | | f| |||| d|} || } t#| | S) Nz:You must specify exactly one of input_ids or inputs_embedsr_rr!)r8)r&rnrrrg)rg)rrgrror)last_hidden_stater)rZr r&rdget_seq_lengthrGrHr^r8rsliding_windowrrrjrhrgrir)r7rmrrgrrnrorpast_seen_tokens mask_function causal_maskrr decoder_layers r:rnzPhimoeModel.forwardys  -t"; < [YZZ Z  ?0*$+>>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L.2k.H.P**Vw #m;')+%    & "oom,oWW![)H4;+H)HI  M)M*) /#$7 MM -00 %++    r;)NNNNNN)r[rqrrr"r-rr rrGrBrsr FloatTensorboolrrrrnryrzs@r:rZrZgs |  .2.204(,26!%4 4 #d*4  t+4 &- 4  4 (4/ 4 $;4 +,4  4 4 4 ^_ 4 4 4 4 4 r;rZ gate_logitsrc|t|tsdSt|tr/|djtjfd|Dd}tjj|d}tj||d\}}tjj ||}|@tj | d} tj |d} n.|j \} } |j d| | zz} |dddddddf | | | ||fd||}tj| |zdtj|dz } |ddddddf | | | |fd|}tj||zdtj|dz } tj| | dz}||zS)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrc:g|]}|Sr)rJ)r] layer_gatecompute_devices r:r^z,load_balancing_loss_func..s&-j-j-jPZjmmN.K.K-j-j-jr;rYrT)r_rwr8rGrcrrrtopkrrHrKr^r]rrJrr)rzrrrconcatenated_gate_logitsr)r3rrtokens_per_expertrouter_prob_per_expertr1r2rgexpert_attention_mask router_per_expert_attention_mask overall_lossr~s @r:load_balancing_loss_funcrs:*[%"@"@q+u%%s$Q.#(9-j-j-j-j^i-j-j-jpq#r#r#r h)112JPR1SSO*_eDDDA(%--.> LLK!J{'8'8':':BBB"'O!C!C!C&4&:# O4:1=*B^_ 4AAAtT1 2 V& OUKX Y Y WR , , R   "Ik&7&7&9&9>> from transformers import AutoTokenizer, PhimoeForCausalLM >>> model = PhimoeForCausalLM.from_pretrained("mistralai/Phimoe-8x7B-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Phimoe-8x7B-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rmrrgrrnror)lossaux_lossrrrrFr(r)r&rrErqr_rvslicer loss_functionrbrr(rr&rrJr8rrrrF)r7rmrrgrrnrrorrroutputsr slice_indicesrrrs r:rnzPhimoeForCausalLM.forwardsnN%9$D $+Jj  +5$* + )%+'!5 + +  + +  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%ffdoPPPPD  M/% ( H !1HKK 4L4LLL(#3!/)!/    r;Tc |rYt|jdrD|jd|jjdzkr&|} | |jjkrd}t jd|||||||d|} | S)NrQr!)rmrrrnrgrorr)hasattrr&r^rQrrr,prepare_inputs_for_generation) r7rmrrrnrgrorr past_length model_inputsr9s r:rz/PhimoeForCausalLM.prepare_inputs_for_generationls"  ' %GHH '"dk&RUV&VVV)88::KdkJJJ"&rs,%$$$$$ &&&&&&!!!!!!........))))))eeeeeeeeeeRRRRRRRR[[[[[[[[QQQQQQQQKKKKKKKKFFFFFFFF&&&&&&IIIIIIIIIIGGGGGGGGEEEEEEEE......L0L0L0L0L0BIL0L0L0^(((*++,+2 UU\ U# U%, U U U U&%% I% <% % < % L4' %  %%'(%%%%2)**@)@)@)@)@)bi@)@)+*@)F; ; ; ; ; u~.; ; ; |$#$#$#$#$#BI$#$#$#Nxxxxv@@@@@ry@@@(!T!T!T!T!T29!T!T!TH'''''3'''T;;;;;O;;;:H H H H H 'H H H Z# *. O&O&el 33d:O&tO&L4' O&  \C O&O&O&O&dIIIII-IIIXdcccc&FH]ccc k j jr;