tj=dZddlmZddlZddlmZddlmZddlmZddl m Z dd l m Z d d l mZd d lmZmZmZmZmZmZd dlmZGddeZGddeZGddejjZd'dZGddeZGddejZ Gddej!Z"GddeZ#GddeZ$Gd d!eZ%Gd"d#eZ&Gd$d%ee$Z'gd&Z(dS)(zPyTorch Phimoe model.)CallableN)nn) GenericForSequenceClassification)ROPE_INIT_FUNCTIONS)maybe_autocast)OutputRecorder)LlamaAttention)MixtralDecoderLayerMixtralExpertsMixtralForCausalLM MixtralModelMixtralPreTrainedModelMixtralRotaryEmbedding) PhimoeConfigc$eZdZddefdZddZdS)PhimoeRotaryEmbeddingNconfigctj|j|_|j|_||_|jjd|_|j |_ |jdkrt|j|_ | |j|\}|_ | d|d| d|ddS)N rope_typedefaultinv_freqF) persistentoriginal_inv_freq)rModule__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersrcompute_default_rope_parameters rope_init_fnrattention_scalingregister_bufferclone)selfrdevicers g/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/phimoe/modular_phimoe.pyrzPhimoeRotaryEmbedding.__init__)s "("@$*$B! 4[A&*&J >Y & & 3DN CD +/+<+*ll_illl )L))A- ; &{ 3y @ @W @T[89[\\\ +M::[0@  '+&7&7 QXw&W&W##&,n""&$T111d]399;;BB > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C % %&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))f$C''))f$C  % % % % % % % % % % % % % % % vvagqw//s=BH  HHN)NN)r: __module__ __qualname__rrrRr+r*rr(sNVV|VVVV 000000r+rceZdZdS)PhimoeAttentionNr:rTrUrVr+r*rXrXUDr+rXc eZdZedejdejdejdejdejf dZedejfdZd S) PhimoeMultiplierscores multiplierselected_experts masked_gates mask_for_onec:||||||zS)a Forward pass for the custom autograd function. Args: ctx: Context object to save information for backward computation. scores (torch.Tensor): Input scores tensor. multiplier (torch.Tensor): Multiplier tensor. selected_experts (torch.Tensor): Tensor of selected experts. masked_gates (torch.Tensor): Masked gates tensor. mask_for_one (torch.Tensor): Mask for one tensor. Returns: torch.Tensor: Result of the forward pass. )save_for_backward)ctxr]r^r_r`ras r*rRzPhimoeMultiplier.forwardZs(. j*:LIIIL((r+grad_at_outputc|j\}}}||z}||dz}|d|||ddddfS)aB Backward pass for the custom autograd function. Args: ctx: Context object with saved tensors from the forward pass. grad_at_output (torch.Tensor): Gradient at the output. Returns: tuple[torch.Tensor, None, None, None, None]: Gradients for the inputs. r1)r7indexsrcN) saved_tensorsmul scatter_add_)rdrer^r_r`grad_at_scores_expandeds r*backwardzPhimoeMultiplier.backwardtsx695F2 $l'*4".1C1CB1G1G"G,," -    $       r+N)r:rTrU staticmethodr;TensorrRrmrVr+r*r\r\Ys) )L) , ) l ) l )))\)2     \   r+r\cBtj5|dd\}}||}||z |z d|zk}dddn #1swxYwY||t d}|ru|tj|tj z d d  d}n|}tj |d }| d| } |r|dd\} }tj||ktj| d k} tjd | d|} t$|| ||| } n| } tj|d|t d} tj5| dd\}}||}||z |z d|zk}dddn #1swxYwY| |t d}|ru|tj|tj z d d  d}n|}tj |d }| d| }|r|dd\} }tj||ktj| d k}tjd |d|}t$|||||}n|}tj| |fd } tj||fd }| |fS)ud Sparse mixer function to select top-k experts and compute multipliers. Based on the paper: https://huggingface.co/papers/2409.12136 We first replace the TopK(·) function as random sampling of discrete variables in model training. Then, following Liu et al. (2023a) and Liu et al. (2023b), we apply Heun's third order method to approximate the expert routing gradient and construct a modified back-propagation to give a mathematically sound gradient estimation for expert routing. Args: scores (torch.Tensor): Input scores tensor. jitter_eps (float): Jitter epsilon for numerical stability. training (bool): Flag indicating if the model is in training mode. top_k (int): Number of top experts to select. Returns: tuple[torch.Tensor, torch.Tensor]: Multiplier and selected experts tensors. r1T)r7keepdim)minr Nz-inf) memory_formatr6r)r7rgg?gioT?gK=U?)alpha)r;no_gradr<absclamp masked_fillr= empty_likelegacy_contiguous_format exponential_log unsqueezesoftmaxgather logical_or rand_likeaddtype_asr\applyscatteruniform_concat)r] jitter_epstrainingtop_kmask_logits_thresholdmax_indfactorr`r_ multiplier_o max_scoresrar^ masked_scoresmasked_gates_top2selected_experts_top2multiplier_top2_omask_for_one_top2multiplier_top2s r* sparsemixerrs$ __)/D)I)I&w##(=#>>"7&"@F!JqS]~ ^ _______________%%&;U6]]KKL #">"7&"@F!JqS]~ ^ _______________&112GvWW (""#4EDbccc SRS[[  Yr]] !( &7R@@@)00R?T0UU,/33D3II G!, !W , OJ ' ' 0 0 2 2T 9   "If.?vNNNVVWhii*00   !     ,z?;DDDJ|%57L$MSUVVV  s%AA22A69A6AI11I58I5ceZdZdS) PhimoeExpertsNrYrVr+r*rrrZr+rcfeZdZdeffd Zdejdeejejfffd ZxZ S)PhimoeTopKRouterrct|j|jd|j|_|j|_|j|_|j|_dS)NFbias) superr hidden_sizenum_local_expertsrouter_jitter_noiseinput_jitter_noisenum_experts_per_tokr num_expertsr(rr9s r*rzPhimoeTopKRouter.__init__sY +V-EERRR#)#= "(";/ !3r+ hidden_statesreturnc4|jrF|jdkr;|tj|d|jz d|jzz}t |}t||j|j|j \}}|||fS)Nr?)rrr) rrr;ryrrrRrrr)r(r router_logitsrouting_weightsr_r9s r*rRzPhimoeTopKRouter.forwards = T4q88 U-m<<EEd--sT5L/L M 66 ,7 d&>^b^h- - - ))o/???r+) r:rTrUrrr;rotuplerR __classcell__r9s@r*rrs4|444444 @U\ @eEL%,FDWlpqqq(*  F$7D) ) ) %%%r+)r:rTrUrintrrrs@r*rrNsK |           r+rc0eZdZeedeedZdS)PhimoePreTrainedModelr)rg)rr attentionsN)r:rTrUr rrrX_can_record_outputsrVr+r*rrYs6'(8BBB+%r+rc$eZdZdeffd ZxZS) PhimoeModelrct|tj|j|jd|_dSr)rrrrrrnormrs r*rzPhimoeModel.__init__bs>    L!39Laefff r+)r:rTrUrrrrs@r*rrasOg|ggggggggggr+rc6eZdZfdZ dfd ZxZS)PhimoeForCausalLMct|tj|j|j|jj|_dS)Nr) rrrLinearr vocab_sizer lm_head_biaslm_headrs r*rzPhimoeForCausalLM.__init__hsB    y!3V5FT[Mefff r+NTc |rYt|jdrD|jd|jjdzkr&|} | |jjkrd}t jd|||||||d|} | S)Nr.r) input_idspast_key_valuesattention_mask inputs_embedsrJ use_cachelogits_to_keeprV)hasattrrr?r.get_seq_lengthrprepare_inputs_for_generation) r(rrrrrJrrkwargs past_length model_inputsr9s r*rz/PhimoeForCausalLM.prepare_inputs_for_generationms"  ' %GHH '"dk&RUV&VVV)88::KdkJJJ"&rs"$$$$$$ 766666++++++444444111111/.....*0*0*0*0*02*0*0*0Z     n   ; ; ; ; ; u~.; ; ; |xxxxv     N   @@@@@ry@@@(!T!T!T!T!T29!T!T!TH     ,   2ggggg,ggg )))))*)))Xdcccc&FH]ccc   r+