tjhj<dZddlZddlmZddlmZddlZddlmcm Z ddl m Z ddlmZddl mZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZddlmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6e(j7e8Z9e'rddl:m;Z;e&de GddeZ<Gdde/Z=Gdde0Z> dAd ej?d!ej@d"ej@d#ej@d$eej@d%fd&eAdzd'eAdzd(eBej@ej@ffd)ZCe ZDeCeDd*<Gd+d,ej?ZEGd-d.e-ZFGd/d0ej?ZGGd1d2eZHGd3d4e.ZIGd5d6e6ZJ dBd7ej@eBej@zdzd8eKdzd9eKdzd:eKd$ej@dzd(ej@eKzf d;ZLGd<d=e5ZMGd>d?e,ZNgd@ZOdS)CzPyTorch Doge model.N)Callable)Union)strict)nn)initialization)ACT2FN)Cache)PreTrainedConfig)compile_friendly_flex_attention)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)RopeParameters)AttentionInterfacePreTrainedModel)Unpack)TransformersKwargsauto_docstringis_torch_flex_attn_availablelogging)OutputRecorder)LlamaForSequenceClassificationLlamaMLPLlamaPreTrainedModel LlamaRMSNormLlamaRotaryEmbeddingapply_rotary_pos_embeager_attention_forward repeat_kv)MixtralForCausalLM MixtralModel) BlockMaskzSmallDoge/Doge-320M) checkpointc @eZdZUdZdZdgZdddddddddddd Zd gd gfd d gd gfd gd gfd ZdZe e d<dZ e e d<dZ e e d<dZ e e d<dZee ze d<dZee d<dZee d<dZee d<dZee d<d Zee d!<dZe e d"<d#Zeezd#ze d$<d%Ze e d&<d#Ze d#ze d'<d Zee d(<dZed#ze d)<d Zee d*<d#Z e d#ze d+<dZ!e e d,<d Z"ee d-<d.Z#e e d/<d0Z$e e d1<d Z%ee d2<d Z&ee d3<d4Z'ee d5<d#Z(e d#ze d6<d#Z)e d#ze d7<d#Z*e e+e zd#ze d8<fd9Z,xZ-S): DogeConfiga keep_window_size (`int`, *optional*, defaults to 2048): The window size of tokens that are not dynamically masked, and dynamic masking is only performed when the sequence length exceeds this value. is_moe (`bool`, *optional*, defaults to `False`): Whether to use the Cross Domain Mixture of Experts, if `True`, the MoE will inherit the MLP to initialize. ```python >>> from transformers import DogeConfig, DogeModel >>> # Initializing a Doge-320M style configuration >>> configuration = DogeConfig() >>> # Initializing a model from the Doge-320M style configuration >>> model = DogeModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```dogepast_key_valuescolwiserowwisecolwise_gather_outputrowwise_split_input) zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.dt_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_projzlayers.*.mlp.router_gatezlayers.*.mlp.down_embedzlayers.*.mlp.up_embed input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnormi vocab_sizei hidden_sizeintermediate_size num_hidden_layershidden_dropoutsilu hidden_actg{Gz?initializer_rangegư> rms_norm_epsT use_cacheFtie_word_embeddingsmax_position_embeddingsNrope_parametersnum_attention_headsnum_key_value_headsattention_biasattention_dropoutmlp_biassliding_windowkeep_window_sizeis_moei@ num_experts@num_experts_per_toknorm_topk_proboutput_router_logitsgMbP?router_aux_loss_coef pad_token_id bos_token_id eos_token_idc `|j |j|_tjdi|dS)N)rGrFsuper __post_init__)selfkwargs __class__s c/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/doge/modular_doge.pyrZzDogeConfig.__post_init__s:  # +'+'?D $''''''').__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planr5int__annotations__r6r8r:r<floatr>strr?r@rAboolrBrCrDrdictrFrGrHrIrJrKrLrMrNrPrQrRrSrTrUrVlistrZ __classcell__r]s@r^r'r':s  &J#4"5&/%.%.&/%."+ )"+$;#8!6  &(9:#%568IJ!"_$56 JK!s!!!s"%NECK%%%J#u###L%It %%%%#'S'''48O^d*T1888    &*t*** ND   &)ut|)))Hd!%NC$J%%% c   FDK!!!! ND   !&$&&&"'%'''#L#*####L#*###+/L#S /D(///(((((((((r_r'ceZdZdS) DogeRMSNormNr`rarbrXr_r^rrrrDr_rrceZdZdS)DogeRotaryEmbeddingNrsrXr_r^rvrvrtr_rvmodulequerykeyvaluer1r$scalingsoftcapreturnc P d}d t|tr|}n|  ddddddd|jdf fd} t|||| |d|d\} } | |j} | dd} | | fS)Nc~tj|z z}|||||z}|SN)torchtanh)score batch_idxhead_idxq_idxkv_idx causal_maskr|s r^ score_modz)flex_attention_forward..score_modsJ  ej999E  "K 28&B\$\!}d* !'!9 & 7i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    ek&*DEEFFy  & 68RY_Yn    i  & 68JQWQf    "$-V5HIII !$-V5HIII r_r0position_embeddingsr1r)r}c |jdd}g|d|jR}||||dd}||||dd} |||dd} |\} } t|| | | \}} || | | |j \} } | | dd | jd| jdd} tj|jt#j| zdd} ||| |j|}t+||j}t.|jjt6}|||| | f||jsdn|j|jd|\}}|j g|dR}| |}||fS) Nrrrr)r0 dt_statesrLr1r;)r1dropoutr{)!rrrrviewrrrrrupdaterrreshaperexprFsoftplusprepare_dynamic_maskrLr!rALL_ATTENTION_FUNCTIONS get_interfacer_attn_implementationr trainingrIr{rr)r[r0rr1r)r\ input_shape hidden_shape query_states key_states value_statescossinr attn_maskattention_interfacer attn_weightss r^forwardzDogeAttention.forwards$)#2#. 88b8$-88 {{4;;}#=#=#B#B<#P#PQQ[[\]_`aa [[]!;!;!@!@!N!NOOYYZ[]^__ {{=1166|DDNNqRSTT &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J LL  " "1a ( ( 0 01CA1F HZ[]H^`b c c  Idfqz)'<'<<==GGBOO --'!2) .  i)BCC (?(M(M K ,.E) ) %8$7     % %#}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r_r7rrLc tj|jj}|j}|dddddddfdd|jdd}|t |ts|jtjkr7|j}tj |tj d|j ||}| |ddddddd|jdfdk|}|jd|krktj |||j }tj||ddd j} |d| d }| |dk|}|S) a8 The core idea of DMA is to calculate the dynamic attention mask to mask the tokens that should be masked, so as to form sparse attention. Combine `dt_states` with `attention_mask` to generate the final `attn_mask`. Args: hidden_states (`torch.Tensor`): The input hidden_states, used to determine the minimum value of the current input precision. dt_states (`torch.Tensor`): dt_states of shape `(batch_size, num_heads, key_sequence_length)`. keep_window_size (`int`): The window size of tokens that are not dynamically masked, and dynamic masking is only performed when the sequence length exceeds this value. attention_mask (`torch.Tensor`, *optional*): attention mask of shape `(batch_size, 1, query_sequence_length, key_sequence_length)`. Nrrr;)devicerrrrTF)dimlargestsortedg?)rfinforminexpandrrr$rlwheretensorr masked_fill zeros_liketopkindicesscatter) r[r0rrLr1 min_dtyperr active_mask topk_indicess r^rz"DogeAttention.prepare_dynamic_masks$K 3448 #aaaD!!!m,33 M'*B    %j.S.S %#uz11%+!&"EL^=RZ_$`$`$`bk"""--nQQQ111F[ XZH[F[=[.\`a.aclmmI ?2 !1 1 1*9E)JZ[[[K :i1ArSW`efffnL%--b,DDK!--kS.@)LLIr_rNN)r7N) r`rarbr'rhrrTensortupler rrrorps@r^rrs>JJzJcDjJJJJJJD/3(, 3)3)|3)#5<#=>3) t+ 3)  3) u|U\D0% 2E2LL M3)3)3)3)r!%.2 ##|#<# #  t+ ########r_rceZdZdS)DogeMLPNrsrXr_r^rr?rtr_rcHeZdZdeffd ZdejdejfdZxZS) DogeCDMoErc2t|j|_|j|_t|j|_|j|_tj tj |j|_ |j |_ |j|_tj|j|j|j|_tj|j|j|j|_tj|j|j|j|_tj|j|j dzd|_tj|j|j|_tj|j|j|_dS)NrrF)rYrr6r8r r>act_fnrNmathfloorsqrtnum_keysrPtop_krQrrrJ gate_projup_proj down_proj router_gate Embedding down_embedup_embedr[rr]s r^rzDogeCDMoE.__init__DsA !-!'!9V./ !- 49T-=#>#>?? / $34#3T5KRXRabbby!143IPVP_``` 4#94;KRXRabbb9T%5t}q7HuUUU,t'79IJJ T%5t7GHH r_r0r}c |j\}}}||d||zd}||jd\\}}\} } |d|dz} | d|jz| dz} | jg| jdddR} | jg| jdddR} | |jd\} }| d|}tj | d}|j r|| ddz}| |}| |}tj||||zdd||zd}|||z}tj|||zdd|||d}||||||z}||z}||fS)NrrrrT)rkeepdimr)rrrrr unsqueezergatherrsoftmaxrQsumrrrmatmulrrrr)r[r0r\bszseq_len_ router_logitsscores_xscores_y indices_x indices_y all_scores all_indicesscoresposition_indicesrrouting_weightsrrexperts_weightsexperts_statess r^rzDogeCDMoE.forward[s (-Wa((77<CCCC #-??4:2?#F#F  $$R)9::)F333   I 22r42HH HO__W-- ==)),z=3E3EcGmUWYZ3[3[\\aabehoboqstt++o66Ho&:&:3=!R&P&PRZ[[``adfmoqrrt{{4>>-3P3P'Q'QTXT`T`anToTo'opp %6 m++r_) r`rarbr'rrrrrorps@r^rrCsuIzIIIIII.,|,  ,,,,,,,,r_rceZdZddededzffd Z ddejdeejejfdzdejdzd ej dzd e dzd e dzd e e d eejeejejfdzffdZxZS)DogeDecoderLayerNrrc t|j|_t|j|j|_t|||_tj tj |j|_ t|j|j|_|jst!|nt#||_tj tj |j|_dS)Nr)rr)rYrr<rrr6r@input_layernormr self_attnrrronesinput_residualpost_attention_layernormrMrrmlppost_attention_residualrs r^rzDogeDecoderLayer.__init__}s $3*6+=6CVWWW&f JJJ l5:f6H+I+IJJ(3F4FFL_(`(`(`%*0-N76???Yv=N=N')|EJv?Q4R4R'S'S$$$r_Fr0rr1 position_idsr)rAr\r}c r|}||}|jd||||||d|\}} tj||j|j}|j|z|z}|}||}||}tj||j|j}|j |z|z}|S)N)r0rr1rr)rA)prrX) rrrrr<rrrrr) r[r0rr1rr)rAr\residualself_attn_weightss r^rzDogeDecoderLayer.forwards!,,];; +94>, ' 3)%+ , , , , ( ( -43FQUQ^___ +h6F !55mDD //  -43FQUQ^___ 4x?-O r_r)NNNNF)r`rarbr'rhrrrr LongTensorr rlrr FloatTensorrrorps@r^rr|s, T Tz TcDj T T T T T TIM.204(,!&  | #5<#=>E  t+  &-   $; +,  u %(95;L(L"MPT"TT U        r_rcheZdZdZdZeedeedZ e j dZ dS)DogePreTrainedModelFr)index)rr0 attentionsctj||t|tr-t |drt j|jdSdSt|trTt |drt j |j t |drt j |j dSdSdS)zInitialize the weightsrrrN) r _init_weightsrrhasattrinitzeros_rrones_rr)r[rws r^r$z!DogePreTrainedModel._init_weightss %dF333 fm , , ;vs## & FH%%%%% & &  0 1 1 ;v/00 2 60111v899 ; 69:::::  ; ; ; ;r_N) r`rarb_supports_flash_attn_can_compile_fullgraphrrrr_can_record_outputsrno_gradr$rXr_r^r r sh "' ;;;)# U]__ ; ;_ ; ; ;r_r ceZdZdS) DogeModelNrsrXr_r^r.r.rtr_r. gate_logitsrNrrcT|t|tsdS|dj}|dj}g}g}|D]9} | |} | |d\\} } \} } | d| dz}| d|z| dz}|jg|jdddR}|jg|jdddR}||d\}}| d|}tj |d}| || |;tj|d}tj|d}||d}tj|||}tj|||}|d|||jdz }tj|d}nk|j\}}t'|}|ddddddf||||fd|}|d|}tj|||}tj|||}|d||tj|z }|ddddddf||||fd||}tj||zdtj|dz }tj||z}||zS)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `router_gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [2, batch_size * sequence_length, num_keys]. num_experts: Number of experts num_keys: Number of keys top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrrrrr)rrrrrrrrrrrrappendrcatr ones_like scatter_add_meanlenrrrlr)r/rNrrr1 compute_dtypecompute_deviceall_expert_indicesall_routing_weightslayer_gate_logitsrrrrrrrr expert_indicesr tokens_per_expertpadrouter_prob_per_expert batch_sizesequence_lengthr:expert_attention_mask router_per_expert_attention_mask overall_losss r^load_balancing_loss_funcrEs@*[%"@"@qN(M ^*N(44-00@@7H7M7Mh\^7M7_7_484y)''++h.@.@.D.DD ))"--89;N;Nr;R;RR $Z_@j&6ss&;@R@@@ &k&C (9#2#(>CCCC (ooeo<< $++B0@AA)JB777!!.111""?3333#51===)$7Q???/44R88!K =Q_```o0 n]]]-::1>PRUVVYkYqrsYtt"',?Q!G!G!G&4&:# O ,, 4AAAt+ , V& OUK L L WR[[ R    044R889N9S9S9U9UV"K =Q_```o0 n]]]-::1>PRUVVY^Yb !Z Z   4AAAt+ , V& O[Q R R WR % % R   )"'+>Aa+agh!i!i!ilqlu ,!m m m " 9.1GGHHL + %%r_ceZdZfdZ ddejdzdejdzdejdzdedzdejdzd ejdzd e dzd e ejzd e dzd e e de fdZxZS)DogeForCausalLMct|t||_|j|_dSr)rYrr.modelrNrs r^rzDogeForCausalLM.__init__2s;    v&& !-r_Nrr.r1rr)r/labelsrAlogits_to_keeprRr\r}c ^| | n |jj} |jd||||||d| } | j} t |t rt | dn|} || dd| ddf}d}||j|||j fi| }d}| rrt| j |j tjtj|j |j|}|%||j||jzz }t)|||| j| j| j| j S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, DogeForCausalLM >>> model = DogeForCausalLM.from_pretrained("SmallDoge/Doge-320M") >>> tokenizer = AutoTokenizer.from_pretrained("SmallDoge/Doge-320M") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)r.r1rr)r/rA)lossaux_losslogitsr)r0r"rrX)rrRrIlast_hidden_staterrhslicelm_head loss_functionr5rErrNrrrrPrSrrrr)r0r")r[r.r1rr)r/rJrArKrRr\outputsr0 slice_indicesrOrMrNs r^rzDogeForCausalLM.forward7sH%9$D $+Jj  +5$*+ )%+' + + + +  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%ffdoPPPPD  M/%  49T%56677( H!1HKK 4L4LLL(#3!/)!/    r_) NNNNNNNrN)r`rarbrrrrr rrlrhrrrrrorps@r^rGrG1s.......2.204(,26*.!%-.,0O O #d*O  t+O &- O  O (4/ O  4'O $;O el*O #TkO +,O  #O O O O O O O O r_rGceZdZdS)DogeForSequenceClassificationNrsrXr_r^rWrWrtr_rW)r'rGr.r rWr)NNrN)Prcrcollections.abcrtypingrrtorch.nn.functionalr functionalrhuggingface_hub.dataclassesrrr& activationsr cache_utilsr configuration_utilsr integrations.flex_attentionr modeling_layersr modeling_outputsrrmodeling_rope_utilsrmodeling_utilsrrprocessing_utilsrutilsrrrrutils.output_capturingrllama.modeling_llamarrrrrrr r!mixtral.modeling_mixtralr"r# get_loggerr`logger!torch.nn.attention.flex_attentionr$r'rrrvModulerrjrrrrrrrr r.rhrErGrW__all__rXr_r^rps  $$$$$$ ......&&&&&&!!!!!! 333333JJJJJJ999999QQQQQQQQ111111AAAAAAAA&&&&&&^^^^^^^^^^^^444444                    HGGGGGGG  H % %!!<;;;;;;0111L(L(L(L(L(!L(L(21L(^     ,        .   ! +*+* I+* <+* +* < +* %, 34 +* T\ +*T\+* 5< %&+*+*+*+*\-,..1G-.wwwwwBIwwwt     h   6,6,6,6,6, 6,6,6,r-----1---`;;;;;.;;;.        #*. g&g&el 33d:g&tg&Djg&  g& L4' g&  \C g&g&g&g&TU U U U U (U U U p     $B      r_