\+"S1S2\&55r?\+"S3S4\?55r@SCS5\\Rn\A\RnS4S6\\8S&\\RnS \\Rn\844S7jjrB\+"S8S9\?\55rC"S:S;\\?5rD"S<S=\\?5rE"S>S?\\?5rF/S@QrGg)D)CallableOptionalUnionN)nn)check_model_inputs)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GenericForQuestionAnswering GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)OutputRecorder) MixtralConfigc6^\rSrSrS\4U4SjjrSrSrU=r$)MixtralBlockSparseTop2MLP8configc>[TU]5 URUlURUl[ R"UR URSS9Ul[ R"URUR SS9Ul [ R"UR URSS9Ul [URUl gNFbias)super__init__intermediate_sizeffn_dim hidden_size hidden_dimrLinearw1w2w3r hidden_actact_fnselfr% __class__s d/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/mixtral/modeling_mixtral.pyr+"MixtralBlockSparseTop2MLP.__init__9s //  ,,))DOOT\\F))DLL$//F))DOOT\\FV../ cURURU55URU5-nURU5nU$N)r5r1r3r2)r7 hidden_statescurrent_hidden_statess r9forward!MixtralBlockSparseTop2MLP.forwardDs> $ DGGM,B CdggmF\ \ $(= >$$r;)r5r-r/r1r2r3) __name__ __module__ __qualname____firstlineno__r!r+r@__static_attributes__ __classcell__r8s@r9r#r#8s 0} 0%%r;r#cf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)MixtralSparseMoeBlockJa This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accommodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. c>[TU]5 URUlURUlUR UlURUl [R"URURSS9Ul [R"[UR5Vs/sHn[U5PM sn5UlUR"Ulgs snfr')r*r+r.r/r,r-num_local_experts num_expertsnum_experts_per_toktop_krr0gate ModuleListranger#expertsrouter_jitter_noise jitter_noise)r7r%_r8s r9r+MixtralSparseMoeBlock.__init__Vs  ,,// !33// IIdoot/?/?eL }}QVW[WgWgQh%iQhA&?&GQh%ij #66&js*Cr>returncURup#nUR(aSURS:aCU[R"U5R SUR- SUR-5-nUR SU5nURU5n[R"US[RS9n[R"X`RSS9upgXfRSSS9-nURUR5n[R "X#-U4URUR"S 9n[R$R&R)XpR*S 9R-S SS5n [R."U RS S9S5R15n U Hn UR2U n [R4"XR7S55upUS U4R9SU5nU "U5XnU S 4-nUR;SUURUR55 M UR9X#U5nX4$)rg?r dimdtyper^T)r^keepdim)r_device) num_classes)r\N)shapetrainingrVtorch empty_likeuniform_viewrQFsoftmaxfloattopkrPsumtor_zerosrbr functionalone_hotrNpermutegreaternonzerorTwheresqueezereshape index_add_)r7r> batch_sizesequence_lengthr/ router_logitsrouting_weightsselected_expertsfinal_hidden_states expert_mask expert_hitted expert_idx expert_layeridxtop_x current_stater?s r9r@MixtralSparseMoeBlock.forwardes2?2E2E/ Z ==T..2 U--m<EEcDL]L]F]_beievev_vw wM%**2z:  -0 ))Mq L,1JJ XZ,[)..2t.DD),,]-@-@A#kk  ): 6m>Q>QZgZnZn  hh))112BP`P`1aiijkmnpqr  koo(o&CQGOOQ 'J<< 3L[%<%D%DQ%GHJC*$+6>>r:NM$0$?/Y\^bRbBc$c ! * *1e5J5M5MmNaNa5b c(299*Wab"11r;)rTr-rQr/rVrNrP) rBrCrDrE__doc__r+rhTensorr@rFrGrHs@r9rJrJJs-  7%2U\\%2ell%2%2r;rJRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$)MixtralRMSNormc>[TU]5 [R"[R "U55UlX lg)z- MixtralRMSNorm is equivalent to T5LayerNorm N)r*r+r Parameterrhonesweightvariance_epsilon)r7r.epsr8s r9r+MixtralRMSNorm.__init__s/ ll5::k#:; #r;cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nrdr\T)ra) r_rqrhfloat32powmeanrsqrtrr)r7r> input_dtypevariances r9r@MixtralRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r;c^[URR5SUR3$)Nz, eps=)tuplerrfrr7s r9 extra_reprMixtralRMSNorm.extra_reprs* ))*+6$2G2G1HIIr;)rr)gư>) rBrCrDrEr+r@rrFrGrHs@r9rrs$;JJr;rcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Nr\rdr`)rfrhcat)xx1x2s r9 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r;cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkcossin position_ids unsqueeze_dimq_embedk_embeds r9apply_rotary_pos_embrsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r;r>n_reprYcURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r N)rfexpandrz)r>rbatchnum_key_value_headsslenhead_dims r9 repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr;modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nrdrrer\r])prgr )rnum_key_value_groupsrhmatmul transposerfrrsrmrrqr_rrg contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r9eager_attention_forwardrs3 ; ;[TU]5 XlX l[ USS5=(d UR UR -UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-SS9Ul[R"UR URUR-SS9Ul[R"UR URUR-SS9Ul[R"UR UR-UR SS9Ulg)NrgTFr()r*r+r%rgetattrr.num_attention_headsrrrrattention_dropout is_causalrr0q_projk_projv_projo_projr7r%rr8s r9r+MixtralAttention.__init__s.  " D9mV=O=OSYSmSm=m $*$>$>&B\B\$\!}}d* !'!9!9ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii : :T]] JFL^L^ejk r;r>position_embeddingsrpast_key_valuecache_positionrrYc `URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO URUR [#URSS5S.UD6unnUR$"/UQSP76R'5nUR)U5nUU4$) Nr\r rd)rrreagersliding_window)rrr)rfrrrkrrrrupdaterrr%_attn_implementationrrgrrrrzrr)r7r>rrrrr input_shape hidden_shape query_statesrrrr cache_kwargsattention_interfacerrs r9r@MixtralAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<'*)!. *) ! *) !!1!12 *)-.*) u||Xell3XeELL>Q5RR S*)*)r;rc8^\rSrSrS\S\4U4SjjrSS\RS\ \R\R4S\ \RS\ \RS \ \ \RS \ \RS \ \ S \R4S jjrSrU=r$)MixtralDecoderLayeri(r%rc>[TU]5 URUl[X5Ul[ U5Ul[URURS9Ul [URURS9Ul g)Nr) r*r+r.r self_attnrJblock_sparse_moer rms_norm_epsinput_layernormpost_attention_layernormrs r9r+MixtralDecoderLayer.__init__)sk !--)&< 5f =-f.@.@fFYFYZ(6v7I7IvObOb(c%r;r>rrrrrrrYc UnURU5nUR"SUUUUUUS.UD6upX-nUnURU5nURU5upX-nU$)N)r>rrrrr)rrrr) r7r>rrrrrrresidualrWs r9r@MixtralDecoderLayer.forward3s!,,];  >> ' 3)%))    !0 !55mD 00?  0 r;)rr.rrr)NNNN)rBrCrDrEr!rr+rhrrrrrr FloatTensorr@rFrGrHs@r9rr(sd}dd26378<59 || #5<<#=> !.  u//0  !u||!45 !!1!12 +,      r;rcl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)MixtralRotaryEmbeddingiVr%c>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r*r+hasattr isinstancerdictgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr%r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r7r%rbrr8s r9r+MixtralRotaryEmbedding.__init__Ws  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rr%rrrrrr=) rBrCrDrEr!r+rhno_gradrr@rFrGrHs@r9rrVs6/}//" ]]_ <  attentionsrN)rBrCrDrEr!__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrrJrr_can_record_outputsrFrr;r9r#r#xs\ &*#./#4"5N""&'(=QG,&r;r#c^\rSrSrS\4U4Sjjr\\SS\\ RS\\ RS\\ RS\\ S\\ RS \\S \\ RS \\S \4S jj55rSrU=r$) MixtralModelir%c >[TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nr)r%F)r*r+ pad_token_id padding_idx vocab_sizer Embeddingr. embed_tokensrRrSnum_hidden_layersrlayersrrnormr rotary_embgradient_checkpointing post_initrs r9r+MixtralModel.__init__s  !.. ++LL):):F input_idsrrr% inputs_embeds use_cacherrrYc zUSLUSL- (a [S5eU(a Uc [5nUcURU5nUcDUbUR5OSn [R "XUR S-URS9nUcURS5nURRc[O[n U "URUUUUUS9n Un URX5n URSURRHnU"U 4U U UUUUS.UD6n M UR!U 5n [#U US9$)Nz:You must specify exactly one of input_ids or inputs_embedsrr )rb)r% input_embedsrrr%r)rrrrrDr)last_hidden_stater%) ValueErrorr r:get_seq_lengthrharangerfrbrr%rrrr>r<r;r=r)r7rBrrr%rCrDrrpast_seen_tokens mask_functionrr>r decoder_layers r9r@MixtralModel.forwardsh  -t"; <YZ Z 0*nO   --i8M  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L.2kk.H.H.P*Vw #;;&))+%  & #oomJ![[)H4;;+H+HIM) $7*).#-  MJ -0 %++  r;)r:r?r<r=r7r>r8)NNNNNNN)rBrCrDrEr!r+rrrrhrrr rboolrrrr@rFrGrHs@r9r4r4s} 151537+/59$(59< E,,-< !.< u//0 < "% <   1 12 < D>< !!1!12< +,<  < < r;r4 gate_logitsrNcUb[U[5(dg[U[5(aBUSRn[R"UVs/sHoUR U5PM snSS9n[R RRWSS9n[R"XrSS9up[R RRX5n Uc:[R"U R5SS9n [R"USS9n GOURupURSX--nUSSS2SS2SS4RXXU45RSX!5R W5n[R "U R5U-SS9[R "USS9- n USSS2SS2S4RXX45RSU5R U5n[R "UU-SS9[R "USS9- n [R "XR#S5-5nUU-$s snf)ax Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrr`r\)r rrbrhrrqrrsrmrortrrnrfrrzrpr)rPrNrPrcompute_device layer_gateconcatenated_gate_logitsrrWrrtokens_per_expertrouter_prob_per_expertr|r}r;expert_attention_mask router_per_expert_attention_mask overall_losss r9load_balancing_loss_funcrZs+:*[%"@"@+u%%$Q..#(99^i-j^iPZmmN.K^i-jpq#r hh))112JPR1SO**_DA((%%--.>LK!JJ{'8'8':B"'O!C&4&:&:# 4::1=*B^_ 4AtT1 2 V&OKX Y WR , R  "IIk&7&7&9logitscJ>[TU]U5 [U5UlURUl[ R "URURSS9UlURUl URUl URUl UR5 gr')r*r+r4r$r8rr0r.r]router_aux_loss_coefrMrNrOr@r6s r9r+MixtralForCausalLM.__init__6s  !&)  ++yy!3!3V5F5FUS $*$?$?!!33#)#=#=  r;cXlgr=r$)r7decoders r9 set_decoderMixtralForCausalLM.set_decoderBs r;cUR$r=rdrs r9 get_decoderMixtralForCausalLM.get_decoderEs zzr;rBrrr%rClabelsrDoutput_router_logitsrlogits_to_keeprrYc ~UbUOURRnUR"SUUUUUUUU S.U D6n U Rn [ U [ 5(a [ U *S5OU nURU SS2USS245nSnUbUR"XUR40U D6nSnU(aZ[U RURURU5nUb+UURURUR 5-- n[#UUUU R$U R&U R(U RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MixtralForCausalLM >>> model = MixtralForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rBrrr%rCrDrlr)lossaux_lossr_r%r>r'r~r)r%rlr$rGr rslicer] loss_functionr8rZr~rNrOrarqrbrr%r>r')r7rBrrr%rCrkrDrlrrmroutputsr> slice_indicesr_rorps r9r@MixtralForCausalLM.forwardHsUP%9$D $++JjJj  +/** + )%+'!5) +  +  11 8B>SV8W8W~ot4]k mA}a,?@A  %%fdooPPD /%%  (( H !11HKK 4LLL(#33!//))!//  r;)r]r$rNrOrar8) NNNNNNNNNr)rBrCrDrE_tied_weights_keys_tp_plan_pp_planr+rfrirrrrhrrr rrOrrrrrr@rFrGrHs@r9r\r\0sY*+=)H_-z:;H 151537+/59-1$(/35934R E,,-R !.R u//0 R "% R   1 12 R ))*R D>R 'tnR !!1!12R c5<</0R +,R  #R R r;r\c\rSrSrSrg) MixtralForSequenceClassificationirNrBrCrDrErFrr;r9rzrzr;rzc\rSrSrSrg)MixtralForTokenClassificationirNr{rr;r9r~r~r|r;r~c\rSrSrSrg)MixtralForQuestionAnsweringirNr{rr;r9rrr|r;r)r\rr4r#rzr~)Nr )r)NrdN)Htypingrrrrhtorch.nn.functionalrrsrltransformers.utils.genericr activationsr cache_utilsr r generationr integrationsr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrrrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrconfiguration_mixtralr!Moduler#rJrrrrrrrnrrrrr#r4rrZr\rzr~r__all__rr;r9rst6-, 9!.)7RB RKF&II+0% %$@2BII@2FY'JRYYJ(J((6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%'(%4;)ryy;)|+4+\?O&#O&U\\* O&  5<<  O&dk /k k \ 'GI_  $ACY  "=?U  r;