g)1zPyTorch Mixtral model.)OptionalUnionN)nn)ACT2FN)Cache DynamicCache)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)Unpack)TransformersKwargslogging)OutputRecorder) MistralAttentionMistralForCausalLMMistralForQuestionAnswering MistralForSequenceClassificationMistralForTokenClassification MistralModelMistralPreTrainedModelMistralRMSNormMistralRotaryEmbedding) MixtralConfig gate_logits num_expertsattention_maskreturncUb[U[5(dg[U[5(aBUSRn[R"UVs/sHoUR U5PM snSS9n[R RRWSS9n[R"XrSS9up[R RRX5n Uc:[R"U R5SS9n [R"USS9n GOURupURSX--nUSSS2SS2SS4RXXU45RSX!5R W5n[R "U R5U-SS9[R "USS9- n USSS2SS2S4RXX45RSU5R U5n[R "UU-SS9[R "USS9- n [R "XR#S5-5nUU-$s snf)ax Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrdim) isinstancetupledevicetorchcattor functionalsoftmaxtopkone_hotmeanfloatshapeexpandreshapesum unsqueeze)rr top_kr!compute_device layer_gateconcatenated_gate_logitsrouting_weights_selected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthnum_hidden_layersexpert_attention_mask router_per_expert_attention_mask overall_losss c/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/mixtral/modular_mixtral.pyload_balancing_loss_funcrI6s+:*[%"@"@+u%%$Q..#(99^i-j^iPZmmN.K^i-jpq#r hh))112JPR1SO**_DA((%%--.>LK!JJ{'8'8':B"'O!C&4&:&:# 4::1=*B^_ 4AtT1 2 V&OKX Y WR , R  "IIk&7&7&9[TU]5 URUlURUl[ R"UR URSS9Ul[ R"URUR SS9Ul [ R"UR URSS9Ul [URUl gNF)bias)super__init__intermediate_sizeffn_dim hidden_size hidden_dimrLinearw1w2w3r hidden_actact_fnselfrM __class__s rHrR"MixtralBlockSparseTop2MLP.__init__s //  ,,))DOOT\\F))DLL$//F))DOOT\\FV../ cURURU55URU5-nURU5nU$N)r\rXrZrY)r^ hidden_statescurrent_hidden_statess rHforward!MixtralBlockSparseTop2MLP.forwards> $ DGGM,B CdggmF\ \ $(= >$$ra)r\rTrVrXrYrZ) __name__ __module__ __qualname____firstlineno__rrRrf__static_attributes__ __classcell__r_s@rHrKrKs 0} 0%%rarKcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)MixtralSparseMoeBlocka This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accommodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. c>[TU]5 URUlURUlUR UlURUl [R"URURSS9Ul [R"[UR5Vs/sHn[U5PM sn5UlUR"Ulgs snfrO)rQrRrUrVrSrTnum_local_expertsr num_experts_per_tokr8rrWgate ModuleListrangerKexpertsrouter_jitter_noise jitter_noise)r^rMr=r_s rHrRMixtralSparseMoeBlock.__init__s  ,,// !33// IIdoot/?/?eL }}QVW[WgWgQh%iQhA&?&GQh%ij #66&js*Crdr"cURup#nUR(aSURS:aCU[R"U5R SUR- SUR-5-nUR SU5nURU5n[R"US[RS9n[R"X`RSS9upgXfRSSS9-nURUR5n[R "X#-U4URUR"S 9n[R$R&R)XpR*S 9R-S SS5n [R."U RS S9S5R15n U Hn UR2U n [R4"XR7S55upUS U4R9SU5nU "U5XnU S 4-nUR;SUURUR55 M UR9X#U5nX4$)rg?r&r)r%dtyper$T)r%keepdim)r~r)) num_classesr)r&N)r3trainingrzr* empty_likeuniform_viewruFr.r2r/r8r6r,r~zerosr)rr-r0r permutegreaternonzerorxwheresqueezer5 index_add_)r^rdrBrCrV router_logitsr<r>final_hidden_statesr? expert_hitted expert_idx expert_layeridxtop_x current_stateres rHrfMixtralSparseMoeBlock.forwards2?2E2E/ Z ==T..2 U--m<EEcDL]L]F]_beievev_vw wM%**2z:  -0 ))Mq L,1JJ XZ,[)..2t.DD),,]-@-@A#kk  ): 6m>Q>QZgZnZn  hh))112BP`P`1aiijkmnpqr  koo(o&CQGOOQ 'J<< 3L[%<%D%DQ%GHJC*$+6>>r:NM$0$?/Y\^bRbBc$c ! * *1e5J5M5MmNaNa5b c(299*Wab"11ra)rxrTrurVrzr r8) rhrirjrk__doc__rRr*Tensorrfrlrmrns@rHrprps-  7%2U\\%2ell%2%2rarpc\rSrSrSrg)MixtralRMSNormNrhrirjrkrlrrarHrrrarc\rSrSrSrg)MixtralAttentionrNrrrarHrrrrarc8^\rSrSrS\S\4U4SjjrSS\RS\ \R\R4S\ \RS\ \RS \ \ \RS \ \RS \ \ S \R4S jjrSrU=r$)MixtralDecoderLayerrM layer_idxc>[TU]5 URUl[X5Ul[ U5Ul[URURS9Ul [URURS9Ul g)N)eps) rQrRrUr self_attnrpblock_sparse_moer rms_norm_epsinput_layernormpost_attention_layernorm)r^rMrr_s rHrRMixtralDecoderLayer.__init__sk !--)&< 5f =-f.@.@fFYFYZ(6v7I7IvObOb(c%rardposition_embeddingsr! position_idspast_key_valuecache_positionkwargsr"c UnURU5nUR"SUUUUUUS.UD6upX-nUnURU5nURU5upX-nU$)N)rdrr!rrrr)rrrr) r^rdrr!rrrrresidualr=s rHrfMixtralDecoderLayer.forwards!,,];  >> ' 3)%))    !0 !55mD 00?  0 ra)rrUrrr)NNNN)rhrirjrkrintrRr*rr(r LongTensorrr FloatTensorrfrlrmrns@rHrrsd}dd26378<59 || #5<<#=> !.  u//0  !u||!45 !!1!12 +,      rarc\rSrSrSrg)MixtralRotaryEmbeddingirNrrrarHrrrrarc.\rSrSrSr\"\SS9\\S.r Sr g)MixtralPreTrainedModeliFr)index)rrd attentionsrN) rhrirjrk_can_compile_fullgraphrrprr_can_record_outputsrlrrarHrrs!"'(=QG,&rarc\rSrSrSS\\R S\\RS\\R S\\S\\RS\\ S \\R S \ \ S \ 4S jjrS rg) MixtralModeli N input_idsr!rpast_key_values inputs_embeds use_cacherrr"c zUSLUSL- (a [S5eU(a Uc [5nUcURU5nUcDUbUR5OSn [R "XUR S-URS9nUcURS5nURRc[O[n U "URUUUUUS9n Un URX5n URSURRHnU"U 4U U UUUUS.UD6n M UR!U 5n [#U US9$)Nz:You must specify exactly one of input_ids or inputs_embedsrr)r))rM input_embedsr!rrr)rr!rrrr)last_hidden_stater) ValueErrorr embed_tokensget_seq_lengthr*aranger3r)r7rMsliding_windowr r rotary_emblayersrDnormr)r^rr!rrrrrrpast_seen_tokens mask_function causal_maskrdr decoder_layers rHrfMixtralModel.forward!sh  -t"; <YZ Z 0*nO   --i8M  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L.2kk.H.H.P*Vw #;;&))+%  & #oomJ![[)H4;;+H+HIM) $7*).#-  MJ -0 %++  rar)NNNNNNN)rhrirjrkrr*rrrrboolrrrrfrlrrarHrr s151537+/59$(59< E,,-< !.< u//0 < "% <   1 12 < D>< !!1!12< +,<  < < rarcR^\rSrSrS/rU4SjrSS\\RS\\RS\\RS\\ S\\RS \\RS \\ S \\ S \\RS \ \\R4S\\S\4SjjrSrU=r$)MixtralForCausalLMi`zlm_head.weightc>[TU]U5 [U5UlURUlUR UlURUlgrc)rQrRrmodelrouter_aux_loss_coefrsr rtr]s rHrRMixtralForCausalLM.__init__csF  !&) $*$?$?!!33#)#=#= rarr!rrrlabelsroutput_router_logitsrlogits_to_keeprr"c ~UbUOURRnUR"SUUUUUUUU S.U D6n U Rn [ U [ 5(a [ U *S5OU nURU SS2USS245nSnUbUR"XUR40U D6nSnU(aZ[U RURURU5nUb+UURURUR 5-- n[#UUUU R$U R&U R(U RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MixtralForCausalLM >>> model = MixtralForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rr!rrrrrr)lossaux_losslogitsrrdrrr)rMrrrr'rslicelm_head loss_function vocab_sizerIrr rtrr,r)r rrdr)r^rr!rrrrrrrrroutputsrd slice_indicesrrrs rHrfMixtralForCausalLM.forwardjsUL%9$D $++JjJj  +/** + )%+'!5) +  +  11 8B>SV8W8W~ot4]k mA}a,?@A  %%fdooPPD /%%  (( H !11HKK 4LLL(#33!//))!//  ra)rr rtr) NNNNNNNNNr)rhrirjrk_tied_weights_keysrRrr*rrrrrrrrrr rfrlrmrns@rHrr`s*+>151537+/59-1$(/35934R E,,-R !.R u//0 R "% R   1 12 R ))*R D>R 'tnR !!1!12R c5<</0R +,R  #R R rarc\rSrSrSrg) MixtralForSequenceClassificationirNrrrarHrrrrarc\rSrSrSrg)MixtralForTokenClassificationirNrrrarHrrrrarc\rSrSrSrg)MixtralForQuestionAnsweringirNrrrarHrrrrar)rrrrrr)NrN)?rtypingrrr*torch.nn.functionalrr-rtorch.utils.checkpoint activationsr cache_utilsrr masking_utilsr r modeling_layersr modeling_outputsr rprocessing_utilsrutilsrr utils.genericrmistral.modeling_mistralrrrrrrrrrconfiguration_mixtralr get_loggerrhloggerrr(rrIModulerKrprrrrrrrrrr__all__rrarHrsw(" !.R9Q&0+   1   H % "& -1 O&u||U5<<%8$>?O&#O&U\\* O&  5<<  O&d% %$@2BII@2F ^  ' +4+\ 3 3= <= @\ +\ ~ 'G  $A  "=  ra