/S:Qr?g)=)CallableOptionalUnionN)nn)check_model_inputs)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GenericForQuestionAnswering GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple) MistralConfigc.^\rSrSrU4SjrSrSrU=r$) MistralMLP"c>[TU]5 XlURUlURUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFbias) super__init__config hidden_sizeintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnselfr* __class__s d/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/mistral/modeling_mistral.pyr)MistralMLP.__init__#s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ cURURURU55URU5-5nU$N)r0r2r.r/)r4xr0s r6forwardMistralMLP.forward-s6NN4;;t~~a/@#ADLLQRO#ST r8)r2r*r0r.r+r,r/)__name__ __module__ __qualname____firstlineno__r)r<__static_attributes__ __classcell__r5s@r6r"r""s0r8r"cUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)r;x1x2s r6 rotate_halfrO2sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r8cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerO)qkcossin position_ids unsqueeze_dimq_embedk_embeds r6apply_rotary_pos_embrZ9sS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r8 hidden_statesn_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rJexpandreshape)r[r\batchnum_key_value_headsslenhead_dims r6 repeat_kvreTs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr8modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)NrGrrF)rIdtype)ptrainingr)renum_key_value_groupsrKmatmul transposerJr functionalsoftmaxfloat32torprlrr contiguous) rfrgrhrirjrkrlrm key_states value_states attn_weights causal_mask attn_outputs r6eager_attention_forwardr`s3 ; ;[TU]5 XlX l[ USS5=(d UR UR -UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-SS9Ul[R"UR URUR-SS9Ul[R"UR URUR-SS9Ul[R"UR UR-UR SS9Ulg)NrdgTFr&)r(r)r*rgetattrr+num_attention_headsrdrbrsrkattention_dropout is_causalrr-q_projk_projv_projo_projr4r*rr5s r6r)MistralAttention.__init__}s.  " D9mV=O=OSYSmSm=m $*$>$>&B\B\$\!}}d* !'!9!9ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii : :T]] JFL^L^ejk r8r[position_embeddingsrjpast_key_valuecache_positionrmr]c `URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO URUR [#URSS5S.UD6unnUR$"/UQSP76R'5nUR)U5nUU4$) NrFrrG)rUrTreagersliding_window)rlrkr)rJrdrviewrurrrZupdaterrr*_attn_implementationrrrrrkrr`rzr)r4r[rrjrrrm input_shape hidden_shape query_statesr{r|rTrU cache_kwargsattention_interfacerr}s r6r<MistralAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<'r?r@rA__doc__r intr)rKTensortuplerr LongTensorrrr<rBrCrDs@r6rrzsG l} l l&+/59 *)||*)#5<<#=>*)!. *) ! *) !!1!12 *)-.*) u||Xell3XeELL>Q5RR S*)*)r8rRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$)MistralRMSNormc>[TU]5 [R"[R "U55UlX lg)z- MistralRMSNorm is equivalent to T5LayerNorm N)r(r)r ParameterrKonesweightvariance_epsilon)r4r+epsr5s r6r)MistralRMSNorm.__init__s/ ll5::k#:; #r8cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NrGrFT)keepdim) rpryrKrxpowmeanrsqrtrr)r4r[ input_dtypevariances r6r<MistralRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r8c^[URR5SUR3$)Nz, eps=)rrrJrr4s r6 extra_reprMistralRMSNorm.extra_reprs* ))*+6$2G2G1HIIr8)rr)gư>) r>r?r@rAr)r<rrBrCrDs@r6rrs$;JJr8rc8^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \RS \ \ \R\R4S \\S \ \R4SjjrSrU=r$)MistralDecoderLayerr*rc >[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul g)N)r*rr) r(r)r+r self_attnr"mlpr rms_norm_epsinput_layernormpost_attention_layernormrs r6r)MistralDecoderLayer.__init__sj !--)Mf%-f.@.@fFYFYZ(6v7I7IvObOb(c%r8r[rjrVr use_cacherrrmr]c Un URU5nUR"SUUUUUUUS.UD6upX-nUn URU5nURU5nX-nU$)N)r[rjrVrrrr)rrrr) r4r[rjrVrrrrrmresidual_s r6r<MistralDecoderLayer.forwards!,,]; >>  ')%)) 3     !0 !55mD /  0 r8)r+rrrr)NNNFNN)r>r?r@rAr rr)rKrrrr boolrrrr<rBrCrDs@r6rrsd}dd2637*.$)59KO||!.u//0  !  D> !!1!12&eELL%,,,F&GH+, u|| r8rcR\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr Sr Sr\\S.rSrg ) MistralPreTrainedModelr*modelTrpast_key_values)r[ attentionsrN)r>r?r@rAr __annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsrBrr8r6rrsQ &*#./#4"5N!"&,&r8rcl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)MistralRotaryEmbeddingi r*c>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r(r)hasattr isinstancerdictgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr*r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r4r*devicerr5s r6r)MistralRotaryEmbedding.__init__ s  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rr*rrrrrr:) r>r?r@rAr r)rKno_gradrr<rBrCrDs@r6rr s6/}//" ]]_ < [TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nr)r*F)r(r) pad_token_id padding_idx vocab_sizer Embeddingr+ embed_tokens ModuleListrangenum_hidden_layersrlayersrrnormr rotary_embgradient_checkpointing post_initrs r6r)MistralModel.__init__/s  !.. ++LL):):F input_idsrjrVr inputs_embedsrrrmr]c USLUSL- (a [S5eUcURU5nU(a Uc [5nUcDUbUR5OSn [R "XUR S-URS9nUcURS5nURRc[O[n U "URUUUUUS9n Un URX5n URSURRHnU"U 4U UUUUU S.UD6n M UR!U 5n [#U U(aUS9$SS9$)Nz:You must specify exactly one of input_ids or inputs_embedsrr)r)r* input_embedsrjrrrV)rjrVrrrr)last_hidden_stater) ValueErrorr r get_seq_lengthrKarangerJrrQr*rrrrrr rr)r4rrjrVrrrrrmpast_seen_tokens mask_functionr~r[r decoder_layers r6r<MistralModel.forward?su  -t"; <YZ Z   --i8M 0*nO  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L.2kk.H.H.P*Vw #;;&))+%  & "oomJ![[)H4;;+H+HIM) *).#-$7  MJ -0 &+/8O  >B  r8)r rrrrrr)NNNNNNN)r>r?r@rAr r)rrrrKrrr FloatTensorrrrrr<rBrCrDs@r6rr-s} 151537+/59$(599 E,,-9 !.9 u//0 9 "% 9   1 12 9 D>9 !!1!129 +,9  !9 9 r8rc~^\rSrSrS/rSS0rSS/S/40rU4SjrSrS r \ \ SS \ \ RS \ \ RS \ \ RS \ \S\ \ R"S\ \ RS\ \S\ \ RS\\\ R4S\\S\4Sjj55rSrU=r$)MistralForCausalLMi}zlm_head.weightlm_head colwise_repr[logitsc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr%) r(r)rrrrr-r+r#rr3s r6r)MistralForCausalLM.__init__sU  !&)  ++yy!3!3V5F5FUS  r8cXlgr:r)r4decoders r6 set_decoderMistralForCausalLM.set_decoders r8cUR$r:r)rs r6 get_decoderMistralForCausalLM.get_decoders zzr8rrjrVrrlabelsrrlogits_to_keeprmr]c ~UR"SUUUUUUUS.U D6n U Rn [U [5(a [ U *S5OU n UR U SS2U SS245nSnUb)UR "SXURRS.U D6n[UUU RU RU RS9$)a{ Example: ```python >>> from transformers import AutoTokenizer, MistralForCausalLM >>> model = MistralForCausalLM.from_pretrained("meta-mistral/Mistral-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-mistral/Mistral-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rrjrVrrrrN)r%r0r)lossr%rr[rr) rrrrslicer# loss_functionr*rrrr[r)r4rrjrVrrr0rrr1rmoutputsr[ slice_indicesr%r3s r6r<MistralForCausalLM.forwards@,0:: , )%+') ,  ,  11 8B>SV8W8W~ot4]k mA}a,?@A  %%pVt{{OeOepiopD%#33!//))   r8)r#rr) NNNNNNNNr)r>r?r@rA_tied_weights_keys_tp_plan_pp_planr)r+r.rrrrKrrr r rrrrrrr<rBrCrDs@r6r"r"}s:*+=)H_-z:;H151537+/59-1$(59348 E,,-8 !.8 u//0 8 "% 8   1 12 8 ))*8 D>8 !!1!128 c5<</08 +,8  8 8 r8r"c\rSrSrSrg)MistralForTokenClassificationirNr>r?r@rArBrr8r6r=r=r8r=c\rSrSrSrg) MistralForSequenceClassificationirNr>rr8r6rArAr?r8rAc\rSrSrSrg)MistralForQuestionAnsweringirNr>rr8r6rCrCsX[r8rC)r"rCrrrAr=)Nr)r)@typingrrrrKrtransformers.utils.genericr activationsr cache_utilsr r generationr integrationsr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrrrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrconfiguration_mistralr Moduler"rOrZrrrerrrrrrrrr"r=rArC__all__rr8r6rUs-, 9!.)7RB PKF&II0   (6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%'(%4;)ryy;)|Y'JRYYJ(J((4(V_$