[TU]5 Xl[R"UR SUR -SS9Ul[R"UR UR SS9Ul[URUl g)Nr Fbias) super__init__configrLinear hidden_sizeintermediate_size gate_up_proj down_projr hidden_act activation_fn)selfr __class__s ]/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/phi3/modular_phi3.pyrPhi3MLP.__init__1sn  IIf&8&8!f>V>V:V]bc6#;#;V=O=OV[\#F$5$56 hidden_statesreturncURU5nURSSS9up2X RU5-nURU5$)Nr dim)r#chunkr&r$)r'r, up_statesgates r)forwardPhi3MLP.forward9sH%%m4 #//!/4 2 24 88 ~~i((r+)r&rr$r#) __name__ __module__ __qualname____firstlineno__rtorch FloatTensorr5__static_attributes__ __classcell__r(s@r)rr0s,7)U%6%6)5;L;L))r+rcNURU5nURU5nURSnUSSU24USUS24pUSSU24USUS24p[R"Xr-[ U5U--U/SS9n [R"X-[ U 5U--U /SS9n X4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r/.Nr0) unsqueezeshaper;catr) qkcossin position_ids unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds r)apply_rotary_pos_embrQBs( -- &C -- &C2Jc;J;&'3 +;)<6c;J;&'3 +;)<6ii%++e*[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URUlURS-Ul URUl SUl UR UR-SURUR---n[R"UR UR-UR SS9Ul[R"UR USS9Ulg)Nhead_dimgTr Fr)rrrrUgetattrr!num_attention_headsrWnum_key_value_headsnum_key_value_groupsscalingattention_dropout is_causalrr o_projqkv_proj)r'rrUop_sizer(s r)rPhi3Attention.__init__es  " F4F4F&JdJd4de $*$>$>&B\B\$\!#)#=#= }}d* !'!9!9,,t}}r?s@r)rSrSbsG Kz Khsm K K(+/59 0)||0)#5<<#=>0)!. 0) ! 0) !!1!12 0)-.0) u||Xell3XeELL>Q5RR S0)0)r+rSct^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \RS \ \ \R\R4S \\S \ \R \ \ \R \R 444SjjrSrU=r$)Phi3DecoderLayerrrUc>[TU]X5 Xl[XS9Ul[ U5Ul[R"UR5Ul [R"UR5Ul g)N)rrU) rrrrS self_attnrmlprDropout resid_pdropresid_attn_dropoutresid_mlp_dropout)r'rrUr(s r)rPhi3DecoderLayer.__init__sZ + &fJ6?"$**V-?-?"@!#F,>,>!?r+r,rdrHre use_cacherfrcrgr-c Un URU5nUR"SUUUUUUUS.UD6upXRU5-nUn URU5nUR U5nXR U5-nU$)N)r,rdrHrerrfrc)input_layernormrrpost_attention_layernormrr) r'r,rdrHrerrfrcrgresidualself_attn_weightss r)r5Phi3DecoderLayer.forwards!,,]; +/>> , ')%)) 3 ,  , ( !#:#:=#II  55mD /  #9#9-#HH r+)rrrrr)NNNFNN)r7r8r9r:rrrr;rrrrboolrr r r<r5r=r>r?s@r)rrs@z@c@2637*.$)59KO||!.u//0  !  D> !!1!12&eELL%,,,F&GH-. u  (51B1BEDUDU1U+V"WW Xr+rc\rSrSrSrSrg)Phi3PreTrainedModelz0.0.5rN)r7r8r9r:_versionr=rr+r)rrsHr+rc,\rSrSrSSjrSrg)Phi3ForCausalLMNc $U(aeURR(aJURSURRS-:a USn XRR::aSn[ 5R "SUUUUUUUUS.U D6n U $)Nrr) input_idspast_key_valuesrd inputs_embedsrfrHrlogits_to_keepr)r rope_scalingrB original_max_position_embeddingsrprepare_inputs_for_generation) r'rrrdrrfrHrrrg past_length model_inputss r)r-Phi3ForCausalLM.prepare_inputs_for_generations$  (("dkk&R&RUV&VV(+KkkJJJ"&*,JJ  +)')%)    r+r)NNNNNTN)r7r8r9r:rr=rr+r)rrs%r+rc\rSrSrSrg)Phi3ForSequenceClassificationrNr7r8r9r:r=rr+r)rrr+rc\rSrSrSrg)Phi3ForTokenClassificationirNrrr+r)rrrr+r)r Phi3Modelrrr)Nr),rtypingrrr;torch.utils.checkpointr activationsr cache_utilsrmodeling_flash_attention_utilsr modeling_utilsr processing_utilsr utilsr mistral.modeling_mistralrrrrrrrconfiguration_phi3r get_loggerr7logger_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCModulerrQrSrrrrr__all__rr+r)rs % ! B5&+   H %8)bii)$@B)BIIB)J'*'T0&(*=&R $D  !>  r+