/S:Qr?g)=)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GenericForQuestionAnswering GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)check_model_inputs) SmolLM3ConfigcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s d/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/smollm3/modeling_smollm3.py rotate_halfr-0sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer-)qkcossin position_ids unsqueeze_dimq_embedk_embeds r,apply_rotary_pos_embr97sS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r. hidden_statesn_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r&expandreshape)r:r;batchnum_key_value_headsslenhead_dims r, repeat_kvrDRs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr.modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nr#rr")r%dtype)ptrainingr)rDnum_key_value_groupsr'matmul transposer&r functionalsoftmaxfloat32torOrKrQ contiguous) rErFrGrHrIrJrKrL key_states value_states attn_weights causal_mask attn_outputs r,eager_attention_forwardr_^s3 ; ;[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-URS9Ul[R"UR URUR-URS9Ul[R"UR URUR-URS9Ul[R"UR UR-UR URS9UlUR(UUlUR,(a%UR.US:XaUR0UlgSUlg)NrCgTbiassliding_attention)super__init__rcrdgetattr hidden_sizenum_attention_headsrCrArRrJattention_dropout is_causalrLinearattention_biasq_projk_projv_projo_projno_rope_layersuse_ropeuse_sliding_window layer_typessliding_windowselfrcrd __class__s r,rjSmolLM3Attention.__init__{s  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  --i8 ((V-?-? -JNa-a  ! !  r.r:position_embeddingsrIpast_key_valuecache_positionrLr<c jURSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n UR(aUup[XX5upUb#SU0nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO UR UR"UR$S.UD6unnUR&"/UQSP76R)5nUR+U5nUU4$)Nr"rr#reager)rKrJrz)r&rCrrviewrTrsrtrwr9updaterdr_rc_attn_implementationrrQrnrJrzr?rYru)r|r:rrIrrrL input_shape hidden_shape query_statesrZr[r3r4 cache_kwargsattention_interfacer^r\s r,forwardSmolLM3Attention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST ==*HC';LVY'_ $L  %,n=L'5'<'*)!. *) ! *) !!1!12 *)-.*) u||Xell3XeELL>Q5RR S*)*)r.raRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$)SmolLM3RMSNormc>[TU]5 [R"[R "U55UlX lg)z- SmolLM3RMSNorm is equivalent to T5LayerNorm N)rirjr Parameterr'onesweightvariance_epsilon)r|rlepsr}s r,rjSmolLM3RMSNorm.__init__s/ ll5::k#:; #r.cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nr#r"T)keepdim) rOrXr'rWpowmeanrsqrtrr)r|r: input_dtypevariances r,rSmolLM3RMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r.c^[URR5SUR3$)Nz, eps=)rrr&rr|s r, extra_reprSmolLM3RMSNorm.extra_reprs* ))*+6$2G2G1HIIr.)rr)gư>) rrrrrjrrrrrs@r,rrs$;JJr.rc.^\rSrSrU4SjrSrSrU=r$) SmolLM3MLPc>[TU]5 XlURUlURUl[ R "URURURS9Ul[ R "URURURS9Ul [ R "URURURS9Ul [URUl g)Nrf)rirjrcrlintermediate_sizerrpmlp_bias gate_projup_proj down_projr hidden_actact_fnr|rcr}s r,rjSmolLM3MLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r.cURURURU55URU5-5nU$N)rrrr)r|r)rs r,rSmolLM3MLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r.)rrcrrrlrr)rrrrrjrrrrs@r,rrs0r.rc8^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \RS \ \ \R\R4S \\S \ \R4SjjrSrU=r$)SmolLM3DecoderLayerrcrdc4>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul URUUl g)N)rcrdr) rirjrlra self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormryattention_typer{s r,rjSmolLM3DecoderLayer.__init__s} !--)Mf%-f.@.@fFYFYZ(6v7I7IvObOb(c%$00;r.r:rIr5r use_cacherrrLr<c Un URU5nUR"SUUUUUUUS.UD6upX-nUn URU5nURU5nX-nU$)N)r:rIr5rrrr)rrrr) r|r:rIr5rrrrrLresidual_s r,rSmolLM3DecoderLayer.forwards!,,]; >>  ')%)) 3     !0 !55mD /  0 r.)rrlrrrr)NNNFNN)rrrrr rrjr'rrrr boolrrrrrrrs@r,rrs <} < <2637*.$)59KO||!.u//0  !  D> !!1!12&eELL%,,,F&GH+, u|| r.rcR\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr Sr Sr\\S.rSrg ) SmolLM3PreTrainedModelircmodelTrpast_key_values)r: attentionsrN)rrrrr __annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrra_can_record_outputsrrr.r,rrsQ &*#./#4"5N!"&,&r.rcl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)SmolLM3RotaryEmbeddingi,rcc>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultinv_freqF) persistent)rirjhasattr isinstancerdictgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrcr rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r|rcdevicerr}s r,rjSmolLM3RotaryEmbedding.__init__-s  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+r&rXrrrstrr'autocastrTr(r3rr4rO) r|r)r5inv_freq_expandedposition_ids_expandedrfreqsembr3r4s r,rSmolLM3RotaryEmbedding.forward>sR!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rrcrrrrrr) rrrrr rjr'no_gradrrrrrs@r,rr,s6/}//" ]]_ < [TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlSUR(R*;UlUR/5 gs snf)Nr)rcFrh)rirj pad_token_id padding_idx vocab_sizer Embeddingrl embed_tokens ModuleListrangenum_hidden_layersrlayersrrnormr rotary_embgradient_checkpointingrcryhas_sliding_layers post_initr{s r,rjSmolLM3Model.__init__Ps  !.. ++LL):):FB  r.)rrrrrr rr)NNNNNNN)rrrrr rjrrrr'rrr FloatTensorrrrrrrrrs@r,r r Ns}"151537+/59$(59E E,,-E !.E u//0 E "% E   1 12 E D>E !!1!12E +,E  !E E r.r c~^\rSrSrS/rSS0rSS/S/40rU4SjrSrS r \ \ SS \ \ RS \ \ RS \ \ RS \ \S\ \ R"S\ \ RS\ \S\ \ RS\\\ R4S\\S\4Sjj55rSrU=r$)SmolLM3ForCausalLMizlm_head.weightlm_head colwise_repr:logitsc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 g)NFrf) rirjr rrrrprlr,rrs r,rjSmolLM3ForCausalLM.__init__sU  !&)  ++yy!3!3V5F5FUS  r.cXlgrr)r|decoders r, set_decoderSmolLM3ForCausalLM.set_decoders r.cUR$rr2rs r, get_decoderSmolLM3ForCausalLM.get_decoders zzr.rrIr5rrlabelsrrlogits_to_keeprLr<c ~UR"SUUUUUUUS.U D6n U Rn [U [5(a [ U *S5OU n UR U SS2U SS245nSnUb)UR "SXURRS.U D6n[UUU RU RU RS9$)a{ Example: ```python >>> from transformers import AutoTokenizer, SmolLM3ForCausalLM >>> model = SmolLM3ForCausalLM.from_pretrained("meta-smollm3/SmolLM3-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-smollm3/SmolLM3-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rrIr5rrrrN)r.r9r)lossr.rr:rr) rr rrslicer, loss_functionrcrrrr:r)r|rrIr5rrr9rrr:rLoutputsr: slice_indicesr.r<s r,rSmolLM3ForCausalLM.forwards@,0:: , )%+') ,  ,  11 8B>SV8W8W~ot4]k mA}a,?@A  %%pVt{{OeOepiopD%#33!//))   r.)r,rr) NNNNNNNNr)rrrr_tied_weights_keys_tp_plan_pp_planrjr4r7rrrr'rrr r)rrrrrrrrrrs@r,r+r+s:*+=)H_-z:;H151537+/59-1$(59348 E,,-8 !.8 u//0 8 "% 8   1 12 8 ))*8 D>8 !!1!128 c5<</08 +,8  8 8 r.r+c\rSrSrSrg) SmolLM3ForSequenceClassificationirNrrrrrrr.r,rFrFr.rFc\rSrSrSrg)SmolLM3ForTokenClassificationirNrGrr.r,rJrJrHr.rJc\rSrSrSrSrg)SmolLM3ForQuestionAnsweringi transformerrN)rrrrrrrr.r,rLrLs%r.rL)rr r+rFrJrL)Nr)r)@typingrrrr'r activationsr cache_utilsr r generationr integrationsr masking_utilsr rmodeling_flash_attention_utilsrmodeling_layersrrrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrconfiguration_smollm3r r-r9rrrDModulerr_rarrrrrr r+rFrJrL__all__rr.r,r_s,-, !.)7RB PKF&II/0(6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%'(%4K)ryyK)\Y'JRYYJ(J(   +4+\_$