"S(S)\ Rn5r?"S*S+\ Rn5r@"S,S-\ Rn5rA"S.S/\ Rn5rB"S0S1\ Rn5rC"S2S3\ Rn5rD\#"S4S5\55rE\#"S6S7\E55rF"S8S9\E\5rG\#"S:S;9"S<S=\E55rH/S>QrIg)@zPyTorch Zamba model.N)AnyCallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNc8^\rSrSrSU4SjjrSrSrSrU=r$) ZambaRMSNorm?c>[TU]5 [R"[R "U55UlX lg)z+ ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s `/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/zamba/modeling_zamba.pyr)ZambaRMSNorm.__init__@s/ ll5::k#:; #cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NT)keepdim) dtypetor+float32powmeanrsqrtr.r-)r/ hidden_states input_dtypevariances r3forwardZambaRMSNorm.forwardHsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r5c^[URR5SUR3$)Nz, eps=)tupler-shaper.r/s r3 extra_reprZambaRMSNorm.extra_reprOs* ))*+6$2G2G1HIIr5)r.r-)gư>) __name__ __module__ __qualname____firstlineno__r)rCrI__static_attributes__ __classcell__r2s@r3r%r%?s$;JJr5r%r@n_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rGexpandreshape)r@rRbatchnum_key_value_headsslenhead_dims r3 repeat_kvr[Ts_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr5c \rSrSrSrSrSrSr\RS4Sjr Sr S\ S\ \R\R44S jrSS \RS \RS\ S \\\\4S\ \R\R44 S jjrS\R*4SjrSS\\ S\ 4SjjrS\ \ \R\ \R44Sjr\SS\\ \ \R4SS4Sjj5rSrg)ZambaHybridDynamicCache`a| A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. NFc $X0lSUlURUlSUlURUR -UlURUlURUl URUl /Ul /Ul /Ul0Ul0Ul0Ul[%UR&5HnU=R[(R*"X R URXCS9/- sl UURUR UR-UR4nU=R[(R*"XdUS9/- sl URUS:XdMURR-U5 M [%UR&5Vs/sHn[(R."//U-US9PM snUl[%UR&5Vs/sHn[(R."//U-US9PM snUlgs snfs snf)NFdevicer:hybridra)r:is_compileablelayers_block_typehas_previous_state mamba_expandr0intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr+zerosappendtensor key_cache value_cache)r/config batch_sizer:rai cache_shape_s r3r) ZambaHybridDynamicCache.__init__rs #!'!9!9"'!'!4!4v7I7I!I$22 & 3 3#11"$  v//0A    J(>(>@U@U^dr!  ""&&$*<*<<## K OO Ke TU UO%%a(H4''..q11SXX^XpXpRqrRqQ%,,tj'8HRqrTYZ`ZrZrTstTsqELL" ):6JTststs "H"H c,[UR5$N)lenryrHs r3__len__ZambaHybridDynamicCache.__len__s4>>""r5 layer_idxrSc>URUURU4$r)ryrzr/rs r3 __getitem__#ZambaHybridDynamicCache.__getitem__s!~~i($*:*:9*EEEr5 key_states value_states cache_kwargsc|URURSS:XaXRU'X RU'Ob[R"URUU/SS9URU'[R"URUU/SS9URU'URUURU4$)Nr8rr7dim)ryrGrzr+cat)r/rrrrs r3updateZambaHybridDynamicCache.updates >>) $ * *2 .! 3(2NN9 %*6  Y '(- 4>>)3Lj2Y_`(aDNN9 %*/))T5E5Ei5PR^4_ef*gD  Y '~~i($*:*:9*EEEr5beam_idxc[[UR55GHTnURURnURUR SUR U55URU'UR URnUR UR SUR U55UR U'URURnURUR SUR U55URU'URURnURUR SUR U55URU'GMW g)zDReorders the cache for beam search, given the selected beam indices.rN) rtrryra index_selectr;rzrnro)r/rrras r3 reorder_cache%ZambaHybridDynamicCache.reorder_caches=s4>>23I^^I.55F(,y(A(N(NqRZR]R]^dRe(fDNN9 %%%i077F*.*:*:9*E*R*RSTV^VaVabhVi*jD  Y '%%i077F*.*:*:9*E*R*RSTV^VaVabhVi*jD  Y '__Y/66F)-)C)P)PQRT\T_T_`fTg)hDOOI &4r5cXR;aURSOUn[UR5U::agURURS$)zYReturns the sequence length of the cached states. A layer index can be optionally passed.r)rprryrGrs r3get_seq_length&ZambaHybridDynamicCache.get_seq_lengthsP3[TU]5 XlX lURUlUR UlURUR-Ul URUl UR S- S-Ul SUl URUl [R"URURUR -SS9Ul[R"URURUR -SS9Ul[R"URURUR -SS9Ul[R"URUR -UR&SS9Ulg)Nr7TFbias)r(r)r{rattention_hidden_sizeattention_head_dimrZnum_attention_headsrXrmax_position_embeddingsr is_causalattention_dropoutrLinearq_projk_projv_projr0o_projr/r{rr2s r3r)ZambaAttention.__init__s5  "%+%A%A"11 $*$>$>&B\B\$\!'-'E'E$ )d2 !'!9!9ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii : :T]] JFL^L^ejk r5r@rpast_key_valuerrSc URSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5n UR U5RU5R SS5n UbURXU5up[n URRS:wa[URRn U "UUU U U4UR(dSO URURS.UD6upU R"/UQSP76R!5n UR#U 5n X4$)Nr8rr7eager)rr)rGrZrviewrrrrrr{_attn_implementationrrrrrVrr)r/r@rrrr input_shape hidden_shape query_statesrrattention_interfacerrs r3rCZambaAttention.forwardsk$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  %'5'<'Q5RR S#)#)r5rc^\rSrSrSrS\4U4SjjrS S\RS\ 4Sjjr S S\ 4Sjjr S S\ 4S jjr S r U=r$) ZambaMambaMixeri)u! Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. r{c >[TU]5 XlX lURUlUR UlURUlURUR-Ul URUl URUl URUR-UlURUlUR"Ul[&R("URURUR URURURS- S9UlUR,Ul[0UR,UlUR4Ul[&R8"URURS-UR$S9Ul[&R<"[>R@"URURUR S--UR55Ul![&R<"[>R@"URURUR5S- S-URS-- 5Ul"[&R<"[>R@"URUR55Ul#[>RH"SUR S-[>RJS9SSS24nURMURS5RO5n[&R<"[>RP"U5RSURURS55Ul*[&R<"[>RV"URUR55Ul,[&R8"URURUR$S9Ul-[\(d[^RaS5 gg) Nr) in_channels out_channelsr kernel_sizegroupspaddingr7rg?r:r8apThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r(r)r{rr0rirjrkrlrgrh mamba_dt_ranktime_step_rankrmmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrin_projr*r+rv x_proj_weightdt_proj_weight dt_proj_biasaranger<rUrlogrVA_logr,Dout_projis_fast_path_availablelogger warning_once)r/r{rAr2s r3r)ZambaMambaMixer.__init__6s   "!--$22 & 3 3!'!4!4v7I7I!I$22#11"448J8JJ#33.. ii..//##--))))A-   !11&112 & 8 8yy!1!143I3IA3MTXTaTab  \\ KK""##d&9&9A&==##  !ll [[++T-@-@$BUBU VY\ \ !!3& '  LLT5G5GI\I\)]^ LLD//!35== I$PQ' R HHT++R 0 ; ; =\\%))A,"6"6t7I7I4K^K^`b"cd ejj););T=P=PQR $"8"8$:J:JQUQ^Q^_ %%   ^ &r5r@ cache_paramsc URupEnUSL=(a UR=(a US:HnURU5RSS5nUR USSU5R SSS9upUR S5R5nU R S5n U RX@RSU5RSS5n URRR URRRS5URRRS55n U(ae[UR S5URURU URR UR"5nUR%S5nOUb1[&R("US:H5(dXR%S5-nUbc[*R,R/XR0URS- S45n URURR3U 5 [5XURR UR"S9nUb1[&R("US:H5(dXR%S5-nURSURUR6U5RSS5nUR8SS2SSS2SS24U-RSS5n [&R:"XR<UR>UR>/SS9upnUR@SS2S4U RSS5-n[&RB"URDRG55*nURHbURHRG5OSn[&RJ"USU4URLURNS9nU(a[QUR5Hn[SURTURSS2U4UUS S4UUS S4UUUUSS2S4UUSS2S4URVUU US S4UUS S 9 R%S5n[&RX"UU4SS9nM GO<[&RJ"USUR6UR>4URLURNS9n[QUR5Hn[[UUUUUUUURSS5UURSS5URVURG5U UUUS S S 9 unn[&RX"UU4SS9R5n[&RX"UUR%S54SS9nM Ub+Ub(URTURR3U5 UR]URSS55nU$) Nrr7r8rr)rrr`.T) dt_softplus)delta_softplusreturn_last_state)/rGrfrrrchunksqueezerrVrmrr-sizer"rnrrr unsqueezer+allrrpadrlcopy_r!rrsplitrrjrexprfloatremptyrar:rtr rorrrr)r/r@r rr|seq_lenruse_precomputed_statesprojected_statesgate conv_weightsrnssm_parameters time_stepBCdiscrete_time_stepr time_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r3cuda_kernels_forward$ZambaMambaMixer.cuda_kernels_forwardssn"/!4!4 Q!-T!9!nl>]>]!nbimnbn << 6@@AF.33JAwOUUVW]^U_ %--a0;;= ||A||J(:(:BHRRSTVWX{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)((8    M*33B7M)%))Na>{K,]$++JZJZgkgvgvwM)%))NaRAXRBU RS- S45n XR&UR'UR7UR/U 5SSU245n UbH[(R:"US:H5(d*XSS2U RS*S24R9S5-n O[(RD"X@RURFURH4U R"US9n Ub1[(R:"US:H5(dXR9S5-n UR7UR/U 5SSU245n Ub1[(R:"US:H5(dXR9S5-n U RSURURFU5RSS5n URJSS2SSS2SS24U -RSS 5n[(RL"XRNURHURH/SS9unnnURPSS2S4URSS 5-URRSS2SSS2S4-n[<R>RUU5n[(RV"URXR[55*n[(RV"USS2SSS2SSS24USS2SS2SS2SS2S4-5nUSS2SS2SS2SS2S4USS2SS2SSS2SS24R[5-nUU SS2SS2SS2SS2S4R[5-n/n[]U5HnUSS2SS2SS2USS24RSS5U -USS2SS2SS2USS24RSS5-n [(R^"U RSS5R!U5USS2SS2USS24R9S55nURaUSS2SS2SS2S45 M [(Rb"USS9nUXRdSS2SSS2S4--nUUR7U 5-nU (aXRUR'URgURSS5RUSU5RSS55nU$) Nrr7r8rr)shiftsdims.r`r)4rGr:rrrrrrrVrm isinstancer]rorrcloner;rarfrnr+rollsumrr-rrrrrrrrrlrvrrjrrrrrsoftplusrrrrtrrwstackrr)r/ input_statesr rr|rrr:rr@r use_cacher) conv_stater r!r"r#r$r  discrete_A discrete_BdeltaB_ur&r} scan_outputr+s r3 slow_forwardZambaMambaMixer.slow_forwards!-!3!3 Q""<< 5??1E.33JAwOUUVW]^U_ %--a0;;= ||A||J(:(:BHRRSTVWX|-DE 00@FFqIZW}}(33DNNCIIK (33DNNC ! ]%9%9:I///qL ,,T^^<BB1ES)55dnnE "ZZ 2BG '4Q1W'= 1a8$;E((8 % *{{7I7I!QPQ'7R*RXZ [ %%![[%5%55M $ 7 : :5 A K KB O !-eiiRS@S6T6T$11}GZGZ[]G^F^F`C`4a4k4klm4n$nM]]..}?T?TWdWjWjkmWn?npq>rs ;E((8 $])CC'M)R S !-eiiRS@S6T6T$11}GZGZ[]G^F^F`C`4a4k4klm4n$nM //1D1DdFYFYZ$++I )%))NaN ? E E GG wA"1aAq=1;;AqAIMPXYZ\]_`bcefYfPgPqPqrsuvPwwI,,y':':1a'@'C'CE'JAaQRTUWXjMLcLcdfLghK    Aq!QJ 7 8 kk,B7 !]VVAtQ=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrratype ValueErrorr,r>)r/r@r rs r3rCZambaMambaMixer.forward1sl  ))V4;M;M;T;T;Y;Y-Y i ,,]Yg,h h  ^ \\r5)rrrrr{rrlrrr0rrhrrrmrrjrrrrrr#)rKrLrMrNrrr)r+rr]r,r>rCrOrPrQs@r3rr)s^ ;{;|im_%"\\_%9P_%B[%7N[%z ]3J ] ]r5rc.^\rSrSrU4SjrSrSrU=r$)ZambaMLPi>c>[TU]5 XlURUlURUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFr) r(r)r{r0rhrr gate_projup_proj down_projr hidden_actact_fnr/r{r2s r3r)ZambaMLP.__init__?s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r5cURURURU55URU5-5nU$r)rKrMrIrJ)r/xrKs r3rCZambaMLP.forwardIs6NN4;;t~~a/@#ADLLQRO#ST r5)rMr{rKrIr0rhrJ)rKrLrMrNr)rCrOrPrQs@r3rFrF>s0r5rFc(^\rSrSrSS\S\\4U4SjjjrSS\RS\RS\S\\RS\\ S \\ S \\ S \ \ S \\R\\\R\R444S jjrSrU=r$)ZambaAttentionDecoderLayeriNr{rc>[TU]5 [X5Ul[ U5Ul[ URURS9Ul [ URURS9Ul g)Nr1) r(r)r self_attnrF feed_forwardr%r rms_norm_epsinput_layernormr0pre_ff_layernormrs r3r)#ZambaAttentionDecoderLayer.__init__Os] ':$V,+F,H,HfNaNab ,V-?-?VEXEX Yr5r@original_hidden_statesrroutput_attentionsr8rrSc [R"X/SS9nURU5nUR"SUUUUUUS.UD6upUR U5nUR U5nU4n U(aX4- n U $)ai Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r8r)r@rrrr^r8r)r+ concatenaterZrWr[rX) r/r@r]rrrr^r8rself_attn_weightsoutputss r3rC"ZambaAttentionDecoderLayer.forwardWs>))=*QWYZ ,,]; +/>>, '))/ , , ( --m< ))-8  "  + +Gr5)rXrZr[rWr)NNFF)rKrLrMrNrrrr)r+rr]boolrrrFrrCrOrPrQs@r3rTrTNsZ{Zx}ZZ26<@,1$)3||3!& 3 3 !. 3 !!89 3$D>3D>3-.3 u  (51B1BEDUDU1U+V"WW X33r5rTc^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \S\ \RS\ \RS \ \ S \ \ S \ \ S \ \RS \ \RS\ \R\ \ \R\R444SjjrSrU=r$)ZambaMambaDecoderLayerir{rc>[TU]5 [XS9Ul[ UR UR S9UlX lg)N)r{rrV) r(r)rmambar%r0rYrZrrs r3r)ZambaMambaDecoderLayer.__init__s: $FH +F,>,>FDWDWX"r5r@r]rrrr^r8cache_positiontransformer_hidden_statesrSc  Un U bX-OUnURU5nURUUUS9nSn X-nU4nU(aX4- nU(aX4- nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r@r r)rZrh)r/r@r]rrrrr^r8rjrkrresidualrarbs r3rCZambaMambaDecoderLayer.forwards<! :S9^M 5dq ,,];  '')# !!0  "  + +G  ( (Gr5)rZrrh) NNNNNFFNN)rKrLrMrNrrr)r+rrr]rdrrFrrCrOrPrQs@r3rfrfs#{#s#:>#'15.2<@,1$)59<@:||:!) 6:C= : !. : ell+ :!!89:$D>:D>:!!1!12:$,ELL#9: u  (51B1BEDUDU1U+V"WW X::r5rfc|^\rSrSrS\S\R S\4U4SjjrSS\ RS\ \ RS\ \ S \ \ RS \ \ RS \ \ S \ \S \ \S\ \ RS\\ R"\ \\ R"\ R"444SjjrSrU=r$)ZambaHybridLayeri shared_transflinearrhcF>[TU]5 XlX lX0lgr)r(r)rqrr mamba_decoder)r/rqrrrhr2s r3r)ZambaHybridLayer.__init__s * "r5r@r]rrrrr^r8rjrSc URUUUUUUUU S9n U Sn U(aU Sn URU 5n URUU UUUUU S9n U(a U SW 4U SS-n U $)a\ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )r]rrrr^r8rjrr)rkrrr^r8rjr7N)rqrrrt) r/r@r]rrrrr^r8rj layer_outputsrkras r3rCZambaHybridLayer.forwards>** #9&)/)+  %2!$4!  -a 0 $(KK0I$J!** &?))/)+  *1-/@AMRSRTDUUMr5)rrrtrq)NNNNNFFN)rKrLrMrNrTrrrfr)r+rrrr]rdrrFrrCrOrPrQs@r3rprps#&@#"))#\r#:>#'15.2<@,1$)59>||>!) 6>C= > !. > ell+ >!!89>$D>>D>>!!1!12> u  (51B1BEDUDU1U+V"WW X>>r5rpcF\rSrSr%\\S'SrSrSS/rSr Sr Sr Sr S r S rg ) ZambaPreTrainedModelir{modelTrTrfrFctURRn[U[R[R 45(aWUR RRSUS9 URb%URRR5 gg[U[R5(adUR RRSUS9 URb2UR RURR5 gg[U[5(a&UR RRS5 g[U[5(GaUR RRSUS9 URR"S-n[R$R'UR(U*U5 URR*URR,-URR.-n[0R2"[0R4"URR.U5[6R8"URR:5[6R8"URR<5- -[6R8"URR<5-5R?URR@S9nU[0R8"[0RB"U*5*5-nURDRRGU5 [0RH"SURJS-[0RLS9SSS24nUROURPS5RS5nURTRRG[0R8"U5RWUR.URXS55 URZRRS5 gg) Nr)r>stdg?r)minrrr8).r{initializer_ranger1rrrr-datanormal_rzero_ Embedding padding_idxr%fill_rrrinituniform_rrgr0rmr+rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrrjr<rUrhrrrVrr)r/rr} dt_init_stdrdtinv_dtr s r3 _init_weights"ZambaPreTrainedModel._init_weights%skk++ fryy"))4 5 5 MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - MM   $ $S )  0 0  % % - -3C - @++33T9K GG  V22[L+ N![[55 8O8OOSWS^S^SlSllN 4;;44nE88DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F    $ $ * *6 2 Q 5 5 9OPTVWPWXA1126AACA LL   # #EIIaL$8$89M9MvOdOdfh$i j HHMM   $%1r5rN)rKrLrMrNr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulrrOrr5r3rzrzs; &*#57OP"3 NL%r5rzc<^\rSrSrSrS\4U4Sjjr\SS\\ RS\\ RS\\ RS\\ S \\ RS \\S \\S \\S \\S\\ RS\\\44Sjj5rSrSrU=r$) ZambaModeliFz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig r{c >[T U]U5 URUlURUl[ R "URURUR5Ul[U5n/n/nURUl [UR5HnURUS:XaUR[XS95 M0URUS:XdMEUR[ R"UR RUR RSS95 UR[XS95 M [#U5n[#U5n/n/Ul['UR5HupxUS:XaaSUS3n /SQn /UR$QU V s/sHoU -PM sn QUlUR[)U[+U5[+U555 MlUR[+U55 M [ R,"U5UlUR0Ul[3URUR4S 9UlSUlUR;5 gs sn f) Nrh)rrbFrzlayers..) z%shared_transf.self_attn.q_proj.weightz%shared_transf.self_attn.k_proj.weightz%shared_transf.self_attn.v_proj.weightz%shared_transf.self_attn.o_proj.weightz+shared_transf.feed_forward.gate_proj.weightz)shared_transf.feed_forward.up_proj.weightz+shared_transf.feed_forward.down_proj.weightz$shared_transf.input_layernorm.weightz%shared_transf.pre_ff_layernorm.weightrV)r(r) pad_token_idr vocab_sizerrr0 embed_tokensrTrertrurwrfrr{iter_tied_weights_keys enumeraterpnext ModuleListlayersrr%rYfinal_layernormgradient_checkpointing post_init) r/r{block mamba_layers linear_layersr}rlayer_id layer_type prefix_name tied_keysrr2s r3r)ZambaModel.__init__Os  !.. ++LL):):F,>FDWDWX&+# Gos+ I6 input_idsr position_idsr inputs_embedsr8r^output_hidden_states return_dictrjrSc  UbUOURRnUbUOURRnUbUOURRnU bU OURRn USLUSL- (a [ S5eUR (a/UR(aU(a[RS5 SnUcURU5nUn [R"U5n U(aUc[RS5 U c,[R"U RSU RS9n UcU R!S5nUR#X%U 5n U(aSOSnU(aSOSn[%UR&5HunnU(aX4- nUR (a6UR(a%UR)UR*U U UUU UUUU 5 nOU"U U UUU UUUU S 9 nUSn U(dMuUScM}UUS4- nM UR-U 5n U(aX4- nU(aUR.(dS Ul[1U U(aUOSUUS 9nU (aU$UR35$) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rrcrr)r]rrrrr^r8rjT)last_hidden_staterr@ attentions)r{r^rr8use_return_dictrCrrrrrr+r2rrGrar_update_causal_maskrr_gradient_checkpointing_func__call__rrfrto_tuple)r/rrrrrr8r^rrrjr@r]rall_hidden_statesall_self_attnsrlayerrwoutputs r3rCZambaModel.forward~s2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKk<$D>k<'tnk<d^k<!!1!12k< u-- .k<k<\!!r5rc^\rSrSrS\4U4SjjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ S \ \ RS \ \ RS \ \S \ \S\ \S\ \S\ \ RS\\\ R4S\\\44Sjj5rSSjrSrU=r$)ZambaForCausalLMir{c >[TU]U5 [U5UlS/URRQUlUR Ul[ R"URUR SS9Ul UR5 g)Nzlm_head.weightFr) r(r)rr{rrrrr0lm_headrrNs r3r)ZambaForCausalLM.__init__so  ' #3"Tdjj6S6S"T ++yy!3!3V5F5FUS  r5cXlgrr{)r/decoders r3 set_decoderZambaForCausalLM.set_decoders r5cUR$rrrHs r3 get_decoderZambaForCausalLM.get_decoder s zzr5rrrrrlabelsr8r^rrrjlogits_to_keeprSc .UbUOURRnU bU OURRn U bU OURRn UR UUUUUUUU U U S9 nUSn[ U [ 5(a [U *S5OU nURUSS2USS245nSnUbUR"UX`R40U D6nU (dU4USS-nUbU4U-$U$[UUURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrrrr8r^rrjrrrlosslogitsrr@r)r{r^rrr{r1rslicer loss_functionrrrr@r)r/rrrrrrr8r^rrrjrrrbr@ slice_indicesrrrs r3rCZambaForCausalLM.forward#sKP2C1N-TXT_T_TqTq%9$D $++JjJj &1%SV8W8W~ot4]k mA}a,?@A  %%ffooPPDY,F'+'7D7V# CV C%#33!//))   r5c USLn U (d]UcUSURS:aUSS2URS*S24nOaURSURS:wa USS2U4nO7[URURSURURS9nUbZUcWUR 5R S5S- nURUS:HS5 U (dUSS2URS*S24nUb U (aSU0n OSUR50n U RUUUUURRUS.5 U $)Nr8rr)r:rarr)rrr8rrrj) rGr]r{r:ralongcumsum masked_fill_rrnum_logits_to_keep) r/rrrrrjrr8r empty_past_kv model_inputss r3prepare_inputs_for_generation.ZambaForCausalLM.prepare_inputs_for_generationusd(4/ )!"%);;%a.*>*>q*A)A)C&CD #~';';A'>>%a&78 5 Y__Q/tzz$++O  %,*>)..077;a?L  % %n&91 = +A 0B/B/D,DE   $+];L')=)=)?@L ,#2&"0"&++"@"@"0   r5)rrr{r) NNNNNNNNNNNr)NNNNNT)rKrLrMrNrr)rrrrr+rrr]rrdrrrFrrCrrOrPrQs@r3rrsp{151537=A59-1$(,0/3&*5934O E,,-O !.O u//0 O ""9: O   1 12 O ))*O D>O $D>O 'tnO d^O !!1!12O c5<</0O  u,, -O O h99r5ra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcN^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\ \ \ \R4S\\RS\\RS \\S \\S \\S \\S \ \\44Sjj5rSrU=r$)ZambaForSequenceClassificationic>[TU]U5 URUl[U5UlURR Ul[ R"URURSS9Ul UR5 grH) r(r) num_labelsrr{rrrr0scorerrNs r3r)'ZambaForSequenceClassification.__init__se   ++' "&**"?"?YYv114??O  r5rrrrrrr8r^rrrSc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UbURSnOURSnURR cUS:wa [ S5eURR cSnOUbXRR :gRU R[R5n[R"URSU R[RS9nUU-RS5nO.Sn[RURR S35 U [R"XRS 9U4nSnUGbURU R5nURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOU"UU5nOURR"S :Xa=[15nU"UR3SUR$5UR3S55nO-URR"S :Xa[55nU"UU5nU (dU4U SS-nUbU4U-$U$[7UUU R8U R:U R<S 9$)ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrr8r^rrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r8r`z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rc regressionsingle_label_classificationmulti_label_classificationr)r{rr{rrGrrCr;rar+int32rargmaxrrr2rK problem_typerr:rrr rr rrrrr@r)r/rrrrrrr8r^rrtransformer_outputsr@rr|last_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r3rC&ZambaForSequenceClassification.forwards(&1%!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r5)rr{rrr)rKrLrMrNr)rrr+rrrr listrrdrFrrCrOrPrQs@r3rrs151537KO59-1$(,0/3&*[ E,,-[ !.[ u//0 [ "%tE4E4E/F(F"GH [   1 12 [ ))*[ D>[ $D>[ 'tn[ d^[  u66 7[ [ r5r)rrrrz)r)Jrrtypingrrrrr+torch.utils.checkpointrtorch.nnrr r activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrutils.import_utilsrrconfiguration_zambar&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1dr!r"rr get_loggerrKrModuler%rrr[r]rrrrrFrTrfrprzrrr__all__rr5r3r%sL( 11 AA!.)>BqqF&,T,XR@P=-~DD-7**.0@BVXfg   H %J299J* UU\\ U# U%,, Uffeff`% II% <<% % << % U\\* %  %%4C)RYYC)LQ]biiQ]j ryy  <<~ARYYAHEryyEP)%?)%)%XG%GGV\+_\~  g %9g  g T gr5