tj) dZddlmZddlZddlmcmZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZddlmZddlm Z er#ddl!m"Z#ddl!m$Z$m%Z%m&Z&dZ'Gddee$Z(n-ddl)m*Z*ddl+m,Z,ddl-m.Z.ddlm/Z/e0ej1ej1ej1fZ2e3e4e2fZ%dZ'dedej1de5ej1zdzdej1fdZ& dfd ej1d!ej1d"ej1d#ej1d$ej1dzd%ej1dzd&ej1dzd'ej1dzd(ej1dzd)ej1dzd*e5dzd+e4d,e4de0ej1ej1ej1ffd-Z6 dgd/ej1d ej1d!ej1d$ej1d%ej1d&ej1d#ej1d"ej1d*e5d+e4d,e4d0e5de0ej1ej1ej1ffd1Z7 dhd2ej1d3ej1d4ej1d5ej1d6ej1d7ej1dzd8ej1dzd9ej1dzd*e5dzd:e8d;e8d+e4d0e5de0ej1ej1ej1e0ej1ej1ej1fdze0ej1ej1ej1fdzffd<Z9 did2ej1d3ej1d4ej1d5ej1d6ej1d=ej1dzd>ej1dzd?ej1dzd:e8d0e5d+e4dej1e0ej1e0ej1ej1ej1ffzfd@Z:d.ej;fd2ej1d3ej1d4ej1d5ej1d6ej1d7ej1d8ej1d9ej1d0e5dAej<de0ej1e0ej1ej1ej1fffdBZ=ddddd.ej;fd2ej1d3ej1d4ej1d5ej1d6ej1d=ej1dzd>ej1dzd?ej1dzd:e8d0e5dAej<de0ej1ej1ej1e0ej1ej1ej1fdze0ej1ej1ej1fdzffdCZ>ddddd.ej?dfdDe*d2ej1d3ej1d4ej1d6ej1d5ej1d=ej1dzd>ej1dzd?ej1dzd:e8d0e5dEej<d+e4dej1e0ej1e0ej1ej1ej1ffzfdFZ@ddddd.ej?ddfdDe*dGe*dHe*d2ej1d3ej1d4ej1d6ej1d5ej1d=ej1dzd>ej1dzd?ej1dzd:e8d0e5dEej<d+e4dIe8dej1e0ej1e0ej1ej1ej1ffzf"dJZAGdKdLejBZCGdMdNejBZ#GdOdPejBZDGdQdRejBZEGdSdTejBZFGdUdeZ(dVZGdWZHGdXdYeZIGdZd[ZJeeGd\d]eZKeGd^d_eIZLeeGd`daeZMeGdbdceIeZNgddZOdS)jzPyTorch xLSTM Model.) dataclassN)nn)CrossEntropyLoss)initialization)GenerationMixin)GradientCheckpointingLayer)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleis_xlstm_available)merge_with_config_defaults)capture_outputs) xLSTMConfig)RMSNorm) mLSTMBlockmLSTMStateTypesoft_capTceZdZdS) xLSTMBlockN)__name__ __module__ __qualname__f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/xlstm/modeling_xlstm.pyrr(s rr)Callable)partial)Literal)round_up_to_next_multiple_ofFvalues cap_valuereturnc>||S|tj||z zS)a Soft caps a tensor to a value. Performs a tanh operation on the logits and scales the result to the cap value. Common technique in attention and output language heads to prevent large logits from dominating the softmax. See for example Gemma2: https://huggingface.co/papers/2408.00118 Args: values: The tensor to cap. cap_value: The value to cap the values to. If None, no cap is applied. Returns: The capped values. )torchtanh)r%r&s r rr7s)  M5:fy&89999r@matKmatVvecBvecI matC_states vecN_statesscaMinter_states matC_initial vecN_initialscaMinter_initialqk_scale chunk_size num_chunksc g|j|jdR\} }}}}| }|j|j}}| |dz} |!tj| ||dz|z|f||}| tj| ||dz|zf||}|tj| ||dzf||}|tj| |||f||n|}|tj| ||f||n|}| tj| |df||n| }|d|z |z}|d}|dj}|d}td| D]u}||dddd||z|dz|zddf<||dddd||z|dz|zf<||dddd|f<|dddd|f}|dddd|f}tj||z|}|dddd|| z|dz| zddf}|dddd|| z|dz| zddf} |dddd|ddf}!tj |!|dz dddddddf}"||"z}#tj ||z|z dddddf}$|$d|z|# d d| zz}%|$|z|# d d dz}&|}|%}|&}w||dddd| dddf<||dddd| df<||dddddf<|||fS) Nrdtypedevice).r:N).r:r).N) shaper=r>r)zerosmaxr%squeezerangeexp transposesum)'r,r-r.r/r0r1r2r3r4r5r6r7r8 batch_sizenh_dhqkdhhvnc_dtype_devicematC_kvecN_k scaM_inter_kvecAscaGscaA_maxkey scaA_max_kscaG_kscaM_inter_k_next matK_chunk matV_chunkvecA_k vecAbar_kmatK_chunk_gated scaGbar_k matC_k_next vecN_k_nexts' r mlstm_chunkwise_recurrent_fw_CrbJsP)D(CTZ^(C(C% B4 *dk  TzH  +z2Q$&MU[dklllK  +z2Q$&Gv^efffK  #${JR!V+EV\cddd  # KRt4F7 S S S S  R^QeEKR.fW M M M Mkw  !( KR+6' J J J J"  M"T)D0G}88B<<&#++B// J'' ! !CCIK111cDjC!Gt+;;QQQ> ??EK111cDjC!Gt+;;; <*6 QQQ3Y '"!!!QQQ),J!!!QQQ)_F % &<*? L L aaaC*$4a:7M$MqqqPQJaaaC*$4a:7M$MqqqPQJ!!!QQQQQQ,'F &+matQepsc |j} | } |j\}}}}|| z}|||| ||}|||| |}|}|||| | |}|||| | |}|||| | |}tjtj| | ftj| }|dddddddddf|dddddddddfz }tj||td }||dddddddddfz}tj |ddj }||dddddddfz}tj ||}|dddddddddf}|dddddddddf}||z }tj |}|| ddz|z}||z} tj ||z }!||!z|z}"|"|z| |zz}#|"|dz| ddz}$tj tj|$tj | }%|#|%| zz }&|&||| | z|}'|%||| | z}(|||| | z})|'|(|)fS)Nr<infr:Fdimkeepdimr?T)r>r@viewr)trilonesboolwherefloatrBr%maximumrErF unsqueezerGabsreshape)*rdr,r-r0r1r2r/r.r6r7r8rerOrMrHrIdqkdhvrK matC_k_states vecN_k_statesscaMinter_k_statesltrmatF_logsig_chunkmatF_logsig_mask_chunk matLogD_chunk vecMintra_k vecM_b_intervecM_k_combinematLogD_stabilized_chunk matD_chunk matS_chunk matM_chunkvecBbarmatQ_chunk_gatedmatNumerator_commonvecDenom_l_commonvecDenom_max_common matH_k_chunkmatH_outvecN_outvecM_outs* r mlstm_chunkwise_parallel_fw_Hrs]+ #.#4 BSby#((RT3GG #((RTBB -yyRZ>>yyRZ>>yyRZ==j JZ(j     !AAAqqq!!!T!12T!!!QQQ4:J5KK!&S2CeEll]!S!S.aaaAAAtQQQ6F1GG i 2uEEEL 0AAAqqq$?? |[AA'111aaaD(89#AAAqqq!!!QQQ$45 #0>#A Y788 T^^B333x? *, )L>9::'>H4.>dARR,}/F/Fr/J/JJZ^^`blp^MqMqq#mEI6G,H,H%)UcTcJdJdee*.AC.GH $$ZR*_cJJ'..z2rJOO!))*b"z/JJ8++rqueryrVvalueigatefgatecstatenstatemstatereturn_last_statesreturn_all_statesc |j\} }}}|| zdkrtd|d| d|| z}|| ||| }|| ||| }||}|d}||dz}t ||||||||| | \}}}t ||||ddddd| ddf|ddddd| f|ddddddf|||| ||  \}}}|||f}| r<||dddd| dddf|dddd| df|ddddddfffz }n|d z }| r ||||ffz }n|d z }|S) NrSequence length is not divisible by chunk size .r:r; r,r-r.r/r3r4r5r6r7r8 rdr,r-r0r1r2r/r.r6r7r8reN)r@ ValueErrorrk logsigmoidcumsumrbr)rrVrrrrrrr6rrr7rerHrIsequence_lengthrKrMr/vecF vecF_logsigr.rwrxryrrr ret_tuples r mlstm_chunkwise_fwrsJ*16 - B Z '1 , ,nnnaknnnoo o  *zz*b"j99zz*b"j99&&t,, !!"%%  TzHj"dD9 9 9NhnNNOO O >j"d3 3 3NhnNNOO O >j"a0 0 0NhnNNOO O ;:r1- - -G%+GGHH H ;:r1- - -G%+GGHH H8&11%888h#6>>9X0>ABB9U^344t~. !!!!QQQ4-08>>3C3CChqqqRSRSRSUVUVUVX\}F] 111aaa qqq!!!T111}!5 5G  "HNN$4$44x#~EAAAqqq$M*^->->Y->-O-OO a  ##+#66#AAAqqq$M2^AAAqqq!!!TM5R5U5U\e5U5f5ff %--a00 )^O,,=!2!2!4!4g>>DHH{H[[ GO DDyD ! !'***=='***=='***==>>>BBBrc  2|j\} } }}|jd}|j}|i||td||td|| || || }}}nOt j| | ||f| |}t j| | |f| |}t j| | df| |}g}t |D]}|dddd|df|dddd|df}}|dddd|ddf|dddd|ddf|dddd|ddf}}}td ||||||||| | d | \}\}}}||t j |d}|r||||ffS|S) Nr:z)Initial states must be provided together.rr<r) rrrrrVrrrrerr?rir) r@r>rrr)rArDrappendstack)rrVrrrrrrrrerrrHrIrrKrvr> matC_state vecN_state vecM_state vecH_listtvecF_tvecI_tvecQ_tvecK_tvecV_tvecHmatHs r mlstm_recurrent_sequence_nativersG(16 - Bk"o   I$5 !LMMM I$5 !LMMM ; // ; // ; //%/ JJj"dC%@ \bcccJj"d%;;W]^^^Jj"a%8 TZ[[[J '' # #A"111aaaD=15AAAq$3GFF&+111aaaAAA:%6AAAqqq!QQQJqqqRSRSRSUVXYXYXYzIZFFF:U :!!!' : : : : 6D6:z:   T " " " "{9"---  *j*== =Krmlstm_chunkwise_kernelautocast_kernel_dtypec  | rtdd|j\}}}}|}|| zdkr|| zdz | z| z}|||||jd}|||||jd}|||||jd}||||}||||}||ddddd|ddf<||ddddd|ddf<||ddddd|ddf<||ddddd|f<||ddddd|f<n |}|}|}|}|}|d||||||||| | | | d | }|ddddd|ddf}|S)Nz6We are padding zeros, so we cannot return last states,z*as they would be not the true last states.rrr) rrVrrrrrrrrerr7r)rr@ new_zeros)rrrVrrrrrrrrerr7rrHrIrrK S_unpaddedS_paddedq_padk_padv_padi_padf_padrs r wrap_chunkwise_pad_zerosr s  H<  16 - B$ Z '1 , ,(:59jHJVHOOJHek!nMMEMM*b(CIaLIIEOOJHek!nMMEOOJH==EOOJH==E*/E!!!QQQ  QQQ& '*-E!!!QQQ  QQQ& '*/E!!!QQQ  QQQ& '',E!!!QQQ  # $',E!!!QQQ  # $ $EEEEE%% 1"7!    AAAqqq+:+qqq() rmlstm_sequence_kernelmlstm_step_kernelenable_loggingc|j\}}}}|jd}||n(tj|||||jtj}| | n'tj||||jtj}| | n'tj||d|jtj}|dkrg}d}||z }||z}|dkr||z}||z}||d||ddf|d||ddf|d||ddf|d||f|d||f||||d| |  \}\}}}||z }||||z }|dkr||d||ddf|d||ddf|d||ddf|d||f|d||f|||d| \}\}}}||tj|d }n|dkrtd |d || d | d | d ||||||  \}\}}}|dddddddf}| r||||ffS|S)af This function computes the last hidden state and matH outputs of the mLSTM, independently of the sequence length. For this it uses three kernels: - mlstm_chunkwise_kernel: mlstm chunkwise kernels that processes chunks of a given chunk size in parallel. - mlstm_sequence_kernel: mlstm kernel that processes the remaining sequence length in a single step recurrence. - mlstm_step_kernel: mlstm kernel that processes a sequence length of 1 in a single step. It tries to maximize the chunksizes to improve performance. It will start with the given chunk size and then divides the chunksize by 2 until the chunk size is smaller than 16. At every chunksize it will process the maximal number of chunks that fit into the remaining sequence length. E.g. for chunk_size = 64, this function will try the chunksizes [64, 32, 16] if necessary. For the remaining sequence length, which is smaller than 16, we use a different kernel that computes the mLSTM in a single step and loop over this in pytorch. Args: mlstm_chunkwise_kernel: The mLSTM chunkwise kernel that processes chunks of a given chunk size in parallel mlstm_sequence_kernel: The mLSTM kernel that processes the remaining sequence length in a single step recurrence query: The query tensor (batch_size, nh, sequence_length, dhqk) key: The key tensor (batch_size, nh, sequence_length, dhqk) value: The value tensor (batch_size, nh, sequence_length, dhhv) fgate: The forget gate tensor (batch_size, nh, sequence_length) igate: The input gate tensor (batch_size, nh, sequence_length) c_initial: The initial cell state tensor (batch_size, nh, dhqk, dhhv) n_initial: The initial hidden state tensor (batch_size, nh, dhqk) m_initial: The initial memory state tensor (batch_size, nh, 1) return_last_states: If True, the function will return the last states of the mLSTM eps: The epsilon value used for numerical stability autocast_kernel_dtype: The dtype used for the kernel computation chunk_size: The chunk size used for the chunkwise kernel enable_logging: If True, the function will log debug information. Default is False. Returns: The last hidden state tensor (batch_size, nh, sequence_length, dhhv) or a tuple containing the last hidden state tensor and the last states of the mLSTM Last states are (cstate (batch_size, nh, dhqk, dhhv), nstate (batch_size, nh, dhqk), mstate (batch_size, nh, 1)). r:Nr>r=rr.T) rrVrrrrrrr7rrre) rrVrrrrrrrrerrz)Received empty sequence (sequence_length=z3), require at least single element in the sequence.) rrVrrrrrrre) r@r)rAr>float32 contiguousr concatenaterrC)rrrrrVrrrrrrrrerr7rrHrIrrKrLc_staten_statem_stateh_outsseq_len_start_idxremaining_seq_lenr8 iter_seq_len seq_len_idxh_outs r (wrap_chunkwise_arbitrary_sequence_lengthrJsp14 - B{2$ IZT4 RWR_``` $ IZT#*EMZZZ $ IZQszWWW  Q  F !  /2C C *j8JA~~)J6 /,> 5K5K%6{%BAAA EFQQSSC!2;!>ABMMOO%6{%BAAA EFQQSS%6{%B BCNNPP%6{%B BCNNPP%%%)'+*? 6 6 622'"\1! e$$$ /2C C  1$$5J5J%6%F IJUUWWC!2?!BAAAEFQQSS%6%F IJUUWW%6%F FGRRTT%6%F FGRRTT%%%'+ 6 6 622' e$$$%f!444EE!## EEEE2C1BmmA&&KKNNmmA&& 2 2 2 .E.GWg!!!QQQaaa-(E  7GW55 5LrcLeZdZdZeZdeffd Z ddejdejdejdejd ejd ejdzd ejdzd ejdzd e dzde ddzdeje eje ejejejffzfdZ de fdZxZS) xLSTMBackendzxLSTM Backend Module for PyTorch. This module wraps the xLSTM kernels and provides a high-level interface for training and inference. configc t||_t|_t |_t|_tt|jt|jtt|j t|jtt|j |j|jtt|jd|_t|jtt|j|j|j}d|jvrtt&|}||_dS)N)rT)rrrr7rerr)rrer7 with_padding)r)super__init__rrchunkwise_kernel_fnrsequence_kernel_fnrstep_kernel_fnr"rgetattrr)inference_state_dtyper7rer _inference_fnmoder _train_fn)selfrtrain_kernel_fn __class__s r rzxLSTMBackend.__init__s/ GG      DK'FD $&ED #"=D !(8'+'?&-+ 'v/K L L'''#*' 'v/K L L###",J&-eV5Q&R&R#'"""D "&(&-eV5Q&R&RJ!, O ,,")*B[j"k"k"k,DNNNrNrrVrrrrrrrr)train inferencer'c :| |jj} d| vrM| |jj} |jjdkr| rtd||||||||||  Sd| vr|||||||||Std|jj) aForward pass of the mLSTM backend. Depending on the configured mode, this method will call the appropriate kernel function. Args: query: The query tensor of shape (batch_size, nh, sequence_length, dhqk). key: The key tensor of shape (batch_size, nh, sequence_length, dhqk). value: The value tensor of shape (batch_size, nh, sequence_length, dhhv). igate: The input gate preactivation tensor of shape (batch_size, nh, sequence_length). fgate: The forget gate preactivation tensor of shape (batch_size, nh, sequence_length). c_initial: The initial cell state tensor of shape (batch_size, nh, dhqk, dhhv). Defaults to None. n_initial: The initial hidden state tensor of shape (batch_size, nh, dhqk). Defaults to None. m_initial: The initial memory tensor of shape (batch_size, nh, 1). Defaults to None. return_last_states: Whether to return the last states of the sequence. Defaults to None. If None, the value from the config is used. Returns: hidden states of shape (batch_size, nh, sequence_length, dhhv) hidden states and last states the last states are the cell state cstate (batch_size, nh, dhqk, dhhv), the normalizer state nstate (batch_size, nh, dhqk), and the max state mstate (batch_size, nh, 1) Nrtrain_with_paddingzFreturn_last_states=True is not supported with train_with_padding mode.) rrVrrrrrrrr rrVrrrrrrzUnknown mode: )rrrrrr) rrrVrrrrrrrrs r forwardzxLSTMBackend.forward sF|{'$%-)-)G&;#';;;)s()qrrr~~''''9&   $$))'''*   !!D$+2B!D!DEEErc|jSrrrs r extra_reprzxLSTMBackend.extra_reprRs k# #rNNNNN)rrr__doc__r config_classrr)Tensorrnr#tupler strr __classcell__rs@r rrs}  #  -; - - - - - -R.2-1-1.29=G FG F<G FG F< G F < G F < G F|d*G F|d*G F|d*G F!%t G F./$6G F\E%,elELRWR^6^0_"_` `G FG FG FG FR $ $ $ $ $ $ $ $ $rrc eZdZdZ ddedededed ef fd Zd ej d ej fd Z d ej d ej fdZ d ej d ej fdZ xZ S) xLSTMRMSNorma3Root mean square normalization layer implementation similar to https://pytorch.org/docs/stable/generated/torch.nn.RMSNorm.html. It normalizes the input tensor by the root mean square of the last dimension. Args: num_features: The number of features in the input tensor. eps: A small value to avoid division by zero. use_weight: Whether to use a learnable weight. use_bias: Whether to use a learnable bias. force_float32_reductions: Whether to force float32 reductions. rcTF num_featuresre use_weightuse_biasforce_float32_reductionscHt||_||_||_|r,t jtj||_ nd|_ |r-t jtj ||_ dSd|_ dSr) rrrrerr Parameterr)rmweightrAbias)rrrerrrrs r rzxLSTMRMSNorm.__init__cs GG      ,D DH,DD ) # l5:l+C+CDD "  !L\)B)BCC   rxr'cJ|j ||jz}|j ||jz}|Srr"r#rr$s r _apply_weight_biaszxLSTMRMSNorm._apply_weight_biasz-{& Oy$ MHrc|j}|jr|}|tj|ddd|jzz}||S)Nrr:Trh) r=rrpr)rsqrtpowmeanrer)rr$in_dtypes r _rms_normalizezxLSTMRMSNorm._rms_normalizesfwH, GGIIEKa "d C Cdh NOOOA44>> !rcZ||}||}|Sr)r/r(r's r r zxLSTMRMSNorm.forwards-##A&&A''**AHrrcTFT)rrrrintrprnrr)rr(r/r rrs@r rrUs #"-1  ! ! ! !  !   ! '+  ! ! ! ! ! !.        "EL "U\ " " " " U\ el        rrc eZdZdZ ddedededed ed ef fd Zd ej d ej fdZ d ej d ej fdZ d ej d ej fdZ xZ S)xLSTMMultiHeadLayerNormamMulti-head version of the LayerNorm layer. It normalizes the last dimension of the input tensor. The input is assumed to have the shape (batch_size, sequence_length, nh, DH), where: batch_size: batch size sequence_length: sequence length nh: number of heads DH: head dimension The normalization is applied over the last dimension (DH) of the input tensor. Args: num_heads: The number of heads. head_dim: The head dimension. eps: A small value to avoid division by zero. use_weight: Whether to use a learnable weight. use_bias: Whether to use a learnable bias. force_float32_reductions: Whether to force float32 reductions Returns: The normalized tensor with the shape (batch_size, sequence_length, nh * DH). rcTF num_headshead_dimrerrrc|t||z|_||_||_|r1t jtj|j|_ nd|_ |r1t jtj |j|_ nd|_ ||_ ||_ dSr)rrrrerrr!r)rmr"rAr#r5r6)rr5r6rerrrrs r rz xLSTMMultiHeadLayerNorm.__init__s GG      )H 4D DH,DD ) # l5:d6G+H+HII "  !LT5F)G)GHH   &DN$DMMMrr$r'cJ|j ||jz}|j ||jz}|Srr&r's r r(z*xLSTMMultiHeadLayerNorm._apply_weight_biasr)rc|j}|jr|}||ddz }|t j|ddd|jzz}||S)Nr:TrhF)rirjunbiased) r=rrpr-r)r+varrer)rr$r. x_centeredys r _layer_normalizez(xLSTMMultiHeadLayerNorm._layer_normalizeszwH, GGIIQVVDV999JU[2te)T)TW[W_)_```A44>> !rcT|j\}}}}||jkr"td|jd|d|j|j|kr"td|jd|d|j||}|||d}||}|S)Nz Expected z heads, got z, input shape: z head dimension, got r:)r@r5rr6r>rtr()rr$rHrrIDHs r r zxLSTMMultiHeadLayerNorm.forwards34' /JRT^## !eT^!e!e!e!e\]\c!e!efff}"" !mT]!m!mQS!m!mdedk!m!mnnn%%a((A *or::A''**AHrr1)rrrrr2rprnrr)rr(r>r rrs@r r4r4s   8#"-1 % % % %  %   %   %'+ % % % % % %4        "el "u| " " " " | \        rr4cHeZdZdeffd ZdejdejfdZxZS)xLSTMFeedForwardrct||_t|j|jz|j|_|jjdkratj |j|j|jj |_ tj |j|j|jj |_ nC|jjdkr3tj |jd|jz|jj |_tj |j|j|jj |_tj|_dS)Nsingle in_features out_featuresr#fusedr)rrrr$ hidden_sizeffn_proj_factorffn_round_up_to_multiple_of up_proj_dim weight_moderLinearr proj_up_gateproj_upproj_up_gate_z proj_downSiLUact_fnrrrs r rzxLSTMFeedForward.__init__s: GG      DK;"V%;;2  D  {&(22$&I & 2!%!1-%%%! "y & 2!%!1-    (G33&(i & 2!"T%5!5-'''#  Y ,#/[)DN '))DKKKrr$r'c|jjdkr?|||||z}n]|jjdkrM||}t j||jfd\}}|||z}| |}|S)NrDrHr:r) rrMrTrOrPrQr) tensor_splitrLrR)rr$gatezr=s r r zxLSTMFeedForward.forwards{&(22KK 1 1! 4 455 QG(G33''**,Q1A0CLLLaKK%%)q!!AHr) rrrrrr)rr rrs@r rBrBsk! $;! $! $! $! $! $! $F U\ el        rrBc leZdZdeffd Z ddejdedzdeejedzffdZ xZ S) xLSTMLayerrct||_t|j|jz|_t|j|jz|_|jj dkr5tj |jj|j|jj |_ tj |jj|j|jj |_tj |jj|j|jj |_tj |jj|j|jj |_tj |jj|jjd|_tj |jj|jjd|_n|jj dkrvtj |jjd|jzd|jzz|jj |_tj |jjd|jjzd|_tj|_t1|j|_t5|jj|j|jjz|jjd|jj |jj|_tj |j|jj|jj |_dS)NrDrETrHrr)r5r6rerrr)rrrr2rI v_dim_factorv_dim qk_dim_factorqk_dimrMrrNrqkv ogate_preactr5 igate_preact fgate_preact qkv_opreact ifgate_preactSigmoid ogate_act_fnr mlstm_backendr4norm_epsnorm_reduction_force_float32multihead_normout_projrUs r rzxLSTMLayer.__init__s GG      DKV/&2EEFFDJf063GGHHDK{&(22 $ 7!%-  $ 7!%-  $ 7!%- %'I $ 7!%-%%%! %'I $ 7!%!6%%%! %'I $ 7!%!6%%%!! (G33#%9 $ 7!"T[1tz>!A-$$$ &(Y $ 7!"T[%:!:&&&" !# D !-T[!A!A!AD "9+/t{'<<K(-)-)Q ###D I J![4[)DMMMrNr$stater'c |jdkrtd|j|j\}}}|jjdkr||}||}||}||} t| ||jj } t| ||jj } n|jjdkr| |} tj| |jd|jzd|jz|jzfd\}}}} t|||jj } tj| |jjfd\} } ||||jjdd d}||||jjdd d}||||jjdd d}| d d} | d d} |d \}}}n|\}}}||||| | ||| \}}||jj||j|jjzf}|j|krtd |jd ||d d}||}|||d}|| |z}||}||fS)Nrz=Input must have shape [batch_size, sequence_length, HD], got rD)r&rHrr:rr)NNNr zGot z , expected )ndimrr@rrMrarbrcrdrre gate_soft_caprfrgr)rWr`r^rhr5rtrFrkrnrjro)rr$rprHrrJrrVro_preacti_preactf_preactrg if_preactrrrrexpected_h_shapeh_normrr=s r r zxLSTMLayer.forwardTs`v{{ !jabah!j!jkkk-.W *J{&(22q ffQiiq ,,Q//#D$5$5a$8$8DKD]^^^#D$5$5a$8$8DKD]^^^(G33"..q11 .3.@ DKDK$*4 ///+sE8%T%7%7%:%:dkF_``` %*%7 DKDYC[ac%d%d%d"(MM*ot{?TVXYYccdeghiiE++j/4;;PRTUU__`acdeeCMM*ot{?TVXYYccdeghiiE))!Q//H))!Q//H}2B/ 9ii27/ 9i))###*  HAu % dk33    w*** !N!N!N Adapted from: https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/init_functions.py Fills the input Tensor with values according to the method described in Transformers without Tears: Improving the Normalization of Self-Attention - Nguyen, T. & Salazar, J. (2019), using a normal distribution.r?c2tj|dSNg)r-stdinitnormal_tensorrs r init_z small_init_method..init_|F#6666rr)rirrs @r small_init_methodrs7 C=e $C77777 Lrc(d|z |dzz fd}|S)zh Adapted from https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/init_functions.py rrc2tj|dSrrrs r rzwang_init_method..init_rrr)n_layersrirrs @r wang_init_methodrs7 h, 'C77777 LrcpeZdZdZeZdZdgZdZdZ de iZ dZ e jfdZxZS)xLSTMPreTrainedModelzR An abstract class for an interface to loading a pre-trained xLSTM model. backbonerT hidden_statescJ|D] \}}||ur|cSdS)N) named_modules)rmodulenamemods r _module_name_mapz%xLSTMPreTrainedModel._module_name_maps=++--  ID#f}} rrc t|t|tjr.t |jj|jj dSt|tj rh|j tj |j |jjdkrd||vrtj |j d||vr6tj|j dt#j|j zdSd||vritj|j t#jdd|j jd|j j|j j dSdS|jjd krZd||vrBtj |j tj|j d|jj|j d|jj|j d|jjz d t#j|j zz tj|j d|jj|j d|jj|j |jjdz t#jdd|j jd|j j|j j zdSd ||vr;t3|j jd |jj|j dSd||vr5t3|jj|jj|j dS|j +t |jj|j dSdSdS)NrDrXrg$rg@g@r:rrHg$@rRr)rirro)r _init_weights isinstancer EmbeddingrrrI embeddingsr"rNr#rzeros_rMrcopy_r) ones_likelinspacer@rr>r=r5rnum_hidden_layers)rrrs r rz"xLSTMPreTrainedModel._init_weightss f%%% fbl + +1 J 6 dk5 6 6t7M N N N N N  * */ J{& FK((({&(22vAVAVW]A^A^7^7^ FM***d33F;;;;Jv{EEOFK4P4P,PQQQQQ 5 5f = ===J "K-b1"#);#5"(+"3     >=(G33$BWBWX^B_B_8_8_ FM*** K 7$+"7 78K 7$+"7 78k"9DK$9"9:;U_V[999:  K 7$+"7 78K 7$+"7 78k$+"7"9"9:;n )"-b%{1$k/       5 5f = ===d V]%8%;dkFcdddekerssssst44V<<<<e T[%>> from transformers import AutoTokenizer, xLSTMForCausalLM, xLSTMCache >>> model = xLSTMForCausalLM.from_pretrained("NX-AI/xLSTM-7b") >>> tokenizer = xLSTMTokenizer.from_pretrained("NX-AI/xLSTM-7b") >>> inputs = tokenizer(text="I am an xLSTM", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> cache_params = xLSTMCache(config=model.config, max_batch_size=1, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, cache_params=cache_params, use_cache=True) >>> outputs.cache_params xLSTMCache() Nrmax_batch_sizer=r>c tjdgt|_|_|_fdt jD|_dS)Nrr<c i|]o}|tjjjjgtjjjgtjjdgfpS)r<r)r)rAr5 qk_head_dim v_head_dim).0layerrr>r=rs r z'xLSTMCache.__init__..Ns      #V%5v7I6K\]!  ^V-=v?QRZ_hnooo ^V-=qAW]^^^    r) r)rr2 seqlen_offsetr=rrDr rnn_state)rrrr=r>rs ```` r rzxLSTMCache.__init__?s{#\1#SHHH          v788    rc8fdjD_dS)Nc i|]q}|tjj|dtjj|dtjj|dfrS)rrr)r) zeros_liker)rrrs r rz$xLSTMCache.reset..\s~     !6q!9:: !6q!9:: !6q!9::   r)rrs`r resetzxLSTMCache.reset[s5        r) rrrrr)bfloat16rr2r=rrrrrr rrs}  L#^!     {  d     8     rrcleZdZUdZejdzed<dZedzed<dZ e ejdzed<dS) xLSTMOutputz cache_params (`xLSTMCache`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Nlast_hidden_state cache_paramsr) rrrrr) FloatTensor__annotations__rrrrrrr rrfsd (4////&*L*t#***59M5*+d299999rrceZdZfdZdZdZeee d de j dzde j dzde dzde dzd e ed eezf d ZxZS) xLSTMModelcxttjjj|_tjfdtj D|_ tj j |_d|_|dS)Nc.g|]}tSr)r)rrJrs r z'xLSTMModel.__init__..zs!$Z$Z$ZAZ%7%7$Z$Z$Zr)reF)rrrr vocab_size embedding_dimr ModuleListrD num_blocksblocksrrIrlout_normgradient_checkpointing post_initrUs `r rzxLSTMModel.__init__vs    ,v'8&:NOOm$Z$Z$Z$ZvGXAYAY$Z$Z$Z[[ $V%7V_MMM &+# rc|jSrrrs r get_input_embeddingszxLSTMModel.get_input_embeddingss rc||_dSrr)r new_embeddings r set_input_embeddingszxLSTMModel.set_input_embeddingss 'rN input_ids inputs_embedsr use_cacherr'c H|d}| |jj}|du|duz rtd|||}|r7|5t |j|d|j|j}|}|j s|jj |j dkr|sd}tj 5|!t |j|j d}tj|} ||j dkr|dd|t||jj z|j df} t!|jD]\} } | | |j| \} } t't)|j| D]0}| |}|j| ||1d |_| | dd|t||jj z|j df<||jj z }||j dk| }dddn #1swxYwYnt!|jD]\} } | || |j| nd\}} |r_t't)|j| D]0}| |}|j| ||1d |_|r|xj|j dz c_||}t3|| S) r cache_params (`xLSTMCache`, *optional*): The xLSTMCache that carries the RNN states. output_hidden_statesNz:You must specify exactly one of input_ids or inputs_embedsrrr)rr)rpF)rr)getrrrrrsizer>r=trainingmax_inference_chunksizer@r)rrmin enumeraterrrDlenrrnn_state_initialrrr)rrrrrrrroffset final_statehidden_states_chunk layer_idx xlstm_blockr state_idxlocal_rnn_states r r zxLSTMModel.forwards" &zz*@AA  '#';#C  -t"; < [YZZ Z   OOI66M  -% ]//22=;OWdWjL&  ' ; 3m6I!6LLL(MF , ,'#-T[Q^QdefQg#h#h#hL#.}== }21555*76C1T(TVcVijkVl$m$mmm+'3 66, 6C1T(TVcVijkVl$m$mmmdkAAF!}21555"!, + , , , , , , , , , , , , , , ,.+4DK*@*@ ; ;& ;+6;!9E9QL*955W[,,( y  ;%*3|/Ei/P+Q+Q%R%R\\ *3I*>$.y9)DJJ?[[[[5:L2  A  & &-*=a*@ @ & & m44 +%    s E&H<<II)NNNN)rrrrrrrrrr) LongTensorrrnr r rrr rrs@r rrts ((( .215*.!% P P #d*P '$.P !4' P $; P +, P   P P P ^_ P P P P P rrceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dS)xLSTMCausalLMOutputaP loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). cache_params (`xLSTMCache`, *optional*, carrying the RNN states): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Nlosslogitsrr) rrrrrr)rrrrrrrrrr rrs&*D% d ")))'+FE  $+++&*L*t#***59M5*+d299999rrceZdZfdZdZdZdZdZee dde j dzde j dzd e dzd e j dzd edzd eed eezfdZxZS)xLSTMForCausalLMct|t||_t j|j|jd|_| dS)NF)r#) rrrrrrNrIrlm_headrrUs r rzxLSTMForCausalLM.__init__s^    "6** y!3V5FUSSS  rc|jSrrrs r get_output_embeddingsz&xLSTMForCausalLM.get_output_embeddingss |rc||_dSrrrnew_embeddingss r set_output_embeddingsz&xLSTMForCausalLM.set_output_embeddingss % rc4|jSr)rrrs r rz%xLSTMForCausalLM.get_input_embeddingss}11333rc6|j|Sr)rrrs r rz%xLSTMForCausalLM.set_input_embeddingss}11.AAArNrrrlabelsrrr'c |j|f|||d|}|d}|||jjj} |js|jj| j dkrd} tj 5| | j dkrt| dd| t| |jjz| j df|jj| dd| t| |jjz| j df<| |jjz } | | j dkdddn #1swxYwYnt| |jj} d} ||| j}| dddddf} |dddf} t#}|| d| d| d} t)| | |j|jS)r)rrrrrN.r:)rrrr)rrrr"r=rprrrr@r)rrroutput_logit_soft_capr>rrrkrrrr)rrrrrrr xlstm_outputsrrrr shift_logits shift_labelsloss_fcts r r zxLSTMForCausalLM.forwards&  %'      &a( m..t|/B/HIIJJPPRR} I!Dv|TU!V!VF B Bv|A..muqqq&3v 8[/[]c]ijk]l+m+m"mmn 9nnF111fs6DK4W+WY_YefgYh'i'iiijdkAAF v|A.. B B B B B B B B B B B B B B Bfdk&GHHF  YYv}--F!#ssAAA+.99;;L!#qrr'?5577L'))H8L--b,2C2CB2G2GHH,J[J[\^J_J_``D"&3'5     sB-EEEr)rrrrrrrrrrr)rrrrnr r rrr rrs@r rrs)&&&444BBB.226*.*.!% 4 4 #d*4 (4/4 !4' 4  4' 4 $; 4 +,4  $ $4 4 4 ^4 4 4 4 4 rr)rrrr) NNNNNNNr+r)r+rrc)NNNNFFr+rc)NNNFrcr+)Pr dataclassesrr)torch.nn.functionalrrrtorch.nnrrrr generationrmodeling_layersr modeling_utilsr processing_utilsr utilsr r rrr utils.genericrutils.output_capturingrconfiguration_xlstmrxlstm.xlstm_large.modelrrrrrexternal_xlstmrcollections.abcr! functoolsr"typingr#r$rrrzdictr2rprbrrnrrrr=rrrrrModulerr4rBr[rrrrrrrr__all__rrr rs[ !!!!!! %%%%%%&&&&&&))))))999999------&&&&&&jjjjjjjjjjjjjj777777555555,,,,,,S??????LLLLLLLLLLN     /    )(((((!!!!!!AAAAAA elEL HI#223NN:::%%,2F2M:Y^Ye::::0,0+/04,0,015!%X:X:lX:lX:lX:l X: \D( X: \D( X: ,-X:lT)X:lT)X:!<$.X:$,X:X:X: u|U\5<7 8X:X:X:X:JL,L,lL,lL,lL, \ L, \ L, ,L,lL,lL,L,L,L,L, u|U\5<7 8L,L,L,L,h'+&*&*!%#("'OO|O \O|O| O | O  t# O t#O t#O$,O!O OOO     elEL%,67$> elEL%,67$>  @ OOOOn*.)-)-#(??|? \?|?| ? | ? <$& ?<$&?<$&?!??? elE%, el2Z,[[\ \????T#(==C=C|=C \=C|=C| =C | =C  =C =C =C=C[=C u|U5<u|#KLL M=C=C=C=CJ*.)-)-#(#(=GG|G \G|G| G | G <$& G<$&G<$&G!GG[G     elEL%,67$> elEL%,67$>  @ GGGG`*.)-)-#(-2^<< (<|<\<| < | < | <<$&<<$&<<$&<!<< %{<< elE%, el2Z,[[\ \<<<<N*.)-)-#'-2^$!SS (S'S$S| S \ S | S|S|S<$&S<$&S<$&S!SS %{SS !S" elE%, el2Z,[[\ \#SSSSjt$t$t$t$t$ryt$t$t$l77777ry777rPPPPP")PPPd-----29---^FFFFFRYFFFP/B      HJHJHJHJHJ?HJHJHJVG G G G G G G G T  : : : : :+ : :  :d d d d d %d d d N :::::+:: :"J J J J J +_J J J Z   r