tjAz ~ddlmZmZddlmZddlZddlmZddlm Z m Z m Z m Z m Z mZe rddlmZe dd ZejeZGd d eZGd deZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGdd eZ Gd!d"e Z!Gd#d$e!eZ"Gd%d&e!eZ#Gd'd(e!eZ$Gd)d*e!eZ%eeee!e!e!e"e#ed+ Z&eeee!e!e!e$e%ed+ Z'Gd,d-Z(d.ed/e)e*e+e,ffd0Z-Gd1d2e(Z.Gd3d4e(Z/Gd5d6e(Z0Gd7d8e(Z1e/Z2Gd9d:e.Z3dS);)ABCabstractmethod)IterableN)PreTrainedConfig)is_hqq_availableis_optimum_quanto_availableis_quanto_greateris_torch_greater_or_equalis_torchdynamo_compilinglogging) Quantizerz2.7T accept_devc eZdZUdZdZdZdZedzed<fdZ dZ dZ e d e jd e jd dfd Ze d e jd e jd ee je jffd Ze ded eeeffdZe d efdZe d efdZdZdZddZde jd dfdZd efdZxZS)CacheLayerMixinz0Base, abstract class for a single layer's cache.FTN _layer_typec tjdi||j7t|tr|t |j<dS|t |j<dSdS)N)super__init_subclass__r issubclass StaticLayerSTATIC_LAYER_TYPE_MAPPINGDYNAMIC_LAYER_TYPE_MAPPING)clskwargs __class__s V/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/cache_utils.pyrz!CacheLayerMixin.__init_subclass__#si!!++F+++ ? &#{++ B=@)#/:::>A*3?;;; ' &c 0d|_d|_d|_dSNF)keysvaluesis_initialized)selfrs r__init__zCacheLayerMixin.__init__+s)- +/ #r c|jjSNr__name__r&s r__repr__zCacheLayerMixin.__repr__0.)++r key_states value_statesreturncdSr)rr&r/r0s rlazy_initializationz#CacheLayerMixin.lazy_initialization3sadadr cdSr)rr&r/r0argsrs rupdatezCacheLayerMixin.update6s -0Cr query_lengthcdSr)r)r&r9s rget_mask_sizeszCacheLayerMixin.get_mask_sizes;sDGCr cdSr)rr,s rget_seq_lengthzCacheLayerMixin.get_seq_length>%(Sr cdS)a Returns the maximum sequence length the layer can hold. A value of `-1` means no maximum, or an undefined maximum, for example a dynamic attention layer that grows indefinitely or a linear attention layer that has no sequence length dimension. Nrr,s rget_max_lengthzCacheLayerMixin.get_max_lengthAs  r c|jrD|jdd|_|jdd|_dSdSz(Offload this layer's data to CPU device.cpuT non_blockingN)r%r#tor$r,s roffloadzCacheLayerMixin.offloadJsP   C U >>DI+..T.BBDKKK C Cr c|jrc|jj|jkrP|j|jd|_|j|jd|_dSdSdSzcIn case of layer offloading, this allows to move the data back to the layer's device ahead of time.TrDN)r%r#devicerFr$r,s rprefetchzCacheLayerMixin.prefetchPsk   I49#3t{#B#B T[t DDDI+..4.HHDKKK I I#B#Br c|jr2|j|jt |dr>t |jtr d|_dS|jdSdS)4Resets the cache values while preserving the objectscumulative_lengthrN)r%r#zero_r$hasattr isinstancerNintr,s rresetzCacheLayerMixin.resetVs   IOO    K      4, - - /$0#66 /)*&&&&,,.....  / /r beam_idxc.|dkr||jd||jj|_|jd||jj|_dSdS)z,Reorders this layer's cache for beam search.rN)r=r# index_selectrFrJr$r&rTs r reorder_cachezCacheLayerMixin.reorder_cachecsy    1 $ $ ..q(++di>N2O2OPPDI+221hkk$+BT6U6UVVDKKK % $r c^td|SNzm`get_max_cache_shape` is deprecated, and will be removed in version 5.16. Please use `get_max_length` instead)loggerwarningr@r,s rget_max_cache_shapez#CacheLayerMixin.get_max_cache_shapeis/ {   ""$$$r r1N)r+ __module__ __qualname____doc__is_compileablesupports_early_initrstr__annotations__rr'r-rtorchTensorr4tupler8rRr;r=r@rGrKrS LongTensorrXr] __classcell__rs@rrrs::N#Kt"""BBBBB$$$ ,,,deld%,d[_ddd^d0,06;l0 u|U\) *000^0G3G5c?GGG^G((((^(    ^ CCC III / / / /We&6W4WWWW %S%%%%%%%%r rceZdZdZdZdejdejddfdZdejdejdeejejffdZ d e dee e ffd Z de fd Z de fd Z d e ddfdZde ddfdZdejddfdZdS) DynamicLayerz A cache layer that grows dynamically as more tokens are generated. This is the default for generative models. It stores the key and value states as tensors of shape `[batch_size, num_heads, seq_len, head_dim]`. Fr/r0r1Nc|j|jc|_|_tjg|j|j|_tjg|j|j|_d|_dSNdtyperJT)rqrJrftensorr#r$r%r3s rr4z DynamicLayer.lazy_initializationxs^","2J4E DKL4:dkJJJ l2TZ LLL "r c|js|||tj|j|gd|_tj|j|gd|_|j|jfS)i Update the key and value caches in-place, and return the necessary keys and value states. Args: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. Returns: tuple[`torch.Tensor`, `torch.Tensor`]: The key and value states. dim)r%r4rfcatr#r$r6s rr8zDynamicLayer.update~sn" ?  $ $Z > > >Ity*52>>> il ;DDD y$+%%r r9c<d}||z}||fS)zDReturn the length and offset of the cache, used to generate the maskrr=r&r9 kv_offset kv_lengths rr;zDynamicLayer.get_mask_sizess( ''))L8 )##r cr|jr|jdkrdS|jjdS)1Returns the sequence length of the cached states.rru)r%r#numelshaper,s rr=zDynamicLayer.get_seq_lengths7" dioo&7&71&<&<1yr""r cdS)zeReturns the maximum sequence length of the cache object. DynamicLayer does not have a maximum length.rr,s rr@zDynamicLayer.get_max_lengthrr max_lengthc|dkr$|t|z }||krdS|jdd|ddf|_|jdd|ddf|_dS)z Crop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be negative to remove `max_length` tokens. rN.)r=absr#r$r&rs rcropzDynamicLayer.crops ??,,..Z@J    J . . FIc;J;12 k#{ {AAA"56 r repeatsc|dkrD|j|d|_|j|d|_dSdS)z8Repeat the cache `repeats` times in the batch dimension.rrvN)r=r#repeat_interleaver$r&rs rbatch_repeat_interleavez$DynamicLayer.batch_repeat_interleaves]    1 $ $ 33G3CCDI+77Q7GGDKKK % $r indicesc|dkr*|j|df|_|j|df|_dSdS)z ##J ===&*&A&D&DT[&Q&Q###r cv|js||||xj|jdz c_t j|j|gd}t j|j|gd}|dddd|j dzdddf|_|dddd|j dzdddf|_||fS)rtrurvNr) r%r4rNrrfrxr#r$r)r&r/r0r7rfull_key_statesfull_value_statess rr8z DynamicSlidingWindowLayer.updates" ?  $ $Z > > > *"22"66 )TY $;DDD!It{L&ArJJJ#AAAqqq4+>*>*B*D*Daaa$GH '111t/B.BQ.F.H.H!!!(KL  111r r9c|j|jk}t|j|jz dzd}|r|jdz |z}n |j|z}||fSNReturn the length and offset of the cache, used to generate the attention maskrr)rNrmax)r&r9is_fullr|r}s rr;z(DynamicSlidingWindowLayer.get_mask_sizesse(D,??.1DDqH!LL  >+a/,>II.=I)##r c|jSrrNr,s rr=z(DynamicSlidingWindowLayer.get_seq_length %%r c|jSz+Return the maximum cache shape of the cacherr,s rr@z(DynamicSlidingWindowLayer.get_max_lengths ""r rc||jkrtdt||jjd|_dS)z Crop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be negative to remove `max_length` tokens. zCannot `crop` a `DynamicSlidingWindowLayer` after it has seen more tokens than itssliding window (otherwise some states are lost)ruN)r=r ValueErrorrrr#rrN)r&rrs rrzDynamicSlidingWindowLayer.cropsf    D$7 7 7B   Z   !%!4r )r+r_r`rarrRr'rfrgr4rhr8r;r=r@rrjrks@rrrsg JZsZZZZZZ RelR%,R[_RRRRRR2,26;l2 u|U\) *2222: $3 $5c? $ $ $ $&&&&&##### 5s 5t 5 5 5 5 5 5 5 5 5 5r rceZdZdZfdZdejddfdZdejdejfdZfdZ fd Z dfd Z d ej ddffd Z d eddffd Zdeddffd Zdejddffd ZxZS)DynamicIndexedLayera{ A cache layer that extends `DynamicLayer` with an extra indexer key cache for Dynamic Sparse Attention (DSA) models (e.g. GLM MoE DSA, DeepSeek V32). The main K/V cache stores tensors of shape `[batch_size, num_heads, seq_len, head_dim]` (inherited). The indexer key cache stores a tensor of shape `[batch_size, seq_len, index_head_dim]` (3D, single-head). c dtd|_d|_dSr")rr' indexer_keysis_indexer_initialized)r&rrs rr'zDynamicIndexedLayer.__init__s/ 15,1###r indexer_key_statesr1Nc|j|jc|_|_t jg|j|j|_d|_dSro)rqrJ indexer_dtypeindexer_devicerfrrrrr&rs rlazy_initialization_indexerz/DynamicIndexedLayer.lazy_initialization_indexersH2D2JL^Le/D/!L43EdNabbb&*###r c|js||tj|j|gd|_|jS)a` Update the indexer key cache by concatenation, and return the full indexer keys. Args: indexer_key_states (`torch.Tensor`): New indexer keys, shape `[batch_size, seq_len, index_head_dim]`. Returns: `torch.Tensor`: The full cached indexer keys, shape `[batch_size, total_len, index_head_dim]`. rrv)rrrfrxrrs rupdate_indexerz"DynamicIndexedLayer.update_indexer"sO* A  , ,-? @ @ @!It'8:L&MSTUUU  r ct|jr#|jdd|_dSdS)NrCTrD)rrGrrrFr&rs rrGzDynamicIndexedLayer.offload1sS   & O $ 1 4 4U 4 N ND    O Or ct|jr=|jj|jkr*|j|jd|_dSdSdS)NTrD)rrKrrrJrFrs rrKzDynamicIndexedLayer.prefetch6so   & U4+<+Ct{+R+R $ 1 4 4T[t 4 T TD    U U+R+Rr ct|jr|jdSdSr))rrSrrrOrs rrSzDynamicIndexedLayer.reset;sD    & &   # # % % % % % & &r rTct||jr\|jdkrA|jd||jj|_dSdSdSNr)rrXrrrrVrFrJ)r&rTrs rrXz!DynamicIndexedLayer.reorder_cache@s h'''  & i4+<+B+B+D+Dq+H+H $ 1 > >q(++dN_NfBgBg h hD    i i+H+Hr rcPt||jr|jdkrdS|dkr|n!|jjdt |z }|jjd|kr|jddd|ddf|_dSdS)Nrr)rrrrrrr)r&r effectivers rrzDynamicIndexedLayer.cropEs  Z   * d.?.E.E.G.G1.L.L F",//JJt7H7Nq7QTWXbTcTc7c   "1 % 1 1 $ 1!!!ZiZ2B CD    2 1r rct||jr@|jdkr%|j|d|_dSdSdS)Nrrv)rrrrrr)r&rrs rrz+DynamicIndexedLayer.batch_repeat_interleaveMsw ''000  & T4+<+B+B+D+Dq+H+H $ 1 C CGQR C S SD    T T+H+Hr rct||jr3|jdkr|j|df|_dSdSdS)Nr.)rrrrr)r&rrs rrz(DynamicIndexedLayer.batch_select_indicesRsl $$W---  & @4+<+B+B+D+Dq+H+H $ 1'3, ?D    @ @+H+Hr r^)r+r_r`rar'rfrgrrrGrKrSrirXrRrrrrjrks@rrrs22222 +el+t++++ ! !%, ! ! ! !OOOOO UUUUU &&&&&& ie&6i4iiiiii DsDtDDDDDDTsTtTTTTTT @EL@T@@@@@@@@@@r rceZdZdZdZdZdeffd Zdej dej dd fd Z dej dej de ej ej ffd Z d ede eeffd Z defdZdefdZxZS)ra A static cache layer that stores the key and value states as static tensors of shape `[batch_size, num_heads, max_cache_len), head_dim]`. It lazily allocates its full backing tensors, and then mutates them in-place. Built for `torch.compile` support. Args: max_cache_len (`int`): Maximum number of tokens that can be stored, used for tensor preallocation. TF max_cache_lenc t||_tjdt |_dSr)rr'rrfrrrRrNr&rrrs rr'zStaticLayer.__init__es> *!&as!;!;!;r r/r0r1Nc|j|jc|_|_|jdd\|_|_|jd|_|jd|_tj|j|j|j |jf|j|j|_ tj|j|j|j |jf|j|j|_ |j |j|_ tsltj|j tj|j tj|j d|_dS)a Lazy initialization of the keys and values tensors. This allows to get all properties (dtype, device, num_heads in case of TP etc...) at runtime directly, which is extremely practical as it avoids moving devices, dtypes etc later on for each `update` (which could break the static dynamo addresses as well). If this is unwanted, one can call `early_initialization(...)` on the Cache directly, which will call this function ahead-of-time (this is required for `torch.export` for example). It is also required whenever the prefill itself ends up in a compiled region (with chunked prefill for instance). NrrpT)rqrJr batch_size num_heads v_head_dim k_head_dimrfzerosrr#r$rNrFr _dynamomark_static_addressr%r3s rr4zStaticLayer.lazy_initializationksG#-"2J4E DK*4*:2A2*>'&,R0$*2.K _dnd.@$/ R*;    k _dnd.@$/ R*;    "&!7!:!:4;!G!G ()) F M - -di 8 8 8 M - -dk : : : M - -d.D E E E"r c|js||||jd}tj||j|jz}|j| |j d|||j d||n2#t$r%||jdddd|f<||j dddd|f<YnwxYw|j|j fS)rtrurJrN) r%r4rrfarangerJrNadd_r# index_copy_r$NotImplementedError)r&r/r0r7rr}cache_positions rr8zStaticLayer.updates " ?  $ $Z > > >$R( i DDDtG]] ##I... = I ! !!^Z @ @ @ K # #A~| D D D D" = = =.8DIaaaN* +0???%&"""r r/r0r1c|js||||jd}|j}||jk}|xj|z c_|r%|jddkr|jdd}|jdd} tj dgt|j } ||dddd| f<|| dddd| f<|j ||j | |j|jfStj |jddddddddf|fd} tj |jddddddddf|fd} n0||z|jkrk|dkr|} |} ntj |jddddd|ddf|fd} tj |jddddd|ddf|fd} ntj||j |jz} |jd | ||jd | |n2#t"$r%||jdddd| f<||jdddd| f<YnwxYw|j||j|jfS|j | dddd|j dddf|j | dddd|j dddf| | fS) rtrurr)dimsrpNrvrrr)r%r4rrrr#rollr$rfrrrRrJcopy_rxrrNrrr)r&r/r0r7rr}current_lengthrnew_keys new_valuesindexrrrs rr8zStaticSlidingWindowLayer.updates" ?  $ $Z > > >$R( 3 D$66 ""i/"" 0 *#q((9>>"2>66![--br-::  bTT[III(2AAAu%*6 111aaa;' ))) !!*---y$+--#()TYqqq!!!QRR{-CZ,PVX"Y"Y"Y$)It{111aaaQQQ;/G.V\^$_$_$_!! i '$*< < <""",$0!!"')TYqqq!!!_n_aaa7O-PR\,]ce"f"f"f$)It{111aaa.RSRSRS;S/TVb.cik$l$l$l!!#\)DKHHH4KaaN A %%aDDD ''><HHHH& A A A2< !!!QQQ./4@ AAAqqq.0111 A  " ' ' 2 2 29dk) ) 111t/A.A.C.CQQQ(FGHHH +AAAqqq43E2E2G2G,JKLLL 111s8I ,I:9I:r9c|j}|j|jk}t|j|z dzd}|r ||zdz }n|j|z|kr |j|z}n|}||fSr)rrr)r&r9rrr|r}s rr;z'StaticSlidingWindowLayer.get_mask_sizes&s+,0BB2^CaGKK  '&59II  ', 6 G G2\AII'I)##r c|jSr)rr,s rr=z'StaticSlidingWindowLayer.get_seq_length8s ))r cVtd|_dSr)rrSrrs rrSzStaticSlidingWindowLayer.reset<s"  %&"""r )r+r_r`rarrRr'rfrgrhr8r;r=rSrjrks@rrrs  J'c'3'''''' M2,M26;lM2 u|U\) *M2M2M2M2^$3$5c?$$$$$*****'''''''''r rcreZdZdZdeffd ZdejddfdZdejdejfdZ d fd Z xZ S) StaticIndexedLayera A `StaticLayer` with an additional statically-allocated indexer key cache for Dynamic Sparse Attention (DSA) models (e.g. GLM MoE DSA, DeepSeek V32). This is the static, `torch.compile`-friendly counterpart of `DynamicIndexedLayer`: the indexer key buffer is preallocated once and mutated in-place. The main K/V cache is inherited from `StaticLayer` (`[batch_size, num_heads, max_cache_len, head_dim]`). The indexer key cache stores a tensor of shape `[batch_size, max_cache_len, index_head_dim]` (3D, single-head). rc t|d|_d|_t jdt |_dS)NrFrr)rr'rrrfrrrRindexer_cumulative_lengthrs rr'zStaticIndexedLayer.__init__KsM }55515,1#*/as)C)C)C&&&r rr1Nc|j|jc|_|_|j\}}}t j||j|f|j|j|_|j |j|_ tsHt j |jt j |j d|_dSro)rqrJrrrrfrrrrrFr rrr)r&rr_index_head_dims rrz.StaticIndexedLayer.lazy_initialization_indexerSs2D2JL^Le/D/(:(@% A~!K +^ <$&    *.)G)J)J4K^)_)_&')) N M - -d.? @ @ @ M - -d.L M M M&*###r cT|js|||jd}tj||j|jz}|j| |j d||n#t$r||jdd|f<YnwxYw|jS)a. Update the indexer key cache in-place at the current positions, and return the full static buffer. Args: indexer_key_states (`torch.Tensor`): New indexer keys, shape `[batch_size, seq_len, index_head_dim]`. Returns: `torch.Tensor`: The full static indexer key cache, shape `[batch_size, max_cache_len, index_head_dim]`. Unfilled positions are masked out downstream by the indexer's attention mask, exactly as the main `StaticLayer` returns its full preallocated K/V. rrN) rrrrfrrrrrrr)r&rseq_lenrs rrz!StaticIndexedLayer.update_indexerbs* A  , ,-? @ @ @$*1-gd6IJJJTMkk &++G444 F   ) )!^=O P P P P" F F F3ED aaa/ 0 0 0 F  s(BB B ct|jr4|j|jdSdSr))rrSrrrOrrs rrSzStaticIndexedLayer.reset}sY    & 3   # # % % %  * 0 0 2 2 2 2 2 3 3r r^) r+r_r`rarRr'rfrgrrrSrjrks@rrrAsDcDDDDDD +el +t + + + +!!%,!!!!63333333333r rc eZdZdZ ddededed ed ef fd Zd ejd ejdeejejffdZ e dZ e dZ defdZ xZS)QuantizedLayera A quantized layer similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://huggingface.co/papers/2402.02750). It allows the model to generate longer sequence length without allocating too much memory for the key and value caches by applying quantization. The cache has two types of storage, one for original precision and one for the quantized cache. A `residual length` is set as a maximum capacity for the original precision cache. When the length goes beyond maximum capacity, the original precision cache is discarded and moved into the quantized cache. The quantization is done per-channel with a set `q_group_size` for both Keys and Values, in contrast to what was described in the paper. r@nbitsaxis_key axis_value q_group_sizeresidual_lengthct||_||_||_||_||_d|_dSr)rr'rrrrrrNr&rrrrrrs rr'zQuantizedLayer.__init__sN    $(.!"r r/r0r1c|xj|jdz c_|js||||||j|_|||j|_ ||fS| |j}| |j }tj ||j |gd}tj ||j|gd}|j dkr|j jddz|jkr|||j|_|||j|_ tjg|j|j|_ tjg|j|j|_nDtj |j |gd|_ tj |j|gd|_||fS)rtru)axisrvrrrp)rNrr%r4 _quantize contiguousr_quantized_keysr_quantized_values _dequantizerfrxr#r$rwrrrrqrJ) r&r/r0r7r dequant_keysdequant_valueskeys_to_returnvalues_to_returns rr8zQuantizedLayer.updates  *"22"66" ,  $ $Z > > >#'>>*2G2G2I2IPTP]>#^#^D %)^^L4K4K4M4MTXTc^%d%dD "|+ +''(<== ))$*@AAL$)Z#HbQQQ 9ndk<%PVXYYY 9==??a  DIOB$7!$;t?S$S$S#'>>.2K2K2M2MTXTa>#b#bD %)^^4D4O4O4Q4QX\Xg^%h%hD " Rz/? HYZZZDI,r1A*J[\\\DKK 49j"9rBBBDI)T[,$?RHHHDK///r cdSr)r)r&rrrs rrzQuantizedLayer._quantizes'*sr cdSr)r)r&q_tensors rr zQuantizedLayer._dequantizer>r c|jSrrr,s rr=zQuantizedLayer.get_seq_lengthrr rrrrr)r+r_r`rarRr'rfrgrhr8rrr r=rjrks@rrrs  " #### #  #  ###### #0,#06;l#0 u|U\) *#0#0#0#0J**^*((^(&&&&&&&&&r rc LeZdZ d dedededed ef fd Zd Zd ZxZS)QuantoQuantizedLayerrrrrrrrrrct|||||tstdt ddr ddlm}m}m}ntd|j d vrtd |j |j d vrtd |j |j d vrtd |j |j dkr|n||_ ||_dS)NrrrrrzYou need to install optimum-quanto in order to use KV cache quantization with optimum-quanto backend. Please install it via with `pip install optimum-quanto`z0.2.5Trr) MaxOptimizerqint2qint4ziYou need optimum-quanto package version to be greater or equal than 0.2.5 to use `QuantoQuantizedLayer`. )rrzA`nbits` for `quanto` backend has to be one of [`2`, `4`] but got )rrzE`axis_key` for `quanto` backend has to be one of [`0`, `-1`] but got zG`axis_value` for `quanto` backend has to be one of [`0`, `-1`] but got r)rr'r ImportErrorr optimum.quantorrrrrrrqtype optimizer) r&rrrrrrrrrs rr'zQuantoQuantizedLayer.__init__sR !%+    +,, T w4 8 8 8  A A A A A A A A A A A{  :V # #maeakmmnn n = ' 'teierttuu u ?' ) )kZ^Zikk #jAooUU5 %r cddlm}|||j||j\}}|||j||||j}|S)Nr)quantize_weight)rr rrr)r&rrrr scale zeropointqtensors rrzQuantoQuantizedLayer._quantizesX222222>>&$*dDDUVVy!/&$*dE9dN_``r c*|Sr)) dequantize)r&r#s rr z QuantoQuantizedLayer._dequantizes!!###r rr+r_r`rRr'rr rjrks@rrrs" )()()()( )(  )(  )()()()()()(V$$$$$$$r rc LeZdZ d dedededed ef fd Zd Zd ZxZS)HQQQuantizedLayerrrrrrrrrrcft|||||tstd|jdvrt d|j|jdvrt d|j|jdvrt d|jt|_ dS)NrzYou need to install `HQQ` in order to use KV cache quantization with HQQ backend. Please install it via with `pip install hqq`)rrrzM`nbits` for `HQQ` backend has to be one of [`1`, `2`, `3`, `4`, `8`] but got )rrzA`axis_key` for `HQQ` backend has to be one of [`0`, `1`] but got zC`axis_value` for `HQQ` backend has to be one of [`0`, `1`] but got ) rr'rrrrrr HQQQuantizer quantizerrs rr'zHQQQuantizedLayer.__init__s !%+     !! @  :_ , ,l`d`jll  = & &paeanppqq q ?& ( (tcgcrttuu u%r c|j|||jj|jj|j|j\}}|jj|d<|j|||jj|d|j|d<|d|j|d<||fS)N)rrJ compute_dtyper group_sizer/)metarJr!zero) r-quantizer#rJrqrrcudarF)r&rrrr#r1s rrzHQQQuantizedLayer._quantize+s// 9#)/*( 0  !% _ G$ty7GHHHW ((88W F|w~66V }r cF|\}}|j||}|Sr))r-r%)r&r# quant_tensorr1rrs rr zHQQQuantizedLayer._dequantize:s($ d**<>> r rr&rks@rr(r(s" !&!&!&!& !&  !&  !&!&!&!&!&!&F   r r(c :eZdZdZdZdZddefdZdZe dd e j dzd e j dzd ed dfdZ e dd e j d ed e j fdZ e dd e j d ed e j fdZdZdZddZde jfdZdZdefdZd efdZdS)LinearAttentionCacheLayerMixinzABase, abstract class for a linear attention single layer's cache.TFrnumber_of_statesc T||_tt||_tt||_tt|d|_tt|d|_tt|d|_tt||_ d|_ d|_ d|_ dSr") r9dictfromkeysrange conv_statesrecurrent_statesis_conv_states_initializedis_recurrent_states_initializedhas_previous_stateconv_kernel_sizerJrq record_pastr&r9rs rr'z'LinearAttentionCacheLayerMixin.__init__Hs 0;?==O_I`I`;a;a@D eTdNeNe@f@f*.-->N8O8OQV*W*W'/3}}UCS=T=TV[/\/\,"&--6F0G0G"O"O $ e4D.E.E F F   r c|jjSr)r*r,s rr-z'LinearAttentionCacheLayerMixin.__repr__Ur.r Nrr>r? state_idxr1cdSr)r)r&r>r?rGs rr4z2LinearAttentionCacheLayerMixin.lazy_initializationXs sr cdSr)r)r&r>rGs rupdate_conv_statez0LinearAttentionCacheLayerMixin.update_conv_state`s`c`cr cdSr)r)r&r?rGs rupdate_recurrent_statez5LinearAttentionCacheLayerMixin.update_recurrent_statecsjmjmr ct|jD]p}|j|r*|j|dd|j|<|j|r*|j|dd|j|<qdSrB)r=r9r@r>rFrAr?r&is rrGz&LinearAttentionCacheLayerMixin.offloadfst,-- a aA.q1 W&*&6q&9&<&rJrFrAr?rNs rrKz'LinearAttentionCacheLayerMixin.prefetchnst,-- g gA.q1 ]d6Fq6I6PTXT_6_6_&*&6q&9&<&rOrAr?rBrNs rrSz$LinearAttentionCacheLayerMixin.resetvst,-- / /A.q1 , #))+++3A6 1%a(..000).D #A & &  / /r rTclt|jD]}|j|rA|j|d||j|j|<|j|rA|j|d||j|j|<dS)DReorders the cache for beam search, given the selected beam indices.rN) r=r9r@r>rVrFrJrAr?)r&rTrOs rrXz,LinearAttentionCacheLayerMixin.reorder_cachest,-- n nA.q1 d&*&6q&9&F&Fq(++VZVaJbJb&c&c #3A6 n+/+@+C+P+PQRT\T_T_`d`kTlTl+m+m%a(  n nr cd|_dS)a Calling this function will activate past state recording, meaning that a call to `update_conv_states` will wait for a call to `crop` before restricting the size of the `conv_states` to `conv_kernel_size`, to be able to retrieve previous full states. TN)rDr,s ractivate_past_recordingz6LinearAttentionCacheLayerMixin.activate_past_recordings  r tokens_to_removecj|jstd|dkrtdt|jD]r}t |}|dkr,|j|d|j| df|j|<C|j|d| |j|z | f|j|<sdS)Nz`crop` was called, but the current layer does not track past states. Call `activate_past_recording` before `crop` to be able to rollback the cache.rzkLinear attention layers can only be cropped by passing a negative int, to specify how many tokens to remove.)rD RuntimeErrorr=r9rr>rC)r&rVrOs rrz#LinearAttentionCacheLayerMixin.crops ;  a  } t,--  A"#344  1$$&*&6q&9#@UVW@X?X?Z?Z:Z&[ ##&*&6q&9**T-B1-EEIYHYYY' ##  r cdS)Nrrr,s rr@z-LinearAttentionCacheLayerMixin.get_max_lengthrr r)NNrrr^)r+r_r`rarbrcrRr'r-rrfrgr4rJrLrGrKrSrirXrUrr@rr rr8r8@sKKN ! ! ! ! ! !,,,,004 \D( ,-   ^ccU\cccRWR^ccc^cmmu|mPSm\a\hmmm^maaaggg////ne&6nnnn   S*r r8c eZdZ d dejdzdejdzdededzddf dZ d dejdededzdejfd Zd dejdedejfd ZdS)LinearAttentionLayerNrr>r?rGrCr1cL||j|j|jc|_|_| |jdn|}||j|<t jg|jdd|R|j|j|j|<ts1|js*tj |j|d|j |<|`t j ||j |<ts*tj |j |d|j|<dSdS)NrrpT)rJrqrrCrfrr>r rDrrr@ zeros_liker?rA)r&r>r?rGrCs rr4z(LinearAttentionLayer.lazy_initializationsF  "{"*5*;[=O' DK9I8P{044Vf /?D !) ,*/+;+#CRC(;*:;;!'")+++D Y ' ,-- Od6F O 11$2B92MNNN9=D +I 6  '/4/?@P/Q/QD !) ,+-- T 11$2G 2RSSS>BD 0 ; ; ; ( 'r c (|j|s|||||j|st|}d|j|<|js`|jd|j|krD|j||jdz }t jj ||dfd}n#t j |j ||gd}|js7|j | |d|j| dfn ||j |<|S) a Update the linear attention cache in-place, and return the necessary conv states. Args: conv_states (`torch.Tensor`): The new conv states to cache. Returns: `torch.Tensor`: The updated conv states. )r>rGrCTrr)valuerv.N) r@r4rBrDrrCrfnn functionalpadrxr>r)r&r>rGrCrfull_conv_statespadding_lengths rrJz&LinearAttentionLayer.update_conv_statesI.y9 v  $ $ dt $ u u u&y1 ]* 15D #I .# k(8(>r(BTEZ[dEe(e(e!%!6y!ADTDZ[]D^!^#(8#6#:#:;Kn^_M`hi#:#j#j  %y$*:9*E{)SY[\\\  ;  Y ' - -.>sTEZ[dEeDeDgDg?g.h i i i i+;D Y ' r c |j|s||||j|||j|S)z Update the linear attention cache in-place, and return the necessary ssm states. Args: smm_states (`torch.Tensor`): The new ssm states to cache. Returns: `torch.Tensor`: The updated ssm states. )r?rG)rAr4r?r)r&r?rGrs rrLz+LinearAttentionLayer.update_recurrent_states[3I> ]  $ $6FR[ $ \ \ \ i(../?@@@$Y//r )NNrN)rNr[) r+r_r`rfrgrRr4rJrLrr rr]r]s,004'+ CC\D(C ,-C C * C  CCCCD]a& &  <& 47& ORUYz& & & & & P00u|0PS0fkfr000000r r]cbeZdZdZddefdZddZdZd Zdd Z d e j fd Z d eddfdZ dS)$LinearAttentionAndFullAttentionLayerFrr9c rt|t||dSNr9)rmr'r]rEs rr'z-LinearAttentionAndFullAttentionLayer.__init__ s6d###%%d=M%NNNNNr r1Nct|dkr%t|dkrtj|g|Rt|dkr%t|dvrtj|fi|dSdSdSNrr)rrr*)lenrmr4r]r&r7rs rr4z8LinearAttentionAndFullAttentionLayer.lazy_initialization s t99>>c&kkQ..  ,T 9D 9 9 9 9 t99>>c&kkY66 4T D DV D D D D D >66r cnt|t|dSr))rmrGr]r,s rrGz,LinearAttentionAndFullAttentionLayer.offloads0T"""$$T*****r cnt|t|dSr))rmrKr]r,s rrKz-LinearAttentionAndFullAttentionLayer.prefetchs0d###%%d+++++r cnt|t|dSr))r]rSrmr,s rrSz*LinearAttentionAndFullAttentionLayer.resets0""4(((4     r rTcrt||t||dSrSN)r]rXrmrWs rrXz2LinearAttentionAndFullAttentionLayer.reorder_cache"s4**4:::""422222r rcrt||t||dSr))r]rrmrs rrz)LinearAttentionAndFullAttentionLayer.crop's4!!$ 333$ +++++r rZr^)r+r_r`rbrRr'r4rGrKrSrfrirXrrr rririsNOOOOOOEEEE+++,,,!!!!3e&63333 ,s,t,,,,,,r ricZeZdZdZddedefdZddZdd Zd ej fd Z d eddfd Z dS)-LinearAttentionAndSlidingWindowAttentionLayerFrrr9c vt||t||dS)Nrrl)rr'r])r&rr9rs rr'z6LinearAttentionAndSlidingWindowAttentionLayer.__init__0s;!**4*OOO%%d=M%NNNNNr r1Nct|dkr%t|dkrtj|g|Rt|dkr%t|dvrtj|fi|dSdSdSrn)rorr4r]rps rr4zALinearAttentionAndSlidingWindowAttentionLayer.lazy_initialization4s t99>>c&kkQ.. % 9$ F F F F F t99>>c&kkY66 4T D DV D D D D D >66r cnt|t|dSr))r]rSrr,s rrSz3LinearAttentionAndSlidingWindowAttentionLayer.reset=s0""4(((!''-----r rTcrt||t||dSru)r]rXrrWs rrXz;LinearAttentionAndSlidingWindowAttentionLayer.reorder_cacheAs4**4:::!//h?????r rcrt||t||dSr))r]rrrs rrz2LinearAttentionAndSlidingWindowAttentionLayer.cropFs4!!$ 333!&&tZ88888r rZr^) r+r_r`rbrRr'r4rSrfrirXrrr rrxrx,sNOOsOcOOOOEEEE....@e&6@@@@ 9s9t999999r rxcReZdZd dedefdZddZdZd Zdd Zd e j fd Z dS)*LinearAttentionAndStaticFullAttentionLayerrrr9c tt||t||dSrk)rr'r])r&rr9rs rr'z3LinearAttentionAndStaticFullAttentionLayer.__init__Ls8T=111%%d=M%NNNNNr r1Nct|dkr%t|dkrtj|g|Rt|dkr%t|dvrtj|fi|dSdSdSrn)rorr4r]rps rr4z>LinearAttentionAndStaticFullAttentionLayer.lazy_initializationPs t99>>c&kkQ..  +D 84 8 8 8 8 t99>>c&kkY66 4T D DV D D D D D >66r cnt|t|dSr))rrGr]r,s rrGz2LinearAttentionAndStaticFullAttentionLayer.offloadYs0D!!!$$T*****r cnt|t|dSr))rrKr]r,s rrKz3LinearAttentionAndStaticFullAttentionLayer.prefetch]s0T"""%%d+++++r cnt|t|dSr))r]rSrr,s rrSz0LinearAttentionAndStaticFullAttentionLayer.resetas0""4((($r rTcrt||t||dSru)r]rXrrWs rrXz8LinearAttentionAndStaticFullAttentionLayer.reorder_cachees4**4:::!!$11111r rZr^) r+r_r`rRr'r4rGrKrSrfrirXrr rrrKsOOcOSOOOOEEEE+++,,,    2e&6222222r rcJeZdZd dededefdZd dZd d Zd ejfd Z dS)3LinearAttentionAndStaticSlidingWindowAttentionLayerrrrr9c xt|||t||dS)N)rrrl)rr'r])r&rrr9rs rr'z ))$m\j)kkk%%d=M%NNNNNr r1Nct|dkr%t|dkrtj|g|Rt|dkr%t|dvrtj|fi|dSdSdSrn)rorr4r]rps rr4zGLinearAttentionAndStaticSlidingWindowAttentionLayer.lazy_initializationps t99>>c&kkQ.. $ 8 E E E E E t99>>c&kkY66 4T D DV D D D D D >66r cnt|t|dSr))r]rSrr,s rrSz9LinearAttentionAndStaticSlidingWindowAttentionLayer.resetys0""4((( &&t,,,,,r rTcrt||t||dSru)r]rXrrWs rrXzALinearAttentionAndStaticSlidingWindowAttentionLayer.reorder_cache}s4**4::: ..tX>>>>>r rZr^) r+r_r`rRr'r4rSrfrirXrr rrrksOOcO3ORUOOOOEEEE----?e&6??????r r) full_attentionsliding_attentionchunked_attentionconvmoelinear_attentionhybridhybrid_slidingdeepseek_sparse_attentionc eZdZdZ d:deeezdzdeeezdzdedefd Z d Z d Z d;d e d efdZ d;d e d efdZdejdejd e deejejffdZ dd e dzde dzdefd$Zd%e d e dee e ffd&Zd6>q !' 2ff (@%$ ? r$>>^c^h^o^o^q^qD  r rr c0|jjd|jdS)Nz(layers=))rr+rr,s rr-zCache.__repr__s .)AA4;AAAAr c*t|jS)zN This value corresponds to the number of layers in the model. )rorr,s r__len__z Cache.__len__s 4;r layer_idxrcfdt|j|jD} |||ddz}n%#t$r|d}YnwxYwt r|jn#tj |j5|j | ddddS#1swxYwYdS)aG Prefetch the next offloaded layer on its device, starting at `layer_idx` and circling back to the beginning if needed. Linear-attention layers are never offloaded and are skipped, as are sliding layers when `only_non_sliding`. Note that we use a non-default stream for this, to avoid blocking. c&g|] \}}| oo| Srr).0 is_linearrrs r z"Cache.prefetch..s=   % :M C#3#B C   r NT) ziprrrrrrrfr4streamrrK)r&rr is_offloadeds ` rrKzCache.prefetchs:    ),T^T_)M)M    1!L$<$B$B4$H$HHII 1 1 1$**400III 1&I uT ! !ejN_N_`d`tNuNu . . K " + + - - - . . . . . . . . . . . . . . . . . .s# A A,+A,! CCCcf|r |j|s!|j|dSdS)a Offload a given `layer_idx`. If `only_non_sliding` is True, it will offload `layer_idx` only if it is a non-sliding layer. Note that we do it on the default stream, so that we ensure all earlier computation in the layer's `update` methods are finished. N)rrrG)r&rrs rrGz Cache.offloadsC ! -T_Y%? - K " * * , , , , , - -r r/r0r1c|j\t|j|krD|j|t|j|kD|jrZt j|j |j | |dz|j |j|j ||g|Ri|\}}|jr|||j ||fS)a Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. Return: A tuple containing the updated key and value states. Nr)rrorappendrrfr4default_streamrJ wait_streamrrKrr8rG)r&r/r0rr7rr#r$s rr8z Cache.updates $  ( 4dk""i// ""4#@#@#B#BCCCdk""i// ? @ J % %j&7 8 8 D DTEY Z Z Z MM)a-)> ? ? ?4t{9-4Z_PT___X^__ f ? ; LLD$9 : : :V|r rr>rGc t|j|tstd|j|j||fi|}|S)ak Updates the cache with the new `conv_states` for the layer `layer_idx`. Parameters: conv_states (`torch.Tensor`): The new conv states to cache. layer_idx (`int`): The index of the layer to cache the states for. Return: `torch.Tensor`: The updated conv states. ?Cannot call `update_conv_state` on a non-LinearAttention layer!)rQrr8rrJ)r&r>rrGrs rrJzCache.update_conv_statesX"$+i02PQQ `^__ _>dk),>{I``Y_`` r r?c t|j|tstd|j|j||fi|}|S)am Updates the cache with the new `recurrent_states` for the layer `layer_idx`. Parameters: smm_states (`torch.Tensor`): The new ssm states to cache. layer_idx (`int`): The index of the layer to cache the states for. Return: `torch.Tensor`: The updated ssm states. r)rQrr8rrL)r&r?rrGrs rrLzCache.update_recurrent_state5s["$+i02PQQ `^__ _H4;y1HIY[doohnoor rc t|j|ds3td|dt|j|jd|j||S)a Updates the indexer key cache for layer `layer_idx`. Parameters: indexer_key_states (`torch.Tensor`): The new indexer key states to cache, shape `[batch_size, seq_len, index_head_dim]`. layer_idx (`int`): The index of the layer to cache the states for. Return: `torch.Tensor`: The updated indexer key states (full cache). rz&Cannot call `update_indexer` on layer z which is a zY; it has no indexer key cache (expected a `DynamicIndexedLayer` or `StaticIndexedLayer`).)rPrrtyper+r)r&rrs rrzCache.update_indexerKst{9-/?@@ OOO I.//8OOO  {9%445GHHHr rrhead_dimrqrJct|tr|gt|z}t|tr|gt|z}t|t|jkr5t dt|dt|jdt|t|jkr5t dt|dt|jdt |j||D]F\}}}|jr|jrtj ||d|f||} | | | GdS)z Initialize all the layers in advance (it's otherwise lazily initialized on the first `update` call). This is useful for our `export` recipes, as `export` needs everything in advance. z,`num_head` was provided as a list of length z, but the Cache currently has z layersz,`head_dim` was provided as a list of length rrpN) rQrRrorrrrcr%rfrr4) r&rrrrqrJlayerlayer_num_headslayer_head_dimfake_kv_tensors rearly_initializationzCache.early_initialization`s i % % 0" c$ii/I h $ $ . zCII-H y>>S-- - -Gs9~~GGmpquq|m}m}GGG  x==C ,, , ,Gs9~~GGmpquq|m}m}GGG 7:$+yRZ6[6[ F F 2E?N, 0D #[*oq.)YafouvvvN  % %nn E E E E F Fr c|tjkrdStj|tsm|dkrt d|d t fdt tD}n#t$rt dwxYwj|S)z=Returns the sequence length of the cache for the given layer.rz+You called `get_seq_length` on layer index R, but this layer is a LinearAttention layer, which does not track sequence length.c3\K|]&}tj|t"|V'dSr)rQrrridxr&s r z'Cache.get_seq_length..; r rJt{[^O_apDqDq r r r r r r rr z{`get_seq_length` can only be called on Attention layers, and the current Cache seem to only contain LinearAttention layers.) rorrQrrnextr= StopIterationr=r&rs` rr=zCache.get_seq_lengths DK(( ( (1$+i0/BB A~~ 6)666  r r r rc$ii0@0@ r r rrr      .  {9%44666s 5B B&c||t|jkrdS|td|jDS|j|S)a Returns the maximum length of the cache. If `layer_idx` is not provided (default), this returns the maximum accross all layers. Otherwise, return the maximum supported value for the given layer. A value of `-1` means no maximum, or undefined maximum, e.g. for dynamic attention layers that can grow indefinitely, or linear attention layer that do not have a sequence length dimension. Nrc3>K|]}|VdSr))r@rrs rrz'Cache.get_max_length..s.GG%u++--GGGGGGr )rorrr@rs rr@zCache.get_max_lengthsa  Y#dk2B2B%B%B2  GG4;GGGGG G;y)88:: :r c||tjkrdS|Y tfdttdz ddD}nP#t$rt dwxYwt j|tst d|d|1tj|j Sj|j |S) zYReturns whether the LinearAttention layer at index `layer_idx` has previous state or not.NFc3\K|]&}tj|t"|V'dSr))rQrr8rs rrz+Cache.has_previous_state..sO!!!$+c"24RSS!!!!!!!r rrz`has_previous_state` can only be called on LinearAttention layers, and the current Cache seem to only contain Attention layers.z/You called `has_previous_state` on layer index zJ, but this layer is an Attention layer, which does not support calling it.) rorrr=rrrQr8allrBr$)r&rrGs` rrBzCache.has_previous_states5  Y#dk2B2B%B%B5    !!!!$SYY]B;;!!! !    5  DK 24RSS /)///   t{9-@GGIIJJ J{9%8CCs :AA6r9c|tjkr|dfStj|tsm|dkrt d|d t fdt tD}n#t$rt dwxYwj||S)a Return a tuple (kv_length, kv_offset) corresponding to the length and offset that will be returned for the given layer at `layer_idx`. The masks are then prepared according to the given lengths (kv_length, kv_offset) and patterns for each layer. rz+You called `get_mask_sizes` on layer index rc3\K|]&}tj|t"|V'dSr)rrs rrz'Cache.get_mask_sizes..rr z{`get_mask_sizes` can only be called on Attention layers, and the current Cache seem to only contain LinearAttention layers.) rorrQrrrr=rr;r&r9rs` rr;zCache.get_mask_sizess DK(( ( (? "$+i0/BB A~~ 6)666  r r r rc$ii0@0@ r r rrr      .  {9%44\BBBs 5BB(c.||S)zReturns the current offset of the query for the given `layer_idx`. It's always equal to the cache length, i.e. `get_seq_length(layer_idx)`, except for MTP layers. )rrzrs rget_query_offsetzCache.get_query_offsets ""Y"777r ctt|jD]!}|j|"dS)z$Recursively reset all layers tensorsN)r=rorrSrs rrSz Cache.resetsIs4;//00 + +I K " ( ( * * * * + +r rTctt|jD]"}|j||#dS)z!Reorder the cache for beam searchN)r=rorrX)r&rTrs rrXzCache.reorder_cachesKs4;//00 ; ;I K " 0 0 : : : : ; ;r rctt|jD]"}|j||#dS)z"Crop the cache to the given lengthN)r=rorr)r&rrs rrz Cache.cropsKs4;//00 4 4I K " ' ' 3 3 3 3 4 4r rctt|jD]"}|j||#dS)zRepeat and interleave the cacheN)r=rorr)r&rrs rrzCache.batch_repeat_interleavesOs4;//00 D DI K " : :7 C C C C D Dr rctt|jD]"}|j||#dS)zSelect indices from the cacheN)r=rorr)r&rrs rrzCache.batch_select_indicessOs4;//00 A AI K " 7 7 @ @ @ @ A Ar ctt|jD]<}t|j|dr|j|=dS)a Calling this function will activate past state recording, meaning that cache with fixed size such as a linear cache will wait for a call to `crop` before restricting the size of its cached states, in order to be able to retrieve previous full states. rUN)r=rorrPrUrs rrUzCache.activate_past_recording sh s4;//00 A AIt{9-/HII A I&>>@@@ A Ar cd|jD}|sdStt|dkrtd||dS)zReturn the batch size of the cache, or ``-1`` if no layer has been initialized yet (e.g. an all-linear-attention cache queried before the first forward).c<g|]}t|d|jS)r)rPrrs rrz$Cache.batch_size..s*\\\uwul?[?[\%"\\\r rrz0The batch size is not consistent across layers: r)rrosetr)r&r$s rrzCache.batch_sizesb ]\ \\\ 2 s6{{  a  XPVXXYY Yayr crt|jdkrdStd|jDS)z&Return whether the cache is compilablerFc3$K|] }|jV dSr))rbrs rrz'Cache.is_compileable..%s%AAE5'AAAAAAr )rorrr,s rrbzCache.is_compileables= t{  q 5AAT[AAAAAAr c|d|jD}t|dkotd|DS)z,Return whether the cache data is initializedc g|] }|j | Sr)rcrs rrz(Cache.is_initialized..*s NNNEE4MN%NNNr rc3$K|] }|jV dSr))r%rs rrz'Cache.is_initialized..+s%&P&Pu';&P&P&P&P&P&Pr )rror)r&rs rr%zCache.is_initialized'sFONT[NNN6{{QP3&P&P&P&P&P#P#PPr c$d|jDS)z9Return whether the layers of the cache are sliding windowc0g|]}t|ddS)rF)getattrrs rrz$Cache.is_sliding..0s$MMM|U33MMMr rr,s rrzCache.is_sliding-sNMMMMMr c$d|jDS)zReturn whether the layers of the cache are linear attention (Mamba/SSM) layers. Note that layers containing both linear and full attention states will return False by this functioncdg|]-}t|tot|t .Sr)rQr8rirs rrz#Cache.is_linear..6sL    u< = = Lu&JKKK   r rr,s rrzCache.is_linear2s'      r c`td||SrZr[ warning_oncer@rs rr]zCache.get_max_cache_shape<3 {   ""9---r c^td|S)Nzi`max_cache_len` is deprecated, and will be removed in version 5.16. Please use `get_max_length()` insteadrr,s rrzCache.max_cache_lenBs1 w   ""$$$r cDtd|jS)Nzp`max_batch_size` is deprecated, and will be removed in version 5.16. Please use the simpler `batch_size` instead)r[rrr,s rmax_batch_sizezCache.max_batch_sizeIs' ~   r )NNFT)Tr[r))NN).r+r_r`ralistrr8rboolr'r-rrRrKrGrfrgrhr8rJrLrrqrJrr=r@rBr;rrSrirXrrrrUpropertyrrbr%rrr]rrrr rrrsI*QUbf )- rr_'EEFMr#'9W'W"X[_"_r r #' rrrr0BBB   ..#......------- , 6;l OR  u|U\) *    FKL <47DG .PQ  %  9< IL     ,II#IRWR^IIII*#F#Fc?#FS / #F { #F  #F#F#F#FJ777C77770;;d ;c;;;; DDC$JD#PT*D`dDDDD<C3C3C5c?CCCC<88#8c8888+++ ;e&6;;;; 4s4444 DsDDDD AELAAAA AAA C   X BBBBXBQQQQXQ NDJNNNXN 4:   X ..S..... %s%%%X% Xr rconfigr1c6t|dd}|~t|dddt|jD}nNt|dddt|jD}ndt|jD}t|dd}||d kr|d|j }i}d |vsd |vr |j|d<d |vr |j|d<d |vsd|vr||d<t d|Drt|dd|d<||fS)z From a `config`, extract the layer types if not present already, as well as the kwargs needed to initialize the corresponding layer caches. layer_typesNrcg|]}dS)rrrrs rrz.get_layer_types_and_kwargs..ZXXX1.XXXr attention_chunk_sizecg|]}dS)rrrs rrz.get_layer_types_and_kwargs..\rr cg|]}dS)rrrs rrz.get_layer_types_and_kwargs..^sUUU+UUUr num_kv_shared_layersrrrrheavily_compressed_attentioncompressed_sparse_attentionrc3K|]}|dvV dS))rrrrNr)r layer_types rrz-get_layer_types_and_kwargs..os) p pV`:Q Q p p p p p pr number_of_conv_statesrr9)rr=num_hidden_layersrrrany)rrr layer_kwargss rget_layer_types_and_kwargsrQs &-66K 6+T 2 2 >XXf>V8W8WXXXKK V3T : : FXXf>V8W8WXXXKKUUU6;S5T5TUUUK#6+A4HH',@1,D,D!"@V%@$@"@A Lk))-=-L-L)/)> %&k)))/)D %&%448UYd8d8d!' X p pdo p p pppW+26;RTU+V+V '(  $$r c |eZdZdZ d deeejdzdfdzdedzde de ffd Z d Z xZ S) DynamicCachea* A cache that grows dynamically as more tokens are generated. This is the default for generative models. It stores the key and value states as a list of `CacheLayer`, one for each layer. The expected shape for each tensor in the `CacheLayer`s is `[batch_size, num_heads, seq_len, head_dim]`. If a config is passed, it will additionally check for sliding or hybrid cache structure, greatly reducing the memory requirement of the cached tensors to `[batch_size, num_heads, min(seq_len, sliding_window), head_dim]`. See `Cache` for details on common methods that are implemented by all cache classes. Args: ddp_cache_data (`Iterable[tuple[torch.Tensor, torch.Tensor]]`, *optional*): It was originally added for compatibility with `torch.distributed` (DDP). In a nutshell, it is `map(gather_map, zip(*caches))`, i.e. each item in the iterable contains the key and value states for a layer gathered across replicas by torch.distributed (shape=[global batch size, num_heads, seq_len, head_dim]). Note: it needs to be the 1st arg as well to work correctly config (`PreTrainedConfig`, *optional*): The config of the model for which this Cache will be used. If passed, it will be used to check for sliding or hybrid layer structure, greatly reducing the memory requirement of the cached tensors to `[batch_size, num_heads, min(seq_len, sliding_window), head_dim]`. offloading (`bool`, *optional*, defaults to `False`): Whether to perform offloading of the layers to `cpu`, to save GPU memory. offload_only_non_sliding (`bool`, *optional*, defaults to `False`): If `offloading` is `True`, this further decides if only the non-sliding layers will be offloaded (because usually the sliding layers are small in size, so there is no need to offload them, and skipping it is faster). Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> past_key_values = DynamicCache(config=model.config) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation ``` NFddp_cache_data.rrrc g}|6|d}t|\} fd|D}|t|D]\}} |~t| dkr| dnd} | >| d} |t | n!|t||| d| d\} } t|dkr+t t|| dSt ||| dS) NTdecoderc4g|]}t|diSr)rrrrs rrz)DynamicCache.__init__..s.kkkQ[0<LL|LLkkkr r*rrrr)rrrrrr) get_text_configr enumerateroitemrrrmr8rr')r&rrrrrdecoder_configrrkv_and_optional_slidingsliding_window_tensorrrrrs @rr'zDynamicCache.__init__s  #33D3AAN(B>(R(R %Kkkkk_jkkkF  %6?6O6O h h2 2>KNNeJfJfjkJkJk,CA,F,Fqu),8)>q)A)F)F)H)H &?~&^&^&^____ lnn555i(//0G0JLcdeLfgg11 v;;!   GG  )5%)A       GG  Fz\t  u u u u ur c#^K|jD]"}|j|jt|ddfV#dS)Nr)rr#r$r)r&rs r__iter__zDynamicCache.__iter__sM[ [ [E*elGE;SUY,Z,ZZ Z Z Z Z [ [r )NNFF) r+r_r`rarrhrfrgrrr'rrjrks@rrrus((XLP*. ). *v*v u|d':C'?!@ADH*v!4'*v *v #' *v*v*v*v*v*vX[[[[[[[r rc :eZdZdZ d dedededeffd ZxZS) StaticCachea Static Cache class to be used with `torch.compile(model)` and `torch.export()`. It will check the `config` for potential hybrid cache structure, and initialize each layer accordingly. See `Cache` for details on common methods that are implemented by all cache classes. Args: config (`PreTrainedConfig`): The config of the model for which this Cache will be used. It will be used to check for sliding or hybrid layer structure, and initialize each layer accordingly. max_cache_len (`int`): The maximum number of tokens that this Cache should hold. offloading (`bool`, *optional*, defaults to `False`): Whether to perform offloading of the layers to `cpu`, to save GPU memory. offload_only_non_sliding (`bool`, *optional*, defaults to `True`): If `offloading` is `True`, this further decides if only the non-sliding layers will be offloaded (because usually the sliding layers are small in size, so there is no need to offload them, and skipping it is faster). Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, StaticCache >>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf") >>> inputs = tokenizer(text="My name is Llama", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = StaticCache(config=model.config, max_cache_len=max_generated_length) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation StaticCache() ``` FTrrrrc t|d\}|d<fd|D}t|||dS)NTr rc4g|]}t|diSr )rr s rrz(StaticCache.__init__..s-fffJ+J7GG,GGfffr r)rrrr') r&rrrrrrrrrs @rr'zStaticCache.__init__st%?v?U?U^b?U?c?c$d$d! \(5 _%ffffZefff :Xpqqqqqr )FT) r+r_r`rarrRrr'rjrks@rrrs$$V!)- r r  r r r #' r r r r r r r r r rr rcLeZdZdZ ddededed ed ed ed effd ZxZS)QuantizedCachea A quantizer cache similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://huggingface.co/papers/2402.02750). It allows the model to generate longer sequence length without allocating too much memory for keys and values by applying quantization. The cache has two types of storage, one for original precision and one for the quantized cache. A `residual length` is set as a maximum capacity for the original precision cache. When the length goes beyond maximum capacity, the original precision cache is discarded and moved into the quantized cache. The quantization is done per-channel with a set `q_group_size` for both keys and values, in contrast to what was described in the paper. See `Cache` for details on common methods that are implemented by all cache classes. Args: backend (`str`): The quantization backend to use. One of `("quanto", "hqq"). config (`PreTrainedConfig`): The config of the model for which this Cache will be used. nbits (`int`, *optional*, defaults to 4): The number of bits for quantization. axis_key (`int`, *optional*, defaults to 0): The axis on which to quantize the keys. axis_value (`int`, *optional*, defaults to 0): The axis on which to quantize the values. q_group_size (`int`, *optional*, defaults to 64): Quantization is done per-channel according to a set `q_group_size` for both keys and values. residual_length (`int`, *optional*, defaults to 128): Maximum capacity for the original precision cache rrrrbackendrrrrrrc |dkrt n!|dkrt ntd|d|d}t |\}} t |dhz } t | dkrtd |  fd t|jD} t | dS) NquantohqqzUnknown quantization backend ``Tr rrz{`QuantizedCache` is only supported for models with only full attention layers. We found the following invalid layer types: c .g|]}Srr)rrrr layer_classrrrs rrz+QuantizedCache.__init__..@s;    Kx\? S S   r r) rr(rrrrror=rrr')r&rrrrrrrrrinvalid_layer_typesrr#rs ````` @rr'zQuantizedCache.__init__'s. h  .KK   +KKHgHHHII I'''553F;; Q!+..2B1CC " # #a ' '0-00          6344    '''''r r) r+r_r`rardrrRr'rjrks@rrrsD"(((!( (  (  ((((((((((((r rc eZdZdZddZdZdefdZdZdd e de fd Z dd e dzde fd Z d Z d e jfdZdefdZde fdZde fdZde jfdZde d e dee e ffdZedZedefdZdZdd e de fdZdS) EncoderDecoderCachea Base, abstract class for all encoder-decoder caches. Can be used to hold combinations of self-attention and cross-attention caches. See `Cache` for details on common methods that are implemented by all cache classes. Args: caches (`Iterable`): Usually an iterable of length 2, containing 2 `Cache` objects, the first one for self-attention, the second one for cross-attention. Can optionally also be an iterable of length 1, containing a `tuple[tuple[torch.Tensor]]` (usually used for compatibility with torch dp and ddp). Example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM, DynamicCache, EncoderDecoderCache >>> model = AutoModelForCausalLM.from_pretrained("openai/whisper-small") >>> processor = AutoProcessor.from_pretrained("openai/whisper-small") >>> inputs = processor(audio=YOUR-AUDIO, return_tensors="pt") >>> # Prepare cache classes for encoder and decoder and pass it to model's forward >>> self_attention_cache = DynamicCache(config=self.config) >>> cross_attention_cache = DynamicCache(config=self.config) >>> past_key_values = EncoderDecoderCache(self_attention_cache, cross_attention_cache) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation EncoderDecoderCache() ``` r1Nc Nt|dkrgg}}|dD]}t|dkr;||dd||ddPt|dkr;||dd||ddtdt|d|t||_t||_nt|dkrt |dtrt |dts;td t|dd t|d|d|_|d|_ntd t|i|_ tt|jD]5}t|j |dk|j |<6dS) Nrrr*rrz$Expected len(combined_cache_data) = z% to be 4 or 6. combined_cache_data = z;One of the two arguments is not a Cache: type(caches[0]) = z, type(caches[1]) = zExpected 1 or 2 arguments, got )rorrrself_attention_cachecross_attention_cacherQr TypeErrorr is_updatedr=rr=)r&cachesself_attention_cache_datacross_attention_cache_datacombined_cache_datars rr'zEncoderDecoderCache.__init__hsB v;;!  DF'A %'-ay x x#*++q00-445H!5LMMM.556I!""6MNNNN,--22-445H!5LMMM.556I!""6MNNNN$%v5H1I1I%v%v^q%v%vwww(45N(O(OD %)56P)Q)QD & & [[A  fQi// xz&)U7S7S x vDQWXYQZOO v vbfgmnogpbqbq v vwww(.q D %)/D & &Ls6{{LLMM Ms4#=>>?? h hI)-d.H.W.WXa.b.bef.f)g)gDOI & & h hr c#XKt|j|jD] \}}||zV dS)zuReturns tuples of style (self_attn_k, self_attn_v, self_attn_sliding, cross_attn_k, cross_attn_v, cross_attn_sliding)N)rr)r*)r&self_attention_layercross_attention_layers rrzEncoderDecoderCache.__iter__sL;>t?XZ^Zt;u;u ? ? 7 "7&)>> > > > > ? ?r c@|jjd|jd|jdS)Nz(self_attention_cache=z, cross_attention_cache=r)rr+r)r*r,s rr-zEncoderDecoderCache.__repr__s=~& - -d>W - -) - - - r c*t|jS)z Support for backwards-compatible `past_key_values` length, e.g. `len(past_key_values)`. This value corresponds to the number of layers in the model. )ror)r,s rrzEncoderDecoderCache.__len__s 4,---r rrc6|j|S)zYReturns the sequence length of the cached states. A layer index can be optionally passed.)r)r=rs rr=z"EncoderDecoderCache.get_seq_length(77 BBBr c6|j|S)zKReturns the maximum sequence length (i.e. max capacity) of the cache object)r)r@rs rr@z"EncoderDecoderCache.get_max_lengthr7r c|j|j|jD] }d|j|< dSr")r)rSr*r,rs rrSzEncoderDecoderCache.resetsV !''))) "((*** / /I).DOI & & / /r rTcn|j||j|dSru)r)rXr*rWs rrXz!EncoderDecoderCache.reorder_caches6 !//999 "00:::::r methodc t|jtrt|jtsGt d|d|jd|jddS)Nr!z)` is only defined for dynamic cache, got z" for the self attention cache and z for the cross attention cache.)rQr)rr*r+__str__)r&r;s rcheck_dynamic_cachez'EncoderDecoderCache.check_dynamic_caches t0, ? ? 45|DD mFmmTE^EfEfEhEhmm'+'A'I'I'K'Kmmm    r maximum_lengthcx||jj|j|dS)z Crop the past key values up to a new `maximum_length` in terms of tokens. `maximum_length` can also be negative to remove `maximum_length` tokens. This is used in assisted decoding and contrastive search (on the Hub). N)r>rr+r))r&r?s rrzEncoderDecoderCache.crops:   !3444 !&&~66666r rc||jj|j||j|dS)zaRepeat the cache `repeats` times in the batch dimension. Used in contrastive search (on the Hub).N)r>rr+r)r*rs rrz+EncoderDecoderCache.batch_repeat_interleavesP   !=!FGGG !99'BBB "::7CCCCCr rc||jj|j||j|dS)zeOnly keep the `indices` in the batch dimension of the cache. Used in contrastive search (on the Hub).N)r>rr+r)r*rs rrz(EncoderDecoderCache.batch_select_indicessP   !:!CDDD !66w??? "77@@@@@r r9c8|j||Sr))r)r;rs rr;z"EncoderDecoderCache.get_mask_sizess(77 iPPPr c|jjSr))r)rr,s rrzEncoderDecoderCache.is_slidings(33r c|jjSr))r)rbr,s rrbz"EncoderDecoderCache.is_compileables(77r c8|jdSr))r)rUr,s rrUz+EncoderDecoderCache.activate_past_recordings !99;;;;;r c`td||SrZrrs rr]z'EncoderDecoderCache.get_max_cache_shaperr r^r[r))r+r_r`rar'rrdr-rrRr=r@rSrfrirXr>rrrgrrhr;rrrrbrUr]rr rr&r&Gs%@hhhh<???  #    ...CCCCCCCCCCd CcCCCC/// ;e&6;;;; #737777DsDDDD AELAAAA Q3Q3Q5c?QQQQ44X48888X8<<<..S.......r r&cNeZdZddeffd Zdededeeefffd ZxZS)MtpCacherrcV|dz}t||zSNr)rr)r&r mtp_offsetrs rrzMtpCache.get_query_offsets)] ww'' 22Z??r r9r1cf|dz}t||\}}|||zfSrK)rr;)r&r9rrLr}r|rs rr;zMtpCache.get_mask_sizess:] $ww55lINN 9)j000r r[)r+r_r`rRrrhr;rjrks@rrIrIs@@#@@@@@@ 131315c?1111111111r rI)4abcrrcollections.abcrrfconfiguration_utilsrutilsrr r r r r hqq.core.quantizerr,r get_loggerr+r[rrmrrrrrrrr(r8r]rirxrrrrrrhrrdr;rrrrr&SlidingWindowCacherIrr rrUs########$$$$$$ 111111<;;;;;;&?&?RV&W&W&W#  H % %S%S%S%S%S%cS%S%S%lK4K4K4K4K4?K4K4K4\N5N5N5N5N5 N5N5N5bF@F@F@F@F@,F@F@F@Rg"g"g"g"g"/g"g"g"T|'|'|'|'|'{|'|'|'~@3@3@3@3@3@3@3@3FI&I&I&I&I&\I&I&I&X4$4$4$4$4$>4$4$4$n66666666rgggggSgggTX0X0X0X0X09X0X0X0v$,$,$,$,$,+?$,$,$,N999994HJc999>222221E{222@?????:NPh???4#22 ! ,2C!4$"11 ,8I!3"hhhhhhhhV !%'7!%E$s)T/