hEzddlmZddlmZmZmZddlmZmZddl Z ddl m Z ddl m Z ddlmZmZd d lmZmZmZd d lmZmZd e d eeeeeeffdZeGddZGddZy))deque)floorgcdsqrt)OptionalUnionN)PretrainedConfig)GenerationConfig) attach_tracertraced)CacheAllocatorFullAttentionCacheAllocatorSlidingAttentionCacheAllocator)get_device_and_memory_breakdownloggerconfigreturnc$t|dd}|3t|dddnd}t|jDcgc]}|}}i}t|D]\}}|j |g|gz||< t |j Dcgc] }t|c}}g} |jD]7\}}tdt||D]}| j||||z9| D cgc] } || d } } | | fScc}wcc}wcc} w)a Group layers depending on the attention mix, according to VLLM's hybrid allocator rules: - Layers in each group need to have the same type of attention - All groups have the same number of layers For a model with the following layer types: ["sliding", "full", "full", "sliding", "full", "full", "full", "full"] We would get two groups: [0, 3] and [1, 2], [4,5], [6,7]. layer_typesNsliding_windowsliding_attentionfull_attentionr) getattrrangenum_hidden_layers enumerategetrvalueslenitemsappend) rr attn_type_ layer_countsi layer_typeindices group_size layer_groupslg group_typess p/var/www/html/aiagenthome/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/cache.pygroup_layers_by_attn_typer/s=&-6K+26;KT+R+^'dt */0H0H*IJ*IQy*I JL";/ :#/#3#3J#Cqc#I Z 0<3F3F3HI3Hs7|3HIJJL+113 Gq#g, 3A   A N ; <4 41== ";r!u% K=  $$#KJ>s DD.D c:eZdZdZej ddfdededejdejde e e e eeje ffde e d dfd Zed e d ed e fd Zed ed dfdZd e fdZed ede de deee d df dZed ede de deee d df dZed ede de d e ee ffdZedej0dej0de deej0deej0d eej0ej0ff dZy)PagedAttentionCacheu Manages the cache for a paged attention mechanism, inspired by VLLM's hybrid allocator. The cache relies on making groups of layers to reduce the complexity of cache management and fragmentation. The cache uses a three-level hierarchy: - Pages: The smallest unit of cache, a page has a size of [num_heads, head_size], which is the space needed to store the key or value states for one token and one layer. For a model with only full-attention layers, to store the KV cache of one token, we need `2 * num_layers` pages: key and values each take `num_layers` pages. Pages are grouped into blocks: - Blocks: A block is a collection of `block_size` pages, serving as the allocation unit to reduce management complexity and fragmentation. Cache is allocated and freed block by block, not page by page. One block is allocated to one layer group, which only has one attention type, like full-attention or sliding-attention. If all layers in the model have the same attention type, then all layers will be in the same group. There is more than one group if and only if the model has a mixed attention types, like layers with full-attention and layers with sliding-attention. - Cache tensors: The physical supports for the cache. There are as many cache tensors as there are layer in a layer group, and the shape of the cache tensor is `[num_blocks * block_size, num_heads, head_size]`. Grouping layers into groups is useful because when we allocate one block to a group N, the block allocated is the same for all layers in group N, equivalently it is allocated across all cache tensors. This allows us to efficiently allocate and free blocks, and to efficiently read and write key and value states. For instance, imagine we have 8 blocks of cache and a model with two layer groups: a full-attention group with 3 layers and a sliding-attention group with 3 layers. At creation time, the physical cache tensors look like this: cache_tensor_0: □ □ □ □ □ □ □ □ cache_tensor_1: □ □ □ □ □ □ □ □ cache_tensor_2: □ □ □ □ □ □ □ □ where □ means the blocks is not allocated to any layer group yet. We have 3 cache tensors because there are 3 layers per group. We allocate 1 block to each group, after allocation, the cache tensors look like this: cache_tensor_0: ✖ ◉ □ □ □ □ □ □ cache_tensor_1: ✖ ◉ □ □ □ □ □ □ cache_tensor_2: ✖ ◉ □ □ □ □ □ □ where ✖ means the block is allocated to the full-attention group, and ◉ means the block is allocated to the sliding-attention group. Now, if we continue to generate, and the sliding window has been reached, we only need to allocate a new block for the full-attention group, and the cache tensors look like this: cache_tensor_0: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ □ □ □ □ □ And after further generation, when we need a new block allocated: cache_tensor_0: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ ✖ □ □ □ □ This would not have been possible if all layers were in the same group: we would have had to allocate a new block for the sliding-attention group, although it is not needed. Nrgeneration_configdevicedtypelayer_device_maptp_sizerc ||_||_||_t|dd}||n |j|_t|dd}||n|j |jz|_t|dd|_t|\} } t| d} t| |_ i|_ i|_ t| D]N\} } | | dk(r |jnd}t| D]%\}}| |f|j|<||j|<'P|3|dkDr.|j |zdk7rt!d |j d |d |j|j z}t|d dd k(rd}nd| vrdnd}t#|j||j| |j |j$z|}|j't|ddt|ddt|dd|j\}}||_||_t-j.d|j(d|jd|d|j*d| g|_g|_||jzdz|j |jf|_t7| D]}t9j:|j4|j|j}t9j:|j4|j|j}t8j<j?|t8j<j?||j0jA||j2jA|t-j.d|j4d|j0djBd|j0djEtGt7||_$g|_%t| D]q\} }|dk(rtM| |j}n5|dk(r"tO| |j|j}nt!d||jJjA|sy) aInitialize a paged attention cache for efficient memory usage. Args: config: Model configuration generation_config: Generation configuration containing cache parameters device: Device for the cache tensors dtype: Data type of the cache layer_device_map: Optional mapping of layer indices to devices tp_size: Tensor parallelism size num_key_value_headsNhead_dim block_size rrrzNumber of key value heads z+ must be divisible by tensor parallel size .attn_implementationpaged_attentionr: page_size num_groupsr*peak_activation_per_tokennum_attention_masks num_blocksmax_batch_tokens max_memory?)rErFmax_memory_percent cache_dtypez7PagedAttentionCache initialized with self.num_blocks = z, self.block_size = z, page_size = z, self.max_batch_tokens = z num_attention_masks = )r4r3zself.cache_shape = z self.key_cache[0].shape = z self.key_cache[0].numel() = rzInvalid group type: )(rr4r3rnum_attention_headsr8 hidden_sizer9r:r/r!rBsliding_windowslayer_index_to_group_indicesrr ValueErrorPagedAttentionMemoryHandler vocab_size%infer_num_blocks_and_max_batch_tokensrErFrinfo key_cache value_cache cache_shapertorchempty_dynamomark_static_addressr#shapenumelr _free_blocksgroup_cache_managersrr)selfrr2r3r4r5r6kv_headsr9r+r-r*r'grouprjlayerrArDmemory_handlerrErFr%new_layer_key_cachenew_layer_value_cache group_typecms r.__init__zPagedAttentionCache.__init__xsM&   6#8$?4<4HfNhNh 6:t4)1)=X6CUCUY_YsYsCs ""3\2F%>f$E! ka) l+!,.)!,/HAu6A!nH[6[V22abN%e,5<=q611%8.<$$U+-0  7Q;'''1Q6 01I1I0JJuv}u~~AMMD$<$<< 60$ 7;L L"# (;k'I!q 4!'-'9'9FV>VX\XeXefz"A"'++d.>.>djjY]YdYd"e $)KK0@0@ [_[f[f$g ! MM - -.A B MM - -.C D NN ! !"5 6    # #$9 : #  *t''++GT^^A->-D-D,HHf$..YZJ[JaJaJcIghi"% "34:<!&{3MAz--0DOOD223AtH]H]^ #7 |!DEE  % % , ,R 04n_blocks request_idcd}|jD]/}|j|||j}|yt||}1|S)zAllocate cache blocks across all layer groups for a given request. Actual allocation is done by the cache managers, and this method only returns the maximum number of blocks actually allocated across all managers.rN)r^allocate_blocksr]max)r_rkrl max_allocatedrh allocateds r.rnz#PagedAttentionCache.allocate_blockssP ++B**8ZARARSI  y9M , rjc^|jD]}|j||j y)zFree all allocated cache blocks for a given request across all layer groups. Actual deallocation is done by the cache managers.N)r^ free_blocksr])r_rlrhs r.rszPagedAttentionCache.free_blockss(++B NN:t'8'8 9,rjc,t|jS)zHGet the current number of unallocated blocks available for new requests.)r!r])r_s r.get_num_free_blocksz'PagedAttentionCache.get_num_free_blockss4$$%%rj past_length query_length read_indexct|j|D])\}}|j|||}|j|+y)zRetrieve physical cache indices for reading KV states in the cache across all layer groups. This method coordinates with all cache managers to build the complete set of read indices needed for attention computation. N)zipr^get_read_indicesextend)r_rlrvrwrxrh read_indicesr)s r.extend_read_indicesz'PagedAttentionCache.extend_read_indicessC!$D$=$=z J B ))*k<PG    (!Krj write_indexct|j|D])\}}|j|||}|j|+y)zRetrieve physical cache indices for writing new KV states to the cache across all layer groups. This method coordinates with all cache managers to build the complete set of write indices needed to store computed KV states.N)rzr^get_write_indicesr|)r_rlrvrwrrh write_indicesr)s r.extend_write_indicesz(PagedAttentionCache.extend_write_indices sC"%T%>%> !L B **:{LQG   )"Mrjcbi}|jD]}|j|||\}}|||<|S)zRetrieve the key sequence length for the given request_id across all layer types. Returns a dictionary of layer types to their corresponding key sequence lengths.)r^ get_seqlens_k)r_rlrvrw seqlens_krhr$seqlen_ks r.rz!PagedAttentionCache.get_seqlens_ksD ++B"$"2"2:{L"Y Ix#+Ii ,rj key_states value_states layer_idxc |j|\}}||} ||} |j|} |j|} |jddj d}|jddj d}|j |} | dk(r4|| | ddddf<|| | ddddf<| | ddddf}| | ddddf}||fS| dk(}| | ddddf}|||<| | ddddf}|||<|| | ddddf<|| | ddddf<||fS)alUpdate the cache with new key-value states for a specific layer. This method writes new KV states to the appropriate cache locations. The behavior differs based on the layer's attention type: - Full attention: New KV states are written to cache, then complete sequence is read from cache - Sliding window: Old KV is read from cache along with extra spaces for the new KV, then new KV is written to cache. This is because new KV might overwrite the old KV, so we need to read the old KV first. Returns the complete KV states (cached + new) for attention computation. rr?rN)rNrTrU transposesqueezerM)r_rrrrxrkwargs group_idxlayer_idx_in_grouplayer_read_indexlayer_write_indexk_cachev_cacherkey_states_with_cachevalue_states_with_cachemasks r.updatezPagedAttentionCache.updatesv()-(I(I)(T% %%i0' 2..!34""#56))!Q/77: #--a3;;A> --i8 Q /9G%q!+ ,/;G%q!+ ,$+,1.D&E # %&===$r)D$+,1.D&E #,8 #D )/9G%q!+ ,/;G%q!+ ,%&===rj)__name__ __module__ __qualname____doc__rWfloat16r r r3r4rdictintrstrrir rnrsrulistr~rrTensortuplerrjr.r1r1=s26|#]]OS!%n1 n1,n1 n1 {{ n1 #4U3 c3I-J(J#KL n1#n1 n1`        :c:d: : &S& )),/)?B)PTUYZ]U^P_) ) ) **,/*?B*QUVZ[^V_Q`* * * #SUYZ]_bZbUc  3>LL3>ll3> 3> & 3> %,,' 3> u||U\\) *3> 3>rjr1c*eZdZdZej ZejZdZ dZ de de de de de d e d d fd Z e dd ed e fdZd d dej fdee dee d edej$d ee e ff dZdej dfd edej$ded ee e ffdZdej fde d edej$d e fdZdej fde d edej$d e fdZd d ej fdee dee dej$d ee e e ffdZy )rPa[A helper class to determine the best number of pages and maximum number of tokens per batch for the paged attention cache, providing automatic sizing based on available GPU memory. The helper works using the number of pages, which is tied to the number of blocks by: num_blocks = num_pages // block_size The memory footprint consists of three main components: - Cache memory: the space needed to store the cache tensors: 2 * layer_group_size * [num_pages, page_size] * cache_dtype - Activation memory: the space temporarily taken by the largest activation during the model forward pass: peak_activation_per_token * max_tokens_per_batch * activation_dtype_size - Static tensors: the space taken by the input/output buffers and metadata tensors for batch processing, sum of: - inputs_ids + outputs_ids + position_ids + logits_indices: 4 * max_tokens_per_batch * int32_size - attention_mask: num_attention_masks * num_pages * max_tokens_per_batch * activation_dtype_size - cumulative_seqlens_q + cumulative_seqlens_k: (1 + 2) * max_tokens_per_batch * int32_size - write_index_tensor: num_groups * max_tokens_per_batch * int32_size - read_index_tensor: num_groups * (num_pages + max_tokens_per_batch) * int32_size The handler can operate in three modes: 1. Auto-sizing: Determines both number of pages and maximum number of tokens per batch using quadratic optimization 2. Fixed cache: Calculates max batch tokens given a fixed number of pages 3. Fixed batch: Calculates number of pages given a fixed maximum batch size ir:rArBr*rCrDrNcX||_||_||_||_||_||_y)aInitialize the memory handler with the parameters that cannot be automatically inferred. Args: block_size: Size of the cache blocks page_size: Size of the cache pages num_groups: Number of layer groups group_size: Number of layers per layer group peak_activation_per_token: Maximum size of activation tensor per token, = hidden_size + vocab_size num_attention_masks: Number of attention masks, 0 if no attention mask is used, 2 if hybrid model, else 1 Nr@)r_r:rArBr*rCrDs r.riz$PagedAttentionMemoryHandler.__init__us0&%"$$)B rjrIc^t\}}}}|t||z }t||z}|S)aCalculate available GPU memory for cache allocation, accounting for already allocated tensors. This method queries the current memory state and applies the specified percentage limit to determine how much memory can be safely used for the paged attention cache. Args: max_memory_percent: Fraction of available memory to use (0.0-1.0). 1.0 means use all available memory. Returns: int: Available memory in bytes for cache allocation )rror)rIr%totalreservedrqavailable_memorys r.get_available_memoryz0PagedAttentionMemoryHandler.get_available_memorys@)H(I%5(I 3y(#;;/2DDErjrHrErFrJc|||j||\}}n/|||j|||}n|||j|||}|j|}|j |||}||kDrt d|d|||fS)afDetermine optimal number of blocks and maximum number of tokens per batch based on available memory and constraints. Check the class docstring for more details. Naming the number of pages as N and the maximum number of tokens per batch as M, the equation solved is: available_memory = sum([ MN * num_attention_masks * activation_dtype_size, 2N * (layer_group_size * page_size * cache_dtype + 2 * num_group), M * (peak_activation_per_token * activation_dtype + 28 + 4 * num_group), ]) where we already simplified int32_size = 4. )rFrErJzMemory footprint z is more than available memory )'compute_num_blocks_and_max_batch_tokenscompute_max_batch_tokenscompute_num_blocksrcompute_memory_footprint MemoryError)r_rErFrIrJrmemory_footprints r.rRzAPagedAttentionMemoryHandler.infer_num_blocks_and_max_batch_tokenss(  "2":+/+W+W"K, (J( #(8(@#<>1K4H4HH1tK^^ _ Q$0043I3I3R3RRUWWZ[^b^m^mZmm nn M CqfGqfGqfUV!ta!eai' ! I,9JKL LR$|"44Q? q S?P>TUV V+, $//1 44 4 KK=:/)cDD`D`Cde f55J0145 d@@ @ KK.+//o$JlJlIpq r#AA +++rjc|j|}||jz}|}|d|z|j|jz|jzd|j zzzz}|j j||jz|jzz}|dd|j zzz }t||z }||jkDr1tjd|d|j|j}|S)aECalculate maximum batch tokens M given a fixed number of cache blocks. The formula for M is given by: M = (available_memory - 2N * (layer_group_size * page_size * cache_dtype + 2 * num_group)) / (activation_dtype_size * (N * num_attention_masks + peak_activation_per_token) + 28 + 4 * num_group) r?rrrr) rr:r*rArrBrrDrCrrrrS) r_rErIrJrrnumdenumrFs r.rz4PagedAttentionMemoryHandler.compute_max_batch_tokenss001CD 0  q9}$.. @;CWCW WZ[^b^m^mZm mnn&&// 00 043Q3Q Q  a$//))) u- d@@ @ KK.+//o$JlJlIpq r#AA rjcd|j|}|}|||jz|jjzz}||dd|jzzzz}d|j |j z|jzd|jzzz}|||j|jjzzz }|||jjzz }t||z }||jz}||jkDr1tjd|d|j|j}|S)a`Calculate number of cache blocks N given a fixed maximum token per token M. The formula for N is given by: N = (available_memory - M * (peak_activation_per_token * activation_dtype + 28 + 4 * num_group)) / (2 * (layer_group_size * page_size * cache_dtype + 2 * num_group) + M * (num_attention_masks * activation_dtype_size)) rrr?rr) rrCrrrBr*rArDrr:rrrS) r_rFrIrJrrrrrEs r.rz.PagedAttentionMemoryHandler.compute_num_blockss1001CD  $"@"@@4CYCYCbCbbb 2DOO(;#;<<T__t~~5 8L8LLqSWSbSbObbc !T%=%=@V@V@_@_%_`` !D$:$:$C$CCC#+& $//1 44 4 KK=:/)cDD`D`Cde f55Jrjc ||jz}d|jz|z|jz|jz}|j|j jz}||z}d|zdz}|j |j jz}|||zz}d|zdz} |j|zdz} |j||zzdz} t||||| | | g} | S)aCalculate the memory footprint breakdown for a given number of blocks and maximum batch tokens. The memory footprint is given by: available_memory = sum([ MN * num_attention_masks * activation_dtype_size, 2N * (layer_group_size * page_size * cache_dtype + 2 * num_group), M * (peak_activation_per_token * activation_dtype + 28 + 4 * num_group), ]) but is broken down below. r?rr ) r:r*rArrCrrDrBsum) r_rErFrJrcache_memory_footprintactivation_memory_footprint4inputs_outputs_positions_and_logits_memory_footprintattention_memory_footprint#cumulative_seqlens_memory_footprintwrite_index_memory_footprintread_index_memory_footprinttotal_memory_footprints r.rz4PagedAttentionMemoryHandler.compute_memory_footprint2s 0 !"T__!4y!@4>>!QT_ThTh!h&*&D&DtG]G]GfGf&f##'77#?@CS?SVW?W<%)%=%=@V@V@_@_%_""i2B&BB"./2B.BQ.F+'+9I'IA'M$&*ooEU9U&VYZ&Z#!$&+D*3,+  " &%rj)g?)rrrrrWbfloat16rint32 _input_dtyperrrri staticmethodfloatrrrr4rrRrrrrrrjr.rPrPWs0;;L$'!"777 7  7 $' 7!7 74    $%)*.$'#(== (,SM(,#3-(," (, [[ (, sCx (,X%(#(== .,!.,[[.,  ., sCx .,f%(#(==   " [[   >%(#(== "[[  <%)*.#(== ,&SM,&#3-,&[[ ,& sC}  ,&rjrP) collectionsrmathrrrtypingrrrWconfiguration_utilsr generation.configuration_utilsr utils.metricsr r cache_managerrrrrequestsrrrrrrr/r1rPrrjr.rs!!" 3>2ff=%&6%5d3iRVWZR[A[;\%BU>U>U>rG&G&rj