o sÒh˜ ã@s”ddlZddlmZddlmZz eƒrddlmZWn ey$Ynw        ddejj dej dej d ej d ej d ed ej fd d„Z dS)éNé)ÚPagedAttentionCache)Úis_flash_attn_2_available)Úflash_attn_varlen_funcÚmoduleÚqÚkÚvÚattention_maskÚcacheÚreturnc Ksè|j|||jfi| ¤Ž\}}t|ddƒsdn|jdf} | dur"| j}d| vr-d|  d¡ini}|| dd¡ d¡ ¡| dd¡ d¡ ¡| dd¡ d¡ ¡|  t j ¡|  t j ¡  ¡|| f|j d | d œ|¤Ž}t|tƒrp|d}|dfS) aæPerform the forward pass of attention with paged key-value cache. This function handles the cache updates and performs the attention computation using the flash_attn_varlen_func for efficient processing. Args: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. but if there is a block table it can be the full k v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. but if there is a block table it can be the full v cu_seq_lens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seq_lens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. Úsliding_windowF)éÿÿÿÿrrNÚs_auxérT)Ú softmax_scaleÚcausalÚ window_size)ÚupdateÚ layer_idxÚgetattrr rÚgetÚ transposeÚsqueezeÚ contiguousÚtoÚtorchÚint32ÚcloneÚscalingÚ isinstanceÚtuple)rrrr r r Ú cu_seq_lens_qÚ cu_seq_lens_kÚ max_seqlen_qÚ max_seqlen_kÚ block_tablesÚimplementationÚkwargsr rÚ custom_kwargsÚ attn_output©r+úc/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/transformers/integrations/flash_paged.pyÚpaged_attention_forwards.% ùö ô r-)NNNNNNNN) rÚgeneration.continuous_batchingrÚutilsrÚ flash_attnrÚ ExceptionÚnnÚModuleÚTensorr-r+r+r+r,ÚsB   € ÿ ôÿþýüûúò