tj+ddlZddlmZddlmZejeZdZdZ dZ da de fd Z ejd d d ejdejdejdejdejdejdededededededede dejfdZejdZdejdeddfd Zd!Z d)d#ejjdejd$ejd%ejd&ejdzdededzd'ejdzdeejdffd(ZdS)*N)logging)sdpa_attention_forward) attn_implementationc@ttSddlm}|dd}|d\}}}|||pd|rdndd }d D]6}t t ||dstd |d |d 7|atS)awLoad the MSA hub kernel once and verify the expected callables are present. The ``attn_implementation`` string may carry a ``paged|`` prefix and/or an ``@`` pin (e.g. ``kernels-staging/msa@v0``); the build currently lives on the repo's ``v0`` branch. The loaded module is cached in a module-level global so registration happens once, not per call. Nr) get_kernel|@rT)revisionversionallow_all_kernels)sparse_atten_func build_k2q_csrzThe MSA kernel loaded from `z` does not expose a callable `zK`. Make sure you request a compatible build, e.g. `kernels-staging/msa@v0`.) _MSA_KERNEL hub_kernelsrsplit partitioncallablegetattr ImportError)r rrepo_id_revkernelfn_names e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/integrations/msa_attention.pyload_and_register_msa_kernelr$ s''''''!'',,R0G'',,OGQ Z#+s?QttPQei j j jF96677 [w[[V][[[   K ztransformers_msa::sparse_atten) mutates_argsqkvq2k cu_seqlens_q cu_seqlens_ktopk block_sizetotal_k max_seqlen_q max_seqlen_k qheads_per_kvscalingimplreturncDt| }tj|j5||||||| | | \}}|||||||||| | |d|  }dddn #1swxYwY|S)a$Opaque wrapper around the CuTe-DSL CSR build + block-sparse kernel. Registered as a ``torch.library`` custom op so ``torch.compile(fullgraph=True)`` treats the whole CSR-build + attention as a single opaque node (no graph break) and ``reduce-overhead`` CUDA graphs can capture it. The internal ``build_k2q_csr`` output is data-dependent in shape, but it never escapes this op (only the fixed-shape ``[total_q, Hq, D]`` attention output does), so the fake/meta impl below is exact. The op is functional (no input mutation). )r0r2r1 qhead_per_kvT)r,r-r1r2blk_kvcausal softmax_scaleN)r$torchcudadevicerr contiguous)r(r)r*r+r,r-r.r/r0r1r2r3r4r5msa k2q_row_ptr k2q_q_indices attn_outputs r#_msa_sparse_atten_oprD<s2 't , ,C   18 $ $  %(%6%6    %%&&7 & & " ]++   %%%%!,                 4  ! ! # ##sABB Bc*tj|SN)r< empty_like)r(r)r*r+r,r-r.r/r0r1r2r3r4r5s r#_msa_sparse_atten_fakerHus$  A  r%querydropoutc|jjdks.tj|jddkrt d|jdtkrtdtd|j j tkrtdtd|d krtd |j j }|tvrtd td |d dS)aValidate kernel capability, dropout and configured topk once per attention module. Mirrors the flash-attention integration, which checks capability/dropout at model init rather than on every forward. The check is cached on the module so the hot path never re-runs it. There is no SDPA fallback: a sparse layer either runs the MSA kernel or this raises. Serves both prefill (q_len > 1) and single-token decode (q_len == 1) -- decode is just a varlen call with one query slot, so there is no context-length threshold. r=r zMSA block-sparse attention requires an SM100 / Blackwell CUDA device. Select a different `attn_implementation` on unsupported hardware.rz2MSA block-sparse attention only supports head_dim .z4MSA block-sparse attention only supports block_size zYMSA block-sparse attention does not support attention dropout; set `attention_dropout=0`.z1MSA block-sparse attention only supports topk in z, got `z0`. Set `index_topk_blocks` to a supported value.N)r>typer<r=get_device_capability RuntimeErrorshapeMSA_SUPPORTED_HEAD_DIM ValueErrorindexerr/MSA_SUPPORTED_BLOCK_SIZE topk_blocksMSA_SUPPORTED_TOPK)modulerIrJr.s r#_validate_msa_initrZs |F""ej&F&Fu|&T&TUV&W[]&]&] P    {2000gNdggghhh ~ $<<<kPhkkklll#~~tuuu > %D %%% <@R < <[_ < < <   &%r%c|j\}} } } |jd|jd} } | | z}|jdtfdtD}|krE|g|jdd|z Rd}t j||gd}||dd|| z| | }|dd|| z| | }|dd|| z| | }t j d|dz| z| |j tj }|dkrx|v|ddz tj d}t jt j d|j tj |g}n.t j d|dz| z| |j tj }| tj }|dddd| || z}t||||||||| z| | |||jj}||| | | S) Nrrrc3(K|] }|k|V dSrFr&).0tr.s r# z$_sparse_attention..s'BBQT q BBr%)dimr)r>dtype)rRnextrXnew_fullr<cat transposereshaper?aranger>int32tozerospermuterDconfig_attn_implementation)rYrIkeyvaluer4 block_indicesr/cache_positionbsz num_q_headsq_lenhead_dim num_kv_headsk_lenr3 padded_topkpadr(r)r*r,valid_kr-r+rCr.s @r#_sparse_attentionr|s(- %CeX)A, ! %L</M  r "DBBBB"4BBBBBKd$$%T}':3B3'?%TtAS%T%TVXYY =#"6B???  1%%cEk;IITTVVA a##C%KxHHSSUUA 1%%cEk<JJUUWWA<C!Gu#4eAHTYT_```L  axxN.!"%)--ek::BB1EEy%+a "T"T"TV]!^__ |Aa5'8%X]Xcddd   5; ' 'C ++aAq ! ! ) ),e T J J U U W WC'      e    *K   sE; A AAr%rNrYrorpattention_maskrqc ||jddz}|t|||||f||d|St|ddst|||d|_|jj} |d} t||||||| | } | dfS) zn TODO: this opens a door to per-layer attn implementation which is something we might want lalter on. Nrg)rJr4_msa_validatedFTrr) rRrrrZrrUr/getr|) rYrIrorpr}rJr4rqkwargsr/rrrCs r#msa_attention_forwardrs+b/T)% E3~ ?FPW  [a   6+U 3 3%65'222 $*JZZ 011N#FE3w WacqrrK  r%)NrNNN)r<utilsrsdpa_attentionr get_logger__name__loggerrXrVrSrstrr$library custom_opTensorintfloatrD register_fakerHrZr|nnModuletuplerr&r%r#rsa 222222  H % %$ c89KK5$ |5$ |5$ |5$  5$ , 5$ , 5$ 5$5$5$5$5$5$5$ 5$ \5$5$5$LK5$p#$#( el U t    :<B<B<BH+/ )-   HO  <    <  L4'   T\ <$&  5<       r%