tjR >UdZddlmZddlZddlZddlmZerddlmZddl m Z ddl m Z ddl mZmZddlZdd lmZd d lmZd gZdad ed<dad ed<e GddZe dhdZ didjdZdkdZdldZdmd+Zdnd2Zdod9Z ed:Z!dpd=Z"dqd?Z# drdsdJZ$ dtdudPZ% dvddQdQdddddRddS dwdVZ&ddQdQdddddWdxdXZ'ddQdQddddddYdydZZ(dddd[dzd^Z) d{dd`d|daZ* d}dd`d~dbZ+dd`ddeZ,ej-dfegdS)z PROTOTYPE! Flash Attention 3 implementation. For fp8: only supports forward pass right now. For fp16/bf16: supports forward and backward pass. ) annotationsN) TYPE_CHECKING)Callable) dataclass)cache) TypeVarTupleUnpack)Library) _registryregister_flash_attention_fa3zCallable | None _FA3_CUDA_FWD _FA3_CUDA_BWDc"eZdZUded<ddZdS) _FA3HandlezLibrary | NonelibraryreturnNonecRd|_tjddS)NF)rtorch_C_set_sdp_use_fa3)selfs U/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/nn/attention/_fa3.pyremovez_FA3Handle.remove*s%  !!%(((((N)rr)__name__ __module__ __qualname____annotations__rrrrr&s6))))))rrdevice torch.devicerintcJtj|\}}|SN)rcudaget_device_capability)r"major_s r_get_device_majorr+0s z//77HE1 Lrflash_attn_interface module_pathstrct|tjdt t S)z Register FA3 flash attention kernels with the PyTorch dispatcher. Args: module_path: Python module path to the FA3 implementation. T)_fa3_import_modulerrrr_fa3_register_kernelsr-s rr r 6s?{### Hd### +-- . ..rrctj|ttjdst d|dttjjdst d|dttjjdst d|dtjjjatjjj a dS)N flash_attn_3zModule 'z' does not expose FA3 kernelsfwdz%' does not expose FA3 forward kernelsbwdz&' does not expose FA3 backward kernels) importlib import_modulehasattrrops RuntimeErrorr4r5rr6rr2s rr0r0Gs K((( 59n - -RPkPPPQQQ 59)5 1 1  I{ I I I    59)5 1 1  J{ J J J   I*.MI*.MMMrr ctddd}|dtd|dtd|dtd|dt d|dt d|d td|d td|S) NatenIMPLCUDAz"_flash_attention_forward.quantizedz-_scaled_dot_product_flash_attention.quantized_flash_attention_forward+_flash_attention_forward_no_dropout_inplace#_scaled_dot_product_flash_attention_flash_attention_backward,_scaled_dot_product_flash_attention_backward) r impl!_fa3_flash_attention_forward_impl4_fa3_scaled_dot_product_flash_attention_forward_impl)_fa3_flash_attention_forward_impl_default4_fa3_flash_attention_forward_no_dropout_inplace_impl<_fa3_scaled_dot_product_flash_attention_forward_impl_default"_fa3_flash_attention_backward_impl5_fa3_scaled_dot_product_flash_attention_backward_impl)libs rr1r1Xs &&& ) )CHH,.OQWHH7< HH"$MvHH5< HH-D HH (*LfUUUHH6= Jrquery torch.Tensortensorstuple[torch.Tensor, ...] dropout_pfloat cum_seq_qtorch.Tensor | None q_descale k_descale v_descale str | Nonec|dkrdStd|DsdStd|DdkrdS|jtjkr |||t jdt||d krd S||d krd Stj sd St|j dkrdSdS)Nzdropout_p must be 0c3$K|] }|jV dSr&)is_cuda.0ts r z,_fa3_common_support_error..s$**Qqy******rzinputs must be CUDA tensorsch|] }|j Sr!)r"r^s r z,_fa3_common_support_error..s & & &AH & & &rr inputs must share devicezWhen using SDPA with fp8, descale tensor should always be used for accurate dequantization. Please use _scaled_dot_product_attention_quantized and provide the descale tensors.zdense query must be 4Dzragged query must be 3DzCUDA not available z#FA3 requires compute capability 9.0) alllendtyper float8_e4m3fnwarningswarn UserWarningdimr' is_availabler+r")rNrPrRrTrVrWrXs r_fa3_common_support_errorrqysC$$ **'*** * *-,, & &g & & &''1,,)) {e)))Y.)2C  +     UYY[[A--''!1!1(( : " " $ $$##&&!++44 4rkeyvaluereturn_debug_maskbool alibi_slopes seqused_kc t |rdS|dS| |jtjkrdS|jsdStjtjtjf t fd|||hDsd Std|||hDdkrd St||||f|||| | } | | d krd S| SdS) Nzreturn_debug_mask must be Falsezalibi_slopes not supportedzseqused_k must be int32zseqused_k must be CUDAc3*K|] }|jvVdSr&rjr_r`supported_dtypess rraz-_fa3_forward_support_error..s+HHqqw**HHHHHHrinputs must be one of ch|] }|j Sr!rzr^s rrcz-_fa3_forward_support_error..s 1 1 1AG 1 1 1rr #all inputs must have the same dtyperdz(query, key, value must be on same device) rjrint32r]rkfloat16bfloat16rhrirq) rNrrrsrRrtrvrwrTrVrWrXerrorr|s @r_fa3_forward_support_errorrs100++ ?ek ) ),,  ,+++U]ENK HHHHUC4GHHH H H;:(8::: 1 1eS%0 1 1 122a7744 % U  E  . . .== 4rgrad_outout logsumexpwindow_size_left int | Nonewindow_size_rightc f |jtjkr dS|jtjkrdStjtjf t fd|||||hDsd Std|||||hDdkrdSt|||||||f||ddd} | | SdS)NzHFA3 backward does not support fp8 - use inference only (torch.no_grad())zlogsumexp dtype must be float32c3*K|] }|jvVdSr&rzr{s rraz._fa3_backward_support_error..s+WWqqw**WWWWWWrr}ch|] }|j Sr!rzr^s rrcz._fa3_backward_support_error..s @ @ @AG @ @ @rr r) rjrrkfloat32rrrhrirq) rrNrrrsrrrRrTrrrr|s @r_fa3_backward_support_errorrs {e))) V  %-''00 u~6 WWWWXuc5RU4VWWW W W;:(8::: @ @hsE3? @ @ @AAQFF44 % 5#uc95     E  4rTs Unpack[Ts]tuple[Unpack[Ts]]c4td|DS)Nc3BK|]}|ddVdS)r N) transposer^s rraz#_transpose_dense..s044qQ""444444r)tuple)rPs r_transpose_densers 44G444 4 44rxcd|-|ddkr|n|S)z2Ensure tensor is contiguous in the last dimension.Nr )stride contiguous)rs r_maybe_contiguousrs,]qxx||q/@/@1<<>>>aGrcu_seq_qcu_seq_kmax_qmax_kscale float | None is_causal block_table num_splits!tuple[torch.Tensor, torch.Tensor]cttdt|}t|}|jtjkrF|ddkr-|ddkr|nt|}t|}t|}t| } t|}tg|||ddd| ||dd| |||ddddd| ||||| | nd| | nddddd|pt jrdnddtj pdR\} }}}| |fS) zF Run the FA3 forward pass by calling the C++ kernel directly. NFA3 not registeredrr rr[T) rr;rrjrrkrr$are_deterministic_algorithms_enabledr_get_sm_carveout_experimental)rNrrrsrrrrrrrrrwrrVrWrXrrqkv cu_seqlens_q cu_seqlens_k softmax_lse out_accumsoftmax_lse_accums r_fa3_run_forwardrs./000%  A#A ;%- - - LL   ! ! LL   ! ! u % % %X..L$X..L!),,I#K00K5B$6 $6 $6 $6 $6 $6 $6 $6 $6 $6 $6 $6 $6 $6 $6 $6 !$6" #$6$ %$6& '$6( )$6* +$6, -$6. /$60 1$62 3$64-8b5$66/:7$68 9$6: ;$6< =$6> ?$6@  F;== DAA1C$6D E$6F ..005AG$6$6$62Ci!2J  &&(( ((rF max_seqlen_q max_seqlen_k deterministic/tuple[torch.Tensor, torch.Tensor, torch.Tensor]cttdt|}|ddkr|n|}|ddkr|n|}|ddkr|n|}t|}t|}t j|}t j|}t j|}t|||||||||||dd|| | | | | d|t jpd|||fS)Nrrr r[r) rr;rrrr empty_likerr)rrNrrrsrrrrrrrrrrrdoutrrrolsedqdkdvs r_fa3_run_backwardrLsi"/000 X & &D#ll2..!33AJJrNNa//SA#ll2..!33A#A I & &C  !  B  !  B  !  B              ..005A-0 r2:rrT rrrrwrvrrcompute_auxiliaryr cum_seq_krc t||||| |||| | | }|td|t|||||||| |||||| | | ||\}}|rntjdtj|j}tjdtj|j}tjd|j|j}nd}d}d}|||||fS)Nz)FA3 flash_attention forward unsupported: )r)rjr"r!r) rr;rrzerosuint64r"emptyrj)rNrrrsrTrrrrRrrtrVrWrXrrrrwrvrrrrrr rng_state philox_offset debug_masks rrFrFs2 '     E NuNNOOO       %HC(KELNNN  Bel5<PPP [%+elKKK    Y z 99r)rrrrwrvrrc Tt||||||||| | dddf | | | ||||d|d \}}}}}|S)NFrrF)rrNrrrsrTrrrrRrrtrrrrwrvrrr*rs rrIrIsr*8        )+! -OAsAq!0 Jr)rrrrwrvrrrc Jt|||||||||| ddd| | | | ||||S)N)rrrrwrvrrrr)rNrrrsrTrrrrRrrtrrrrwrvrrrs rrHrHsZ* -        )+! +   r)rrrrunusedct||||||| ||| }|td|tj}t |||||||||| || ||nd||nd|\}}}|||fS)z0FA3 implementation of _flash_attention_backward.Nz*FA3 flash_attention backward unsupported: r)rr;rrr)rrNrrrsrrrTrrrrRrrrrrrrrrrrs rrKrK,s* (      E OOOPPP>@@M"       ,8b.:JBB" r2:rr[rc 4t|||||ddd||| } | td| t|||\} } } |jtjkr tjn|j}t j||}|dd}| d}| d}t| | | dd|||||| ||||\}}}}}| d}| d}||dd|||||f S)NzFA3 SDPA forward unsupported: rzr r)rrrVrWrX) rr;rrjrrkrrrsizerF)rNrrrsrVrWrXrRrrtrrrrr out_dtypeout_bhsdout_bshd max_q_flash max_k_flashr*rrrrrrs rrGrGgsa '        E CECCDDDuc511GAq! #(+1D"D"D%+IY777H!!!Q''H&&))K&&))K3T      4440AsI}j" JJqMME HHQKKE       rc 4t|||ddd|||| S)Nr)rG)rNrrrsrRrrtrs rrJrJs: @          r philox_seedrct||||||| ddd }|td|t|||||\}}}}}t||||||dd|| | | | | |\}}}t|||\}}}|||fS)zCFA3 implementation of _scaled_dot_product_flash_attention_backward.NzFA3 SDPA backward unsupported: r)rr;rrK)rrNrrrsrrrTrrrrRrrrrr grad_out_tq_tk_tv_tout_trrrdq_outdk_outdv_outs rrLrLs& (%eS)YdD  E DUDDEEE(8%eS(($JS#u4        JBB&.b"b99FFF 66 !!rFA3) register_fn)r"r#rr$)r,)r-r.rr)r-r.rr)rr )rNrOrPrQrRrSrTrUrVrUrWrUrXrUrrY)rNrOrrrOrsrOrRrSrtrurvrUrwrUrTrUrVrUrWrUrXrUrrY)rrOrNrOrrrOrsrOrrOrrOrRrSrTrUrrrrrrY)rPrrr)rrUrrU)NNNNNN)&rNrOrrrOrsrOrrUrrUrr$rr$rrrrurrrrrwrUrrUrVrUrWrUrXrUrrUrrrr)F) rrOrNrOrrrOrsrOrrOrrOrrUrrUrrrrrrrrurr$rr$rrurr)NNN),rNrOrrrOrsrOrTrUrrUrr$rr$rRrSrrurtrurVrUrWrUrXrUrrrr$rr$rwrUrvrUrrUrrUrrurr)$rrOrNrOrrrOrsrOrTrUrrUrr$rr$rRrSrrurtrurrrr$rr$rwrUrvrUrrUrr)$rNrOrrrOrsrOrTrUrrUrr$rr$rRrSrrurtrurrrr$rr$rwrUrvrUrrUrrUrr)"rrOrNrOrrrOrsrOrrOrrOrTrUrrUrr$rr$rRrSrrurrOrrOrrrrrr)NNNr[FF)rNrOrrrOrsrOrVrUrWrUrXrUrRrSrrurtrurr)r[FF)rNrOrrrOrsrOrRrSrrurtrurr)rrOrNrOrrrOrsrOrrOrrOrTrUrrUrr$rr$rRrSrrurrOrrOrr).__doc__ __future__rr7rltypingrcollections.abcr dataclassesr functoolsrtyping_extensionsrr r torch.libraryr r __all__rr rrr+r r0r1rqrrrrrrrrFrIrHrKrGrJrLregister_flash_attention_implr!rrrs#""""" )((((((!!!!!!22222222 !!!!!!#  "& %%%%!% %%%% ))))))) ) ./////"////"B""""J((((V####L\$5555HHHH$ $%)%)%)'+!%N)N)N)N)N)@ 88888L&*%)%)D:%)(,#'+"!/D:D:D:D:D:D:h%)(,'+!'------x%)(,'+#!'++++++|#'$(%888888~&*%)%)#DDDDDDDV# P!2"2"2"2"2"2"j( ';WXXXXXXr