tjTJddlZddlZddlZddlmcmZddlmZm Z m Z m Z m Z m Z mZmZddlmZddlmZejeZ d-dejd ejd ejd ejdzd df d Zd.ded efdZd.ded efdZd.ded efdZ d.dejded efdZd.d efdZd.ded efdZ d.ded efdZ!d.ded efdZ"d.ded efdZ#dZ$dejd e%eje&e&ffdZ'dejd efdZ(dejde&d e&d!e&d ejf d"Z)d#Z*dejd$e&d%ed ejfd&Z+d'Z,d(ejfd)Z-d*Z.dejd ejd ejd ejdzd e%ejejejejdzff d+Z/ d/dejd ejd ejd ejdzfd,Z0dS)0N)can_use_cudnn_attentioncan_use_efficient_attentioncan_use_flash_attentioncudnn_sdp_enabledflash_sdp_enabledmath_sdp_enabledmem_efficient_sdp_enabled SDPAParams) SDPBackend) NestedTensorFquerykeyvalue attn_maskreturnc t|tr*t|trt|ts(td|jd|jd|jd|j|jks|j|jkr(td|jd|jd|jd|j|jks|j|jkr(td|jd |jd |jd|d ks0|d ks|d krOtd |d |d|d|j|jks|j|jkr(td|jd|jd|jd|tddS)NzNExpected query, key, and value to be nested tensors, but got query.is_nested: z, key.is_nested: z, and value.is_nested: z instead.zLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: zTExpected query, key, and value to all be at least 3 dimensional, but got query.dim: z , key.dim: z and value.dim: z[Expected query, key, and value to all be ragged on the same dimension, but got ragged dims z, z, and z, respectively.zMasks are not yet supported!) isinstancer ValueError is_nesteddtypedevicedim _ragged_idxtorchbool)rrrr dropout_p is_causalscales Y/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/nested/_internal/sdpa.py_validate_sdpa_inputr#s ul + +  #|,,  %..   ?(- ? ?JM- ? ?$)O ? ? ?    {ci5;%+#=#= 7$)K 7 7>Ai 7 7 %  7 7 7    |sz!!U\U\%A%A 9%*\ 9 9AD 9 9!& 9 9 9    yy{{Q#''))a--599;;?? Yyy{{ Y Y'*wwyy Y YBG))++ Y Y Y    CO++u/@EDU/U/U c% c c), c c@E@Q c c c   7888paramsc|jd}|jd}|jd}||ko||kS)Nr)rsizerr)r%debug q_batch_size k_batch_size v_batch_sizes r"_check_batch_size_nestedr,LsX<$$Q''L:??1%%L<$$Q''L < ' HLL,HHr$c"d}|jd}|jd}|jd}||ko||k}|r|dzdkr||ks!|rtd|||dSdS)NrzFor NestedTensor inputs, Flash attention requires q,k,v to have the same last dimension and to be a multiple of 8 and less than or equal to 256. Got Query.size(-1): %d, Key.size(-1): %d, Value.size(-1): %d instead.FTrr'rrlogwarningr%r(max_sizequery_size_last key_size_lastvalue_size_lastsame_head_dim_sizes r"!_check_head_dim_size_flash_nestedr:YHl''++OJOOB''Ml''++O=(O_-O  q A % %  ( (   KKX     u 4r$c"d}|jd}|jd}|jd}||ko||k}|r|dzdkr||ks!|rtd|||dSdS)Nr/r0rzFor NestedTensor inputs, cuDNN attention requires q,k,v to have the same last dimension and to be a multiple of 8 and less than or equal to 128. Got Query.size(-1): %d, Key.size(-1): %d, Value.size(-1): %d instead.FTr1r4s r"!_check_head_dim_size_cudnn_nestedr>sr;r$param param_namect|tstd|jdkr|rtd|dS|dkr|rtd|dSdS)Nzparam should be a jagged NTr zMFused kernels do not support ragged num_head_dims, %s has a ragged num_heads.FrzAFused kernels do not support seq_len == 0, %s has a seq len of 0.T)rr AssertionErrorrr2r3_get_min_seqlen)r?r@r(s r":_check_for_seq_len_0_and_consistent_head_dim_nested_helperrDs e\ * *<:;;; A   KK_   u !##   KKS   u 4r$c t|||}||kr|dks||kr|dks ||kr+|dkr%|r!td|||||||dSdS)Nr zzBoth fused kernels require query, key and value to have broadcastable %s, got Query %s %d, Key %s %d, Value %s %d instead.FT)maxr2r3)q_sizek_sizev_sizer@r(r5s r"_try_broadcast_param_sizerJs666**H 8  ! h  6Q;; h  6Q;;   KKC   u 4r$cX|jjrt|jd|nd}|sdS|jjrt|jd|nd}|sdS|jjrt|jd|nd}|sdS|jd}|jd}|jd}||ko||k}|sU|jjs|jjs |jjr|rtddSt|||d|SdS) NrTFrrr zFBoth fused kernels do not support training with broadcasted NT inputs.z num heads) rrrDrrr' requires_gradr2r3rJ) r%r( q_is_safe k_is_safe v_is_safe q_num_heads k_num_heads v_num_headssame_num_headss r"_check_for_seq_len_0_nestedrTs < ! B L'5     u :  B Ju     u < ! B L'5     u,##A&&K*//!$$K,##A&&K K/NK;4NN   L & z' |)    \5( k;    4r$cVtttf}|D]}|||sdSdSNFT)r,r:rTr%r( constraints constraints r"_can_use_flash_sdpa_jaggedrZsH )#K " z&%(( 55  4r$cJttf}|D]}|||sdSdSrV)r,rTrWs r"_can_use_efficient_sdpa_jaggedr\sE #K" z&%(( 55  4r$c|jddrZ|jddr-|jdds|rt ddS|jr|rt ddSdS)Nr zGIf inputs are nested tensors they must be contiguous after transposing.FzENested tensors for query / key are not supported when is_causal=True.T)r transpose is_contiguousrrr2r3r )r%r(s r"_can_use_math_sdpa_jaggedras L " "1a ( ( 6 6 8 8 z##Aq))7799 |%%a++99;;    KKY   u    KKW   u 4r$c ts6ts(tsts tjStjtjtjtj f}t|||||||}|D]} | tj krt|rtj cS| tjkr,t|rt|rtjcS| tjkr,t|rt|rtjcS| tjkr+trt!|rtjcSt"dt|dt|dt"dt|dt|dt"dt!|dt"dt|dtjS)Nz)Memory efficient kernel not used because:T)r(z(Flash attention kernel not used because:z'Math attention kernel not used because:z(cuDNN attention kernel not used because:)rr rrr ERRORFLASH_ATTENTIONEFFICIENT_ATTENTIONMATHCUDNN_ATTENTIONr rrrZrr\rar2r3) rrrrdropoutr enable_gqaorderingr%backends r"_select_sdp_backendrl%sB    )++ !"" "##   "&" HsE9gy* U UF'' j0 0 0&v.. 2!1111 j0 0 0&v.. 23Mf3U3U 2!1111 j4 4 4*622 67U88 6"5555 jo % %!! '&?&G&G '!&&&KK;<<<d3333"66666KK:;;;F$////vT2222KK9:::fD1111KK:;;;F$////  r$qkvc`t|tstd|l|t j|j}| }| j d}n| dt j|j}| }t|d}|||fS)Nz<===nnGoG Q!#IA~~t!*K!""+ & 55  4r$Nnz num_headshead_dimcf|jr|S||||SN)rrtview)r}rrrs r"_view_as_densers3}} ;;sIx 0 00r$c|d}|d}|d}|d}|d}|d}||kr||kr ||kr||kstd|d} |d} |d} |dd} |dd} |dd}t| \}}}t| \}}}| s#t | s| } | s#t | s| } |s#t |s|}t| || | }t| || | }t||| | }| | | | d}||||||||fS)Nrr zz+_is_computing_meta_flops..sI   GGu{/@ @      r$F) rjit is_scriptingrrr_python_dispatch _get_current_dispatch_mode_stackany)rtorch_dispatch_mode_stacks r"_is_computing_meta_flopsrsx 9 ! ! # #  (?(? K ( I I K K "  .       5r$c|jjt|stjs||||fSfd}||||||||fS)a* [Autocasting SDPA for NJT] Normal autocasting doesn't work for NJT+SDPA right now: * NJT intercepts the __torch_function__ call for scaled_dot_product_attention, which happens before we get to any aten ops or dispatcher logic; then the torch_function logic calls into efficient attention or flash attention. So, autocasting on the scaled_dot_product_attention op won't work because we never see that aten op. * If we put autocasting on `_flash_attention_forward`, then we'll get autocasting to run, but the kernel selection logic in torch_function handling (ie. jagged_scaled_dot_product_attention) won't work correctly: the kernel selection logic will run before autocasting, and choose a kernel based on the un-autocasted dtypes; but then autocasting will run and the actual attention computation will happen in a different dtype. An alternative is to just change the backend selection logic for SDPA+NJT to be autocast-aware and rely on autocasting to do the actual conversions for flash attention / efficient attention. However, by manually doing the actual autocast before the backend selection, we ensure that the autocast handling for backend selection doesn't diverge from the autocast handling for the actual dtype conversions. c||Stj}|jjr |j|ks|jtjkr|S||Sr)rget_autocast_dtyperis_floating_pointfloat64rq)r target_dtype device_types r"cvtz_autocast..cvts] 9H/ << * w,&&w%-''HttL!!!r$)rrrris_autocast_enabled)rrrrrrs @r" _autocastrs4,#K&&,e.G .T.T,c5)++ " " " " " 3u::ss3xxUSS^^ ;;r$ct t||||\}}}}t|||||||t|tr*t|trt|tst dddlm}|dkr|dkr|dkr|jdkrtj | | | t|tr| n||||} || | | |j|jS|jp |jp|j} t#|||||||} t%|r t&j} | t&jkr|d} t-|d d } t-|d d }t-|d d }t/||}t1| ||\}}}}}}}}t2jj|||||||||d | \}}}}}||fi|dd }t=|| S| t&jkrt1|||\}}} }}}}}t2jj |!d|!d| !dd|||||tE|| | \}}!}"}#}$}||#dfi|dd S| t&j$krrt1|||\}}} }}}}}t2jj%||| |||||| ||d | \ }}%}&}'}$}(}"}#})||fi|dd S| t&j&kr | }*| }+|j},|j}-|j'd}.|j'd}/d }0|0|}|0|}|0|}t3j(||||||| d}1|1dd ) }1|1*d|.|/}1||1|*|+|,|-dd }1|1StWd)Nz8query, key, and value must all be NestedTensor instancesr)'nested_view_from_values_offsets_lengthsrr )rrr r!)rrzr/r0F)r!r^c|jdd|jddz }tj|dd}|t |d}tjt |}|dd}|S)Nr r/r^r)r) _offsetsrr_rtsplitlistnestedas_nested_tensorr)jagged_layout_ntror_ tensor_list strided_nts r" get_strided_layout_nested_tensorzMjagged_scaled_dot_product_attention..get_strided_layout_nested_tensors&/36F6OPSQSPS6TTG(8!Q??I#**,,224==a2HHK66tK7H7HIIJ#--a33>>@@J r$)rorrzz=No viable backend for scaled_dot_product_attention was found.),rr#rr rB$torch.nested._internal.nested_tensorrrrFscaled_dot_product_attentionrtrpro_maybe_min_seqlen_maybe_max_seqlenrLrlrr rdr'rrrropsaten_flash_attention_forwardr_rre_efficient_attention_forward unsqueezerwsqueezerg_cudnn_attention_forwardrf_size"_scaled_dot_product_attention_mathrrr)2rrrrrr r!riroutputcompute_logsumexpbackend_choicer query_padded key_padded value_paddedog_scalerrrrrrrr attention _logsumexp _philox_seed_philox_offset_debug_attn_maskquery_reshaped key_reshapedvalue_reshaped log_sumexpseedoffset max_seqlen_q logsumexp cum_seqlen_q cum_seqlen_kv max_seqlen_kv_rp q_lengthsrrzd1d2rattn_outs2 r"#jagged_scaled_dot_product_attentionrsC$-UC #J#J E3yUIy)USSS 5,''Y sL ) )Y ul + +Y WXXX yy{{Q37799q==UYY[[1__ARVWAWAW/ LLNN JJLL LLNN&0L&I&IX   """y    76  MMOO MMOO..     +Ws/@WEDW( sE9iJN &&4 $3333**R..$UAu55 "3511 $UAu55 #E511 '|Z N N !  ! ( )    IN 3 3 !  ! ( )      4      <;      )Aq// *)W=== :9 9 9 'uc5 9 9    ( )    IN 7 7  $ $Q ' '  " "1 % %  $ $Q ' '  ( )     NN  8        76   a      )Aq//  :5 5 5 'uc5 9 9    ( )    IN 3 3     ( )       4           76      )Aq//  :? * *--//MMOO , , [^ [_   1077..s330077; 3y)Ye     %%a++6688??AA==R,,::  !!    )Aq//  K   r$)NrFN)F)NrFNF)1loggingrtorch.nntorch.nn.functionalrrrtorch.backends.cudarrrrrrr r torch.nn.attentionr nested_tensorr getLogger__name__r2Tensorr#rr,r:r>strrDrJrTrZr\rarltuplerwr|rrrrrrrrrr$r"rs                     *)))))''''''g!!&* 00 <0 0 <0|d" 0 0000f I IZ I I I I Ij$4j$616 <%( 6RV.88 8D8888v  z 4    :tj$(,,,^11% cSV@V:W11116elt@1 L1"1/21>A1 \1111lGGGT L*-6: \& EL   +< <+< +< <+<|d" +<  5<u|U\D5H HI +<+<+<+