tj dddlZddlmZmZddlmZejeZeZ dej dej j dej fdZ ddej j dej d ej d ej d ej dzd ededzdedzdedzdedzdej dzdeej dffdZdS)N)_flash_attention_forward!flash_attn_supports_top_left_mask)loggingquerymodulereturncD|jtjkr|jj}tj|rtj|St|jdr |jjStd| Dj jSdS)ziIf the query is in float32, return a target dtype compatible with flash attention. Return None otherwise. _is_quantizedc3XK|]%}t|tjj!|V&dS)N) isinstancetorchnnLinear).0layers g/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/integrations/flash_attention.py z#get_target_dtype..s8bb%z%QVQYQ`?a?abbbbbbbN) dtyperfloat32devicetypeis_autocast_enabledget_autocast_dtypehasattrconfignextmodulesweight)rr device_types rget_target_dtyper" s {em##l'  $[ 1 1 p+K88 8 V]O 4 4 p=& &bb6>>+;+;bbbbbio o 4rkeyvalueattention_maskdropoutscalingsliding_windowsoftcap is_causals_auxc  T| ddrtd|jd} t d|jDrt d|dd}|dd}|dd}t||} | | n|j} t||||f| | ||||t| |j j t|dr|jnd| | |jndd | }|dfS) Noutput_attentionsFzFlash Attention does not support `output_attentions=True`. Please set your attention to `eager` if you want any of these features.rc3"K|] }|dkV dS)rN)rdims rrz*flash_attention_forward..1s& + +3!8 + + + + + +rzTensor query has shape with a zero dimension. FlashAttention does not support inputs with dim=0. Please check your input shapes or use SDPA instead. layer_idx) query_lengthr+r' softmax_scaler)r*use_top_left_mask target_dtypeattn_implementationr3r,)getlogger warning_onceshapeany ValueError transposer"r+r_use_top_left_maskr_attn_implementationrr3tor)rrr$r%r&r'r(r)r*r+r,kwargsseq_lenr7 attn_outputs rflash_attention_forwardrFswzz%u--  W   k!nG + +u{ + + +++  B   OOAq ! !E --1  C OOAq ! !E$E622L'2 8HI*     %,!"M>&-fk&B&BL&""  HHU[ ! ! !%( )K.  r)r#NNNNN)rmodeling_flash_attention_utilsrrutilsr get_logger__name__r:r@TensorrModulerr"floatintbooltuplerFr0rrrQs~ hhhhhhhh  H % %6688 EL %(/ ek    ( !% !!%CC HOC <C C < C L4' C  CT\C$JCT\Cd{C <$ C 5< CCCCCCr