S\'S(\'S)\*S*\2S+\\2S,\\'S-\*S.\\2S/\\*S0\\RJS1\\3\\*44S2jjr4S?S3\RJS4\RJS5\RJS\\RJS\'S)\*S*\2S6\\RJS+\\2S,\\'S-\*S.\\2S/\\*S7\\RjS8\\RjS9\\'S:\\'S"\\R^S \\4&S;jjr6g)@N)partial)Optional TypedDict)is_flash_attn_2_availableis_flash_attn_3_available#is_flash_attn_greater_or_equal_2_10is_torch_npu_availableloggingcx[5(ag[5(a[5(+$SSKJn U"5$)NFr'is_npu_fa2_top_left_aligned_causal_mask)rrr integrations.npu_flash_attentionrr s c/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/modeling_flash_attention_utils.py!flash_attn_supports_top_left_maskr$s/ "" ""6888Y 2 44cZ[5=(d [5=(d [5$N)rrr rris_flash_attn_availabler0s $ & a*C*E aI_Iaar dropout_p window_size)dropoutsliding_windowimplementationc`[5n[5nUS:XdUcU(aU(dSSKJnJn SSKJnJn Oo[[peUS:Xd UcU(a SSK JnJn OK[5(a SSK J n SS K Jn O/[US S5n[US S5nUbUc[!S US 35eX4XV4$)a Lazy loads the respective flash attention implementations. Return: flash_attn_func: The base flash attention function. flash_attn_varlen_func: The flash attention function supporting variable sequence lengths, e.g. for padding-free training. pad_input: The function to pad inputs into one sequence and returning the respective kwargs. unpad_input: The function to unpad outputs based on the kwargs (from pad_input). flash_attention_2Nr)flash_attn_funcflash_attn_varlen_func) pad_input unpad_inputflash_attention_3r)npu_flash_attn_func)npu_flash_attn_varlen_funcrrzJCould not find the currently requested flash attention implementation at `z^`.Make sure that you request a valid kernel from the hub, e.g. `kernels-community/flash-attn`.)rr flash_attnrrflash_attn.bert_paddingr r! _pad_input _unpad_inputflash_attn_interfacer rr#r$getattr ValueError)ris_fa2is_fa3rrr r!s r _lazy_importsr.Cs' (F & (F,,1GF[aFBB!+\; 0 0^5KPV T T # % % ` n&n6GNO%,^=UW[%\ "%-1H `ao`pqst I JJrc[R"U5Rn[R"[5Rn0nUHn[R XD5nXQ;X5'M [ [US9$)a Depending on the version and kernel some features are not supported. Due to limitations in `torch.compile`, we opt to statically type which (optional) kwarg parameters are supported within `_process_flash_attention_kwargs`. NOTE: While all supported kwargs are marked as `True`, everything else is marked as `False`. This might be confusing for kwargs that we use in any case, e.g. `is_causal`. )supports_mapping)inspect signature parameters_process_flash_attention_kwargs_hf_api_to_flash_mappinggetr)flash_functionflash_parametersprocess_parametersr0paramfa_params r_lazy_define_process_functionr<gsp((8CC **+JKVV#+//=%-%A"$ 2EU VVrc[S[[[[455(a[ U5uqqqq[ c[[5q[[[[4[ 4$)a  Lazy loading flash attention and returning the respective functions + flags back NOTE: For fullgraph, this needs to be called before compile while no fullgraph can can work without preloading. See `_check_and_adjust_attn_implementation` in `modeling_utils`. c3&# UHoSLv M g7frr).0ks r .lazy_import_flash_attention..s PO9O)any _flash_fn_flash_varlen_fn_pad_fn _unpad_fnr._process_flash_kwargs_fnr<)rs rlazy_import_flash_attentionrJ}s] Py*:GYO PPP:G:W7 #Wi '#@AQ#R ') <>V VVrcJUR"S/URSSQ76nX!$)z A local implementation of the PyTorch indexing operation `tensor[indices]` on the first axis, after flattening the first two dimensions of the tensor. This is functionally equivalent to FA2's `index_first_axis` and replaces the need to import it. N)reshapeshape)tensorindicesreshaped_tensors r_index_first_axisrSs+nnR;&,,qr*:;O  ##rcUbX-OUnURS[RS9nURS[RS9n[R"UR 5SS9R 5nUR 5R 5n[R"[R"US[RS9S5n[X5UUUU4$)a unpad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. rLdimdtypeFas_tuplerrr) sumtorchint32nonzeroflattenmaxitemFpadcumsumrS) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchrQmax_seqlen_in_batch cu_seqlenss rr(r(s 3>2I-~I }}5;;}?*..2U[[.ImmI--/%@HHJG*..0557u||$4!5;;OQWXJ -1  rcURSSn[R"X#-/UQ7URURS.6nXU'UR "X#/UQ76$)a pad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) rNdevicerW)rOr\zerosrorWview)rerQbatchseqlenrVoutputs rr'r'sZ   ab !C [[%. hC h 8L8LTaTgTg hF#7O ;;u +s ++rrfreturncXURS[RS9n[R"UR 5SS9R 5nUR 5R 5n[R"[R"US[RS9S5nUUU4$)aA Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. rLrUFrXrrZ) r[r\r]r^r_r`rarbrcrd)rfrirQrkrls r_get_unpad_datarws &))b )DmmN224uEMMOG+..0557u||$4!5;;OQWXJ r query_layer key_layer value_layer query_lengthc[U5upgnURSURS=n :a!USS2SU 2SS2SS24USS2SU 2SS2SS24p!URupp[X5n[X&5nXK:Xa[X5nUnUnUnOhUS:XaJSn[R"U S-[R UR S9nUSSnURS5nOUSS2U*S24nU"X5tnnpnUUUUX4X44$)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. unpad_input_func: The function to use for unpadding the input tensors. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rrLNrWro)rwrOrSr\aranger]rosqueeze)rxryrzrfr{unpad_input_func indices_k cu_seqlens_kmax_seqlen_in_batch_kseq_len batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_q_s r _upad_inputrsER6E^5T2I2q(<((9:JZ[fJwG Y a  $ 6  ris_packed_sequencec U(Gdg[RURS.nUSS2S4nURSS:Xa&[R"UR S540UD6OUR S5n[R"[R"S0UD6URS5R[R5/S5n[R"[R"S0UD6UR S5RS5R[R5/S5n[UR55n[UR55S-nOUR5n[R"UR S5UR[RS9n [R"XS:H[R"UR 5UR[RS945nUnUR!5R5nUR#5nUnXV4Xx44$)a This function returns all the necessary kwargs to call `flash_attn_varlen_func` extracted from position_ids. The `position_ids` can be either packed sequence or the usual padded position ids, for example in inference time. Arguments: position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. is_packed_sequence (`bool`, *optional*, defaults to `True`): Whether the input position ids are a packed sequence or not. Return: (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). r}NrLrrrn)r)r\r]rorOonessizeaddcatrprdtointr`r_r~rPdiffra) position_idsr tensor_kwargslast_position_idsq_len cu_seq_lens_q cu_seq_lens_k max_length_q max_length_krs r#prepare_fa_kwargs_from_position_idsr:s. "'++9L9LM (B/!!"%* JJ|((+ =} ="&&q)   5;;#BM#BELLQRODVDVW\WbWbDc"dfgh  [[ ,m ,.?.C.CA.F.M.Ma.P.S.STYT_T_.` acd 599;' ,0023a7 #++- LL!2!21!5l>Q>QY^YdYde  !+, \..09L9LTYT_T_`  & %))+//1 $((* #  )L+G GGrcURSnXE:HnUR5RSURS5URS55nUR5RSURS5URS55nUR5RSURS5URS55n[ X6S9uupxupXX'U4X44$)a$ This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cumulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cumulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Sequence length of the input queries. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rrL)r)rO contiguousrqrr) querykeyvaluerr{ kv_lengthrrrrrs r_prepare_from_posidsr|sB ! I%2     # #B 2 2 GE ..   CHHRL#((2, ?C     # #B 2 2 GECfD@"]$@\ }= ?[ \\rcP[R"S[5 [XX#5$)NzThe function `_prepare_flash_attention_from_position_ids` in `transformers.modeling_flash_attention_utils` is deprecated and will be removed in a future version. Please use `_prepare_from_posids` instead.)warningswarn FutureWarningr)rrrrs r*_prepare_flash_attention_from_position_idsrs& MM W E @@rcUcg[R"URSURS9UR 5-nUS:H=(a. X - R 5R 5R5$)a Check the position ids whether packed sequences are indicated or not 1. Position ids exist 2. Flattened sequences only are supported 3. Compile-friendly `not (torch.diff(position_ids, dim=-1) >= 0).all()`, i.e. we have multiple increasing sequences Fr)ro)r\r~rOrominabsr[bool)rrincreasing_position_sequencess r_is_packed_sequencersq  \''*<3F3FG,JZJZJ\\" ? ` = LQQSWWY^^``rqr@v target_dtypecU(aiUR[R:XaK[R SUS35 UR U5UR U5UR U5p!nXU4$)aM PEFT usually casts the layer norms in float32 for training stability reasons therefore the input hidden states gets silently casted in float32. Hence, we need cast them back in float16 / bfloat16 just to be sure everything works as expected. This might slowdown training & inference so it is recommended to not cast the LayerNorms! zCasting fp32 inputs back to z for flash-attn compatibility.)rWr\float32logger warning_oncer)rr@rrs rfa_peft_integration_checkrs^5==0:<.Hfgh$$|$add<&8!$$|:La 7Nrc\rSrSr%Sr\\R\S'\\R\S'\\ \S'\\ \S'Sr g) FlashAttentionKwargsia Keyword arguments for Flash Attention with Compile. Attributes: cumulative_seqlens_q (`torch.LongTensor`, *optional*) Gets cumulative sequence length for query state. cumulative_seqlens_k (`torch.LongTensor`, *optional*) Gets cumulative sequence length for key state. max_length_q (`int`, *optional*): Maximum sequence length for query state. max_length_k (`int`, *optional*): Maximum sequence length for key state. cumulative_seqlens_qcumulative_seqlens_krrrN) __name__ __module__ __qualname____firstlineno____doc__rr\ LongTensor__annotations__r__static_attributes__rrrrrs? #5#3#344"5#3#3443-3-rrF)total key_length is_causalr softmax_scaleruse_top_left_masksoftcap deterministics_auxr0c  ,U=(a U=(a US:H(+US.n U S(aX=2.4.1 is enabled. s_aux (`torch.Tensor`, *optional*): Attention sink auxiliary that adds a `bias` to the attention calculation via an additional head. Return: flash_kwargs (`dict`): A dict of kwargs that are requested and supported. r)causalrrrrFLASH_ATTENTION_DETERMINISTIC01rr)osgetenv) r{rrrrrrrrrr0kwargs flash_kwargss rr4r4s\M%6%L<1;L M&L  $$+[! &>+E*Je'5&F ]#(*6MBIIFegj