tj5dZddlmZmZddlmZddlZddlmcm Z ddl m Z m Z mZmZddlmZddlmZmZmZmZgdZejeeje eje ejeGd d eZGd d ejZd efdZd efdZdS)zCDefines bias subclasses that work with scaled_dot_product_attention)autoIntEnum)warnN)can_use_efficient_attentioncan_use_flash_attentionis_flash_attention_available SDPAParams)_raise_kernel_warnings)_calculate_scale_input_requires_grad_postprocess_flash_output_validate_sdpa_input)causal_upper_leftcausal_lower_right CausalVariant CausalBiasc:eZdZdZeZeZdS)ra+ Enum for causal variants used in attention mechanisms. Defines two types of causal biases: ``UPPER_LEFT``: Represents upper-left triangular bias for standard causal attention. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with ``shape=(3,4)``, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] ``LOWER_RIGHT``: Represents lower-right triangular bias, the include values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with ``shape=(3,4)``, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Note that these variants are equivalent to each other when the sequence lengths of the query and key/value tensors are equal since the triangular matrix is square. .. warning:: This enum is a prototype and subject to change. N)__name__ __module__ __qualname____doc__r UPPER_LEFT LOWER_RIGHTU/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/nn/attention/bias.pyrr!s1..`J$&&KKKrrcDeZdZdZdedededdffd Zdejdej fd Z dejdej fd Z ddejdzdej fd Z e ddej dej dej dddedededzdedej fdZedfd ZdefdZxZS)raP A bias representing causal attention patterns. For an overview of the bias structure, see the :class:`CausalVariant` enum. This class is used for defining causal (triangular) attention biases. For constructing the bias, there exist two factory functions: :func:`causal_upper_left` and :func:`causal_lower_right`. Example: .. code-block:: python from torch.nn.attention.bias import causal_lower_right bsz, num_heads, seqlen_q, seqlen_kv, head_dim = 32, 8, 4, 12, 8 # Create a lower-right causal bias attn_bias = causal_lower_right(seqlen_q, seqlen_kv) q = torch.randn( bsz, num_heads, seqlen_q, head_dim, device="cuda", dtype=torch.float16 ) k = torch.randn( bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16 ) v = torch.randn( bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16 ) out = F.scaled_dot_product_attention(q, k, v, attn_bias) .. warning:: This class is a prototype and subject to change. variant seq_len_q seq_len_kvreturnNc:t|ts$tdt|jt ||_||_||_ ||kr#|tj krtdddSdSdS)a Initializes the CausalBias instance with a specified variant and sequence lengths. Args: variant (CausalVariant): The type of causal bias to use (either UPPER_LEFT or LOWER_RIGHT). seq_len_q (int): The sequence length of the query tensor. seq_len_kv (int): The sequence length of the key/value tensor. Raises a warning if the LOWER_RIGHT variant is used with seq_len_q > seq_len_kv, as it may produce NaNs. z%variant must be a CausalVariant, got zTLower right causal bias will produce NaNs in the output when seq_len_q > seq_len_kv!) stacklevelN) isinstancerAssertionErrortypersuper__init__rrr rr)selfrrr __class__s rr)zCausalBias.__init__ws'=11  PW 8NPP   "$ z ! !g1J&J&J f       " !&J&Jrdevicec~tjtj|j|j|tjS)zUpper left causal biasr,dtype)torchtrilonesrr boolr*r,s r _upper_leftzCausalBias._upper_lefts2z Jt~tvUZ X X X   rc|j|jz }tjtj|j|j|tj|S)zLower right causal biasr.)diagonal)r rr0r1r2r3)r*r,diagonal_offsets r _lower_rightzCausalBias._lower_rightsQ/DN:z Jej   %     rc|tjd}|jtjkr||S|jtjkr||SdS)a Materializes the causal bias into a tensor form. Depending on the variant, this method generates either an upper-left or lower-right triangular matrix to represent the causal bias. Args: device (Optional[torch.device]): The device on which to create the tensor. Defaults to CPU. Returns: torch.Tensor: The materialized bias tensor. Ncpu)r0r,rrrr5rr9r4s r _materializezCausalBias._materializesh >\%((F <=3 3 3##F++ + \]6 6 6$$V,, ,7 6rFquerykeyvalue attn_mask dropout_p is_causalscale enable_gqacx|rtd|j|jks|jtjkrt j|||d|d||S|jtjkrr?r@rArBrCrDrE sdpa_params alignment og_head_sizeog_scale needs_paddingpad_lenoutrSs r _dispatchzCausalBias._dispatchsF  OMNN N  9#7 7 7 M$<<<1#%     -"; ; ; UD)YPU V V V$sE4IzK'{33 D"',"3u"<"F 58&C C C 3=$1&11 1ww))$tVDDDrcN|SN)r<__repr__)r*s rrszCausalBias.__repr__0s   ""++---rrr)r=FNF)rN)rrrrrrar)r0r,Tensorr5r9r< staticmethodfloatr3ri classmethodrkstrrs __classcell__)r+s@rrrVs@ #3SW2 %, 5<     5< EL    --5<$#6-%,----( " oo|o \o|o o  o  ot|oo ooo\obEEEEE[E.#........rrr!ct|dkrtd|\}}ttj||S)a& Creates an upper-left triangular causal bias. This function generates a upper-left triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the upper left corner of the matrix. This equivalent to the `is_causal=True` argument in `scaled_dot_product_attention`. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] Args: size: The size of the bias matrix. Returns: CausalBias: The UPPER_LEFT triangular causal bias variant. r#z*causal_upper_left only supports 2D tensors)lenr&rrrrXrr s rrr4sA8 4yyA~~IJJJ Iz m. : F FFrct|dkrtd|\}}ttj||S)a: Creates a lower-right triangular causal bias. This function generates a lower-right triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Args: size: The size of the bias matrix. Returns: CausalBias: The LOWER_RIGHT triangular causal bias variant. r#z+causal_lower_right only supports 2D tensors)r{r&rrrr|s rrrVsA> 4yyA~~JKKK Iz m/J G GGr)renumrrwarningsrr0torch.nn.functionalrYrZrVtorch.backends.cudarrrr torch.nn.attentionr torch.nn.attention._utilsr r r r__all___dynamoallow_in_graphrrtrrrrrrrsII  655555 U T T 9::: 4555 8999 Z(((22222G222j[.[.[.[.[.[.[.[.|G GGGGD"H"H"H"H"H"H"Hr