tj[ JddlZddlmZddlmZddlmZddlZddlm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZmZgdZedd ZGd d ejZd ejfd ejdededefdZ GddeZ!GddeZ"d a#dZ$d a%dZ&dS)N) namedtuple)Callable)Any))sparse_semi_structured_from_dense_cutlass'sparse_semi_structured_to_dense_cutlass) fallback_dispatchersemi_sparse_addmmsemi_sparse_clonesemi_sparse_detachsemi_sparse_indicessemi_sparse_linearsemi_sparse_mmsemi_sparse_scaled_mm semi_sparse_tsemi_sparse_tosemi_sparse_to_copysemi_sparse_valuessemi_sparse_view)SparseSemiStructuredTensor!SparseSemiStructuredTensorCUTLASS$SparseSemiStructuredTensorCUSPARSELTto_sparse_semi_structured_SEMI_STRUCTURED_SPARSE_CONFIGz=sparse_min_rows sparse_min_cols dense_min_rows dense_min_colscXeZdZUdZdZeed<eej e fed<dZ e ed<dZ e ed<dZe ed<eed <eeefed <ejd zed <ejd zed <ejd zed<ejd zed<ejd zed<e ed<eed<gdZe d'dejd ejd zd ejd zdejd zdejd zdejd zde dede fdZdefdZdeeeeeje ee fffdZedeeje ee fdejfdZejjZedefdZ ed(d)dZ!edejdd fdZ"d Z#eefdejd!eddfd"Z$d d#d$ejd%ejd zdejfd&Z%d S)*ra This class implements semi-structured sparsity as a Tensor subclass. Semi-structured sparsity describes a sparsity pattern where n in every 2n elements are sparse, depending on the datatype. It is also referred to as 2:4 sparsity or fine-grained structured sparsity. There are two backends available for semi_structred sparsity, either cuSPARSELt or CUTLASS. This class is meant to serve as a base class for both implementations. SparseSemiStructuredCUTLASS and SparseSemiStructuredCUSPARSELT both inherit from this class and define three backend-specific items. Note that as such, this class cannot be instantiated directly. -`_DTYPE_SHAPE_CONSTRAINTS` - A dictionary holding backend specific dense/sparse min shape constraints - `def from_dense()` - backend specific compression routines - `def _mm()` - backend specific mm op (either torch._cslt_sparse_mm or torch._sparse_semi_structured_(mm|addmm)) r_DEFAULT_ALG_ID_DTYPE_SHAPE_CONSTRAINTSF_FORCE_CUTLASS_FUSE_TRANSPOSE_PROTOTYPE_WARNING_SHOWNBACKENDSPARSE_DISPATCHNpackedmetapacked_tmeta_tcompressed_swizzled_bitmaskfuse_transpose_cusparseltalg_id_cusparselt)r"r#r$r%r&shape requires_gradc |jsVtjdtdd|_|t j|||} n||} ntdt j ||| j | j | j | } || _|| _|| _|| _|| _|| _|| _| S)a0 Create a new instance of the tensor subclass from the compressed sparse representation. We have the option to create the subclass with the compressed representations of both X and X', for training. For inference, we only need a single representation (either X or X'), while the corresponding other set will be None. Depending on the backend selected, certain fields will be set to None. (CUSPARSELT vs CUTLASS) Args: shape: The shape of the original dense tensor packed: The compressed representation of the original dense tensor meta: The metadata of the original dense tensor, if it is stored separately packed_t: The compressed representation of the transposed original dense tensor meta_t: The metadata of the transposed original dense tensor, if it is stored separately compressed_swizzled_bitmask: The masks used by the CUTLASS backend to determine which threads should participate in the computation. Used for pointwise ops. fuse_transpose_cusparselt: When running with cuSPARSELt, we have the option to fuse a transposition with a matmul, which is useful in the case of 2:4 sparse training. alg_id_cusparselt: The algorithm id to use when using cuSPARSELT, will have effect on performance Returns: torch.Tensor: A torch.Tensor wrapper subclass. Raises: ValueError: If all of the tensor arguments are None. zThe PyTorch API of SparseSemiStructuredTensor is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.sparse module for further information about the project. stacklevelTNz3At least one of packed or packed_t must be provided)devicedtypelayoutr*)rwarningswarn UserWarning_load_dispatch_tabletorch_dynamoallow_in_graph ValueErrorTensor_make_wrapper_subclassr/r0r1r"r#r$r%r&r'r() clsr)r"r#r$r%r&r'r(r*previous_tensortensors Z/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/sparse/semi_structured.py__new__z"SparseSemiStructuredTensor.__new__OsN+ . MH     ,0C (  $ $ & & & M ( ( - - -  $OO  !&OORSS S44  ")!'")' 5    " -H*+D(#4  returncnt|dstd|jjd|jdS)Nr)ztensor has no shape attributez(shape=))hasattrAssertionError __class____name__r))selfs r?__repr__z#SparseSemiStructuredTensor.__repr__sBtW%% B !@AA A.)??$*????rActtfdj}jjjjf}||fS)Nc(t|duSN)getattr)xrIs r?z?SparseSemiStructuredTensor.__tensor_flatten__..sWT1--T9rA)listfilter __slots__r)r'r(r*)rI inner_tensors tensor_metas` r?__tensor_flatten__z-SparseSemiStructuredTensor.__tensor_flatten__sZ 99994> J J   J  *  "    k))rArUc |\}}}}|||dd|dd|dd|dd|dd||| S)Nr"r#r$r%r& r)r"r#r$r%r&r'r(r*)get) r<rTrU outer_size outer_strider)r'r(r*s r?__tensor_unflatten__z/SparseSemiStructuredTensor.__tensor_unflatten__sNYJ(*;]s $$Xt44""6400"&&z488 $$Xt44(5(9(9-t))'@/'    rAc|j|jvrt|jd|jd|j|j||||S)NzI only supports a specific set of operations, can't perform requested op (rD)_overloadpacketr!NotImplementedErrorrH)r<functypesargskwargss r?__torch_dispatch__z-SparseSemiStructuredTensor.__torch_dispatch__sp  s': : :%<@@/3}@@@ 9s"4#78udFSSSrAct|ddtjjjt tjjjttjjjttjjj ttjjj ttjjj ttjjjt tjjjt$tjjjt$tjjjt*tjjjt.tjjjt2tjjjt6tjjjt:tjjjt>i|_ ||j !|dSdSdS)zT Loads the op overload sparse dispatch table for the current class. r!N)"rNr6opsatenvaluesrindicesr is_same_sizerdetach_detachr trviewrmmrmatmuladdmmr linearr _to_copyr _scaled_mmrcloner torr!update)r<custom_dispatch_tables r?r5z/SparseSemiStructuredTensor._load_dispatch_tables 3)4 0 0 8 %'9 &(; +-@ &(; %'9  - #%5 !> %~ $&7 %'9 ')< )+@ $&7 !>#C "%0#**+@AAAAA' 9 8$10rAoriginal_tensorc .|jstd|jd|dkr%td|d|std|j|jvrtd|jd|d |j\}}|j|jj}|j|jj }||ks||zs ||ks||zrtd |jd |d |d dS)z_ Assert that the given tensor is valid for semi-structured sparse compression. zError original_tensor.device= z= is not supported! Only CUDA tensors are currently supported.r,zError original_tensor.dim = z; is not supported! Only 2d tensors are currently supported.zXError original_tensor is not contiguous!Only contiguous tensors are currently supported.zError original_tensor.dtype z is not a supported dtype for !zError original_tensor.shape zS is not supported! Both dimensions must be larger or equal than and a multiple of (z, rDN) is_cuda RuntimeErrorr/dim is_contiguousr0rr)sparse_min_rowssparse_min_cols)r<rymnmin_rowsmin_colss r? _validate_device_dim_dtype_shapez;SparseSemiStructuredTensor._validate_device_dim_dtype_shapes & =1G===     A % %;/B/B/D/D;;;  ,,.. C   (D D Dj/Djjdgjjj  $1/0EFV/0EFV x<<1x<<1x<<1x<<k/DkkS[kk_gkkk ,8>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> A = torch.Tensor([0, 0, 1, 1]).tile((128, 32)).half().cuda() tensor([[0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], ..., [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.]], device='cuda:0', dtype=torch.float16) >>> A_sparse = to_sparse_semi_structured(A) SparseSemiStructuredTensor(shape=torch.Size([128, 128])) >>> A_sparse.values() tensor([[1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], ..., [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.]], device='cuda:0', dtype=torch.float16), >>> A_sparse.indices() tensor([[-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], ..., [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370]], device='cuda:0', dtype=torch.int16)) zSetting transpose from `to_sparse_semi_structured` is deprecated and will be removed in a future release. `SparseSemiStructuredTensor` only support contiguous input tensors.r,r-)r) r2r3 FutureWarningrrr6sparserrr)ryrrSPARSE_SUBCLASSs r?rr5ssh   R       & 4 ? 66 \ >  % %of % E EErAc peZdZdZdZejeddddejeddddej eddddej eddddiZ e e jfd ejd ed dfd Zfd Ze dd ejd dfdZddddejdejdzded ejfdZxZS)ra This class implements semi-structured sparsity for the CUTLASS backend. In this implementation, the specified elements and metadata are stored separately, in packed and meta respectively. When _FORCE_CUTLASS is set, or when cuSPARSELt is not available, this subclass calls into _sparse_semi_structured_(mm|addmm) and sparse_semi_structured_from_dense for conversion to the compressed format. cutlass @ryrrBc ||t|\}}||j||ddd|jS)Nr"r#r$r%r&r*)rrr)r*)r<ryrsparse_tensor_cutlassmeta_tensor_cutlasss r?rz,SparseSemiStructuredTensorCUTLASS.from_densesf ,,_=== 6o F F ! s  !($(,)7    rAc|j|jtd|jjdkrt |j|jnt S)Nz meta and packed must not be Noner,)r#r"rFndimrsuperr)rIrGs r?rz*SparseSemiStructuredTensorCUTLASS.to_densesj 9  3 !CDD D y~"" 4      !!##  rArc jtj||d\}}}}}||j|||||dS)a This function takes in a unpruned dense tensor and runs a (branchless) static sort across a 4x4 tile. It greedily picks the largest values in the tile, upholding the 2:4 sparsity constraint across both rows and columns. The algorithm used to prune the matrix is implemented in `_sparse_semi_structured_tile`. Then it creates the packed and meta tensors for the compressed sparse representation of the pruned dense tensor. It also calculates the packed_t and meta_t tensors for the compressed sparse representation of the transposed pruned dense tensor. Since we cannot transpose the compressed representations, we store both for the fw/bw pass respectively. Finally, this function also computes a compressed swizzled bitmask that encodes the sparsity pattern. This can be used in the backward pass to mask the gradients. :: [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to CUTLASS semi-structured -> packed [1 2 6 2] [0 0 6 2] -> metadata -> pack to transposed CUTLASS -> packed_t semi-structured representation -> metadata_t -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same five outputs from the dense tensor can be found below:: from torch.sparse import SparseSemiStructuredTensorCUTLASS from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cutlass, meta_cutlass = sparse_semi_structured_from_dense_cutlass( pruned ) packed_t_cutlass, meta_t_cutlass = ( sparse_semi_structured_from_dense_cutlass(pruned.t().contiguous()) ) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUTLASS( dense.shape, packed_cutlass, meta_cutlass, packed_t_cutlass, meta_t_cutlass, bitmask, ) T algorithm use_cutlassFr)r6_sparse_semi_structured_tiler)r<ryrr"r#r$r%r&s r?prune_dense_static_sortz9SparseSemiStructuredTensorCUTLASS.prune_dense_static_sortsh~  . yd        ' s  !(C    rANFrshould_transpose_denserrrc t|trtd|jj}|jdks |jdkrt d|d|j|jt d|dt|j |j }tj j||j|j||jd|j|j|S)NZ`SparseSemiStructuredTensor @ SparseSemiStructuredTensor` is not supported by the hardwarer,`)` matmul: Broadcasting is not implemented$` matmul: operation is not supportedr) isinstancerr9rGrHrr_r"r#_ensure_cutlass_mm_registeredrr0r6rfsemi_structured cutlass_mmr)dense_min_rowsdense_min_cols)rIrrrrccls_name constraintss r?rz%SparseSemiStructuredTensorCUTLASS._mms a3 4 4 l >* 9>>QVq[[%GHGGG  ; $)"3%BHBBB  * + + +7@K9,77   1 **&   rAr)rHrrrr r6int8rfloat16bfloat16float32rrrrr:rrrrrr __classcell__)rGs@r?rr|s  G 222sBCC 55b"aCC 66r2q!DD 55b"aCC  1@     -    [ *      68K K #lK %K K K [K b%)', !!! <!lT! ! !% ! !!!!!!!!rArc `eZdZdZdZejeddddejeddddej eddddej eddddiZ e e jfdejdeddfd Ze ddejdd fd Zd dddejdejd zdedejfdZd S)ra The cuSPARSELt backend expects the specified elements and the metadata to be stored in a single tensor: packed = [ specified elements of original tensor | metadata ] For an original tensor of size (m, k) we expect the first m * k // 2 elements to be the kept elements The rest of the tensor is metadata. Since there is only one tensor, we only use the packed and packed_t attributes respectively. cuSPARSELt also supports transposition fusion, which is necessary for performant 2:4 sparse training, as well as specifying alg_id, a config that affects the performance of the matmul depending on matmul sizes. cusparseltrrrryrrBc ||||jtj|ddddtj||j S)NrX)rr)r6_cslt_compressrrr*rs r?rz/SparseSemiStructuredTensorCUSPARSELT.from_dense8s` ,,_===s!''88(,&@&P$)7    rArrc tj||d\}}}}}||jdd}||jdd}||j|||||dS)a~ This function does the same thing as described in SparseSemiStructuredCUTLASS, but uses the cuSPARSELt metadata layout and sparse matmul. The only functional difference is that cuSPARSELt stores `metadata` and `packed` together into a single tensor. :: [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to cuSPARSELT semi-structured -> packed [1 2 6 2] [0 0 6 2] -> pack to transposed cuSPARSELt -> packed_t semi-structured representation -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same three outputs from the dense tensor can be found below:: from torch.sparse import SparseSemiStructuredTensorCUSPARSELT from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cusparselt = torch._cslt_compress(pruned) packed_t_cusparselt = torch._cslt_compress(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUSPARSELT( dense.shape, packed_cutlass, None, packed_t_cutlass, None, bitmask ) Frrrr)r6rrnr)rs r?rz>s  !(C    rANFrrrrc 6t|trtd|jdks |jdkrt d|jjd|j|jkrWt d|jjdt|j dt|j d|jd|jd |g|j|jkrWt d|jjdt|j dt|j d |jd |jd |jtj krOt d|jjdt|j dt|j d |jd |j t d|jjdt|j|j}tjj||j ||j d|j|j||j| S)Nrr,rrz` matmul: trying to do `A=z @ B=z`, with A.dtype=z and B.dtype=zH. This operation is only supported when A and B have the same data type.z + C`, with A.dtype=B.dtype=z and C.dtype=zK. This operation is only supported when A, B and C have the same data type.z`, with A.dtype=B.dtype=zO. mm is not supported for float8_e4m3fn, please use `torch._scaled_mm` instead.rr)rrr9rr_rGrHr0rr)r6 float8_e4m3fnr" _ensure_cusparselt_mm_registeredrrfr cusparselt_mmrrr()rIrrrrcrs r?rz(SparseSemiStructuredTensorCUSPARSELT._mms a3 4 4 l  9>>QVq[[%VDN+VVV  7dj %YDN+YYuTZGXGXYY_defel_m_mYY $ YY9:YYY    dj 8 8%\DN+\\uTZGXGX\\_defel_m_m\\(, \\AB\\\  :, , ,%`DN+``uTZGXGX``_defel_m_m``(, ```  ; %QDN+QQQ  - . . .7@K9,::  1 **&&&   rAr)rHrrrr r6rrrrrrrrrr:rrrrrrrAr?rr$so  G ;;BBKK 222r2rBB 55b"aCC 66r2q!DD  1@     0    [ &68> > #l> %> > > [> H%)', 444 <4lT! 4 !% 4 444444rArctrdSdaddlm}|dddtjd tjd tjd tjdzd t d t dt dt dtjfd}|jdtjd tjd tjd tjdzd t d t dt dt dtjfd}dS)zLazily register the cutlass_mm custom op. Registration is deferred to avoid importing torch.library at module load time, since torch.sparse is imported early during ``import torch``. NTr custom_opzsemi_structured::cutlass_mmr mutates_argsdenser"r#r out_featuresrrrrBcJ|j\}} | |z} | |z} | dkp| dk} |} | r)tjj|d| d| f} |r| n| }|tj|||}ntj||||}| ra|r|n| }|d|dd|}|r&| n| S|r&| n|S)Nrr) r)r6nn functionalpadrm_sparse_semi_structured_mm_sparse_semi_structured_addmmnarrow contiguous)rr"r#rrrrrrrto_pad_mto_pad_nneed_pad dense_paddedmm_inputresout_colss r?rz1_ensure_cutlass_mm_registered..cutlass_mms;{1B(?B(?q=1HM  V 8.2251h8:TUUL'=O<>>###< <264JJCC5dFD(SSC  X29qqHm|m$++Aq(;;C+AW35577%%'''s~~GWGW W'=Fsuuww!!###3FrAtranspose_densec|r |jdn |jd}|r||fn||f} tj| |j|jSNrrrr)r6emptyr0r/) rr"r#rrrrrr output_shapes r?_cutlass_mm_fakez7_ensure_cutlass_mm_registered.._cutlass_mm_fakesb&5H5;q>>%+a.(7 UX| $ $lH=U { +<    rA)_cutlass_mm_registered torch.libraryrr6r:rr register_fake)rrrs r?rrs]!''''''Y,2>>>G|G GlGlT! G  G  GG!%G GGG?>G8 |   l lT!               rActrdSdaddlm}|dd dd tjd tjd tjdzd t d t dt dt dt dt dtjfd}|jd tjd tjd tjdzd t d t dt dt dt dt dtjfd}dS)z,Lazily register the cusparselt_mm custom op.NTrrzsemi_structured::cusparselt_mmrrFrr"rrrrfuse_transposerrrBc |j\} } | |z} | |z} | dkp| dk} |}| r)tjj|d| d| f}|r|n|}tj|||||}| rZ|r| n| }|r)|dd|S|dd|S|S)Nr)rtranspose_resultrr) r)r6rrrrm_cslt_sparse_mmrr)rr"rrrrrrrrrrrrrrrrs r?rz7_ensure_cusparselt_mm_registered..cusparselt_mms{1B(?B(?q=1HM  V 8.2251h8:TUUL'=O<>>###<#  +      ?29qqH ?zz!Q11<<>>>zz!Q11<<>>> rAc |r |jdn |jd} |r| |fn|| f} tj| |j|jSrr) rr"rrrrrrrrrs r?_cusparselt_mm_fakez=_ensure_cusparselt_mm_registered.._cusparselt_mm_fake6sb&<O5;q>>Q(6 TX| $ $\8rsn""""""$$$$$$ "   ",$C"" H"H"H"H"H"H"H"H"Z,<DFDF\DFDF DF DFDFDFDFNeeeee(BeeeP]]]]]+E]]]@= = = @"@ @ @ @ @ rA