tjpdZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl Z ddl mZddlmZdd lmZmZmZmZmZd d lmZd d lmZejeZe d GddZejd_dZ ejd`dZ!ejdadbdZ"e j#j$dadcdZ%dadddZ&ejded Z'dfd#Z(dgd%Z)dhdid(Z*djd/Z+dkd6Z,dld:Z-dmd<Z.dndBZ/dodLZ0edMdNO dpdqdUZ1drdXZ2drdYZ3dsd[Z4 dhdtd^Z5dS)uuSDeepGEMM integration: fused grouped GEMM kernels from `kernels-community/deep-gemm`. Provides: - `deepgemm_bf16_experts_forward`: BF16 M-grouped experts forward. - `deepgemm_fp8_fp4_linear`: end-to-end FP8/FP4 linear (output dtype follows the input). - `deepgemm_fp8_fp4_experts_forward`: FP8 (or FP4 on SM100+) M-grouped experts forward. - `deepgemm_fp8_fp4_megamoe_experts_forward`: FP8xFP4 Mega MoE forward (SM100+). Requirements: CUDA, Hopper (SM90+), CUDA runtime ≥ 12.3, kernels-community/deep-gemm ≥ 2.5 (Mega MoE symbols required). Mega MoE additionally needs SM100+ at call time. ) annotationsN)Callable) dataclass)logging)deprecate_kwarg)KERNELS_MAX_VERSIONKERNELS_MIN_VERSIONis_kernels_availableis_torchdynamo_compilingresolve_internal_import)lazy_load_kernel)to_localT)frozenceZdZUdZded<ded<ded<ded<ded<ded<ded <ded <ded <ded <d ed<dS)DeepGEMMz>Curated entry points exposed by `kernels-community/deep-gemm`.rfp8_fp4_matmulgrouped_fp8_fp4_matmul_ntgrouped_fp8_fp4_matmul_nngrouped_bf16_matmul_ntgrouped_bf16_matmul_nnper_token_cast_to_fp8!transform_sf_into_required_layouttransform_weights_for_mega_moeget_symm_buffer_for_mega_moefp8_fp4_mega_moeint m_alignmentN)__name__ __module__ __qualname____doc____annotations__`/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/integrations/deepgemm.pyrr9sHH''''''''$$$$$$$$####////,,,,****r&rreturn str | Nonecptjdptjd}|r|Stjd}|r::L w~~l## l## *q!Yq\\  * * * * * * * * * * * * * * * nn[*..2L2LMMQQR[]_``G"==--bqb1LE5u::s5zz) )^4    D ',,y-88K w~~k"" k"" Ja ">II J J J J J J J J J J J J J J J @5;;q>>**C A,?,??? @$    D W\\)Y 9 9F w~~f f K "?JJ K K K K K K K K K K K K K K K I"5;;q>>22 #t+lT.Ab-HHH I$    D  4sD "B7 D BD B A>D D$#D$'G<6(F* G<*F..G<1F.2AG<<HHK#(J KJKJ3KK('K(Frequires_sm100boolDeepGEMM | strctsytsdtdtdtdStjsdStj\}}|rdnd}||vr|rdnd }d |d ||d S|d krdnd}t}|d|dd |ddStj tj |ddsd|d|dd |ddSt}|d|d|dd |ddS||krAd||d|dd |dd|dd |dd |d!|dd |ddStd"}|d#St|d$d} t|d%d} t|d&d} t|d'd} t|d(d} t!|d)*}t|d+d}t|d,d}t|d-d}t|d.d}t|d/d}d0d$| fd%| fd&| fd'| fd(| fd)|fd+|fd,|fd-|fd.|fd/|ff D}|r1d1d2 |d3tdtdtd St#| | | | | ||||||4 S)5aLoad DeepGEMM once or returns an error message if env or any required symbol is missing. This is wrapped in a function that will raise an `ImportError` with the error message. The reason we raise in the wrapper rather than here is that @functools.cache will only cache a return value, not an exception. `requires_sm100` raises a Blackwell-specific error for callers (FP4 / Mega MoE) that won't work on Hopper, instead of the generic SM90+ message. zUDeepGEMM kernel requires the `kernels` package. Please install a compatible version (z <= version < z), e.g. `pip install kernels==`z9DeepGEMM kernel requires CUDA, but CUDA is not available.)r@) r@zBlackwell (SM100)z"Hopper (SM90) or Blackwell (SM100)zDeepGEMM requires z; current device is SMr>r@) r])r^Nu(DeepGEMM's JIT needs a CUDA toolkit ≥ rrz8, but none was found. Set `CUDA_HOME` to a CUDA toolkit.binr-z:DeepGEMM's JIT compiles with nvcc, but none was found in `u,/bin`. Point `CUDA_HOME` at a full CUDA ≥ z& toolkit (not a runtime-only install).zDeepGEMM found nvcc in `u/bin` but could not read its CUDA version (no parseable `version.json`, `version.txt`, or `include/cuda.h`). Point `CUDA_HOME` at a complete CUDA ≥ z toolkit.zDeepGEMM on SMu needs a CUDA ≥ z toolkit, but nvcc z in `u.` is too old. Point `CUDA_HOME` at a CUDA ≥ z deep-gemmucFailed to load `kernels-community/deep-gemm` — check that a build matches the current torch/CUDA.fp8_fp4_gemm_nt$m_grouped_fp8_fp4_gemm_nt_contiguous$m_grouped_fp8_fp4_gemm_nn_contiguous!m_grouped_bf16_gemm_nt_contiguous!m_grouped_bf16_gemm_nn_contiguouszutils.per_token_cast_to_fp8) chained_pathrrr&get_mk_alignment_for_contiguous_layoutrcg|] \}}|| SNr%).0nameattrs r' z)_load_deepgemm_kernel..s- D$ <  <bDSSSES5SSS S$rkk77w"$$  58A;55RS555 w~~bgll9eVDDEE tYtt2:1+tt@H ttt )**  J9JJ%-a[JJ3;A;JJJ  ( " "AAAA!AAxXY{AA?AA%1!_AA;DAA$QKAA*21+AAA  k * *F ~ttV%6==N '0VX\ ] ] '0VX\ ] ]$V-PRVWW$V-PRVWW3FIfggg(/8[]a(b(b%%,V5UW[%\%\"#*63QSW#X#X v'OQUVVv'94@@ / 35N O 35N O 02H I 02H I *,A B 02S T -/M N +-I J 57G H !1 2  G"  ADIIg> 15`, `>> 7`, `<< 1`) otherwise leak mantissa bits into adjacent UE8M0 byte slots and silently corrupt the SF. SM90 consumes raw fp32 SFs without going through this path. ii)viewrnint32 bitwise_and_float)rint_views r'_ceil_to_ue8m0r>sAwwu{##H  & 4 45E F F K KEK X XXr& expected_mn int | Nonect|j}|jtjkr|H|d|kr/||dz}||d}|r2|tj }n;| }n&|jtj kr|rt|}| dvr%td| d|s|S|d}|d}d|z}| |z |z}| d krd |fn||zd |f}t!||kr|Stj|j||j|j } | || S) uLay out `sf` as DeepGEMM's dispatch expects, per arch. On SM100 the int-SF path only *checks* the SF (`tma_stride_check`) and never transforms it, so we hand it a TMA-aligned MN-major layout (`stride(-2) == 1`, `stride(-1) == align(mn, 16/esize)`). On SM90 DeepGEMM transforms SFA itself (`get_mn_major_tma_aligned_tensor`) and only *checks* SFB against `sm90_sfb_check`, which rejects TMA padding (`stride(-1)` must equal `size(-2)`, not `align(mn, …)`); a padded weight SF trips `layout.hpp` whenever `mn` isn't a multiple of `16/esize` (e.g. N=576 → mn=5). So on SM90 we return the raw row-major SF and let DeepGEMM lay it out. Inputs come in three flavors: - `float8_e8m0fnu` on SM100: raw UE8M0 bytes — pack 4 K-bytes → int32 (last dim /4) for the kernel's `(INT, 1, gran_k)` path. - `float8_e8m0fnu` on SM90: SM90 dispatch only accepts FP32 SFs, so cast UE8M0 → FP32 (exact upcast — UE8M0 is the biased-exponent half of a pow-of-2 FP32, so `.float()` rebuilds the original FP32 scale exactly). - `float32`: per-token / per-block SFs from `per_token_cast_to_fp8` or on-disk weights — round to UE8M0 on SM100 (see `_ceil_to_ue8m0`). - `int32`: already-packed UE8M0 — pass through. When `expected_mn` is set and the SF's M-dim is smaller (block-quantized UE8M0, e.g. DSv4-Flash compressor weights with `(N/128, K/128)` SFs), we repeat the SF on the M-axis to per-row before packing — the `(INT, 1, gran_k)` DeepGEMM kernel branch is the only UE8M0 path on SM100; for `gran_mn > 1` the kernel only handles FP32 SFs and would otherwise reject our INT SF here. Ndim)rr_z"DeepGEMM SF must be 2D or 3D, got Drrrr)rrrrnfloat8_e8m0fnusizerepeat_interleave contiguousrrrrrrrH element_sizetuplestride empty_stridedshapecopy_) rris_sm100gran_mnmnkfalign_to aligned_mntarget_stridesouts r'_coerce_sf_for_kernelrKs8##H x5'''  "rwwr{{['@'@!RWWR[[0G%%g2%66B  %%ek22BBBB U] " "x " B   vvxxvIbffhhIIIJJJ }} B BR__&&&H3(?#h.J(*A a__BOQPZ;[N RYY[[^++  bhbhry Y Y YCIIbMMM Jr&weightweight_scale_inv block_size tuple | Nonerdictc|jtjkrddddS|tdt |}|dvrtd|d|jtjkr|rdd ddSd d d S) uCPick the `per_token_cast_to_fp8` kwargs from weight dtype + SF dtype + arch. Cases mirror the kernel's recipes: - FP4 weights (`int8`): gran_k=32 packed-UE8M0 SF. SM100+ only. - FP8 weights + UE8M0 SF on SM100: gran_k=128 packed-UE8M0 SF (DSv4). - FP8 weights + UE8M0 SF on SM90: gran_k=128 FP32 SF — the SM90 dispatch in `layout.hpp` only matches FP32 SFs, so we keep act SFs as FP32 (and float the weight SF in `_coerce_sf_for_kernel`; UE8M0 → FP32 is an exact upcast). - FP8 weights + float SF: gran_k=128 float SF (DSv3). T  use_ue8m0gran_kuse_packed_ue8m0Nz]DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.))r)rru?DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got r>rF)rr)rrnint8rHrr)rrrrs r'_select_fp8_cast_kwargsrs|uz!!!RTJJJ k   z""J///j]gjjjkkk!555(5!SdKKK# . ..r&expert_ids_sorted num_expertsr alignmentuse_psum_layout&tuple[torch.Tensor, torch.Tensor, int]c|j}|d}tj||d|dz }||zdz |z|z}|t |||dz zz}||z } tjj | dd} tj ||| |z} |r(| d} nRtj |fd|tj } tj||k|d| | <| | |fS)aBuild the TMA-aligned grouped layout DeepGEMM expects. Returns `(sorted_to_padded, grouped_layout, total_padded_rows)`: - `grouped_layout` is per-row expert id (Hopper, with `-1` for padding / sentinels) or a cumsum of aligned per-expert counts (Blackwell). - EP sentinels (values == `num_experts`) are routed past the last expert block so DeepGEMM skips them. rr)binsminmax)rrrrrr)rrrnhistcrlongrnn functionalpadcumsumarangefullrwhere) rrrrr num_tokenstokens_per_expertaligned_tokens_per_experttotal_padded_rowspadding_per_expertcumulative_paddingsorted_to_paddedgrouped_layouts r'!_build_deepgemm_contiguous_layoutrsq %F"''**J $5$9$9$;$;+STZehiZijjjooqq"3i"?!"C !QU^ ^"S[%A%AYQR]%SS35FF,001C1J1J11M1MvVV|Jv>>>ASTeAffu299!<<@@BB%6$8"VSXS^___+0;7H;7VXiXmXmXoXoqs+t+t'( ^-> >>r&xrrcjtj|g|jddR|j|jd}|||<|S)z;Pad a sorted tensor into the TMA-aligned contiguous layout.rNr)rnemptyrrr)rrrpaddeds r'_pad_for_deepgemmrsD [* YQWQRR[ Y YQRQX Y Y YF F  Mr&x_paddedc||Srir%)rrs r'&_unpad_from_deepgemm_contiguous_layoutrs $ %%r& hidden_states top_k_index top_k_weightsrrcx|d}|d}|d}tj|\} } || |z} || } t | |||\} }}| |kd}| |dz | | | || | ||fS)zSort tokens by expert id and build the M-grouped padded layout. Returns `(sorted_hidden_states_g, sample_weights_g, expert_ids_g, sentinel_mask, perm, sorted_to_padded, grouped_layout, total_padded_rows)`. rr)r)rreshapernsortr unsqueezeclamp_)rrrrrr num_top_k expert_idssample_weights expert_ids_gpermsorted_hidden_states_gsample_weights_grrr sentinel_masks r'_dispatch_routed_inputrs  $$I$$R((J"**2..NJ//L$*49+<=%d+ ;\k;;;7n&7"[0;;B??MK!O,,,   r& out_paddedsorted_weightsrrrr hidden_dim out_dtype torch.dtypec t||} | || jdz} | |dt j|} t j|d| j | |<| |  ||| d|S)uRUnpad → weighted multiply → mask sentinels → restore order → top-k reduce.rgrrrr) rtorr masked_fill_rn empty_likerrrrsum) rrrrrrrrrrweightedinv_perms r'_combine_routed_outputr s 1=M N NC^&&sy11;;B???H ----%%H\$))A,,szBBBHTN H  " ":y* E E I Ia I P P S ST] ^ ^^r& output_dtypezv5.16)r<inputbiastorch.Tensor | Nonetorch.dtype | Noneactivation_scalec |td|jtjtjfvrt d|jt |jtjk}t|||t|j }| d|j d} |j | fi|\} } tj| j d|j d|j |j} |dr d d |d fnd} || t#| | d f|t#||d f| | | |j dd|j dfz} || || S) uEnd-to-end DeepGEMM linear: per-token activation quant + FP8/FP4 matmul. Static (per-tensor) activation quantization is rejected — DeepGEMM needs per-row SFs. Callers should route static activations through the Triton fallback. Nz@DeepGEMM linear does not support static activation quantization.z7DeepGEMM linear requires FP16 or BF16 activations, got r|rrrrrrr)recipe)NotImplementedErrorrrnbfloat16float16rHrrrrrrrrrr0rrradd_)rrrrrrr deepgemm cast_kwargsinput_2d qinput_2dscale_2doutput sf_recipes r'deepgemm_fp8_fp4_linearr#s#!"deee {5>5=999`SXS^``aaa#6<5:3MNNNH)&2BJPYZ_ZfPgPghhKzz"ek"o..H8(8QQ[QQIx [+V\!_U\Y^Yd e e eF2=AS1T1T^A{8,--Z^I  )( q@Q@QRRRS &'7V[[QR^^TTTU  [[SbS)V\!_,>> ? ?F  D Mr&selftorch.nn.Modulec |jtjkrtd|jt }|jr|jn|j}|j}| d}| d}| d} t||||j |j t|\} } } } }}}}t|jr|jn|j}t|j}|jr"t|jr|jn|jnd}|jrt|jnd}|jr |jdn |jd}t1| ||}tj||||j}|||||t||jr|d||| |jr||n||}tj|| ||j}|||||t||jr|d||| t;|| | ||||| |j S)N;DeepGEMM experts path requires bfloat16 hidden states, got rrrr)r)rrnrrHr is_transposedrrrrrrrrrhas_gate gate_up_projup_proj down_projhas_biasgate_up_proj_bias up_proj_biasdown_proj_biasrrr index_add_ _apply_gateact_fnr)rrrrrgrouped_bf16_matmulrrrr sorted_hiddenrrrrrrr weight_up weight_downup_bias down_bias up_out_dimactproj_outrs r'deepgemm_bf16_experts_forwardr2Ms en,,lWdWjllmmm#%%H=A=Ot(99U]Ut  !F  $$I##A&&J##B''J {M43CXEY[dek[l[l     dmM**NNI4>**KZ^ZgqhUt--DDUVVVmqG15H,---DI)-(:R$$ PQ@RJ M+;=N O OC{,j}ObcccHY.R[\bRcRcdddd }HA/1FGGG-1]Ut))) H@U@UH +'F-J] ^ ^ ^C+sNT]^dTeTeffff }E q*Il,CDDD !     r&c |jrtdt|j|jdkrt d|jtjkrtd|jt|j jtj k}|j r|jn|j}|j}|d}|d}|d} t%|jr|jn|j} t%|jr|jn|j} t%|j } t%|j} t1| | |jt5|}t7||||j|jt5|\}}}}}}}}|dr d d |d fnd}|j|fi|\}}tA|||}tA|||}tj!|| j"d |tj }||tG|| f| tG| | d  f|||t5||jr|$|n|%|}|j|fi|\}}tj!|| |tj }||tG|| f| tG| | d  f|||t5|tM|||||||| |j S)NzDeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices. Use `experts_implementation='grouped_mm'`, or run one device per process (TP/EP).staticzJDeepGEMM experts dispatch does not support static activation quantization.rr|rrrrrrr r)r r)'_deepgemm_disabled RuntimeErrorrdown_proj_scale_invactivation_schemer rrnrrHrr!rrrrrrrrrr gate_up_proj_scale_invup_proj_scale_invrrrrrrr0rrrrrr'r(r)rrrrrgrouped_fp8_fp4_matmulrrrrr+weight_scale_upr,weight_scale_downrr*r _expert_ids_grrrrrract_fp8 act_scalesr1proj_fp8 proj_scalesrs r' deepgemm_fp8_fp4_experts_forwardrCs    \   #4#;<<< ))!"noooen,,lWdWjllmmm#4>3G5:3UVVVH.2.@h**hFh !F  $$I##A&&J##B''JdmM**NNIdmgt::QUQghhO4>**K !9::))_doW`agWhWhiiK {M43CXEY[dek[l[l    2=AS1T1T^A{8,--Z^I9(8VV+VVGZ)9;LMMG":/?ARSSJ{,ioa.@W\WefffH ' @QRRRS )/y~~VXGYGYZZZ[!&)) .2]Ut))) H@U@UH;H:8SS{SSHk +'F%. Y Y YC (BSTTTU +,=;K[K[\^K_K_```a !&))  "     r&modulecPtd}t|jj}t|jj}t|jjtj }t|j jtj }|j }|j }|j }|dzdks |dzdkrtd|d|d||d|z|d | } ||||d | } ||| f|| f\\} } \} } tj| d |_tj| d |_tj| d |_ tj| d |_d S)u!One-shot pack + permute of an FP8Experts module's L1/L2 weights into the Mega MoE UTCCP layout. Called lazily on the first megamoe forward; idempotent via the caller's ``_megamoe_transformed`` flag. Steps: 1. Cast UE8M0 SF → FP32 and call ``transform_sf_into_required_layout`` → packed int32 in MN-major TMA-aligned layout. 2. Run ``transform_weights_for_mega_moe``: interleaves gate/up on L1 and transposes both SFs for UTCCP. 3. Overwrite the loader-side parameters in place; the interleave preserves the ``[E_local, 2*I, *]`` leading dims so downstream ``.size(...)`` reads stay valid. Unwraps any ``DTensor`` wrappers FSDP2/EP may have placed around the loader- side Parameters — the kernel takes raw pointers. Tr|rrzwDeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 (FP8 SF granularity); got hidden_dim=z, intermediate_hidden=r>r)rr)r  num_groupsF) requires_gradN)rrr9datar7rrrnrrr!intermediate_dimrrrHrrrr Parameter) rDrgate_up_sf_raw down_sf_raw gate_up_wdown_wintermediate_hiddennum_local_expertsr gate_up_sfdown_sfgate_updowns r'setup_megamoe_weightsrUs3"$4888Hf;@AAN65:;;K,12277 CCNNPPI f&+ , , 1 1%* = = H H J JF 1*"JB!2R71<< m4> m mVi m m m   ;; $ <J88$ 9G.6-T-T J ..*Wj?D' (,,WE,JJF$)H$6$6zQV$6$W$WF!x))$e)DDF!&!3!3G5!3!Q!QFr& process_group%torch.distributed.ProcessGroup | Nonec bt|j|jjtjkrt d|jjd|tdtd}t|ddst|d|_ | d }| d }| d }|j d } |j d d z} | | z} t|d d|j j|kr ||||| || |_ ||ddd\} } |j jd|| |j jd|| |j jd|||j jd||t j||ftj|j}|||j|jf|j|jf|j tt|dddd||jS)uFP8 acts × FP4 weights Mega MoE forward (SM100+). Fuses EP dispatch + L1 + SwiGLU + L2 + EP combine into one kernel, overlapping NVLink with tensor-core compute. The kernel handles the full `(num_tokens, hidden) → (num_tokens, hidden)` MoE forward including the weighted top-k reduction; the caller must NOT all-reduce the output. `process_group` is supplied automatically by `MoeTensorParalellExperts._prepare_input_fn` when the module is wrapped for TP — it's required for the symm-buffer rendezvous on first forward. `top_k_index` is GLOBAL expert ids (`-1` marks skipped slots). Caller-managed `self` attributes: - `gate_up_proj`, `gate_up_proj_scale_inv`: L1 weight + UE8M0 SF. - `down_proj`, `down_proj_scale_inv`: L2 weight + UE8M0 SF. Both pairs must be transformed together via `transform_weights_for_mega_moe((gate_up, gate_up_sf), (down, down_sf))`. - `config.swiglu_limit` (optional): SwiGLU clamp; absent → unclamped. zJDeepGEMM Mega MoE requires FP4-packed expert weights (dtype=`int8`), got `z/`. Use the 'deepgemm' dispatch for FP8 experts.NzDeepGEMM Mega MoE requires a `process_group` for the EP group. The TP wrapping (MoeTensorParalellMegaMoeExperts) supplies it automatically; pass it explicitly otherwise.Tr|_megamoe_transformedFrrrr symm_buffer)hiddennum_topkrnum_max_tokens_per_rankrOrrrconfig swiglu_limit)activation_clamp)rr7rrrnrr6rHrrqrUrYrrZr]rrrrx_sftopk_idx topk_weightsrrrrr9r!r)rrrrrVrrrrrPrOnum_global_expertsx_fp8rays r'(deepgemm_fp8_fp4_megamoe_experts_forwardrg$s2#4#;<<< %*,, Y!' Y Y Y    i   $4888H 4/ 7 7)d###$(!  $$I##A&&J##B''J)..q11+0033q8*]-?-?-A-AAt]D))1T5E5]`j5j5j#@@ *$. 3 A  00$WYlp0qqKE4{ {#))%000+:+&,,T222kzk*00===!+:+.44]CCC  Z,EN=K_```A  D78 12  x!>!>PTUU  44 # $ $$r&)r(r))r(r8)F)rXrYr(rZ)rXrYr(rz)rXrYr(r)rrr(rY)rrr(rz)rrr(rri)rrrrr(r) rrrrrrrrYr(r) rrrrrrrrYr(r)rrrrrrr(r)rrrrr(r)rrrrrrrrrrrrYr(r)rrrrrrrrrrrrrrrrrrr(r)NNNN)rrrrrrrrrr8rrr rr(r) rrrrrrrrr(r)rDrr(rz) rrrrrrrrrVrWr(r)6r# __future__r functoolsrDr.rJr1collections.abcr dataclassesrrnutilsrutils.deprecationrutils.import_utilsr r r r r hub_kernelsrtensor_parallelr get_loggerr loggerrcacher7rWry_dynamoallow_in_graphr}rrrrrrrrrrrrr2rCrUrgr%r&r'rvs  #""""" $$$$$$!!!!!! //////*)))))%%%%%%  H % %  $, $ ++++\ pppppf 9 9 9 9 9 ====, Y Y Y Y:::::z////>????D&&&&////d____4111 !%)-'+,0&&&&21&R>>>>BYYYYx7R7R7R7R~<@ S%S%S%S%S%S%S%r&