tj UddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZd dlmZe rddl m!Z!ddl"m#Z#ddl$m%Z%er ddl&Z&ddl'm(Z(ej)e*Z+dedededZ,ej-.dd/Z0e0evZ1erUddl2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;ddl2mZ?ddl2m@ZAddl2mBZCdeDfdZ@deDfd ZBdaEeFdzeGd!<d"eFfd#ZHd"eFfd$ZId]d%ZJn8d&Z1d'Z@d(ZBGd)d*Z6Gd+d,Z7Gd-d.Z5d/Z:d0Z9d1ZJd2d d3d4d d3d4d d3d5d6d3d7dd3d8d9d:d;ZKeFeDeFeDeDffeGd<<iZLeFeDe dzfeGd=<d>eDdzd"eMfd?ZN d^d>eDd@edzdAeMd"e dzfdBZOeLfdCeDdDeFeDe dzffdEZPd]d_dJZ> d`dCeDdKeDdzdLeQeDzdzdAeMd"e f dMZz-use_kernel_forward_from_hub..[ss)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)r)s r1r'r'TsF  #7 CC C   xYvxx   #? "r3 func_namecxtrt|StdtdS)Nr+c|Sr-r.funcs r1r2z*use_kernel_func_from_hub..dsr3)r4!_kernels_use_kernel_func_from_hubr6r7r8)r9s r1r(r(]sG  %4Y?? ?   xYvxx   %$ $r3_KERNEL_MAPPING_CACHEreturncj dtddditdtddtd d d d idtjtdddtjtdddiidtjtdddtjtdddiidtjtjztdddtjtjztdddiidtjtjztdddtjtjztdddiidtjtjztdddiitjtdddtjtddditjtdddtjtddditjtddditjtddditjtdddtjtdddidtjtdddtjtddditjtdddtjtddditjtddditjtdddid dtjtjztd!d"diidtjtjztd!d#diidtjtjztd!d$diidtjtjztd!d%diidtjtjztd!d&diidtjtjztd!d'diid(}tjtd)d*d+itd)d*d+tjtd,d*d+id-dtjtjztdd.d+iid/}||z}|S)0Ncudaz!kernels-community/deformable-detrMultiScaleDeformableAttentionr)repo_idr)versiony)min_capabilitymax_capability)type propertieszAtlas-Inference/gdnQwen3_5GatedDeltaNet(ef12347fc77d6ddf1cb72c0bd0af1c7d6cc69172T)rDr)revisiontrust_remote_codekernels-community/mamba-ssmcausal_conv1d_fncausal_conv1d_updatezkernels-community/liger-kernelsLigerSwiGLUMLPrLigerTiledSwiGLUMLP LigerGEGLUMLPLigerTiledGEGLUMLP LigerLinear LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)rBrocmxpumpsnpuzkernels-community/megablocksMegaBlocksMoeMLPCPUMegaBlocksMoeMLP)rBrYrZcpuzkernels-community/activationFastGELU QuickGELUNewGELUSiluGeluGeluTanh)rCrKrPrQ SwiGLUMLPGeGLUMLPLinearrXr]r`rarbSiLUGeLUrezkernels-community/rotaryapply_rotary_transformers)rDr9rEzkernels-community/aiter-rope)rZrBrYLigerForCausalLMLoss)rotary_pos_embForCausalLMLoss)rrrr!TRAINING INFERENCE TORCH_COMPILEr)_KERNEL_MAPPING_FUNCTION_KERNEL_MAPPINGs r1_build_kernel_mappingrtls4?>."-SQTUUU#15G&*  %M? =#5 !$$$ NO =#5 !%%%  !M? =#9 !$$$ NO =#9 !%%%  %NT%77 A#3 !::: MD$66 A#8 !999  NT%77 A#2 !::: MD$66 A#7 !999  MD$66 A#0 !999M? A#1 !$$$ NO A#1 !%%% M? A#1 !$$$ NO A#1 !%%% NO ;#, !%%%NO ?#, !%%%M? A#1 !$$$ NO A#1 !%%% Q44nM? >#5 !$$$ NO >#5 !%%% M? >#5 !$$$ NO >#5 !%%% NO >#5 !%%%NO >#8 !%%%A'!'!RNT%77 >#- !:::NT%77 >#. !:::NT%77 >#, !:::NT%77 >6[\:::NT%77 >6[\:::NT%77 >:_`:::Iki ki bNN :Fakl%%% '6B]ghNN >Jeop%%% MD$66 AMcmn999 !$ $  0*,DDr3c:ttatS)zPReturn the default transformers kernel mapping, building it lazily on first use.)r?rtr.r3r1get_kernel_mapping_transformersrvws ! ($9$;$; !$$r3cD|t}t|dSr-)rvr")mappings r1$register_kernel_mapping_transformersry~s& ?577G(((((r3Fc d}|S)Nc|Sr-r.r/s r1 decoratorz.use_kernel_forward_from_hub..decoratorsJr3r.argskwargsr|s r1r'r's   r3c d}|S)Nc|Sr-r.r<s r1r|z+use_kernel_func_from_hub..decoratorsKr3r.r}s r1r(r(s   r3ceZdZdZdZdS)rc td)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeErrorselfr~rs r1__init__zLayerRepository.__init__sopp pr3c td)NzMLayerRepository requires `kernels` to be installed. Run `pip install kernels.NotImplementedErrorrs r1loadzLayerRepository.loads%&uvv vr3N__name__ __module__ __qualname__rrr.r3r1rrs: q q q w w w w wr3rceZdZdZdZdS)r c td)NzSLocalLayerRepository requires `kernels` to be installed. Run `pip install kernels`.rrs r1rzLocalLayerRepository.__init__stuu ur3c td)NzRLocalLayerRepository requires `kernels` to be installed. Run `pip install kernels.rrs r1rzLocalLayerRepository.loads%d r3Nrr.r3r1r r s5 v v v     r3r ceZdZdZdS)rc td)NzMFuncRepository requires `kernels` to be installed. Run `pip install kernels`.rrs r1rzFuncRepository.__init__snoo or3N)rrrrr.r3r1rrs( p p p p pr3rc td)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rr~rs r1r#r#s l   r3c td)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.rrs r1r"r"sstttr3c td)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.rrs r1ryrys q   r3zkernels-community/causal-conv1d)rDrErOz!kernels-community/finegrained-fp8zkernels-community/deep-gemmzkernels-community/sonic-moez ep-support)rDrM)z causal-conv1dz mamba-ssmzfalcon_mamba-ssmzfinegrained-fp8z deep-gemmz sonic-moe_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationc8|duotjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rs r1 is_kernelrs. 4' c I@BU V V^b br3attention_wrapperallow_all_kernelscddlm}ddlm}d|vr|ddn|}t |sdSt sttd|vr-|d\}}| }n|}d}| }| d\}}} | }| r| nd} d} | t|d } | rd nd} t|| | | } n2#t$rt$r} td |d| dd} ~ wwxYwd}t| dr |t }|}n1t| drddlm}||n|}d}n|t'| |}|||||||| S)a Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continuous batching, where we want the `paged` wrapper, which calls the paged cache. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if it is a custom kernel outside of the `kernels-community` hub repository. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|rN:@r)"requested_attention_implementationrErrMrErz-An error occurred while trying to load from 'z': .flash_attention_2flash_attn_varlen_funcsparse_atten_func)msa_attention_forwardsdpa) masking_utilsrmodeling_utilsrsplitrr ImportError_MISSING_KERNELS_MESSAGEstrip partitionrr% ValueError Exceptionhasattrr msa_attentionrgetattrregister)rrrrractual_attn_namerD kernel_name_revrEis_fa4kernelemask_implementationkernel_functionrs r1load_and_register_attn_kernelrsz =<<<<<888888.attach_hidden_kernelsQs(92>>DDFF 1 1HD"4 5 5 7 78888!"bi00$BdBBB &&tR000 1 1r3cnt|diD]"}t||rt||#dS)Nr)rrdelattr)rrs r1detach_hidden_kernelsz(kernelize..detach_hidden_kernels\sMF$5r:: & &Dvt$$ &%%%  & &r3NrBrY)rI)inherit_mapping)devicerT)rrrapplytrainingr!rprorrIrr kernel_configuse_local_kernelr$kernel_mapping_kernels_kernelize _use_kernels)rrrr device_typerrs r1r&r&Ls  ! !42333 1 1 1&&&+ )***%*^`t~~$,\`l' & %5%7%7  K[)))   *"'"5"FFO#E$7$FXghhh D D"5dCCCC D D D D D D D D D D D D D D D uV$ ? ? ? ?! )***** )****s0BDC" D"C&&D)C&*DD5rMrEcddlm}tsttd||d}t |||||S)Nr) __version__ transformers) frameworkrErD)rMrE user_agentrN)rrrrget_kernel_hub)rrMrErrrs r1r%r%wsf   ! !42333-+R]^^J hJbs   r3 module_namescBttrgfd}|S)aO This decorator attaches the target function within the module as a plain attribute (not as a submodule). Keep in mind that this registration is only meant for `kernelize` to recognize its target modules (i.e. function exchanged for a weightless `nn.Module` with the same forward) to then exchange to the kernel variation (in-place) if the conditions are met. We cache each of these function-based registrations: After proper registration and exchange it is removed from the module's `_modules` dict as it does not really act as `nn.Module` but a base function. c2|jfd}||_|S)Nc|g|Ri||jdi}D]N}t|ddp!t|ddpt|dd}|td||||<OdS)Nrrkernel_layer_namer9z)Could not infer kernel function name for )__dict__ setdefaultrr)rr~rhidden_kernelsrrr orig_inits r1new_initz8use_kernelized_func..decorator..new_inits Id ,T , , ,V , , ,"]556GLLN" * *B D116r#6==6r;55 <$%WQS%W%WXXX(*t$$ * *r3)r)r0r rrs @r1r|z&use_kernelized_func..decorators7L  * * * * * *"   r3)rr)rr|s` r1use_kernelized_funcr s?,))&$~ . r3c#*K dadVdadS#dawxYw)z Context manager used to adjust the value of the global `ALLOW_HUB_KERNELS`. This is needed, as this argument cannot be forwarded directly to the `__init__` of the models, where we set the attention implementation. TNF)rr.r3r1allow_all_hub_kernelsr s:"  "E!!!!s parent_cls child_names kernel_clsc~|jfd}td|j|fd|i}d|j|_|S)a6 Create a new class that inherits from `parent_cls` and fuses the child modules specified in `child_names with the provided `kernel_cls`. The first child in `child_names` will be replaced with the `kernel_cls`, and the rest will be replaced with `nn.Identity()` to keep the same interface. cg|Ri|fdD}|}td|ddD]$}t|tj%dS)Nc0g|]}t|Sr.)r).0rrs r1 zMmake_parent_class_for_kernel_fusion..patched_init..s#@@@DGD$''@@@r3rr)setattrrIdentity) rr~rchildrenkernel_instancerrr original_inits ` r1 patched_initz9make_parent_class_for_kernel_fusion..patched_inits d,T,,,V,,,@@@@K@@@$*h/k!no666O / /D D$ . . . . / /r3Fusedr)rrIrr)r rrr patched_clsrs `` @r1#make_parent_class_for_kernel_fusionrsr'M///////4z244zmjR^E_``K@z'>@@K r3r0type[PreTrainedModel]configrrrc  t|drt|jdstd|jd|jj}i}i}d}|jD]\}}t|ttfrd|i}t|tr8t| dkrtd|ntd|d|tt| }t|trd|\} } | d d} | d d} | d d pt } | d | ind | i} | d | iz} | | f}n|} dt d } | | f}| d\}}}|r|std| d||jrt't)||}nt+d+||d| }|}|td|t.j|j}|rt5||jddnd}|2d|jvr)t9j|jd}||_t|tr&||||<a|j|_|||<|||j<nt|tr| td|jd|jd|j|_d|D}d|D}ttA|dkrtd||d!dd}d|D}|7tEj#d 5||}dddn #1swxYwYd }|$D]}\} tKj&||stO fd!|Dstd"|d#|d$|jd%}tQ }tS|||||j<~|std&|d'|jd(tU|d%)t|d*r._noop_forward&sr3z Fused kernel z* requires a companion layout class named 'zLayout' in the same module.cg|] }|d S)rr.)ritems r1rz.Ds<<<T!W<<.Es+JJJqqxxQ//2JJJr3zGAll patterns for a fused kernel must share the same parent module, got r*z\w+cFg|]}|dddS)rrr+r-s r1rz.Ms+FFF1188C++A.FFFr3metac38K|]}t|VdSr-)r)rchildrs r1 z;register_kernel_replacements_and_fusions..Ws-KKe76511KKKKKKr3zModule z* does not have the expected child modules z required for the fused kernel TzNo module matched pattern z for fused kernel z3. Provide the full dotted path from the model root.) overwriteconversion_mappingr.)/rr!rrr"rrrstrtuplerlenvaluesnextiterrrrrr rrrsysmodulesrrr functoolswrapsr%rsetrtorchr named_modulesr fullmatchallrIrr r listr6r )!r0rrr" patch_mapping new_mapping meta_modelr)hub_reporepo_strmetadatarMrErN final_reporDrlayer_name_in_reporepor kernel_mod layout_clsr' glob_patternsparent_patternsparent_patternr matched_anyr module_clsexisting transformsrs! @r1(register_kernel_replacements_and_fusionsrYs 3 ' 'Uws7G/V/VUS#,SSSTTT!,J%'MKJ - < B B D DC6C6 H he - - (h'H h % % X8??$$%%** mckmm+ VVV VVWW WX__..//00 h & & .!) Hh||J55Hll9d33G ( -@% H H ]L] /6/B 7++U]H^H ,.?@ @H"H-JJH#$;LMMH"H-J)1););C)@)@&& b0 b`8``R\``aa a  ) 'w---DD #-D YY[[  V(VVWW W[__Z%:;; R\fWZJ,?)G)G)GNNNbf  !iz7J&J&J _Z/ 0 0  1 0 "/J  j# & &9 !*4 J',6+>J ((2M* %0:K + , , E * *+ ! OJ$7OO(1OOO ,6+>J (<<<<r7r?r__annotations__rtrvryrrboolrrrintrFr rr rIrrY__all__r.r3r1rps; $$$$$$%%%%%% jjjjjjjj444444111111118888885444443666666000000222222LLL  H % %uuu+>uu^quuu !# /@% H H N N P P04HHv                       VUUUUU#####%C%%%%*.4$;---I4IIIIV%T%%%%)))))   wwwwwwwwpppppppp   uuu   "CqQQ:qII$AaPP#FSTUU:qII: UU 22T#tCH~-.8:S*t"3349993:$diKKK19DK\`K$KKKK\Pf1 1 #1 S*t:K5K0L1 1 1 1 h(+(+(+(+(+Z $# Dj3Y    "$d8nx&?$$$$P " " " c 4U/ U/ U/"U/ U/U/U/U/p   r3