tj53DddlmZdZ d dZdZGdd eZdS) )PushToHubMixinc d} t|}n #t$rtd|wxYw|jj}|dkr+ddlm}|rddl}t|dj d S|S) z Infers the device type from the model parameters. Args: model: The model instance. Returns: The device type. z { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } zPCannot determine model device, please provide a device to the mapping. Example: cudar)is_torch_availableNversionrocm) next parameters StopIteration ValueErrordevicetypeutilsrtorchgetattrhip)modelEXAMPLE_MAPPINGparamdev_typerrs ^/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/utils/kernel_config.py infer_devicers O U%%''((     p_n p p    | H6......      LLLui((,8v Os !&ANFc ddlm}|dvrtd||dd} |dd} |||| | |||ii||<dS)NrLayerRepositoryrr xpunpuneurontpuBOnly cuda, rocm, xpu, npu, neuron and tpu devices supported, got: :r)repo_id layer_namerrevisiontrust_remote_code)kernelsrr split) r&r repo_namemodecompatible_mappingrr'r(rrepo_layer_namer%s radd_to_mappingr/:s(''''' DDDf^dffgggooc**1-Oooc""1%G //*!"3   &z"""cddlm}ddlm}|dvrt d||dd}|dd}||||||ii||<dS) Nr)Path)LocalLayerRepositoryrr#r$r) repo_pathr&)pathlibr2r)r3r r*) r&rr+r,r-r2r3r.r4s radd_to_mapping_localr6Ps,,,,,, DDDf^dffgggooc**1-O$$Q'I &&$y//* &z"""r0c8eZdZdZd dZ d dZdZdZd d ZdS) KernelConfigze Kernel configuration class. This class is used to configure the kernel mapping for a model. NFc8||ni|_i|_||_dS)N)kernel_mappingregistered_layer_namesuse_local_kernel)selfr:r<s r__init__zKernelConfig.__init__hs*0>0JnnPR&(# 0r0rc Jddlm} ||| |||||ii|j|<dS)Nrr)r%r&r'rr()r)rr:) r=r%registered_namer&rr,r'rr(rs r update_kernelzKernelConfig.update_kernelmsZ ,+++++ oo#)%#&7  0 O,,,r0cx|D]$\}}t|dr|j|j|<%dS)Nkernel_layer_name) named_moduleshasattrrCr;)r=rnamemodules rstore_registered_layer_namesz)KernelConfig.store_registered_layer_names~sT!//11 M MLD&v233 M4:4L+D1 M Mr0c pd}||t|jts't d|dt |j|jD]\}}||jvrt d|dd}t|ttfrd|i}d}t|trA|D])\}}|}|s|d vrt d |t|tr|\}} t| ts#t d t | d | d | ddx} 't| tst d| | ddx} .| ddx} t d| d| t|trd|vsd|vrt d|d|+t d|d|dS)ar Validates the kernel_mapping to ensure that: 1. Each layer_name in the mapping is registered in the model (i.e., the model contains a module with a matching kernel_layer_name). 2. Each kernel value is - either a string of the form 'org/repo:layer_name' or a tuple with the same as string and a dict of {"revision"/"version/trust_remote_code": ...}, - or a dict mapping device types ("cuda", "rocm", "xpu", "npu") to such values as above. 3. Each device key in a dict is one of "cuda", "rocm", "xpu", or "npu". 5. Each trust remote code key must be a bool. 6. Each revision or version key must exist mutually exclusive if it has been passed explicitly. 7. Each repo_name is a valid repository and layer name in the format 'org/repo:layer_name' (i.e., a string containing both a slash and a colon). 8. If a local path is detected, it should be in the format '/abs/path:layer_name', where the absolute path points to the kernel repository, like "/home/user/layer_norm". Args: model: The model instance whose modules are checked for registered kernel_layer_name attributes. Raises: ValueError: If a layer_name is not registered in the model, if a device is not supported, or if a repo_name is not a valid 'org/repo:layer_name' string. a} For single device form remote { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, For multiple devices form remote { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", "rocm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } You can also pass metadata along to inform about specific kernel information { "RMSNorm": ("kernels-community/layer_norm:LlamaRMSNorm", {"version": 1, "trust_remote_code": True}), ... }, For single device form local { "RMSNorm": "/abs/path:LlamaRMSNorm", ... }, For multiple devices form local { "RMSNorm": { "cuda": "/abs/path:LlamaRMSNorm", "rocm": "/abs/path:LlamaRMSNorm", ... }, ... } z7Kernel mapping must be a dict of the following format: z, got: zLayer z[ is not registered in the model, please register it first using use_kernel_forward_from_hubFNTrr#z^The passed metadata as second entry in a tuple needs to be a dict but found: type(metadata) = z for .r(z6Expected a bool value for `trust_remote_code` but got r'rzExpected valid combination for version/revision (mutually exclusive but one of them) to be passed when passed as tuple, but got revision= z and version= /r$zKernel mapping for 'zl' must be a valid repo name with a layer name (e.g., 'org/repo:layer_name' or '/abs/path:layer_name'), got: z'Kernel mapping must follow the format: ) rH isinstancer:dictr ritemsr;valuesstrtuplegetbool) r=rMAPPING_FORMATr&kernelskip_device_checkrrepor+metadatar(r'rs rsanitize_kernel_mappingz$KernelConfig.sanitize_kernel_mappings2() T ))%000$-t44 |.||aefjfyazaz|| #'"5";";"="=. l. l J!>!G!G;C!G!G!G## 2:>QSW1X1XX-e#-.?#F#F"&0$p]n$p$p'"'"!")1 Z(F(FFHO'/||It'D'DDG!U"#-!kNV!k!k_f!k!k## &i55I9M9MQT\eQeQe(w:wwluwwRf; D!!j>!j!jbh!j!jkkk]. l. lr0c rddlm}i}t|}|jD]\}}|jr|jn|j}|r ||jz}t|ttfr||i}|D]\} } | |kr |j rt|| | ||'t| trH| \} } | dd} | dd}| dd}n| } d} d}d}t|| | |||| | ||_dS) a Transforms a simple kernel_mapping of the form: { "RMSNorm": ("kernels-community/layer_norm:LlamaRMSNorm", {"version": 1, "trust_remote_code": True}), ... }, or for local path: { "RMSNorm": "/home/user/liger_kernels:LigerRMSNorm", ... }, into a nested mapping: { "RMSNorm": { "cuda": { Mode.INFERENCE: LayerRepository( repo_id="kernels-community/layer_norm", layer_name="LlamaRMSNorm", version=1, trust_remote_code=True, ) } } } or for local path: { "RMSNorm": { "cuda": { Mode.INFERENCE: LocalLayerRepository( repo_path=Path("/home/user/liger_kernels"), layer_name="LigerRMSNorm", ) } } } that's compatible with the kernels library. The device is inferred from the model's parameters if not provided. The Mode is inferred from the model's training state. r)Moder'Nrr(Fr)rr+r,r-rr'r()r)r[rr:rNtrainingTRAINING INFERENCE TORCH_COMPILErLrPrQr<r6rRr/)r=rcompiler[r-current_devicer&rUr,rrWr+rXr'rr(s rcreate_compatible_mappingz&KernelConfig.create_compatible_mappingsd !     %e,,"&"5";";"="=& &  J$)NF4==D 1d00&3,// 2(&1 &    ^++((VT4I[\\\dE** .*.'Ix'||J==H&ll9d;;G(0 5H%(P(P%% $I#HG(-%!''9#%&7     ) >1r0)NF)NrF)F) __name__ __module__ __qualname____doc__r>rArHrYrbr0rr8r8cs1111 ot    "MMM slslslj^1^1^1^1^1^1r0r8)rNF)rrrr/r6r8rgr0rrhs#"""""%%%Rjo,&s1s1s1s1s1>s1s1s1s1s1r0