tj3ddlmZddlmZer ddlmZddlmZddlm Z m Z m Z m Z m Z ddlmZe r dd lZdd lmZe jeZd ZGd d eZd S) ) TYPE_CHECKING) HfQuantizer)PreTrainedModel) Mxfp4Config)is_accelerate_availableis_kernels_availableis_torch_availableis_triton_availablelogging)get_module_from_nameN)WeightConverterceZdZUdZdZded<fdZdZdZdd d e d e fd Z dd Z ddd de fdZ dZdZdZdZed e fdZdZdZxZS)Mxfp4HfQuantizerz/ FP4 quantization using fbgemm kernels Frquantization_configc Jtj|fi|d|_dSN)super__init__triton_kernels_hub)selfrkwargs __class__s e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/quantizers/quantizer_mxfp4.pyrzMxfp4HfQuantizer.__init__2s1,77777"&c|j7 ddlm}|dd|_n#t$rtdwxYw|jS)z3Lazy import and initialize kernels only when neededNr) get_kernelz(kernels-community/gpt-oss-triton-kernelsr)versionz2kernels package is required for MXFP4 quantization)rintegrations.hub_kernelsr ImportError)rrs r_lazy_import_kernelsz%Mxfp4HfQuantizer._lazy_import_kernels6sv  " * XAAAAAA*4*5_ij*k*k*k'' X X X!"VWWW X&&s"<ctstd|jjrdSt stdt jpt jd}|j dvrF|j r,t d|dd|j_dStd|d t jr d}t!d }t#}nt jrBt j}|d k}t!d }t#}n1|j dkr d}t!d }t#}nd }d }d }|j r|s(t dd|j_dS|s(t dd|j_dS|s(t dd|j_dSn3|st)d|st)d|st)d|j s||d}|At/|t0r.|j s)d|vrt)ddSdSdSdS)NzqUsing mxfp4 quantization requires torchPlease install the latest version of torch ( pip install --upgrade torch )z9Using mxfp4 requires Accelerate: `pip install accelerate`cpu)cudaxpur$zGUsing MXFP4 quantized models requires model on cuda/xpu/cpu, but found zj, we will default to dequantizing the model to bf16. To use mxfp4, please disable the current accelerator.TzIQuantizing a model using MXFP4 requires model on cuda/xpu/cpu, but found z7. To use mxfp4, please disable the current accelerator.z3.5.0)z3.4.0FuMXFP4 quantization is only supported on GPUs with compute capability >= 7.5 (e.g T4, A100, L4, H100, or B200) or XPUs (e.g Intel® Data Center GPU Max Series). We will default to dequantizing the model to bf16.zMXFP4 quantization requires Triton: CUDA requires Triton >= 3.4.0, XPU/CPU requires Triton >= 3.5.0. Please install triton: `pip install triton`. We will default to dequantizing the model to bf16.zMXFP4 quantization requires the `kernels` package: `pip install kernels>=0.12.0`. We will default to dequantizing the model to bf16.uMXFP4 quantization is only supported on GPUs with compute capability >= 7.5 (e.g T4, A100, L4, H100, or B200) or XPUs (e.g Intel® Data Center GPU Max Series) or CPUzMXFP4 quantization requires Triton: CUDA requires Triton >= 3.4.0, XPU/CPU requires Triton >= 3.5.0. Please install triton: `pip install triton`zPMXFP4 quantization requires the `kernels` package: `pip install kernels>=0.12.0` device_mapdiskzYou are attempting to load an FP4 model with a device_map that contains a disk device.This is not supported when the model is quantized on the fly. Please use a quantized checkpoint or remove the disk device from the device_map.)r r!r dequantizer torch acceleratorcurrent_acceleratordevicetype pre_quantizedlogger warning_once RuntimeErrorr& is_availabler r r%get_device_capability ValueErrorr"get isinstancedictvalues) rargsrr/is_device_supported_mxfp4triton_availablekernels_installedcompute_capabilityr)s rvalidate_environmentz%Mxfp4HfQuantizer.validate_environmentAs!## ]   # .  F&(( [YZZ Z"6688OEL))#^^,=r2rJJFF#^^,={BPRSSF)6>ETT$19>)B##7= d22T22236< d22T22233 ="8##rcdS)NTrs ris_serializablez Mxfp4HfQuantizer.is_serializablestrc:tddS)NzMXFP4 quantization don't support training, please consider dequantizing the model first by passing quantization_config=Mxfp4Config(dequantize=True) to .from_pretrained()F)r2r3rs r is_trainablezMxfp4HfQuantizer.is_trainables' x   urc$ddlm}||S)Nr) Mxfp4Quantize)integrations.mxfp4r)rrs rget_quantize_opsz!Mxfp4HfQuantizer.get_quantize_opss$666666}T"""rc*ddlm}m}|jrI|jjr=t ddgd||gt ddgd g||ggSt ddgd ||gt ddgd||ggS) Nr)Mxfp4DequantizeMxfp4Deserializedown_proj_blocksdown_proj_scalesz down_proj$)source_patternstarget_patterns operationsgate_up_proj_blocksgate_up_proj_scalesz gate_up_proj$)rrrr1rr+r)rrrs rget_weight_conversionsz'Mxfp4HfQuantizer.get_weight_conversionss JJJJJJJJ   $":"E %79K$L$1 / 5 56  %:rs7! 80000007777773222225LLL444444  H % %Q Q Q Q Q {Q Q Q Q Q r