o sh5@sddlZddlmZmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlm?Z?ddl@mAZAddlBmCZCddlDmEZEddlFmGZGide%de)de+d e7d!e!d"e?d#eAd$e5d%e/d&e9d'e;d(e-d)e1d*eEd+e'd,eGd-eCe3e#e=d.ZHide de de d%ed ed!e d"ed#ed$ed'ed(ed)ed&ed*ed+e d,ed-eee ed.ZIeJeKZLGd/d0d0ZMGd1d2d2ZNd3eOfd4d5ZPd6eOfd7d8ZQd9d:ZRdS);N)OptionalUnion) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8Config FPQuantConfig GPTQConfig HiggsConfig HqqConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FPQuantHfQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerawqbitsandbytes_4bitbitsandbytes_8bitgptqaqlmquantoquarkfp_quanteetqhiggshqqzcompressed-tensors fbgemm_fp8torchaobitnetvptqspqr)fp8z auto-roundmxfp4c@s.eZdZdZedefddZeddZdS)AutoQuantizationConfigz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictcCs|d}|dds|ddr!|ddrdnd}tj|}n|dur)td|tvr;td|d ttt|}||S) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsrErFsuffix target_clsrYZ/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/transformers/quantizers/auto.pyrUys    z AutoQuantizationConfig.from_dictcKsVtj|fi|}t|dddurtd|d|j}||}|jdi||S)Nquantization_configz)Did not found a `quantization_config` in z2. Make sure that the model is correctly quantized.rY)rfrom_pretrainedgetattrrPr[rUupdate)rVpretrained_model_name_or_pathkwargs model_configrEr[rYrYrZr\s  z&AutoQuantizationConfig.from_pretrainedN)__name__ __module__ __qualname____doc__ classmethoddictrUr\rYrYrYrZrDss rDc@sdeZdZdZedeeeffddZeddZ edeeefde efdd Z e d d Z d S) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. r[cKsxt|tr t|}|j}|tjkr|jr|d7}n|d7}|tvr0t d|dt t t|}||fi|S)NrJrIrLrM) isinstancergrDrUrFrrOrGrSrPrRrT)rVr[r`rFrXrYrYrZ from_configs     zAutoHfQuantizer.from_configcKstj|fi|}||S)N)rDr\rj)rVr_r`r[rYrYrZr\s zAutoHfQuantizer.from_pretrainedquantization_config_from_argscCs|durd}nd}t|trt|trt|}nt|}|dur9|jj|jjkr9td|jjd|jjdt|tt tt t t frg|durg| }|D] \}}t|||qP|dt|d7}|dkrwt|t swt||St||S) z handles situations where both quantization_config from args and quantization_config from model config are present. NzYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)rirgrrUrD __class__rbrPrr rr rget_loading_attributesitemssetattrrRrTwarningswarnloggerinfo)rVr[rk warning_msgloading_attr_dictattrvalrYrYrZmerge_quantization_configss8       z*AutoHfQuantizer.merge_quantization_configscCs|dd}|dds|ddr"|ddrdnd}tj|}n|dur*td|tvr@td|d tt d dSd S) NrFrGFrHrIrJrKrLrMz~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) rNrrOrPrQrswarningrRrSrT)rErFrWrYrYrZsupports_quant_methods   z%AutoHfQuantizer.supports_quant_methodN)rbrcrdrerfrrrgrjr\rry staticmethodr{rYrYrYrZrhs   4rhmethodcfdd}|S)z-Register a custom quantization configuration.cs6tvr tddt|tstd|t<|S)NzConfig '' already registeredz*Config must extend QuantizationConfigMixin)rQrP issubclassr TypeErrorrVr}rYrZregister_config_fn   z8register_quantization_config..register_config_fnrY)r}rrYrrZregister_quantization_config   rnamecr~)zRegister a custom quantizer.cs6tvr tddt|tstd|t<|S)Nz Quantizer 'rz!Quantizer must extend HfQuantizer)rSrPrrrrrYrZregister_quantizer_fnrz1register_quantizer..register_quantizer_fnrY)rrrYrrZregister_quantizerrrc Cst|d}|rt|jsd}|s|dur,|r t|j||_n||_tj|j|d} nd} | dur^| j|||||d| |}| |}| |}t | jdds^| jj } t | d| |d<| |||fS)Nr[F) pre_quantized)dtypefrom_tf from_flax device_map weights_only dequantizevaluequant) hasattrrhr{r[ryrjvalidate_environment update_dtypeupdate_device_mapupdate_tp_planr]rF) configr[rrrrr user_agentr hf_quantizerrFrYrYrZget_hf_quantizer*s:      r)Srqtypingrrmodels.auto.configuration_autorutilsrutils.quantization_configrrr r r r r rrrrrrrrrrrrrrbaserquantizer_aqlmrquantizer_auto_roundr quantizer_awqr quantizer_bitnetr!quantizer_bnb_4bitr"quantizer_bnb_8bitr#quantizer_compressed_tensorsr$quantizer_eetqr%quantizer_fbgemm_fp8r&quantizer_finegrained_fp8r'quantizer_fp_quantr(quantizer_gptqr)quantizer_higgsr* quantizer_hqqr+quantizer_mxfp4r,quantizer_quantor-quantizer_quarkr.quantizer_spqrr/quantizer_torchaor0quantizer_vptqr1rSrQ get_loggerrbrsrDrhstrrrrrYrYrYrZs  \                                )n