h]<ddlZddlmZddlmZddlmZmZmZddl m Z ddl m Z ddl mZerd d lmZd d lmZmZmZmZmZmZmZmZer ddlZd d lmZej:eZGd de Z y)N) defaultdict)cached_property) TYPE_CHECKINGOptionalUnion)version) HfQuantizer)get_module_from_name)PreTrainedModel)ACCELERATE_MIN_VERSIONis_accelerate_availableis_bitsandbytes_availableis_torch_availableis_torch_hpu_availableis_torch_npu_availableis_torch_xpu_availablelogging)Conv1Dc4eZdZdZdZdZdZddgZfdZdZ d d Z d e e de e fd Z d d de defdZd d ddde ddfdZdee eee ffdee eee fffdZd!dZdZ d"d d dee e fdZd#dZd"dZedefdZedefdZdZxZ S)$Bnb4BitHfQuantizera 4-bit quantization from bitsandbytes.py quantization method: before loading: converts transformer layers into Linear4bit during loading: load 16bit weight and pass to the layer object after: quantizes individual weights in Linear4bit into 4bit at the first .cuda() call saving: from state dict, as usual; saves weights and `quant_state` components loading: need to locate `quant_state` components and pass to Param4bit constructor TF bitsandbytes acceleratec 2t||fi||jj|jj|_d|jj ddg|_|jjr|j jddgyy)Nzquant_state.bitsandbytes__absmax quant_map nested_absmaxnested_quant_map) super__init__quantization_configllm_int8_skip_modulesmodules_to_not_convertbnb_4bit_quant_typebnb_keysbnb_4bit_use_double_quantextend)selfr"kwargs __class__s i/var/www/html/aiagenthome/venv/lib/python3.12/site-packages/transformers/quantizers/quantizer_bnb_4bit.pyr!zBnb4BitHfQuantizer.__init__As ,77  # # 9 9 E*.*B*B*X*XD '))A)A)U)U(V W     # # = = MM /3E!F G >crtstdtdtds tdt s tdt j tjj dt j dkr)tjjs td d d l m }d d lm}|}|d |j!dds|j!ddr t#d|j!d}|t%|t&r|j(j*sr|Dcic]}||j,vs|||}}t/|j1dhk(r|ryd|j1vsd|j1vr t#dyyyycc}w)NzWUsing `bitsandbytes` 4-bit quantization requires Accelerate: `pip install 'accelerate>=z'`T)check_library_onlyzrUsing `bitsandbytes` 4-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes`zThe bitsandbytes library requires PyTorch but it was not found in your environment. You can install it with `pip install torch`.rz0.43.1zThe installed version of bitsandbytes (<0.43.1) requires CUDA, but CUDA is not available. You may need to install PyTorch with CUDA support or upgrade bitsandbytes to >=0.43.1.r )!validate_bnb_backend_availability)'is_bitsandbytes_multi_backend_available)raise_exceptionfrom_tfF from_flaxzConverting into 4-bit or 8-bit weights from tf/flax weights is currently not supported, please make sure the weights are in PyTorch format. device_mapcpudiskaSome modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. )r ImportErrorrrrrparse importlibmetadatatorchcuda is_available integrationsr0utilsr1get ValueError isinstancedictr" llm_int8_enable_fp32_cpu_offloadr$setvalues) r)argsr*r0r1bnb_multibackend_is_enabledr5keydevice_map_without_lm_heads r,validate_environmentz'Bnb4BitHfQuantizer.validate_environmentPs&(ikAjBBDE )DAE "#?  ==++33NC Dw}}U]G^ ^::**,!m EC&M&O#)$? ::i '6::k5+I;  ZZ -  ":t,,,MM1;*0:cIdId>dZ_$  '*:$$&'E727R4;;==KeKlKlKnAn )BoN- #*s F4F4returnctjtjjdtjdkDr:ddlm}|t jk7rtjd|jStd)Nrz0.19.0r) CustomDtypezXtarget_dtype {target_dtype} is replaced by `CustomDtype.INT4` for 4-bit BnB quantizationaUYou are using `device_map='auto'` on a 4bit loaded version of the model. To automatically compute the appropriate device map, you should upgrade your `accelerate` library,`pip install --upgrade accelerate` or install it from source to support fp4 auto device mapcalculation. You may encounter unexpected behavior, or pass your own device map) rr9r:r;accelerate.utilsrOr<int8loggerinfoINT4rB)r) target_dtyperOs r,adjust_target_dtypez&Bnb4BitHfQuantizer.adjust_target_dtypesi ==++33LA BW]]S[E\ \ 4uzz) vw## #b r-unexpected_keyscn|Dcgc]#tfd|jDr"%c}Scc}w)Nc3@K|]}j|ywNendswith).0xks r, z.s5[]ajjm])anyr&)r)modelrWr_s `r,update_unexpected_keysz)Bnb4BitHfQuantizer.update_unexpected_keyss,*\?a#5[T]]5[2[?\\\s#22rcr param_namec ddl}tfd|jDryt|\}}t ||j j xr|dk7S)Nrc3@K|]}j|ywrZr[r]r^res r,r`z>Bnb4BitHfQuantizer.param_needs_quantization..s=}!z""1%}raTbias)rrbr&r rCnn Linear4bit)r)rcrer*bnbmodulenames ` r,param_needs_quantizationz+Bnb4BitHfQuantizer.param_needs_quantizationsM" =t}}= =+E:> &#&&"3"34GGr- param_valuez torch.Tensor target_devicez torch.devicec ddl}tfd|jD}}|r.dvrjdddnjdddt |\} } t |t rtrd|}|jrjddd} t|dstt|_ |j| j||it|j| t|jdzk(ri} |jr| | d <|j| j!| d } |j"j$j&d| |j| d |d | }|| j(| <|j| =yy|j+d }t-| | }t/| j0t2r |j4}|j6}|j!dd|j"j$|fdd i|j+|}|| j(| <y)Nrc3@K|]}j|ywrZr[rhs r,r`z.sJMqJ//2Mraz quant_state..r r npu:param_quant_statsrmz.weightF)dataquantized_stats requires_graddevicer6_is_hf_initializedry)rrbr&rsplitr rCintr pre_quantizedhasattrrrDrvupdatelen$is_bnb_supports_quant_storage_modulepoprj Params4bitfrom_prequantized _parameterstogetattr issubclass source_clsrT__dict__)r)rcrprerqr*rl is_quant_stat full_namerm tensor_name module_name param_kwargsweight new_value old_values ` r,create_quantized_paramz)Bnb4BitHfQuantizer.create_quantized_params? #JDMMJJ  0>j0P !!#q)!,V`VgVghkmnVopqVr 35*E  mS ).D.F"=/2M   $++C3A6K4!45)4T):&  " "; / 6 6 ;7O P4))+673t}};MPQ;QQ! <<-3L*// <@@K=PWAXY?CFF--??$($:$:;$G"'(  #  3<"";/**;7!R$$u-I 4I&++V4%KK ''F JJ+T 2))))S5SFSVVWdeI.7F  { +r- max_memoryc^|jDcic] \}}||dz }}}|Scc}}w)Ng?)items)r)rrJvals r,adjust_max_memoryz$Bnb4BitHfQuantizer.adjust_max_memorys96@6F6F6HI6H(#sc3:o6H IJs)cV|&tjd|tj}|S)NzOverriding dtype=%s with `dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in 8-bit or 4-bit. Pass your own dtype to specify the dtype of the remaining non-linear layers or pass dtype=torch.float16 to remove this warning.)rRrSr<float16)r)dtypes r, update_dtypezBnb4BitHfQuantizer.update_dtypes- = KK?  MME r-c|tjjr!dtjji}nt r$ddtj ji}n]t r$ddtjji}n/tr!dtjji}nddi}tjd|d|S)Nruzhpu:r6z:The device_map was not initialized. Setting device_map to zL. If you want to use the model for inference, please set device_map ='auto' ) r<r=r>current_devicernpurhpurxpurRrS)r)r5s r,update_device_mapz$Bnb4BitHfQuantizer.update_device_maps  zz&&( %**";";"=> ') D)A)A)C(D"EF ') D)A)A)C(D"EF ') %))":":"<=  %[ KK))3 5]]  r-keep_in_fp32_modulesc &ddlm}|jj}|j ||jj ||_t|tryt|jdkDr]|jDcgc] \}}|dvs |} }}t| dkDr |s td|j j| |||j |j}|j|j_ycc}}w)Nr )replace_with_bnb_linearr )r7r6rzIf you want to offload some keys to `cpu` or `disk`, you need to set `llm_int8_enable_fp32_cpu_offload=True`. Note that these modules will not be converted to 8-bit but kept in 32-bit.)r$r")r?rr"rEget_modules_to_not_convertr#r$rCrDrkeysrrBr(config) r)rcr5rr*rrErJvalue keys_on_cpus r,$_process_model_before_weight_loadingz7Bnb4BitHfQuantizer._process_model_before_weight_loading s ;+/+C+C+d+d(&*&E&E 4++AACW' # j$ 'C 0A,BQ,F1;1A1A1C`1C:3uP_G_31CK`;!#,L >  ' ' . .{ ;' $*E*E[_[s[s ,0+C+C (as  D D c >d|_|j|_|SNT)is_loaded_in_4bitis_serializableis_4bit_serializable)r)rcr*s r,#_process_model_after_weight_loadingz6Bnb4BitHfQuantizer._process_model_after_weight_loading,s "&%)%9%9%;" r-ctjtjjdtjdk\}|stj dyy)Nrz0.41.3zYou are calling `save_pretrained` to a 4-bit converted model, but your `bitsandbytes` version doesn't support it. If you want to save 4-bit models, make sure to have `bitsandbytes>=0.41.3` installed.FT)rr9r:r;rRwarning)r)safe_serialization_is_4bit_serializables r,rz"Bnb4BitHfQuantizer.is_serializable1sQ ' i.@.@.H.H.X Y]d]j]jks]t t$ NNh r-ctjtjjdtjdk\S)z determines if the current version of bitsandbytes supports the `module` parameter in `Params4bit.from_prequantized` :return: rz0.43.3)rr9r:r;r)s r,rz7Bnb4BitHfQuantizer.is_bnb_supports_quant_storage_module=s3}}Y//77GHGMMZbLcccr-cyrr|rs r, is_trainablezBnb4BitHfQuantizer.is_trainableFsr-cPddlm}|||j|j}|S)Nr )dequantize_and_replace)r")r?rr$r")r)rcrs r, _dequantizezBnb4BitHfQuantizer._dequantizeJs)9& 4..DD\D\  r-)rU torch.dtyperMr)rrrMrrZ)rcr )!__name__ __module__ __qualname____doc__use_keep_in_fp32_modules requires_parameters_quantizationrequires_calibrationrequired_packagesr!rLrVliststrrdboolrorrDrr~rrrrrrrrrpropertyrr __classcell__)r+s@r,rr0s[ $'+$ '6 H6p ]T#Y]4PS9]H.?HSH_cH=8 =8$=8 =8 & =8@DeCHo1E,F4PSUZ[^`c[cUdPdKe 059 D D'tCy1 DD dddddr-r)!r: collectionsr functoolsrtypingrrr packagingrbaser quantizers_utilsr modeling_utilsr r@rrrrrrrrr< pytorch_utilsr get_loggerrrRrr|r-r,rsd#%1120   &   H %``r-