tj) hdZddlmZmZddlmZddlmZmZer ddl Z ddl m Z ej e ZdadZGdd e jZ dd eedzd efd Zde jdedefdZde jde jde jdedef dZGddeZGddeZdS)a Metal affine quantization integration for transformers. This module provides: - ``MetalLinear``: a drop-in replacement for ``nn.Linear`` that stores weights as affine-quantized uint32 packed tensors and uses the ``quantization-mlx`` Metal kernels for the forward pass. - ``replace_with_metal_linear``: walks a model and swaps every eligible ``nn.Linear`` with ``MetalLinear``. - ``MetalQuantize`` / ``MetalDequantize``: weight conversion operations that participate in the new ``WeightConverter`` pipeline. Weight layout (transposed, matching ``affine_qmm_t``): - ``weight``: ``[N, K_packed]`` (``uint32``) -- K is the packed dimension. - ``scales``: ``[N, K // group_size]`` (``float16 / bfloat16``) - ``qbiases``: ``[N, K // group_size]`` (same dtype as scales) The kernel call is ``affine_qmm_t(x, weight, scales, qbiases, group_size, bits)`` which computes ``y = x @ dequant(weight).T``, identical to ``nn.Linear``. ) ConversionOps _IdentityOp)should_convert_module)is_torch_availableloggingNct; ddlm}|ddan&#t$r}t d|d|d}~wwxYwtS)z>Lazily load the quantization-mlx kernel from Hugging Face Hub.N) get_kernelz0kernels-community/mlx-quantization-metal-kernels)versionz9Failed to load the quantization-mlx kernel from the Hub: zm. Make sure you have `kernels` installed (`pip install kernels`) and are running on an Apple Silicon machine.) _metal_kernel hub_kernelsr Exception ImportError)r es j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/integrations/metal_quantization.py_get_metal_kernelr3s  / / / / / /&J'YcdeeeMM   ?A???    s A;Ac feZdZdZdejddfdedededed ef d Zd ej d ej fd Z dS) MetalLinearz A quantized linear layer that stores weights in affine uint32 packed format and uses the ``quantization-mlx`` Metal kernels for the forward pass. Parameters match ``nn.Linear`` with additional quantization metadata. F in_features out_featuresbiasbits group_sizectj|||_||_||_||_d|z}||z}||z} |tjkr;tj tj ||tjd|_ n0tj tj |||d|_ |tjkr tj nd} tj tj || | d|_ tj tj || | d|_|r-tj tj ||_dS|dddS)N )dtypeF) requires_gradr)nnModule__init__rrrrtorchuint32 Parameterzerosweightfloat32scalesqbiasesrregister_parameter) selfrrrrrr elems_per_intk_packedn_groups scales_dtypes rr#zMetalLinear.__init__Qsc 4   &( $d -/*, EL ,u{<QVQ]'^'^'^nstttDKK,u{<TY'Z'Z'ZjopppDK(-(=(=u}}4 l5;|X\#Z#Z#Zjoppp |EK hl$[$[$[kpqqq  2 U[%>%>??DIII  # #FD 1 1 1 1 1inputreturnc|jjtjkr+tj||j|jSt}| ||j|j |j|j |j|j |j}|j ||jz}|SN)r(rr$r%r! functionallinearrr affine_qmm_tr*tor+rr)r-r3kerneloutputs rforwardzMetalLinear.forwardss ;  , ,=''t{DIFF F"$$$$  K KNN5; ' ' LOOEK ( ( O I    9 di'F r2N) __name__ __module__ __qualname____doc__r$r%intboolr#Tensorr=r2rrrIsl 2 2 2 2 2  2 2 2 2 2DU\elr2rFmodules_to_not_convert pre_quantizedc |jr|S|j}|j}d}|D]s\}}t ||st |t jrC|rinddi} td|j |j |j du||d| } | || d}t|std|S)a` Replace every eligible ``nn.Linear`` with ``MetalLinear``. Args: model: the ``PreTrainedModel`` (on the meta device at this point). modules_to_not_convert: module names to leave untouched. quantization_config: the ``MetalConfig`` instance. pre_quantized: ``True`` when loading from a quantized checkpoint. FrN)rrrrrTzYou are loading a model with Metal quantization but no nn.Linear modules were found. Please double check your model architecture.rE) dequantizerr named_modulesr isinstancer!Linearrrrr set_submoduleloggerwarning) modelrFquantization_configrGrrhas_been_replaced module_namemodule module_kwargs new_modules rreplace_with_metal_linearrWs%  #D$/J$2244%% V$[2HII   fbi ( ( %"/DBBgt_M$".#0[,%   J    Z 8 8 8 $    ;   Lr2r(rrc2|j\}}d|z}d|zdz }||z}||||}|dj} |dj} | | z |z d} | } || dz | dz } | d| tj ||} ||z}tj ||tj |j }t|D]}|| d d |d |f||zzz}| tj| | fS) aP Quantize a 2-D float weight ``[N, K]`` into packed uint32 + scales + biases. Returns ``(w_packed, scales, biases)`` with: - ``w_packed``: ``[N, K // (32 // bits)]`` uint32 - ``scales``: ``[N, K // group_size]`` float32/float16/bfloat16 - ``biases``: ``[N, K // group_size]`` float32/float16/bfloat16 rr )dimg:0yE>)minrrdeviceN)shapefloatreshaper[valuesmaxclamp unsqueezeroundr:r$int32r'r]ranger%)r(rrNKr.max_valr0 w_groupedw_minw_maxr*biasesw_intr/w_packedis r_affine_quantize_tensorrrs a 49&:E #:5*d#S#S &&/2j/@/@z  a((++b(,:t$$$$( (,;d$$$$)[  vyy,, fii ++  r2N)r>r?r@rAr#dictrrEr2rryrysO))) $ T      r2rycLeZdZdZdZd dededzdefdZed d Z dS) MetalDequantizez Dequantize (weight, scales, qbiases) back to a full-precision tensor. Used when ``dequantize=True`` is set in the config to fall back to a normal ``nn.Linear`` on devices without MPS. c||_dSr6r{r}s rr#zMetalDequantize.__init__r~r2Nrfull_layer_namer4c 2|jjj}|jjj}t |dkr ||diS|dd}|dd}|dd}t |||||} || |jiS)Nrzweight$rr*r+)r|rQrrlenrwr:r) r-rrrrr quantizedr*r+rvs rrzMetalDequantize.converts 49&:E z??Q  #Z %:; ;y)!, H%a(Y'*))VWjRVWW&,!7!788r2rctSr6)r)r-s r reverse_opzMetalDequantize.reverse_op*s }}r2r6)r4r) r>r?r@rAr#rstrrpropertyrrEr2rrrs))) 9 9$ 9t 9Y] 9 9 9 9Xr2r)NNF)rAcore_model_loadingrrquantizers.quantizers_utilsrutilsrrr$torch.nnr! get_loggerr>rNr rrLrrrrCrWrDrBrrrwryrrEr2rrs*<;;;;;;;??????////////LLL  H % % ,;;;;;");;;@04 // I,/ ////d5EL5c55555Bl$)L:?,TW_b.     M   @mr2