tj&;ddlmZddlmZmZerddlZddlmZddlmcm Z ej e Z dZdejdejfdZejd ejd ejdejfd ZGd d ejZGddejjZGddejjZGddejZddeedzfdZGddZdS))should_convert_module)is_torch_availableloggingNquantized_weightsreturnc|j}|dtzdz tz}t|dkr|f}n|g|ddR}|dz }tj||jtj}|tj}tt|d|zdz}t|D]C}||z}t||z|d} |d| |z xx||| d|zzzcc<D|S)a Packs a tensor of quantized weights into a compact format using 2 bits per value. Parameters: ----------- quantized_weights : torch.Tensor A tensor containing ternary quantized weights with values in {-1, 0, 1}. These values are adjusted to {0, 1, 2} before being packed. Returns: -------- torch.Tensor A packed tensor where each element stores 4 quantized values (each using 2 bits) in an 8-bit format. rNdevicedtyper) shapeVALUES_PER_ITEMlentorchzerosr uint8tominrange) roriginal_shaperow_dimpacked_tensor_shapepackedunpackeditistartends ^/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/integrations/bitnet.py pack_weightsr"s0 ',Na ?2Q6?JG >a&j&<);<< [,5F5MUZU` a a aF ##EK00H _~a0G;q@ A AB 2YY@@G %'/>!#455#+8E#I#6!a%#?? Mrrc|j}t|dkr|dtz}|f}n|dtz}|g|ddR}tj||jtj}ttD]0}||dz}||dz}dd|zz} || zd|zz |||<1||dz S)u2 Unpacks a tensor of quantized weights that were stored in a packed format using 2 bits per value. Parameters: ----------- packed : torch.Tensor A tensor containing packed weights where each element represents 4 quantized values (using 2 bits per value). dtype : torch.dtype The dtype of the returned Tensor Returns: -------- torch.Tensor A tensor of unpacked weights, where each value is converted from its packed 2-bit representation. Example: -------- packed = torch.tensor([[0b10100001, 0b00011000], [0b10010000, 0b00001010]], dtype=torch.uint8) # Unpack the values unpacked = unpack_weights(packed) # Resulting unpacked tensor print(unpacked) # Output: tensor([[ 0, -1], [-1, 1], [-1, 1], [-1, 1], [ 1, 0], [ 0, -1], [ 1, -1], [ 1, -1]]) Explanation of the example: --------------------------- Let's take the first value for example 0b10100001, we will only focus on the first column, because every element is unpacked across the first dimension - First 2 bits: `01` → 0 at [0][0] - Second 2 bits: `00` → -1 at [0][2] - Third 2 bits: `10` → 1 at [0][4] - Fourth 2 bits: `10` → 1 at [0][6] the second value of the same row (0b10010000) will give the values for [0][1], [0][3], [0][5], [0][7] We subtract 1 because during the packing process, it's easier to work with values like 0, 1, and 2. To make this possible, we add 1 to the original ternary weights (which are typically -1, 0, and 1) when packing them. When unpacking, we reverse this by subtracting 1 to restore the original ternary values. r rNr r) rrrrrr rrr) rr packed_shapeoriginal_row_dimunpacked_shaperrrr masks r!unpack_weightsr*7sb\!""-=>>{>&-u{SSSH ? # #99LO#l1o%QU|%}!a%8s ;;u   !!r#c eZdZ ddededededef fd Zejdd Z ejd Z d Z xZ S) BitLinearNFư> in_features out_featuresbias use_rms_norm rms_norm_epsct||_||_||_|dt j|tz|ft j ||dt j d|||r,|dt j|||nd|_ d|_ |rddl m}||||_ dSdS) Nweightrr weight_scaler r0r LlamaRMSNormeps)super__init__rr.r/register_bufferrrrronesr0rms_normmodels.llama.modeling_llamar8) selfr.r/r0r rr1r2r8 __class__s r!r<zBitLinear.__init__}s6  &(   K0+>k         J          l5Y_)`)`)` a a a aDI  H B B B B B B(L,GGGDMMM H Hr#cLd|dz z }d|dz zdz }||ddjdz }||z||}|t j|fS)a\ Activation function : Performs symmetric, per-token quantization on the input activations. Parameters: ----------- input : torch.Tensor Input activations to be quantized. num_bits : int, optional (default=8) Number of bits to use for quantization, determining the quantization range. Returns: -------- result : torch.Tensor Quantized activation tensor, with values mapped to an `int8` range. scale : torch.Tensor The per-channel scaling factors used to quantize the tensor. rr Tdimkeepdimh㈵>r)absmaxvaluesclamproundrrint8)rAinputnum_bitsQnQpscaleresults r!activation_quantzBitLinear.activation_quants$X\" # 8a< 1 $UYY[[__T_::AGGDGQQQ%-&&((..r266yy$$e++r#c|||zz }|SN)rArQ input_scaler6outs r!post_quant_processzBitLinear.post_quant_processs{\12 r#c|j||}|j}t||j}||\}}t j||j|}|||j |}|j 1||j dd |z }|S)Nrr rE) r?r4r*rrWFlinearrr]r6r0view expand_as)rArQww_quant input_quantr[ys r!forwardzBitLinear.forwards = $MM%((E K $*555#'#8#8#?#? [ H[^^DJ// 9 9  # #At'8+ F F 9 2&&0033 3Ar#)NNFr-)rC) __name__ __module__ __qualname__intboolfloatr<rcompilerWr]rh __classcell__rBs@r!r,r,|s ""(H(H(H(H (H(H(H(H(H(H(H(HT ],,,],. ]]       r#r,cXeZdZdZeejdZedZdS) WeightQuanta Implements a custom autograd function for weight quantization. This performs ternary quantization (-1, 0, 1) based on scaling by the mean absolute value of the weights. It uses the Straight-Through Estimator (STE) for the backward pass. c8|j}|}d|dz }||zdd|z }||S)Ng?rIrJrEr )rrnrKmeanclamp_rOrNr)ctxr4rrUs r!rhzWeightQuant.forwards fjjll''))00T0:::5.''))//A66>yyr#c.|}|SrYclonerw grad_output grad_inputs r!backwardzWeightQuant.backward &&(( r#N rirjrk__doc__ staticmethodrrorhr~rZr#r!rsrss` ]  ]\ \r#rscXeZdZdZeejdZedZdS)ActQuanta8 Implements a custom autograd function for activation quantization. This performs symmetric 8-bit quantization (to the range [-128, 127]) based on the maximum absolute value along the last dimension (per-token/row scaling). It uses the Straight-Through Estimator (STE) for the backward pass. cH|j}|}d|ddjdz }||zdd|z }||S)NrETrFrIrJi) rrnrKrLrMrvrOrNr)rw activationrrUs r!rhzActQuant.forwards %%'' jnn&&**r4*@@GNNSWNXXX 5(//1177cBBUJ }}U###r#c.|}|SrYryr{s r!r~zActQuant.backwardrr#NrrZr#r!rrs` ]$$]\$\r#rc ReZdZ ddedededed ed ef fd Zd Zd ZxZ S) AutoBitLinearTNFr-r.r/r0 online_quantr1r2c 0t|||||_d|_|rddlm} | |||_|sG|dtjd||| |j dSdS)Nrr7r9r6r r5) r;r<rr?r@r8r=rr>"_register_load_state_dict_pre_hook load_hook) rAr.r/r0r rrr1r2r8rBs r!r<zAutoBitLinear.__init__s lD999(  H B B B B B B(L,GGGDM D   !     3 3DN C C C C C D Dr#c|dz|vrH||dzj|jjkr*t||dz|jj||dz<|S)Nr4r_)rr4r*)rA state_dictprefixargskwargss r!rzAutoBitLinear.load_hook se X * , ,FX+K+^+^!.$*$6%+%8#[4%}3$m1&9&Kx&W%8%E%8%E " " "J+< II"11%888!*$*$6%+%8#[4%}3$m1I\%g%8%E%EbgI\%f%8%E%Ebf"""J--e444##K<<<$(!7 ) ) ) ) ) ) ) ) ) ) ) ) ) ) ):      LsDE  E$ 'E$ c eZdZdZ ddeeeejfdej j dzdedzdeeejffdZ dS) BitNetDeserializec||_dSrY) hf_quantizer)rArs r!r<zBitNetDeserialize.__init__vs(r#N input_dictrfull_layer_namer c |D]%\}}t|tr |d||<&d}||}ddlm} d} |j} |q|o| ||\} } t| drPt| dr@| j}|j d}|tz|krd} t| d r | j j} | r0| tj}t|| }||iS) Nrr4r)get_module_from_nameFr/r.Tr6r_)itemsrlistpopquantizers.quantizers_utilsrrhasattrr/rrr6rrrr*)rArrrrkeyvalue key_weightr4rneeds_unpacking target_dtyper_ expected_out actual_out weight_uint8s r!convertzBitNetDeserialize.convertys@%**,, + +JC%&& +"'( 3  ++FFFFFF|  !<,,UODDIFAv~.. A76=3Q3Q A &2 #\!_ /<??&*Ov~66A'-':'@  F!99U[11L#L EEEFF##r#NN) rirjrkr<dictstrrrTensorrModulerrZr#r!rrus))) )-&* !$!$d5<001!$x%!$t !$ c5< !$!$!$!$!$!$r#rr) rrutilsrrrtorch.nnrtorch.nn.functional functionalr` get_loggerrirrrr"rorr*rr,autogradFunctionrsrrrrrrrrZr#r!rs??????////////$LLL#########  H % %#EL#U\####LA"5<A" A" A"A"A"A"HTTTTT TTTn%.).u~&.77777BI777t77d3i$>N7777t%$%$%$%$%$%$%$%$%$%$r#