tjkuddlZddlZddlZddlmZddlmZddlmZej e Z Gddej Z Gdd ej ZGd d ej ZGd d ej ZGddej ZGddej Zd dZd dZd dZGddeZGddeZGddeZd!dZGddeZdS)"N)nn)Function)loggingc>eZdZdZ d fd Zd dZxZS) QuantEmbeddinga Quantized version of `torch.nn.Embedding`. Adds quantization-specific arguments on top of `torch.nn.Embedding`. Args: weight_bit (`int`, *optional*, defaults to `8`): Bitwidth for the quantized weight. momentum (`float`, *optional*, defaults to `0.95`): Momentum for updating the activation quantization range. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. N@Fffffff?c t||_||_||_||_||_||_||_tj tj ||g|_ |dtj d|dtj|j | |_| |_| |_d|_t(j|_dS)Nweight_scaling_factorweight_integerF)super__init__num_dim padding_idxmax_norm norm_typescale_grad_by_freqsparser Parametertorchzerosweightregister_buffer zeros_like weight_bitmomentum quant_modepercentile_modeSymmetricQuantFunctionapplyweight_function) selfnum_embeddings embedding_dimrrrrr_weightrr r! __class__s e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/ibert/quant_modules.pyrzQuantEmbedding.__init__+s "  &  ""4 l5; /N#O#OPP  4ek!nnEEE -u/? /L/LMMM$  $$5;c |jsEtj||j|j|j|j|j|j dfS|j}|j }| d}| d}t|j||d|_||j|j|j|j|_tj||j|j|j|j|j|j }||jz|jfS)NrF)r!r functional embeddingrrrrrrdatadetachminexpandmax$symmetric_linear_quantization_paramsrr r%r"r) r&x positionsincremental_statew w_transformw_minw_maxemb_ints r+forwardzQuantEmbedding.forwardLs@  ''K$MN+K   Kfmmoo !!((++!!((++%I$/[`bgin%o%o""22 K$*>@Z  -))     M N  # K  33T5OOOr,) NNr FFNr r FNN)__name__ __module__ __qualname____doc__rr> __classcell__r*s@r+rrs    <<<<<;per-channel mode is not currently supported for activation.)rractivation_bitact_range_momentumr! per_channel percentiler#r$ act_functionrrrrIrJNotImplementedError)r&rMrNrO channel_lenr!r*s r+rzQuantAct.__init__s ,"4$&28 e  %+a.. 9 9 9  %+a.. 9 9 9  !5u{1~~ F F F JJ$ JJ JJ$ JJJJ%&cdd dr,c |jjd|jd|jd|jdd|jdd S)Nz(activation_bit=z, quant_mode: z , Act_min: z.2fz , Act_max: ))r*r@rMr!rIitemrJ)r&s r+__repr__zQuantAct.__repr__st~& 1 18K 1 1? 1 17;z7H7H P 1 1 )) 0 1 1 1 r,c||n||z}|jr|jr Jd|jr Jd|j}|j} | dkr*|dks Jd|jdkr<|j dkr|j|z|_|j | z|_ n|j dkr?tj|j||_tj|j | |_ nD|j|j z|d|j z zz|_|j |j z| d|j z zz|_ |j s|dfS||jn|}||j n|} t|j|| |j |_|(|||j|j|j} n)t"|||j|j||} |jd} | | z|jfS) Nz:percentile mode is not currently supported for activation.rLrz5NaN detected when computing min/max of the activationg&|g&|>r)rO)trainingrPrOr0r2r4isnansumrIrJrNrr!r5rMrKrQ FixedPointMulr$view) r&r6pre_act_scaling_factoridentityidentity_scaling_factor specified_min specified_maxx_actrIrJ quant_act_intcorrect_output_scales r+r>zQuantAct.forwardsr%8a< = j d d(d d d&' f f)f f f'JNN$$EJNN$$E;;==$$&&!++ 0A0A0C0Cq0H0H0HG1I0HH z~~'))djnn.>.>.G.G!Z%/ !Z%/ (B.."Ytz599 "Ytz599 !Z$*AAEQQUQhMhDii !Z$*AAEQQUQhMhDii  $; +3 +3 "F  4;K# # #  " ) --a1DdoW[WnooMM)//&#'' M $6;;B??33T5LLLr,)r FNF)NNNNNr@rArBrCrrWr>rDrEs@r+rGrGqs eeeeee&    $ $<M<M<M<M<M<M<M<Mr,rGc8eZdZdZ d fd ZfdZd d ZxZS) QuantLineara8 Quantized version of `torch.nn.Linear`. Adds quantization-specific arguments on top of `torch.nn.Linear`. Args: weight_bit (`int`, *optional*, defaults to `8`): Bitwidth for the quantized weight. bias_bit (`int`, *optional*, defaults to `32`): Bitwidth for the quantized bias. per_channel (`bool`, *optional*, defaults to `False`): Whether or not to use channel-wise quantization. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. Tr Fct||_||_t jt j||g|_| dt j |j| dt j|j|rXt jt j||_ | dt j |j ||_ ||_ ||_||_||_ d|_t"j|_dS)Nrfc_scaling_factor bias_integerF)rr in_features out_featuresrrrrrrrbiasrr!rObias_bitr"r#r$r%) r&rnrorprrqrOr!r*s r+rzQuantLinear.__init__s &(l5; k/J#K#KLL  -u/? /L/LMMM 0%+d>O2P2PQQQ  N U[%>%>??DI  1A$)1L1L M M M$$&  $$5;r,ctt}d|d|jd|jd}|S)N(z weight_bit=z , quant_mode=rU)rrWrr!)r&sr*s r+rWzQuantLinear.__repr__sA GG     O O Ot O OT_ O O Or,Nc|js.tj||j|jdfS| |jdks Jd|j}|j}|j r5tj |dd\}}tj |dd\}}nN| d}| d}t|j|||j |_||j|j|j|j|_|j|z}|j'||j|jd||_|dd}||z } tj| |j|j|z|fS)N)rrp)rzInput activation to the QuantLinear layer should be globally (non-channel-wise) quantized. Please add a QuantAct layer with `per_channel = True` before this QuantAct layerr)routFrY)r!rr.linearrrpshaper0r1rOrr2r4r3r5rrlr%r"rrqrmr^) r&r6prev_act_scaling_factorr9r:r;_r<bias_scaling_factorx_ints r+r>zQuantLinear.forwards U=''$+DI'NNPTT T'27N7TX\7\7\7\ _8]7\\ Kfmmoo   0y!>>>HE1y!>>>HE11OO%%,,Q//EOO%%,,Q//E!EdoW\^ceieu!v!v"22 K$*>@V  #47NN 9 $ 4 4TY uVi j jD "9">">q""E"E++ M t/BIZ [ [^q q   r,)Tr rjFFNrgrEs@r+riris{  ns<<<<<<, # # # # # # # # r,ric2eZdZdZdfd ZdZd dZxZS) IntGELUa} Quantized version of `torch.nn.GELU`. Adds quantization-specific arguments on top of `torch.nn.GELU`. Args: quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "gelu" or "nonlinear" is given. TnonecLt||_|dvr!tdd|_|jst j|_d|_d|_ gd|_ |j dxx|j dzcc<dS) N) nonlineargeluzForce dequantize geluFg -?)g]m{ҿgMrr) rrr!loggerinforGELU activation_fnkconstcoeff)r&r! force_dequantr*s r+rzIntGELU.__init__6s $ 1 1 1 KK/ 0 0 0#DO +!#D  )))  1 A& r,ctj|jd|z }tj|jd|dzz }tj|}tjtj|| }|||zdz|zz}|dz|jdz}t |d|jzz }|d|jzz}||fSNrrr) rfloorrsignr2abs floor_ster$r)r&r|scaling_factorb_intc_intrabs_inty_ints r+int_erfzIntGELU.int_erfFs DJqMN:;; DJqMNA,==>>z%  )EIe,,uf555Q.67'*TZ]:4: 566'!TZ-7n$$r,Nc|js||dfS||z }||||jz \}}d|z}|||zz}||zdz }||z|fS)N?r)r!rrr)r&r6rr| sigmoid_intsigmoid_scaling_factor shift_ints r+r>zIntGELU.forwardUs /%%a(($. .N".2ll5.SWSYBY.Z.Z+ +11 y01'*@@1D~%~55r,)Trr})r@rArBrCrrr>rDrEs@r+rr+sj'''''' % % % 6 6 6 6 6 6 6 6r,rc6eZdZdZdfd ZdZdZdZxZS) IntSoftmaxa Quantized version of `torch.nn.Softmax`. Adds quantization-specific arguments on top of `torch.nn.Softmax`. Args: output_bit (`int`): Bitwidth for the layer output activation. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "softmax" or "nonlinear" is given. Frct||_d|_||_|dvr!t dd|_td|j|_d|_ d|_ gd |_ |j d xx|j d zcc<|j d xx|j d zcc<dS) Nrj)rsoftmaxzForce dequantize softmaxFr!gvq -)gN$?g'|:?rrrr) rr output_bitmax_bitr!rrrGactx0rcoef)r&rr!rr*s r+rzIntSoftmax.__init__qs $ $ 4 4 4 KK2 3 3 3#DOB4?;;; 111  !  ! $  !  ! $ r,c*tj5tj|jd|z }tj|jd|dzz }dddn #1swxYwY||z|z|z}|jd|dzz}||fSr)rno_gradrr)r&r|rrrzs r+int_polynomialzIntSoftmax.int_polynomials ]__ B BK ! ~ =>>EK ! ~q/@ @AAE B B B B B B B B B B B B B B BU]e #e +1(99.  sAA((A,/A,ctj5tj|j|z }dddn #1swxYwYtj||j|z}t ||z }|||zz }|||\}}tj t |d|j|z zzd}|d|jzz }||fS)Nrrr2) rrrrr4rrr$rclamp)r&r|rx0_intqrexp_intexp_scaling_factors r+int_expzIntSoftmax.int_exps ]__ ; ;[>!9::F ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; %f!455 OOEFN + + FQJ &*&9&9!^&L&L##+ioogdj1n8M.MNNTUVVV+am;&&s=AAc |js#tj|ddfS||z }|dd\}}||z }|||\}}|||\}}||z }|dd} t d|j z| z } t || zd|j |j z zz }dd|j zz }||z|fS)NrYrT)rkeepdimrr) r!rr.rr4rrr\rr$rr) r&r6rr| x_int_maxrzrrexp exp_int_sumfactors r+r>zIntSoftmax.forwards :=(((33T9 9N"yyRy66 1 !&*ll5.&I&I###'((74F"G"G **kkb$k77 DL;!>??//'F"2Q4<$/;Y5Z"Z[[Q//'77r,)Fr) r@rArBrCrrrr>rDrEs@r+rrdst  %%%%%%"!!! ' ' '8888888r,rc8eZdZdZd fd ZdZdZd d ZxZS) IntLayerNorma Quantized version of `torch.nn.LayerNorm`. Adds quantization-specific arguments on top of `torch.nn.LayerNorm`. Args: output_bit (`int`, *optional*, defaults to `8`): Bitwidth for the layer output activation. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "layernorm" or "nonlinear" is given. r Frc"t||_||_t jt j||_t jt j||_ ||_ |dvr!t dd|_ | dt jd||_d|_d|_t#|j|j |_dS)N)r layernormzForce dequantize layernormFshiftrrjr)rrnormalized_shapeepsrrrrrrpr!rrrrrdim_sqrtrG activation)r&rrrr!rr*s r+rzIntLayerNorm.__init__s  0l5;/?#@#@AA L-=!>!>?? $ 6 6 6 KK4 5 5 5#DO Wek!nn555$  "4?tOOOr,c  tj5|dz}tj|dd}tjtj|d|jzz }|j}tj|j||_t dt|dt|jddddS#1swxYwYdS)NrTaxisrzDynamic shift adjustment: z -> ) rrr\log2sqrtrceilr4rrrint)r&ry_sq_intvar_intr shift_olds r+ set_shiftzIntLayerNorm.set_shifts. ]__ \ \axHiq$???GZ 7Q _+D E EFFKKMMRRTTE I4:u55DJ KKZS^^ZZTZZZ [ [ [  \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \sCC88C<?C<c||t|d|jzz }|dz}t j|dd}|S)z This fallback function is called when overflow is detected during training time, and adjusts the `self.shift` to avoid overflow in the subsequent runs. rTr)rrr$rrr\)r&r y_int_shiftedrrs r+overflow_fallbackzIntLayerNorm.overflow_fallbacksW u!4: (=>>  !#)H1d;;;r,NcR|jsk|dd}||z }tj|dzdd}|tj|j|zz }||jz|jz}|dfS|j\tj|j dtj }tj| |j |_||z }t|dd}||z } t| d|jzz } | dz} tj| dd} |jrb| d|jzkrB|| } | d|jzdzks Jdttj| d|jzz} td| z }t| |zdz } |jdz }|jj|jjz }t||z }| |z} ||jz}| |z}||fS) NrTr)dtypeg?zfError detected in overflow handling: `var_int` exceeds `self.max_bit` (the maximum possible bit width)li@)r!meanrrrrrprtensorrxfloattodevice round_ster$rrr\rZr4rrr0r1)r&r6rryvarnr|mean_intrrrrstd_intrrpbias_ints r+r>zIntLayerNorm.forwardsl 66q$6//DDA*QT4888CEJtx#~...ADK$)+Ad7N =  QWQZu{;;;A!JqMM,,QX66DMN"??5::1d:#C#CDD !4: (=>>  !#)H1d;;; = {{}}4<//0077{{}}q$,'<<<<X=<< //%*W"5"566DJF11 233.y~$$&&$+*:*A*A*C*CD??4.#899 '$+5 N ".  r,)r Frr}) r@rArBrCrrrr>rDrEs@r+rrs  PPPPPP&\\\   .!.!.!.!.!.!.!.!r,rFcZ|jd}t|d|dzz z}t||zdz}tj||j}|dkr|dz}ntj| |j }|s(|}|}||fS)a Calculate the percentile max and min values in a given tensor Args: input (`torch.Tensor`): The target tensor to calculate percentile max and min. lower_percentile (`float`): If 0.1, means we return the value of the smallest 0.1% value in the tensor as percentile min. upper_percentile (`float`): If 99.9, means we return the value of the largest 0.1% value in the tensor as percentile max. output_tensor (`bool`, *optional*, defaults to `False`): If True, this function returns tensors, otherwise it returns values. Returns: `Tuple(torch.Tensor, torch.Tensor)`: Percentile min and max value of *input* rrg{Gz?)r)rxroundrkthvaluevaluesrV) inputlower_percentileupper_percentile output_tensor input_length lower_index upper_index upper_bound lower_bounds r+get_percentile_min_maxrs";q>L ,??K.+666=K1!Ao ~uf <<<CC )!&&(( !&&((  ##r,c.t|jdkr1|dddd}|dddd}not|jdkr-|dd}|dd}n*|d}|d}|r?|d|z ||St jd|z |z|zS)a? Quantize single-precision input tensor to integers with the given scaling factor and zeropoint. Args: input (`torch.Tensor`): Single-precision input tensor to be quantized. scale (`torch.Tensor`): Scaling factor for quantization. zero_pint (`torch.Tensor`): Shift for quantization. inplace (`bool`, *optional*, defaults to `False`): Whether to compute inplace or not. Returns: `torch.Tensor`: Linearly quantized value of *input* according to *scale* and *zero_point*. rYrrr)lenrxr^mul_add_round_rr)rscale zero_pointinplaces r+linear_quantizer4s$ 5;1 2q!Q''__RAq11 U[  Q   2q!!__R++  2__R((  3;$$Z0077999 ;sU{U*Z7 8 88r,ctj5d|dz zdz }|rmtjtj||gdd\}}tj|d|z }nMt||}tj|d|z }dddn #1swxYwY|S)a/ Compute the scaling factor with the given quantization range for symmetric quantization. Args: saturation_min (`torch.Tensor`): Lower bound for quantization range. saturation_max (`torch.Tensor`): Upper bound for quantization range. per_channel (`bool`, *optional*, defaults to `False`): Whether to or not use channel-wise quantization. Returns: `torch.Tensor`: Scaling factor that linearly quantizes the given range between *saturation_min* and *saturation_max*. rrrg:0yE>rN)rrr4stackrr)num_bitssaturation_minsaturation_maxrOrrrzs r+r5r5Ws*$  5 5 (Q, ! #  5yn.@.@.B.BNDVDVDXDX-Y_`!a!a!aghiiiHE1K400014EE**,,n.@.@.B.BCCEK400014E 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 LsCC((C,/C,c>eZdZdZedZedZdS)r#zw Class to quantize the given floating-point values using symmetric quantization with given range and bitwidth. ctjd|j}d|dz zdz }t|||d}tj|| |dz }||_|S)a6 Args: x (`torch.Tensor`): Floating point tensor to be quantized. k (`int`): Quantization bitwidth. percentile_mode (`bool`): Whether or not to use percentile calibration. scale (`torch.Tensor`): Pre-calculated scaling factor for *x*. Note that the current implementation of SymmetricQuantFunction requires pre-calculated scaling factor. Returns: `torch.Tensor`: Symmetric-quantized value of *input*. g)rrrF)r)rrrrrr)ctxr6rr"rrr new_quant_xs r+r>zSymmetricQuantFunction.forward|sh"\#el;;; !a%L1 %a EJJJ k+r1q599  r,c2|j}t|jdkr|dddd}nDt|jdkr|dd}n|d}||z ddddfS)NrrYrr)rrrxr^clone)r grad_outputrs r+backwardzSymmetricQuantFunction.backwards  { ! !Q & &JJr1a++EE " # #q ( (JJr1%%EEJJrNNE  ""U*D$dBBr,Nr@rArBrC staticmethodr>rr,r+r#r#wsY\2 C C\ C C Cr,r#c>eZdZdZedZedZdS)rz; Straight-through Estimator(STE) for torch.floor() c*tj|Sr})rrrr6s r+r>zfloor_ste.forward{1~~r,c*|Sr}rrrs r+rzfloor_ste.backward  """r,Nr r r,r+rrT\##\###r,rc>eZdZdZedZedZdS)rz; Straight-through Estimator(STE) for torch.round() c*tj|Sr})rrrs r+r>zround_ste.forwardrr,c*|Sr}rrs r+rzround_ste.backwardrr,Nr r r,r+rrrr,rc|}|d}tj|\}}g}|D]o}t tj|d|zz tjdtj }| |ptj |}t||z }tj||j|tj||j|fS)z Decompose the scaling factor into mantissa and twos exponent. Args: scaling_factor (`torch.Tensor`): Target scaling factor to decompose. Returns: ``Tuple(torch.Tensor, torch.Tensor)`: mantisa and exponent rYrr)rounding)sizer^npfrexpcpunumpyrdecimalDecimalquantize ROUND_HALF_UPappendarrayrr from_numpyrr)inputsrshape_of_inputoutput_moutput_etmp_mm int_m_shifteds r+ batch_frexpr.s<[[]]N[[__F&**,,"4"4"6"677Hh E $$ OAG, - - 6 6wq7I7IT[Ti 6 j j    ]####xHW~~(H ""%%fm4499.II ""%%fm4499.II r,cDeZdZdZe ddZedZdS)r]aQ Function to perform fixed-point arithmetic that can match integer arithmetic on hardware. Args: pre_act (`torch.Tensor`): Input tensor. pre_act_scaling_factor (`torch.Tensor`): Scaling factor of the input tensor *pre_act*. bit_num (`int`): Quantization bitwidth. z_scaling_factor (`torch.Tensor`): Scaling factor of the output tensor. identity (`torch.Tensor`, *optional*): Identity tensor, if exists. identity_scaling_factor (`torch.Tensor`, *optional*): Scaling factor of the identity tensor *identity*, if exists. Returns: `torch.Tensor`: Output tensor(*pre_act* if *identity* is not given, otherwise the addition of *pre_act* and *identity*), whose scale is rescaled to *z_scaling_factor*. Nc,t|jdkrd}nd}||_d|dz zdz }tj5||}| ||}||_tj||z } |tj} |tj tj} | | z } || } t| \} }| tj| tjz}tj|d|zz }|tj||z }|tj} |tj tj} | | z } || } t| \}}|tj|tjz}tj|d|zz }||z}tj |tj | dz |cdddS#1swxYwYdS)Nrc|Sr}r r6s r+z'FixedPointMul.forward..sr,c0|dddS)NrrY)r^r2s r+r3z'FixedPointMul.forward..sq!R 0 0r,rrr ) rrxr`rrz_scaling_factorrtypedoublerr.r)rpre_actr_bit_numr5r`rareshaperz_int_A_B new_scaler,eoutputwx_intm1e1output1s r+r>zFixedPointMul.forwards %+ , , 1 1!kGG00G 'A+  " ]__! D! D%,W-C%D%D "#*1'2I*J*J'#3C K*@ @AAE',,U\::B"'' 44::5<HHBRI **Iy))DAqZZ --u|0D0DDF[36!233F#X0G%GHH,11%,??&++EK88>>u|LLG #GI.. $Y//B ++el33bggel6K6KK+gb&9:: 6);v{{5;77!aCCC! D! D! D! D! D! D! D! D! D! D! D! D! D! D! D! D! D! DsH7J  J J cd}|j||jz }||jz dddd|dfSr})r`rr5)rr identity_grads r+rzFixedPointMul.backward.sS < #'--//#2FFM  ""S%994tTS`bfffr,r?r r r,r+r]r]sk, $2D2D2D\2Dhgg\gggr,r])F)r)r rrrrtorch.autogradrutilsr get_loggerr@rModulerrGrirrrrrr5r#rrr.r]r r,r+rKs" ######  H % %PPPPPPPPPPRYPPPPPPfgMgMgMgMgMrygMgMgMTM M M M M ")M M M `6666666666bi666666rD8D8D8D8D8D8D8D8Nb!b!b!b!b!29b!b!b!J!$!$!$!$H 9 9 9 9F@*C*C*C*C*CX*C*C*CZ # # # # # # # # # # # # # # # #DQgQgQgQgQgHQgQgQgQgQgr,