ktj(@ddlmZddlZddlmZddlmZddlmZddl Z ddl Z ddl m Z ddl m Zdd lmZmZdd lmZdd lmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-Gd deZ.eGddZ/GddZ0eGddZ1eGddZ2eGddZ3eGddZ4eGddZ5GddeZ6dS)) annotationsN) dataclass)Enum)Any) TensorProto)onnx_pb) BaseQuantizerQuantizationParams) TensorData)DEQUANT_OP_NAMEONNX_TYPE_TO_NP_TYPE QUANT_OP_NAMEQuantizedValueQuantizedValueType __producer__ __version__add_dequant_output_suffixadd_dequant_suffixadd_quant_input_suffixadd_quant_output_suffixadd_quant_suffixcompute_data_quant_paramscompute_scale_zpcompute_scale_zp_blockedcompute_scale_zp_float8 find_by_nameget_qmin_qmax_for_qType ms_domainnormalize_axisquantize_onnx_initializersnap_zero_point_to_uint8tensor_proto_to_array)CreateQDQQuantizerceZdZdZdZdZdS)QDQQuantTensorTyperr N)__name__ __module__ __qualname__ ACTIVATIONWEIGHTBIASd/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/onnxruntime/quantization/qdq_quantizer.pyr&r&0sJ F DDDr/r&c$eZdZUded<ded<dS)QDQQuantParamProviderstr input_name node_nameNr(r)r*__annotations__r.r/r0r2r29s"OOONNNNNr/r2c(eZdZejdddfdZdS)QDQTensorQuantInfoNcX||_||_||_|du|_|J||_dSN) tensor_typequant_para_provideraxis is_shared data_type)selfr<r=r>r@s r0__init__zQDQTensorQuantInfo.__init__Bs<&#6  ,D8$$$"r/)r(r)r*r&r+rBr.r/r0r9r9As7#5#@VZaequ######r/r9c8eZdZUded<ded<ded<ded<dS)QDQBiasQuantInfor3r5r4 weight_namefloatbetaNr6r.r/r0rDrDLs7NNNOOOKKKKKr/rDc6eZdZUded<ded<ded<d dZd S) QDQTensorQuantParamsr originalzQuantizationParams | None convertedset[str] | Noneconverted_recv_nodesreturnch|j|jS|j|jS||jvr|jn|jSr;rKrJrMrAconsumer_node_names r0get_for_consumerz%QDQTensorQuantParams.get_for_consumer]B > !=  $ ,> ! #58Q"Q"Qt~~X\Xeer/N)rNr r(r)r*r7rSr.r/r0rIrIWsT    (((()))) f f f f f fr/rIc$eZdZUded<ded<dS)QDQScaleZpInitializersrscale zero_pointNr6r.r/r0rWrWks*r/rWc.eZdZUded<ded<ded<dS)QDQTensorScaleZpInitializersrWrJzQDQScaleZpInitializers | NonerKrLrMNr6r.r/r0r[r[ts6$$$$,,,,))))))r/r[c6eZdZUded<ded<ded<d dZd S) QDQTensorQuantizedValuerrJzQuantizedValue | NonerKrLrMrNch|j|jS|j|jS||jvr|jn|jSr;rPrQs r0rSz(QDQTensorQuantizedValue.get_for_consumerrTr/N)rNrrUr.r/r0r]r]~sT$$$$)))) f f f f f fr/r]c0eZdZ dYdZdZdZdejfdZdZdZ d[d Z dZd Z dZ d\dZ d]dZd^dZdZdZd Zd!Zd"Z d_d`d-Z d_dad1Z d_dbd2Zdcd4ZdYd5Zd6Zd7Zd8Zd9ZdZd:Z dYddd?ZdedBZ dfdFZ! dgdhdMZ"didOZ#djdTZ$dkdVZ%dldXZ&dS)m QDQQuantizerNc   tj|||||||||| | i|_i|_g|_| dg|_| dd|_| dd|_| dd|_ i|_ i|_ | di|_ | ddrtnd|_| d d|_| d d|_| d d |_|jd krt&jt&jt&jt&jf t1 fd|jD} |js=|j vs |j vs| r)t9jdtdt|_||_i|_ i|_!dS)N"OpTypesToExcludeOutputQuantizationAddQDQPairToWeightF QuantizeBiasTDedicatedQDQPair QDQOpTypePerChannelSupportToAxisUseQDQContribOpsQDQKeepRemovableActivations"QDQDisableWeightAdjustForInt32Bias BlockSizerc3*K|] }|jvVdSr;)r<).0t opset21_typess r0 z(QDQQuantizer.__init__..s;//34 .//////r/zONNX QuantizeLinear and DequantizeLinear operators do not support 16-bit/4-bit integer quantization types prior to opset 21. The domain of QuantizeLinear and DequantizeLinear operators will be set to 'z' to enable support.)"r rBtensors_to_quantizebias_to_quantizenodes_to_removeget'op_types_to_exclude_output_quantizationadd_qdq_pair_to_weight quantize_biasdedicated_qdq_pairtensor_to_its_receiving_nodestensor_to_producing_dq'qdq_op_type_per_channel_support_to_axisr qdq_op_domainqdq_keep_removable_activations(qdq_disable_weight_adjust_for_int32_bias block_size opset_versionrUINT16INT16UINT4INT4anytensor_quant_override_qtypesactivation_qType weight_qTypeloggingwarningcalc_graph_quant_paramsquantization_paramsinitializer_quant_paramsquantized_value_map) rAmodel per_channel reduce_rangerr tensors_rangenodes_to_quantizenodes_to_excludeop_types_to_quantize extra_optionsoverrides_have_opset21_typesros @r0rBzQDQQuantizer.__init__sc            CE =?!8E7H7HImoq7r7r4 '4&7&78Le&T&T#+..~tDD#0"3"34F"N"NNP*BD#8E7H7HIkmo7p7p4*7*;*;> 8J +AWcl999(555D$<<< 11+>> 8JWbnw8x8x8x(555 y y=r@z9only support per-channel quantization on weight. Tensor: z is not quantized.)rrrr@rrrrr9r&r,rqrr)rArr>rs r0"quantize_weight_tensor_per_channelz/QDQQuantizer.quantize_weight_tensor_per_channel\sk4:+A+A+C+CDD  yJ$:$@*BXB`#aaa8J 2 9PVP`999(555ba OwXcwww x x x x xr/ronnx.TensorProtorNc|j|jdz}|j|}tj}||||_|j||S)zk Duplicates an existing initializer and adds it to the model. Returns the new initializer. r )r#get_largest_initializer_name_suffixnameonnxrCopyFromadd_initializer)rAr name_suffixnew_initializer_namenew_initializers r0_dup_initializerzQDQQuantizer._dup_initializerfs| :II+JZ[[^__ "-"2AKAA*,,  ---3 ""?333r/?c|j|rbtjd|d||d\}}|r|||n||dSt||j }|tj d|ddS|j tj jtj jfvrtjd|ddS|} ||jvrT||} | j} |j|| |htjd |d | d t)|||||j| <dS) a Adds a bias tensor to the list of bias tensors to quantize. Called by op quantizers that want to quantize a bias with bias_zero_point = 0 and bias_scale = input_scale * weight_scale * beta. TODO: Explain the reasoning for using this formula. Args: node_name: name of the node that consumes the bias, input, and weight tensors. bias_name: name of the bias tensor to quantize. input_name: name of the input tensor whose scale is used to compute the bias's scale. weight_name: name of the weight tensor whose scale is used to compute the bias's scale. beta: Multiplier used to compute the bias's scale. zQuantizing bias tensor 'z=' as a weight due to the presence of user-specified overridesr) default_axisNzExpected bias 'z' to be an initializerz%' to be an floating-point initializerzCreated a copy of bias input 'z ' called '')tensor_quant_overridesrtrinfois_tensor_per_channelrrrrrrr@rrrrrrrrreplace_input_of_nodesrD) rAr5 bias_namer4rErGis_per_channelr>bias_initializeractual_bias_namenew_bias_initializers r0quantize_bias_tensorz!QDQQuantizer.quantize_bias_tensorrs  & * *9 5 5  Ls9sss   $(#=#=iVW#=#X#X ND 777 4HHHH++I666 F' 4:3I3I3K3KLL  # OOiOOO P P P F  %j.D.JJLbLj-k k k L[9[[[ \ \ \ F$ - - -$(#8#89I#J#J 38  J - -i9II; W W W Lb)bbO_bbb c c c3C9jZegk2l2l.///r/ input_scale np.ndarray weight_scalerEbias_tprbooltuple[bool, np.ndarray | None]c |jsdSt|}tjtj}d}tj|jtjtj|jdztjz } |j } d} |stj |tjdtj} tj |tjdtj} tj tj | tj | }|d|zz| z }tj| tj}tj| tj}||z}||krJ|dkrD||z }tjd |d |d |jd ||z}|| }d } n*|jr"t'|jdkr |jd}t)|D]}tj ||}|d|zz| z }tj| tj}tj|| tj}||z}||krP|dkrJ||z }tjd|d|d|d |jd ||z}|| ||<d } | |fS)aI Checks if the bias scale (input_scale * weight_scale) that we intend to use is too small. A bias scale that is too small leads to quantized bias values that fall outside the range of a int32 and have to be clipped, which decreases accuracy. If this function detects such a scenario, the weight_scale value will be increased to prevent this from happening. Although the adjustment method and amount differs, the idea to adjust the weight's scale came from the following reference: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/optimize/quantization_utils.cc#L252 :param input_scale: The input's scale. :param weight_scale: The weight scale to potentially adjust. :param weight_name: The weight initializer's name. Used for logging. :param bias_tp: The bias ONNX initializer. :param is_per_channel: True if the bias and weight are quantized per-channel. :return: A tuple with a bool indicating if the weight's scale was adjusted and the new weight scale. FNgqh?dtyper Frg@gzIncreasing scale for weight `z` by the ratio z to ensure bias input `z` has a valid scale.TzIncreased scale[z] for weight `z ` by ratio )sizer#npiinfoint32arraymaxfloat64minrminimummaximumabsitemrrrastypeshapelenrange)rArrrErrbias_float_data int32_infomultiplicative_epsilonqrangeweight_scale_dtypeupdated_an_elemrminrmaxabsmaxbias_smallest_valid_scaleinput_scale_fp64weight_scale_fp64bias_candidate_scaleratio new_scale num_elemsi bias_rmaxs r0#_adjust_weight_scale_for_int32_biasz0QDQQuantizer._adjust_weight_scale_for_int32_biassx2  ;/88Xbh'' !'*. ;;;bhz~XYGYacak>l>l>ll)/( +:o1133RXarz5R5R5RSSD:o1133RXarz5R5R5RSSDZt bfTll;;F(>#,(ORX(X %!x (8(8(:(:"*MMM  "):):)<),>bj#Q#Q#Q $&H\!_-A-A-C-C2:$V$V$V!'7:K'K$(+DDDK_beKeKe58LLELT1TTKTTTYTT18TTT!2E 9I&/&6&67I&J&JLO&*O ,,r/c 8|jrdS|jD]u\}}|j|jvs|j|jvs|j|jvr1|j|j|j }|j|j}tj |dtj |j}|j|j}|d}|tjjtjjfvr|d}|r|d} |dddu} ||| |jt-||j| \} } | r| |d<wdS)a3 Iterates through all bias inputs that should be quantized to int32. If the intended bias scale (equal to input_scale * weight_scale) is too small, this function will increase the associated weight's scale to ensure the bias does not overflow the int32 range when quantized. NrXr quant_typerYr>)r~rritemsr4rrqrErrSr5rasarrayrhelpertensor_dtype_to_np_dtyper@rINT8rrrtrrrr) rAr bias_info input_qparams input_inforweight_quant_paramsweight_quant_typeweight_zero_pointrrdid_update_weight_scalenew_weight_scales r0,_adjust_weight_quant_params_for_bias_tensorsz9QDQQuantizer._adjust_weight_quant_params_for_bias_tensorss  8  F$($9$?$?$A$A& @& @ Iy$D,DDD't/GGG(0MMM!4Y5IJ[[\e\oppM1)2FGJ*g&dk.R.RS]Sg.h.hK#'"? @U"V  3L A  )9)>@P@V(WWW,? ,M  $$&& ':7'CL044VTBB$NN9=8`8`%Y (>(>(@(@AA 99 5 #%5' @/?#G,M& @& @r/c:|j|dSr;)rsappend)rAnodes r0 remove_nodezQDQQuantizer.remove_node&s ##D)))))r/cD|j|jdSr;)r remove_nodesrs)rAs r0rzQDQQuantizer.remove_nodes)s!  455555r/c|jD]}||rat||}||jD]5}||jvr g|j|<|j||6|jtkr|j D] }||j |< | |_ ||||jr|||js|jt,|jj_t0|jj_|jt6kr |jt6d|jjS)Nr )rnodesshould_quantize_noder$quantizeinputryrop_typer outputrz_calc_initializer_quant_paramsrr_quantize_normal_tensors_quantize_sharing_param_tensorsrw_quantize_bias_tensorsrrvclean_initializersr producer_namerproducer_versionr|rset_opset_import)rAr op_quantizerrs r0quantize_modelzQDQQuantizer.quantize_model,sJ$$&& D DD((.. Q1$== %%'''#':QQK"$*LLLJL:;G6{CJJ4PPPP|..#';DDK?CD/ <<(,(K(K(M(M% 99;;; %%''' ,,...   *  ' ' ) ) ) * , J ) ) + + +)5 &,7 )   * * J ' ' 1 5 5 5zr/c||jvr|j|j|j|jt|j|dkrb|j|sH|j|s.|j||||jvr|j|=dSdS)Nr TF) rrKrrinput_name_to_nodesis_graph_outputis_graph_inputreplace_output_of_all_nodesrq)rAupstream_output_namers r0try_replacing_upstream_outputz*QDQQuantizer.try_replacing_upstream_outputKs 43 3 3(5?G()=>HPDJ22445IJKKqPPJ../CDDQJ--.BCCQ J 2 23G U U U#t'???,-AB4ur/rq_inputq_outputquant_node_name scale_namezp_namer> int | Nonerintc||jd}|r||d<tjjt|||g|g|fi|} |j| gdS)zI Creates a QuantizeLinear node and adds it to the model. r>domainrN)r|rr make_noderr add_nodes) rAr.r/r0r1r2r>rkwargs qlinear_nodes r0_create_q_nodezQDQQuantizer._create_q_nodeZs~+/$:L!M!M  .#-F< {,  j' * J        l^,,,,,r/dq_input dq_outputdequant_node_namec||jd}|r||d<tjjt|||g|g|fi|} |j| gdS)zK Creates a DequantizeLinear node and adds it to the model. r6rN)r|rrr8r rr9) rAr=r>r?r1r2r>rr: dequant_nodes r0_create_dq_nodezQDQQuantizer._create_dq_nodess~+/$:L!M!M  .#-F< {,  z7 + K        l^,,,,,r/c | |jd} | r| | d<tjjt|||g|g|fi| } tjjt |||g|g|fi| } |j| | gdS)Nr6r)r|rrr8rr rr9)rAr.r/r0r=r>r?r1r2r>rr:r;rAs r0_create_qdq_nodeszQDQQuantizer._create_qdq_nodess+/$:L!M!M  .#-F< {,  j' * J       {,  z7 + K        lL9:::::r/ weight_protoc |j}||jvrdS|j|}|d}|dd}|||}d}t |}|j|||jr]t|} | || t|| |t||j j|jj|| nt||d|d|d||} |j| | j}|| j|t||j j|jj|| t%|||j j|jjt&j| } t+| dd|j|<dS) a Adds Q/DQ nodes for an initializer. If `self.add_qdq_pair_to_weight` is true, creates the sequence (weight_f32 -> Q -> DQ -> ). Otherwise, this function quantizes the initializer and adds the sequence (weight_quant -> DQ ->). Nr>rr)rrrYrX)r>r)r>)rrrrt_make_scale_zp_initializersrrreplace_input_of_all_nodesrvrrDrrrXrYr!rrBrr Initializerr]) rArErE quant_paramsr>rscale_zp_initializers q_weight_nameweight_dequant_outputweight_quant_output quant_weightquantized_values r0_add_qdq_nodes_for_initializerz+QDQQuantizer._add_qdq_nodes_for_initializers #' $2 2 2 F+/+H+U  $$V,,&**<;; $ @ @l [ [$( 9+ F F --k;PQQQ  &' #:+"F"F   " "# --#%";//%+0%05% #    5\*\*W%% L J & &| 4 4 4(-M  !%";//%+0%05% !   )   ! ' , ! , 1  *    1HY]_c0d0d ---r/c l|jr0||jvr&t|j|dkrt|j|}t|D]}d|dz}t ||z}t ||z} t ||z} t||z} |||| || | |||j||} |j | || |dkr8t|| ||tj |} t| dd|j|<dS|}t |}|j |r-t#|}|}|j ||n|j ||||t |t |t ||t|||t||||tj |} t| dd|j|<dS)Nr _r scale_type)rxryrrrrrrrDrreplace_node_inputrrInputr]rr)rr+rH)rArr1r2r@num_dedicated_qdq_pairrpostfix tensor_name_quant_output_postfix"tensor_name_dequant_output_postfixquant_node_name_postfixdequant_node_name_postfixrrPr.r>s r0_add_qdq_pair_for_activationz)QDQQuantizer._add_qdq_pair_for_activations  #@ itAAAD6{CDDqHH%()KK)X%Y%Y "122 q q%a!e++3J;3W3WZa3a05N{5[5[^e5e2*:;*G*G'*Q',>{,K,Kg,U)&&4+46-   9+FqI --dKAcddd66&4#:"*0#, '''O=TTceiko>Iz))+66 N0=='  66{GLLLL 55k9MMM  " "' 44 --' 44";//   -"($ O5LO]acg4h4hD $[ 1 1 1r/cbd|j|gD}|jr6||jvr-t|j|dkrt d|} ||}t } n| |z } t|t|k} |j|} |} | r*t|} |j || t|} | | | t|||t|}| r| s|}| r"||kr|j||| || |t!||||}| s;t|d}|| |t!|d||t|d}| ||t|d||t|d}| r| r|}|r"||kr|j||||||t!|d||t#||||t$j|}t#||||t$j|}t)||||j|<dS)a Adds Q and DQ ops to a tensor whose quantized data type is converted. That is, some consumers may use the original data type from the producer, while other consumers use the converted data type. This is generally done by adding a sequence of ops that convert from one data type (e.g., uint8) to another (e.g., uint16). T_float ---> Quant(to u8) ---> Convert(to u16) ---> Dequant(to float) ---> T_float' where Convert(to u16) is equivalent to: ---> Dequant(to float) ---> Quant(to u16) ---> This function handles the following scenarios: 1) Tensor T is not a graph output; all consumers use the converted type ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 ---> 2) Tensor T is not a graph output; some consumers use the original type, others use the converted type ---> Q1 -+-> DQ1 ---> | +-> DQ1' ---> Q2 ---> DQ2 ---> 3) Tensor T is a graph output; all consumers use the converted type ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 -+-> | +-> 4) Tensor T is a graph output; some consumers use the original type, others use the converted type ---> Q1 -+-> DQ1 -+-> | | | +-> | +-> DQ1' ---> Q2 ---> DQ2 ---> 5) Tensor T is a graph output that is not consumed by any other nodes. ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 ---> ch|] }|j Sr.)r)rmrs r0 zEQDQQuantizer._add_qdq_ops_for_converted_activation..eskkk4TYkkkr/r z|Do not currently support converted quant_types in TensorQuantOverrides when the `dedicated_qdq_pair` extra_option is enabledN_convert_convert_clonerT)ryrtrxr ValueErrorsetrr)rr+rr<rrrrBrrrrWr]r)rArfirst_scale_name first_zp_namescale_data_typeconvert_scale_nameconvert_zp_nameconvert_recv_nodestensor_recv_nodesoriginal_recv_nodesall_use_convertedr) first_q_inputfirst_q_outputfirst_dq_outputsecond_q_inputsecond_q_outputsecond_dq_outputoriginal_quantized_valueconverted_quantized_values r0%_add_qdq_ops_for_converted_activationz2QDQQuantizer._add_qdq_ops_for_converted_activation5s`lk43U3Y3YZegi3j3jkkk  # tAAAD6{CDDqHHO  0  %!2 "%%%  "58J"J  233s;L7M7MM*44[AA$  O2;??M J 2 2; N N N0==  >+;K+H+HJZ\i   4K@@  *#4 *)O  a?k#A#A J - -k?L_ ` ` `  O-? -L-LN^`m   )  3{4L4L4LMMN  "k#A#A#ABB     2[2J2J2JKK     555 6 6      5 5M5M5MNN  +0 +*   a"2k"A"A J - -k;KM_ ` ` `    +777 8 8      $2      $& $ $ $  %3      $& % % % !1H $&?AS1 1  ---r/c |jD]\\}}||jvr|jsDt ||j}|r||n||j vr |j|=i| |}|std|d|j =| ||jjj|jjj|jn}|j|jjjksJ|||jjj|jjj|j|j jj|j jj|j|j|=^dS)z} Adds Q/DQ ops to tensors (activations and weights) that have been marked for quantization by op quantizers. z4Quantization parameters are not specified for param zb. In static mode quantization params for inputs and outputs of nodes to be quantized are required.N)r@)rqcopyrrr?rrrrQrz"_make_tensor_scale_zp_initializersrdrKr^rJrXrrYr@rwrM)rAr tensor_infortensor_qparam_initializerss r0rz%QDQQuantizer._quantize_normal_tensorss)-(@(E(E(G(G(M(M(O(O0 :0 : $Kd666(, :*; 8N8N8P8PQQ '77 DDDD #d&AAA 4[A 151X1XYd1e1e.5(S^ 2;C99'6?EJ6?JO&1&; : +48R8[8a8kkkkkBB'6?EJ6?JO'16@FK6@KP6K,[9a0 :0 :r/c |jrs|jD]<\}}|j}|r,|j|jvr|j|=|j|j|j}||rtd||j vrtd|dd}d}||j vr7|j |}|j r#| ||j d}|j}|"|||j|j|||j|j|jj|jj|jj|>|jqdSdS)a{ Adds Q/DQ ops to tensors that have been marked for quantization by op quantizers. Only operates on tensors that want to use the quantization parameter initializers from an upstream tensor. For example, a Transpose node's output tensor will typically want to use the same quantization parameter initializers as the Transpose node's input. zBQuantization parameter shared mode is not supported for weight yetz5Quantization parameter sharing is invalid for tensor z& because it has already been quantizedNrb)rqryrr=r4rrSr5is_input_a_initializerrdrzrrKrGrMr^r1r2rwrXr@rrY)rArr{quant_providerrPconverted_qparam_initsrM tensor_paramss r0rz,QDQQuantizer._quantize_sharing_param_tensorss&/ ,0,D,I,I,K,K,Q,Q,S,S. . ( [!,!@!,n&?4C[&[&[0=&*&>~?X&Y&j&j&0''O22;??o()mnnn"d&AAA(DT_DDD.2*+/("d&>>>(,(@(M (2V595U5U +]-Dj6624A3U0-599')C_E\BB'+6+328B28=2=B0O&/ / / / / r/c t|jD]\}}||jvr|||t ||j}|j||j|j}|j dkrt|j ts,tdt|j d|jt!|}t"jd|jg|g||j }n|j dvr|jt"jjt"jjt"jjhvrt5d|jd|j|j|jg}t!|}|j1t"jd ||g||j|j }nCt"jd ||g||j }nt5d |j d |j|dS)zq Adds DQ ops (or Cast) for bias tensors that have been marked for quantization by op quantizers. CastUnexpected type z for input=)rto)NDequantizeLinearzUnexpected quantize type z for DequantizeLinear.Nrr6)r7zUnexpected operator type r) rrrrquantize_bias_staticrrrremove_initializerrJ node_typerr@r4rrr4rrrr8q_name node_qtyperrBFLOAT16r RuntimeErrorr1r2r>r|add_node)rArrinit quant_valuer5rAinputss r0r z#QDQQuantizer._quantize_bias_tensors@sa%)$9$?$?$A$A1 .1 . IyD444  % %i ; ; ; 4:+A+A+C+CDDD J ) )$ / / /29=FK$.."$.#66r#$ptDN7K7K$p$pXaXl$p$pqqq.y99 #{44 '(K"~ 5   &*DDD)$,$-$*. ''q;CY'q'q'qrrr%,k.DkFYZ.y99 #/#';#8#8*" !(-#1 $9$$LL$(;#8#8*" !#1 $9$$LL##Y{?T#Y#Y#YZZZ J   - - - -c1 .1 .r/c&||jvp||jvSr;)rqrrrs r0is_tensor_quantizedz QDQQuantizer.is_tensor_quantizedwsd66^+I^:^^r/rr str | Nonetuple[bool, int | None]c|j|}|dS|j|rdS|j|}|js|sdS|r|j||n|}|r(|j|}|dd}t|j }t||\} }| stj d|d|d|dSd|fS) a Checks if a given tensor is configured to be quantized per-channel. If so, also returns the channel axis. ORT only supports per-channel quantization on static weights (i.e., ONNX initializers). If the user did not provide tensor quantization overrides for this tensor, then the value of self.per_channel determines if the weight is to be quantized per-channel. Params: tensor_name: The name of the tensor to check. default_axis: The default channel axis. This method checks if the normalized axis is within bounds. Can be overridden via the extra_options 'QDQOpTypePerChannelSupportToAxis' and 'TensorQuantOverrides'. op_type: Optional, defaults to None. The operator type that is the only consumer of this weight. Used to access the extra option 'QDQOpTypePerChannelSupportToAxis'. Returns: A tuple (is_per_channel, axis) in which the first element indicates whether the tensor is quantized per-channel and the second element is the channel axis. The returned axis is only None if the tensor is not per-channel or the axis is out of bounds. Nrrr>zAxis z is out-of-range for weight 'z ' with rank T) initializersrtrhas_per_tensor_overrideshas_per_channel_overridesrr{get_per_channel_overridesrdimsr rr) rArrrweight_initializerhas_per_chan_overridesr>per_chan_overrides weight_rank axis_valids r0rz"QDQQuantizer.is_tensor_per_channelzs,2".22;??  %;  & ? ? L L ;!%! ;Zast;??VVVgs ! 1!%!OOPbccnJ ,Z F F  155k4HHG Q$0q1A<$P$P!;L;X$%6777^bbr/rrrDc ||jvr|j|jjS||j|j}|t d|jd|d||j|j}|t d|jd|d|||||j \}}}}} } t||||tj |j dkrdnd| | } t| dd|j|<|S) z] Quantized the bias. Zero Point == 0 and Scale == Input_Scale * Weight_Scale Nz9Unable to get valid quantization scale for weight input 'z' when quantizing bias 'z ' to int32.z2Unable to get valid quantization scale for input 'r r)rr)rrJrrrEr5rdr4quantize_bias_static_implrGrrrIrr]) rArrrrquantized_bias_namequantized_bias_scale_namequantized_bias_zp_namebias_scale_datarrrPs r0rz!QDQQuantizer.quantize_bias_statics 0 0 0+I6?F F::9;PR[Reff  @ILa@@)2@@@  99):NPYPcdd  @YEY@@)2@@@   * *9krrNr'r zWrong scale/zp shapesz,Scale and zero-point must have the same rank _zero_point_scalezUnexpected dtype=z for param_name=)rtrrrrr make_tensorraveltolistrrrrfloat32rrrfloat16rrdrW)rArrJrrYrXzero_point_typer>r is_blockedzero_point_namer1init_zprU init_scales r0rGz(QDQQuantizer._make_scale_zp_initializerss.",/ W%&|4'++F33&**<;; *%%  # ,U[1A1AQ1F1F2G#'#3EK8H8HA8M8M9N#' @P@P@R@R@Y@Y@[@[   ""7+++ ;"* $ $#/5JJ [BJ & &#/7JJ\\\j\\]] ][,,ZU[RWR]R]R_R_RfRfRhRhii  "":...%j':::r/#QDQTensorScaleZpInitializers | Nonec|j ||jvrtjd|ddS|j|}t|ts#t dt |d|d|||j}|j r|||j dnd}t|||j S)a Create and returns all scale/zero_point initializers for a given tensor. If the tensor is converted to a different quantization type, this function creates two pairs of zp/scale initializers. Otherwise, only one pair of zp/scale initializers is created. Nz$Quantization parameters for tensor:"z" not specifiedr for rrb) rrrrrIrrrGrJrKr[rM)rArroriginal_initsconverted_initss r0rzz/QDQQuantizer._make_tensor_scale_zp_initializerss  # +{$BZ/Z/Z L\ \\\ ] ] ]40= -)=>> [YtM/B/BYYYYYZZ Z99+}G]^^& D , ,[-:QS] ^ ^ ^  ,NO]Mopppr/ tensor_datar quant_overridesdict[str, Any]c|j}d|vr |dj}d|vrd|vr|d|d}}n|tjjkrt ||jd\}}n|d|jd}|d|jd}|d|j }|d d } t|| | \} } t||| | ||j \}}|j r3|tjjkr|st||| | |j \}}t!||| S)z Calculates quantization parameters (scale/zero-point) given a tensor's min/max range and optional user-provided overrides. rrXrYr rrr symmetricrF)rr)qminqmaxmin_real_rangerYrXr)rr<rr FLOAT8E4M3FNravg_stdrt range_valueis_activation_symmetricrrr#is_activation_restricted_asymmetricUINT8r"r squeeze) rArrrzerorXrrrrrrs r0calc_quant_paramszQDQQuantizer.calc_quant_params4s * ? * *(6BJ o % %,/*I*I),79Q%DD 4+8 8 81*k>QRS>TUUKD%%"&&v{/Fq/IJJD"&&v{/Fq/IJJD'++K9UVVI*..~uEEL0,bklllJD$*4tT9dNabbKD%7 J$JZJ`<`<`ir<`6$TTEX e"T\\^^5==??_ijjjjr/dict[str, QDQTensorQuantParams]c|jiS|i}|jD]}|j|}t|ts#t dt |d|d|j|i}|||}d}d}d|vr7|||d}|d d}t|||||<|S)z Calculates quantization parameters (scale/zero-point) for all tensors in the graph using each tensor's min/max range and optional user-provided overrides. Nrrr) default_valconvert recv_nodes) radjust_tensor_rangesrr rrrget_per_tensor_overridesrrtrI)rArrtdrrJrKrMs r0rz$QDQQuantizer.calc_graph_quant_paramsPs"   %I !!### - o oK#K0Bb*-- T R488 R R+ R R RSSS"9RRS^lnRooO--b/BBHI#' O++ 222y7QRR '6y'A'E'El'S'S$/CHiYm/n/n  , ,""r/dict[str, QuantizationParams]c t i}|jD]\}}t||j}|s0t |}t |j}|jtj u}|r|j n|j }|j |r|j |} d| dvr| ddj}t|} d| dv} | sZt!t#j| dd| t#j| dd|j|||<ng} g} | D]d}| t#j|d| | t#j|d|je| dd}t+||\}}|st-d|jd |d |t!t#j| t#j| || ||<|j |ig} d| dvr| ddj}| dd|j}|d u} | p|r||n|j}| dd |}| dd|j}d }d }||nd}|rx|jdkrmt+||\}}|st-d|jd|d ||}t=||||j|\}}t!|||||j||<T| szt?| ||||j!| dd| dd\}}t!|||| ||<t+||\}}|st-d|jd |d ||}|j|}g} g} tE|D]}|#||}| r|t | kr| |ni}t?|$||||j!|d|d\}}| || |t#j%| }t#j%| }t!|||| ||<|S)ze Returns quantization parameters (scale/zero_point/quant_type) for all initializers. rrr>rYrrXrzWeight z# has a per-channel axis with value z that is out-of-bounds for rank )rYrXrr>Nrrz" has a block-wise axis with value )rYrXrr>rrr)rr rmin_override rmax_override)&rqrrrrr#rrr<r&r,rrroverrides_scale_zprr rrrrr rdrrtr>is_weight_symmetricrrrrrflattenrrtakerr)rArrr{rinitializer_datainitializer_rank is_weightr overrideszp_dtyperzero_points_list scales_listchan_overrides channel_axis is_axis_validnorm_channel_axisis_symmetric_default is_symmetricrrYrX block_axisnorm_block_axis channel_countrper_channel_datachannel_overrideschannel_zero_point channel_scales r0rz+QDQQuantizer._calc_initializer_quant_paramsms >@(,(@(F(F(H(HP P  $K&{DJ4J4J4L4LMMK 4[AA "#3#9:: $/3E3LLI.7R**T=RJ*==kJJ#  7 D 9Q<//!*1l!;!GJ/ ;!'9QN>TUU#-888' 44 (*$"$K*3ll(// 9UW_0`0`aaa#**28N74KScSi+j+j+jkkkk#,Q<#7L7ElTd7e7e4M#4((Ik&6II[gII6FII 8J#%8,<#=#= h{33#-. 888' 4377 bTJJIy|++&q\,7C $Q<++FK4DEEL)5N$2$8Ac((444tGc !%Q<++K9MNNL$Q<++ND???&&}5555Z(899  ;//3E))% 444#K00#"r/r;)rr3)rr3r4r3r5r3)rrrNr)r) rrrrrEr3rrrrrNr)Nr)r.r3r/r3r0r3r1r3r2r3r>r3rr4)r=r3r>r3r?r3r1r3r2r3r>r3rr4)rr4)rEr)rr3rr4rrrNr)rr3rRr3rNr)rr3rrDrNr3)r)rr3rJr rr3rNrW)rr3rNr)rr rrrNr )rNr)rNr)'r(r)r*rBrrr&r+rrrrrrrrrrrr&r-r<rBrDrQr^rwrrr rrrrrGrzrrrr.r/r0r`r`s&c&c&c&c&J   ,EIVhVsyyyy:XXXX    "TTTTyyy    /m/m/m/mbN-N-N-N-`1@1@1@f***666   >   , -----@ -----F;;;;;@DeDeDeDeLAiAiAiAiF[ [ [ z4:4:4:l666p5.5.5.n____# /////bcccc0.#.#.#.#bZ\(;(;(;(;(;Tqqqq.kkkk8####:X#X#X#X#X#X#r/r`)7 __future__rr dataclassesrenumrtypingrnumpyrrrrrbase_quantizerr r calibrater quant_utilsr rrrrrrrrrrrrrrrrrrr r!r"r#registryr$r&r2r9rDrIrWr[r]r`r.r/r0rs #"""""!!!!!! &&&&&&========!!!!!!2)(((((          ########   fffffff f&   ******* * fffffff f$t#t#t#t#t#=t#t#t#t#t#r/