ktji` ddlZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlmZmZddlmZejeZdZddZdd Zd Zgd Zd gd d gd d gdZGddZ ddZddZdS)N)AttributeProto GraphProto ModelProto NodeProto TensorProtohelper numpy_helper) infer_shapesinfer_shapes_path)versioncd|DS)z| Convert numpy float16 to python int. :param np_list: numpy float16 list :return int_list: python int list c g|]M}tt|dddddNS)HN)intbinviewzfill).0_s ^/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/onnxruntime/transformers/float16.py z%_npfloat16_to_int..$sK D D DqCAFF3KK  $**2.. 2 2 D D D)np_lists r_npfloat16_to_intrs E DG D D DDr"\o>@c d}|tj|dkjddkr|tj|dk}|tj|dk}||kr t d|d|||kr t d|d||tj|dkjddkr|tj|dk}|tj|dk}|| kr!t d|d| || kr!t d|d| tj|d||||}tj|| |d| |}tj|||td||}tj|td|| | |}tj|S)a? Convert float32 numpy array to float16 without changing sign or finiteness. Positive values less than min_positive_val are mapped to min_positive_val. Positive finite values greater than max_finite_val are mapped to max_finite_val. Similar for negative values. NaN, 0, inf, and -inf are unchanged. c<tj||k||kSN)np logical_and)abcs rbetweenz&convert_np_to_float16..between/s~a!eQU+++rrzthe float32 number z will be truncated to infz-inf) r#whereshapemaxminloggerdebugfloatfloat16)np_arraymin_positive_valmax_finite_valr( positive_max positive_min negative_max negative_mins rconvert_np_to_float16r9'sj,,,A&&'-a0144A 6 67;;== A 6 67;;== > ) ) LLc|ccSacc d d d + + + LLe|eeScee f f fA&&'-a0144A 6 67;;== A 6 67;;== N? * * LLd|ddTbSbdd e e e ,, , , LLf|ffTdSdff g g gx8-=>>@PRZ[[Hx!1 18Q??BRART\]]Hx%,,GGYabbHxf x.IIN?\deeH :h  rct|tstdt||jtjkrtj|_|jrOttj |j||}t|}||j dd<g|jdd<|j rEtj|j d}t|||}||_ |S)aConvert tensor float to float16. Args: tensor (TensorProto): the tensor to convert. min_positive_val (float, optional): minimal positive value. Defaults to 1e-7. max_finite_val (float, optional): maximal finite value. Defaults to 1e4. Raises: ValueError: input type is not TensorProto. Returns: TensorProto: the converted tensor. 3Expected input type is an ONNX TensorProto but got Nfloat32dtype) isinstancer ValueErrortype data_typeFLOATFLOAT16 float_datar9r#arrayr int32_dataraw_data frombuffertobytes)tensorr3r4 float16_dataint_list float32_list float16_lists rconvert_tensor_float_to_float16rPIs fk * *_]tTZ||]]^^^ ;,,,&.   &0&:K1L1LN^`nooL(66H#+F aaa #%F aaa ? 5= JJJL0?OQ_``L*2244FO Mrcttj|j}tj|j|j|Sr")r to_arrayr+rmake_tensor_value_infonamerB)rKr+s rmake_value_info_from_tensorrUns/  !& ) ) /E  (f6F N NNr)ArrayFeatureExtractor BinarizerCastMapCategoryMapperDictVectorizerFeatureVectorizerImputer LabelEncoderLinearClassifierLinearRegressor Normalizer OneHotEncoderRandomUniformLike SVMClassifier SVMRegressorScalerTreeEnsembleClassifierTreeEnsembleRegressor TreeEnsembleZipMapNonMaxSuppressionTopKRoiAlignRangeCumSumMinMaxUpsampler)Resize GroupNorm SkipGroupNormc*eZdZdZdefdZdefdZdS)InitializerTrackerz'Class for keeping track of initializer. initializerc0||_g|_g|_dSr")rx fp32_nodes fp16_nodes)selfrxs r__init__zInitializerTracker.__init__s&rnodecv|r|j|dS|j|dSr")rzappendr{)r|r~is_node_blockeds radd_nodezInitializerTracker.add_nodesB  ) O " "4 ( ( ( ( ( O " "4 ( ( ( ( (rN)__name__ __module__ __qualname____doc__rr}rrrrrrwrwsP11K )Y))))))rrwFc  |dks Jd|ttjtjjks Jd|in|} t |t r|} tjtj tjdkrz|sxtj tj| 5} | j} t#| | tj| }d}dddn #1swxYwYntj| }t |t&st)dt+|d}|sBtjtj tjd kr t,}n#wxYw|t.}|g}t1|}t1|}t d |jDd }t5t6}|d krdg|d<t8d|d|dd|d|d|d|g}g}g}g}| ||}||i}t1}t1}d|jj D}d|jj!D}t tDrfd|D}fd|D}nsg}g}tG|jj D]!\}}|j|vrdt |z}|||j<|$|jdt |z}|jj%$}|&|||_tNj(|jj)_*tWj,d|jg|gtNj(|g}|jj-.||||$|#tG|jj!D]\}}|j|vrdt |z} | ||j<|$|jd t |z}|jj%$}|&|| |_tNj(|jj)_*tWj,d| g|jgd|g}|jj-.||||$|i}!|rg}"|D]}#t |#t&r|"|#jt |#t^r|#j0D]9}|j1tNj2kr"|j|!vsJtg||!|j<:|#j-D]e}|j|vr titk|j D],}|j ||vr||j ||j |<-titk|j!D],}|j!||vr||j!||j!|<-|j6|vp|j|v}$tG|j D]a\}} | |!vrX|$p9||7|j6gvo|| 7|j6gv}%|!| 8||%b|$r||J|j6dkr=|j9D]5}&|&jd!kr(|&j:tNj2krtNj(|&_:n6|j6d"vrd#}'|j9D]5}&|&jd$kr(d}'|&j:tNj2krtNj(|&_:6|j6d%vr:|'s8|j9.tWj;d$tNj(g|j6|vs |j6| vr!|j9D]}&|"|&P||gt |#txr|"|#j=|#j>D]}|"||#j?&t|#j?|||#jAD]}t|||}t |#t^rtjC|#j |#j!|#j%D]}|jj)j*tNj2kr9|j|vr0tNj(|jj)_*|||jDd&rl|jjEj*j)j*tNj2krC|j|vr:tNj(|jjEj*j)_*|||"}||!FD]x}(|s|(jGrmt|(j0|||(_0|t|(j0|(jIr$|s"t8Jd'|(jGy|D]})tG|)j D]\}} |||)j6vs|| 7|)j6gvr2|D]}*| |*jkr|jj%$}|&|*| d(z}||_tNj2|jj)_*| d)z}tWj,d| g|gd|g}|jj-.|||)j |<n| r tNjKn tNj2}+|D]})titk|)j D]}|)j |} |D]}*| |*jkr|jj%$}|&|*| d(z}||_|+|jj)_*| d)z}tWj,d| g|g|+|g}|jj-.|||)j |<ntitk|)j!D]}|)j!|},|D]}*|,|*jkr|jj%$}|&|*|,d*z}-|-|_|+|jj)_*|,d+z}tWj,d|-g|,gd |g}|jj-.||-|)j!|<n|S),aConvert tensor float type in the input ONNX model to tensor float16. Args: model (ModelProto or str): The ONNX model or path of the model to convert. min_positive_val (float, optional): minimal positive value. Defaults to 5.96e-08. max_finite_val (float, optional): maximal finite value of float16. Defaults to 65504. keep_io_types (Union[bool, List[str]], optional): It could be boolean or a list of float32 input/output names. If True, model inputs/outputs should be left as float32. Defaults to False. disable_shape_infer (bool, optional): Skips running onnx shape/type inference. Useful if shape inference has been done. Defaults to False. op_block_list (List[str], optional): List of op types to leave as float32. Defaults to None, which will use `float16.DEFAULT_OP_BLOCK_LIST`. node_block_list (List[str], optional): List of node names to leave as float32. Defaults to None. force_fp16_initializers(bool): force converting all float initializers to float16. Default to false, which will convert only the one needed to avoid precision loss. force_fp16_inputs(Dict[str, List[int]]): Force the conversion of the inputs of some operators to float16, even if this script's preference it to keep them in float32. Raises: ValueError: input type is not ModelProto. Returns: ModelProto: converted model. rzginvalid min_positive_val. smallest positive float16 value: subnormal 5.96e-08, and normalized 6.104e-05z4invalid max_finite_val. largest float16 value: 65504Nz1.8.0)dirTz$Expected an ONNX ModelProto but got z1.2.0c36K|]}|jdv |jVdS))zai.onnxN)domainr )ros r z+convert_float_to_float16..s0[[Aqx??Z?Zai?Z?Z?Z?Z[[r )default rrrsz"fp16 parameters: min_positive_val=z max_finite_val=z keep_io_types=z disable_shape_infer=z op_block_list=z node_block_list=z force_fp16_initializers=cZg|](}|jjjtjk!|j)SrrA tensor_type elem_typerrCrTrns rrz,convert_float_to_float16..s0jjja8J8TXcXi8i8i168i8i8ircZg|](}|jjjtjk!|j)Srrrs rrz,convert_float_to_float16..s0lllq!&:L:VZeZk:k:kAF:k:k:krcg|]}|v| Srrrr keep_io_typess rrz,convert_float_to_float16..s#DDDQm1C1Cq1C1C1Crcg|]}|v| Srrrs rrz,convert_float_to_float16..s#FFFa1 3E3E3E3E3Ergraph_input_cast_graph_input_castCast)torTgraph_output_cast_graph_output_castr)EyeLike Multinomial RandomNormalRandomNormalLike RandomUniformrb SequenceEmpty BernoulliFr>)rrr sequence_typezXinitializer is used by both fp32 and fp16 nodes. Consider add these nodes to block list: _cast_to_fp32_cast_to_fp32_node _cast_to_fp16_cast_to_fp16_node)Lr0r#finfor1r,r?strr parseonnx __version__tempfileNamedTemporaryFileospathdirnamerTr loadrr@rAr DEFAULT_OP_BLOCK_LISTset opset_importdictALWAYS_FLOAT_INPUTSr.r/rgraphinputoutputlist enumerateadd value_infoCopyFromrrDrrr make_noder~extendrrxrBrCrwrangelenop_typegetr attributeimake_attributerggraphstrPtensors itertoolschainHasFieldrvaluesr{rUrzinfoBFLOAT16).modelr3r4rdisable_shape_infer op_block_listnode_block_listforce_fp16_initializersforce_fp16_inputs#use_bfloat16_as_blocked_nodes_dtypeforce_fp16_inputs_dict model_pathtmpfileshape_infer_model_pathfunc_infer_shape onnx_opsetalways_float_inputsqueuevalue_info_list node_listmixed_float_type_node_list name_mappinggraph_io_to_skipio_casts fp32_inputs fp32_outputsrr output_name node_namenew_value_infonew_node input_namefp32_initializers next_levelqruse_fp32_weightattr has_dtypevaluer~r accuracy_typeroutput_cast_names. ` rconvert_float_to_float16rsH x ' ' 'q ( ' ' U28BJ#7#7#;<< < < <>t < < <#4# 2 2 2 LL # # #%+,--$$1 6\ ! !-A6J#-L   ( ( (+c!ff4I"[37799N  # #A & & &",N 8C8KN  + 5(*xAT]^^^_H K  # #H - - -  " "> 2 2 2 LL # # #79 b ^ :^ :A!Z(( +!!!'***!Z((@ AJJA{k&777 v->>>>>4Fq4I4I)!&1:A:AAv)) "3qw<<00BB71:55)5agaj)AAGAJ"3qx==11DD8A;,66*6qx{*CAHQK&'i=&@&]AFoD]O)217););WW :%):::.=/ !%8%<% J T / 6 6q 9 9 9E bH#))++ " e&6  ?@QSces t tE   " "#>u?P#Q#Q R R R (?  BotoBB +&tz22  MAz+DL999QBXB\B\]a]ikmBnBn=n=n-   00%*[%;%?%?%A%AN"++J777",">K*5N'@K@QN'3= *-A AI & 0* }YZaj k k klHK$++H555$/DJqME1  &-PfK((U`UfM''s4:''  AAJ-   00%*[%;%?%?%A%AN"++J777",">K*5N'@MN'3= *-A AI & 0* }Yfmv w w wxHK$++H555$/DJqME1 s4;''((  A[^F-   Z_,,%*[%;%?%?%A%AN"++J777'-'?$*:N'@MN'3= &)= =I & 0:J9KfXZ\cl m m mnHK$++H555%5DKNE- " Ls.DDDF$$F&ct|tstdt||jtjkrtdd}|jrtj|j}|j rtj |j d}|tdt|||}tj tj|tj|z S)zSMeasure the maximum absolute difference after converting a float tensor to float16.r;z#Expected tensor data type is float.Nr<r=zexternal data not loaded!)r?rr@rArBrCrEr#rFrHrI RuntimeErrorr9amaxabsr<)rKr3r4 float32_datarLs rfloat_to_float16_max_diffrs fk * *_]tTZ||]]^^^ ;,,,>???L 3x 122  G}V_IFFF 6777(7GXXL 726,L)A)AABB C CCr)rr) rrFFNNFNF)rloggingrrnumpyr#rrrrrrrr onnx.shape_inferencer r packagingr getLoggerrr.rr9rPrUrrrwrrrrrr s  eeeeeeeeeeeeeeeeee@@@@@@@@  8 $ $EEE    D""""JOOO D#$Aq6QPQFSS ) ) ) ) ) ) ) )"!(-AAAAH DDDDDDr