ktj?ddlmZddlZddlZddlmZddlZddlmZm Z m Z ddl m Z ddl mZmZdd lmZd d lmZd d lmZ d1d2d#Zd3d'ZGd(d)Z d4d5d0ZdS)6) annotationsN)Path)fix_output_shapesmake_input_shape_fixedoptimize_model)remove_initializer_from_input) FusionGeluFusionLayerNormalization) ONNXModel)FusionLpNormalization)FusionSpaceToDepthF model_inputstr | Path | onnx.ModelProto model_output str | Pathexclude_initializer_from_inputboolfuse_layernormsave_as_external_dataall_tensors_to_one_fileexternal_data_location str | Noneexternal_data_size_thresholdintexternal_data_convert_attributeinputs_to_make_channel_lastlist[str] | Noneoutputs_to_make_channel_lastdynamic_input_shapeslist[tuple[str, str]] | Nonereturnc d} t|tjr|ntj|} t | d} t | }| r`| D]G\}}d|dD}t|||Ht|j d} |r| t|j z} t|}| rd} t|}| rd} t|}| rd} |rlt!d| jD}|jdkrt'jd|jd n%t+|}| rd} | s| r5d }||d z}t/|j | | || d} d }||d z}|j jjD]H}|jdkr;|js4||}|d z }||_d} t'jd|jd|dI| r/|tj| ||||||| S)a If necessary, this method creates a new "pre-processed" model in preparation for quantization of a model to be used in QNN EP. Returns true if a new model was created. This method perfoms the following operations: - Fuse Erf sequence into a single Gelu node. - Fuse ReduceL2 sequence into a single LpNormalization node (p == 2). - (Optional) Fuse ReduceMean sequence into a single LayerNormalization node. Args: model_input: Path to the input model file or ModelProto. model_output: Path the output model file, which is only created if this method returns True. exclude_initializer_from_input: A bool specifying whether to exclude initializer from input. Defaults to False. fuse_layernorm: True if ReduceMean sequences should be fused into LayerNormalization nodes. Defaults to False. save_as_external_data: True if output model should be saved with external data. Defaults to false. all_tensors_to_one_file: Effective only if save_as_external_data is true. Defaults to false. If true, save all tensors to one external file specified by external_data_location. If false, save each tensor to a file named with the tensor name. external_data_location: Effective only if save_as_external_data is true. Defaults to None. Specify the external file to which all tensors are saved. Path is relative to the model path. If not specified, the model's name is used. external_data_size_threshold: Effective only if save_as_external_data is true. Defaults to 1024. Tensors with a data size >= external_data_size_threshold are converted to external data. To convert every tensor with raw data to external data, set to 0. external_data_convert_attribute: Effective only if save_as_external_data is true. Defaults to false. If true, convert all tensors to external data. If false, convert only non-attribute tensors to external data. inputs_to_make_channel_last: List of graph input names to transpose to be "channel-last". For example, if "input0" originally has the shape (N, C, D1, D2, ..., Dn), the resulting model will change input0's shape to (N, D1, D2, ..., Dn, C) and add a transpose node after it. Original: input0 (N, C, D1, D2, ..., Dn) --> Updated: input0 (N, D1, D2, ..., Dn, C) --> Transpose --> input0_chanfirst (N, C, D1, D2, ..., Dn) --> This can potentially improve inference latency for QDQ models running on QNN EP because the additional transpose node may allow other transpose nodes inserted during ORT layout transformation to cancel out. outputs_to_make_channel_last: List of graph output names to transpose to be "channel-last". For example, if "output0" originally has the shape (N, C, D1, D2, ..., Dn), the resulting model will change output0's shape to (N, D1, D2, ..., Dn, C) and add a transpose node before it. Original: --> output0 (N, C, D1, D2, ..., Dn) Updated: --> output0_chanfirst (N, C, D1, D2, ..., Dn) --> Transpose --> output0 (N, D1, D2, ..., Dn, C) This can potentially improve inference latency for QDQ models running on QNN EP because the additional transpose node may allow other transpose nodes inserted during ORT layout transformation to cancel out. dynamic_input_shapes: A list of tuples specifying model input name to and its static shape in comma seprated format, for example: [('input', '1,3,256,256')]. Defaults to None. FT) shape_inferc,g|]}t|S)r).0is y/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/onnxruntime/quantization/execution_providers/qnn/preprocess.py z(qnn_preprocess_model..fsFFFa3q66FFF,c3FK|]}|jdks |jdk|VdS)zai.onnxN)domain)r*xs r, z'qnn_preprocess_model..s9cc18r>>QXYbMbMb!MbMbMbMbccr.zUnable to fuse ReduceMean sequence into a LayerNormalization node. ONNX model must use an opset >= 17 in order to use LayerNormalization, but found version z9. Please use onnx.version_converter to update your model.Transpose_channel_r)transpose_node_name_prefix transpose_node_name_start_suffixqnn_preproc_node_Constantz Node of type z" does not have a name. Renamed to .)rrlocationsize_thresholdconvert_attribute) isinstanceonnx ModelProto load_modelsave_and_reload_optimize_modelr splitrgraphrmodelr r applyrrnext opset_importversionloggingwarningr get_largest_node_name_suffixupdate_io_to_channel_lastnodeop_typenametopological_sort save_model)rrrrrrrrrr r"r#modifiedrF onnx_model input_nameinput_shape_str input_shape fusion_gelu fusion_lpnorm fusion_s2d onnx_opsetfusion_layernormtranspose_node_prefixtranspose_node_suffixunnamed_node_prefixavailable_suffixrO new_node_names r,qnn_preprocess_modelrcsKPH%k4?CC eKKYdIeIeE *5d C C CE5!!J+? P P 'JFF?+@+@+E+EFFFK ":#3#3#5#5z; O O O O**+++&D1*2BCCCZ((K**55M$J//J  ccU%7ccccc    " " Os%/%7sss     8 CC %%'' # &B  4%/%L%LMb%c%cfg%g!   ' ('<-B     .!>>?RSSVWW &+nn <: % %di %2H4DHHM  ! %DIH OlDLll\illl m m m  ##%%%   "7$;+7=     Or.rFonnx.ModelProtor'ctjd5}t|d}t j||d|r^t|d}tjt|t||}t|d}t||t j |}dd i}|j r| |j tj |||cdddS#1swxYwYdS) Nzort.qnn_preproc.)prefixzqnn_proc_input.onnxT)rzqnn_proc_infer.onnxzqnn_proc_output.onnxz onnx.inferz onnxruntime.tools.qnn.preprocess)tempfileTemporaryDirectoryrjoinpathr@rSshape_inferenceinfer_shapes_pathstrrrBmetadata_propsupdatehelperset_model_props)rFr'qnn_preproc_tmp_dir model_in_pathmodel_infer_pathmodel_out_path ret_model ret_metapropss r,rCrCs  $,> ? ? ?CV011::;PQQ  }DIIII  -#$788AABWXX   2 23}3E3EsK[G\G\ ] ] ],M122;;z-update_io_to_channel_last..sGGGfGGGr.ci|] }|j| Sr)r)r*goutputs r,rz-update_io_to_channel_last..sLLLG',LLLr.z is not a graph inputz is not a graph outputc32K|]}|jD]}||V dSr)input)r*rOrVs r,r4z,update_io_to_channel_last.. s9jjDQUQ[jj:_ijZjjjjjjjr. tensor_typeshapezExpected input z# to have a tensor_type with a shaper z to be of rank >= 3r Transpose)rQinputsoutputspermzExpected output r)setrEroutput ValueErrorrnrOrxrangerrtypeHasFieldrrQrdimr@TensorShapeProto Dimensionrlistro make_nodeextend value_infor)rFrrr7r8rEr{r}rV output_nameryio_maprOr+ g_input_nameg_inputrX input_rank channel_dimtranspose_permtranspose_node g_output_nameg_output output_shape output_ranks r,rNrNs+1r22-344 $5 KEGG5;GGGLLu|LLL'CC . . . AAABB B /)EE 0 0 0 CCCDD D 1S!233444S!344555jjEJjjjjjj 13DFX Y YF E Es4:'' C CAz!} CA2B!B!B & 3 3DJqM B B 1 A C4:a=4E#E#E & 3 3DJqM B B 1 s4;''(( E EA{1~!222!'!4!4T[^!D!D A E ) , , #L1|$$]33 b7<;S;\;\]d;e;e b`w|```aa al.4 )) >>Pw|PPPQQ Q+5577 [_Q/000q*q.)) @ @A OA  ' ' A(> ? ? ? ? Q'00===eJ//00z"" 6 6A%&UUAN1  &Nq.. .T0PTTL>((667 /   )A-( >*+++++,, %m4}%%m44 dHM*++++ F233333r.) FFFFNrFNNN)rrrrrrrrrrrrrrrrrrr r!r"r!r#r$r%r)rFrdr'rr%rd)r6r) rFrdrr!rr!r7rlr8r) __future__rrKrgpathlibrr@tools.onnx_model_utilsrrr#tools.remove_initializer_from_inputr fusionsr r rUr rZrfusion_spacetodepthrrcrCrxrNr)r.r,rsr #""""" ``````````QQQQQQ;;;;;;;;######000000333333 ,1 "'$))-(,,148599=\\\\\~$$0$0$0$0$0$0$0$0V';,- s4s4s4s4s4s4s4r.