ktjB^ddlmZddlZddlmZddlmZddlm Z m Z m Z ddl m Z eeZGddeZdS) ) getLoggerN)Fusion) FusionUtils) NodeProtohelper numpy_helper) OnnxModelceZdZdZdeffd ZdededzfdZdede dzfdZ d e d e de fd Z d ede fd Zd e de fdZdedede fdZde de de fdZde de fdZde de fdZ dde de de de de dedzdefdZdZxZS) FusionMultiHeadAttentionDiTu Fuse MultiHeadAttention for Diffusion Transformer (DiT) models like F5-TTS. Recognizes attention patterns where Q, K, V are pre-computed (e.g., after RoPE) and K is pre-transposed, with optional Cast nodes for mixed-precision (FP16) inference and a custom scalar scale factor before Softmax. Supported patterns (anchored at Softmax): MatMul(Q, K^T) → [Cast(FP16→FP32)] → Mul(scale) → Softmax → [Cast(FP32→FP16)] → MatMul(attn, V) → Transpose(perm=0,2,1,3) → Reshape → output Where: - Q is in BNSH format (post-RoPE or post-projection) - K is pre-transposed to BNHS format (via Transpose(perm=0,1,3,2) or natively) - V is in BNSH format - Scale is an arbitrary scalar constant (e.g., 100.0 for DiT, or 1/sqrt(d_k)) - Cast nodes are optional (present in FP16 models for FP32 Softmax stability) modelcRt|ddgdS)NMultiHeadAttentionSoftmax) fused_op_typesearch_op_types)super__init__)selfr __class__s e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/onnxruntime/transformers/fusion_mha_dit.pyrz$FusionMultiHeadAttentionDiT.__init__%s- .BU^T_`````mul_nodereturnNcdtdD]}|j|j|}|vt |t jr/|jdkr#t| cSrt |ttfrt|cSdS)zExtract the scalar scale constant from a Mul node. The scale can be in either input[0] or input[1]. Returns: float: the scale value, or None if not found. N) ranger get_constant_valueinput isinstancenpndarraysizefloatitemint)rrivalues rget_scale_from_mulz.FusionMultiHeadAttentionDiT.get_scale_from_mul(sq ( (AJ11(.2CDDE eRZ00(zQ$UZZ\\22222'U|44( <<'''trc\ddg}tdD]s}|j|j|}|Jt |t jr|jdk||<Rt |ttfrd||<t|dr |dsdS|dr |dsdSdS)zDetermine which input of Mul is the data (non-constant) input. Returns: int: the input index (0 or 1) of the data input, or None if ambiguous. FrNrTr) rr rrr r!r"r#r&r$)rris_scalar_constantr'r(s rget_data_input_of_mulz1FusionMultiHeadAttentionDiT.get_data_input_of_mul:s $U^q 1 1AJ11(.2CDDE eRZ001,1J!O&q))U|441,0&q) a  );A)> 1 a  );A)> 1tr tensor_nameoutput_name_to_nodecl||vrdS||}d}d}|||kr|jdkr]tj|d}|gdkr@|jd|vr0||jd}||}|dkr|Sn1|jd|vr||jd}nn |dz }|||kdS)uDetect num_heads by walking upstream from a BNSH tensor looking for a Reshape node. Typical upstream patterns: Reshape(shape=[B, S, N, H]) → Transpose(perm=0,2,1,3) → ... → tensor_BNSH Reshape(shape=Concat(..., N, H)) → Transpose(perm=0,2,1,3) → ... → tensor_BNSH Returns: int: number of heads, or 0 if not detected. r N Transposepermrrrr)op_typer get_node_attributer_get_num_heads_from_reshape) rr-r.currentdepth max_depthr2parent num_headss rdetect_num_headsz,FusionMultiHeadAttentionDiT.detect_num_headsOs 1 1 11&k2 !ei&7&7+-- 3GVDD<<<''}Q'+>>>!4W]15E!F$($D$DV$L$L $q==#,,}Q#666-gmA.>? QJE%!ei&7&7(qrnodecjdkrdStjdkra|jjd}|:t |t jr |jdkrt|dStjdkrjdd|j dDvrfd |j dD}|rt|djdkrs|j|djd}|Ft |t jr,|jdkr!t| SdS) zExtract num_heads from a Reshape node's shape parameter. Handles: - Static shape constant: [B, S, num_heads, head_dim] - Concat-based shape: Concat([B_dim], [S_dim], [num_heads], [head_dim]) ReshaperrrNc(h|]}|jdS)r)output).0ns r zJFusionMultiHeadAttentionDiT._get_num_heads_from_reshape..s5u5u5uaahqk5u5u5urConcatcLg|] }|jdjdk|!S)rr)rCr)rDrEr>s r zKFusionMultiHeadAttentionDiT._get_num_heads_from_reshape..s7qqq!TUT\]^T_cgcmnocpTpTpATpTpTpr) r5lenrr rr r!r"r#r&get_nodes_by_op_typer%)rr> shape_value concat_nodesr(s ` rr7z7FusionMultiHeadAttentionDiT._get_num_heads_from_reshapewsn <9 $ $1 tz??a  *77 1 FFK&:k2:+N+N&S^ScghShSh;q>*** tz??a  DJqM5u5u4:KjKjksKtKt5u5u5u$u$uqqqqtz'F'Fx'P'PqqqL 1LO$9 : :a ? ? 55l1o6KA6NOO$!%441q"5::<<000qrc|jjjjD]Y}|j|krL|jjj}|r9t|jdkr!|jd}|j dkr |j cSZ|jjjj D]Y}|j|krL|jjj}|r9t|jdkr!|jd}|j dkr |j cSZdS)zTry to detect num_heads from a BNSH tensor's shape in graph inputs or value_info. For BNSH tensors, the N dimension (index 1) is num_heads. rArr) r graphrnametype tensor_typeshaperJdim dim_value value_info)rr-inprSdim_nvis r!detect_num_heads_from_input_shapez=FusionMultiHeadAttentionDiT.detect_num_heads_from_input_shapes :#)/ / /Cx;&&,2/S^^q00!IaLE**$...*"(3 / /Bw+%%+1/S^^q00!IaLE**$...qr reshape_out transpose_outcB||jdS)zTry to detect num_heads from the output Transpose's input shape. The Transpose converts BNSH -> BSNH. The N dimension gives us num_heads. r)rZr)rr[r\s rdetect_num_heads_from_outputz8FusionMultiHeadAttentionDiT.detect_num_heads_from_outputs 55m6I!6LMMMr input_name output_namecd}|j|}|Ltjt jgdd|}|j||jtj d||g|g|j d}|j ||j|j |j<|S) aAdd a Reshape node to convert 4D BxSxNxH to 3D BxSxD. Args: input_name: input name for the 4D tensor of shape BxSxNxH. output_name: output name for the 3D tensor of shape BxSxD. Returns: str: the output name. bsnh_to_bsd_reshape_dimsN)rrint64)dtype)rPr@inputsoutputsrP)r get_initializerr from_arrayr!arrayadd_initializerthis_graph_namer make_nodecreate_node_name nodes_to_addappendnode_name_to_graph_namerP)rr_r` new_dims_namenew_dims reshape_nodes r reshape_to_3dz)FusionMultiHeadAttentionDiT.reshape_to_3ds3 :--m<<  #.rx '/R/R/RYfgggH J & &x1E F F F'  . M,,Y77      ...:>:N$\%67rc |dz}tjd|g|g|jddgd}|j||j|j|j<|S)z4Add a Transpose node to convert BNSH to BSNH format._BSNHr1Transpose_BNSH_to_BSNH name_prefixr3rPr2 rrnr rorprqrmrrrPrr_r`transpose_nodes rtranspose_bnsh_to_bsnhz2FusionMultiHeadAttentionDiT.transpose_bnsh_to_bsnh 7* )  L M,,[F^,__       000<@||"||| jd| |}%|j|%|j|j|%j<|| | g}&||&||j|&| jdg||st d dS|j !|&d|_"dS)!Nrraxis)rcr4 )CastMatMulr1r@)rrrrr)rr1r@)rrrr2r3)Mulrr)rNrrrz9fuse_dit_attention: failed to extract scale from Mul noderFrTr1) input_indexr.rzBfuse_dit_attention: Q/V element type mismatch (%s vs %s), skippingzBfuse_dit_attention: Q/K element type mismatch (%s vs %s), skippingzlfuse_dit_attention: cast nodes present, V not traced through Cast, types unverifiable (q=%s, v=%s), skippingzLfuse_dit_attention: K Cast source dtype mismatch with Q (%s vs %s), skippingz.fuse_dit_attention: failed to detect num_heads_BSDzDfuse_dit_attention: intermediate %s has multiple consumers, skipping)rrrrCr<rzFfuse_dit_attention: downstream nodes have external consumers, skipping)#r5r find_graph_outputrCr r6get_opset_versionmatch_child_pathrcheck_node_attributematch_parent_pathr)loggerdebugr,r match_parentr get_dtyper=rZr^rrvrqrJrrprmrrrPis_safe_to_fuse_nodesnodes_to_remover prune_graph)'rr>input_name_to_nodesr.softmaxrcast_after_softmax child_nodes matmul_svr\r[cast_before_softmax parent_nodes mul_scale matmul_qkrdata_input_idxexpected_data_sourceparent_nodes_altq_bnshk_transposed_inputv_bnshv_traced_through_cast v_producerk_transpose_nodek_bnshk_traced_through_castk_bnsh_for_dtype k_producerq_dtypek_dtypev_dtyper<q_bsnhq_bsdintermediate_outputsr-rrs' rfusez FusionMultiHeadAttentionDiT.fuses|y(((( : ' 'q(9 : :  F +GV<<  G 3 3 F >  ! FATl29!< *.B B B" 8#':#?#?---N*A.$$ $+FJ!V++#-#3A#6 (,%*&&v..*&&'788*&&v..  7#67g;M;M LL]_fho p p p F  7#67g;M;M LL]_fho p p p F +/A/MWl/M'/ @   ! W%8W=PU\`gUgUg LL^    F ))&2EFF >>--f6IJJI >>>>vFFI >>>>vFFI >>99+}UUI >> LLI J J J F ,,V44""66F?;; !* 0 3Y5Ea5H'.YZJ[\  * ' '(;(B1(E F F F  ) ' '(:(A!(D E E E/  K111c:Mk:Z6[6[^_6_6_ cepqqq 77%a( 8     ***6:6J$X]3%m[A  )  " "#5 6 6 6 z// k0346IK^    LLa b b b F ##O444 r)N)__name__ __module__ __qualname____doc__r rrr$r)r&r,strdictr=r7rZr^rvrrrr __classcell__)rs@rr r s+(aiaaaaaa9$iC$J*&C&d&s&&&&P c6SS0N NR[N`cNNNN##2            *#(( ( (  (  (  (t|( ((((TE E E E E E E rr )loggingrnumpyr! fusion_baser fusion_utilsronnxrrr onnx_modelr rrr rrrs $$$$$$0000000000 8  E E E E E &E E E E E r