ktj=bddlmZddlZddlmZddlmZddlm Z m Z m Z m Z ddl mZeeZGddeZdS) ) getLoggerN)Fusion) FusionUtils) NodeProto TensorProtohelper numpy_helper) OnnxModelc *eZdZdZdeffd Zd!dedefdZded e defd Z d e d e de fd Z dede dzfdZ dede dzfdZdede fdZdedee efde fdZdede dzfdZdede dzfdZde de dzfdZde de de de dedef dZd ZxZS)"FusionMultiHeadAttentionMMDitzO Fuse MultiHeadAttention for Multimodal Diffusion Transformer (MMDiT). modelc`t|ddgi|_dS)NMultiHeadAttentionSoftmax) fused_op_typesearch_op_types)super__init__unsqueeze_update_map)selfr __class__s g/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/onnxruntime/transformers/fusion_mha_mmdit.pyrz&FusionMultiHeadAttentionMMDit.__init__s5 .BU^T_```$&!!!r start_nodereturncD|j|gd|ddg|}|dS|d}t|jdkrdS|j|jd}|dSt|jdkrdSt |dS) a Detect num_heads from Reshape & Transpose of q/k/v for both Stable Diffusion 3.x and Flux 1.x: MatMul .. [-1] [24] .. | | | / / Add Concat(axis=0) | / Reshape | Transpose(perm=0,1,3,2) | (start_node) ) TransposeReshapeConcatroutput_name_to_nodeN)r match_parent_pathleninputget_constant_valueshapeint)rrr" input_indexnodes concat_shapevalues r get_num_headsz+FusionMultiHeadAttentionMMDit.get_num_headss ,, :::[!Q|j|dgdg|}|r||d|SdS)a Detect num_heads from subgraph like the following (num_heads=24 in this example): MatMu .. [-1] [24] .. | | | / / Add Concat | / Reshape | Transpose(perm=0,2,1,3) | SimplifiedLayerNormalization | Transpose(perm=0,1,3,2) Another variant is to an extra Concat node to join two symmetrical subgraphs: | | MatMul MatMul .. [-1] [24] .. | | | | / / Add Concat Add Concat | / | / Reshape Reshape | | Transpose Transpose(perm=0,2,1,3) | | SimplifiedLayerNormalization SimplifiedLayerNormalization | / Concat | Transpose(perm=0,1,3,2) Both patterns are used in stable diffusion 3.5 model. rSimplifiedLayerNormalizationrr r!)r r&r0)rr1r"r2r-s rget_num_heads_from_kz2FusionMultiHeadAttentionMMDit.get_num_heads_from_k:sD # IJ00h(FG!Qex1E I))%(4GHHH IJ00<=sXk1E I))%(4GHHHqr input_name output_namecd}|j|}|Ltjt jgdd|}|j||jtj d||g|g|j d}|j ||j|j |j<|jd S) a+Add a Reshape node to convert 4D BxSxNxH to 3D BxSxD. Args: input_name (str): input name for the 4D tensor of shape BxSxNxH. output_name (str): output name for the 3D tensor of shape BxSxD, where D = N * H. Returns: str: the output name bsnh_to_bsd_reshape_dimsN)rrr#int64)dtype)namerinputsoutputsr<r)r get_initializerr from_arraynparrayadd_initializerthis_graph_namer make_nodecreate_node_name nodes_to_addappendnode_name_to_graph_namer<output)rr6r7 new_dims_namenew_dims reshape_qs r reshape_to_3dz+FusionMultiHeadAttentionMMDit.reshape_to_3dks3 :--m<<  #.rx '/R/R/RYfgggH J & &x1E F F F$  . M,,Y77      +++7;7K$Y^4""rmul_qNc.|j|ddgddg}|dS|\}}tj|dgdsdS|jd|jd<|jd}|dz|jd<||jd|dzS) a MultiHeadAttenion requires query in BSD format. This function adjusts query from BNSH to BSD format. Before: MatMul | Add Concat | / Reshape | Transpose(perm=0,2,1,3) | SimplifiedLayerNorm | Mul After: MatMul | Add Concat | / Reshape | SimplifiedLayerNorm | Reshape (shape=[0, 0, -1]) r4rrNpermrr%r _BSNH_BSD)r r&rcheck_node_attributer(rKrO)rrPr"pathsln_a transpose_a sln_outputs r'adjust_query_from_bnsh_to_bsd_no_concatzEFusionMultiHeadAttentionMMDit.adjust_query_from_bnsh_to_bsd_no_concats:z++  +[ 9 F   <4!{/ V\\\RR 4%*1- A\!_ $w. Q!!%,q/:3FGGGrc2|j|gdgd}|dS|\}}}t|jdkrdS|j|ddgddg}|dS|\}}t j|d gd sdSt j|d gd sdSt j|d dsdS|jd|jd<|jd|jd<t jd |jd|jdg|jdd zg|j d d} |j | |j |j | j<|| jd|jddzS)a MultiHeadAttenion requires query in BSD format. This function adjusts query from BNSH to BSD format. Before: MatMul MatMul | | Add Concat Add Concat | / | / Reshape Reshape | | Transpose(perm=0,2,1,3) Transpose(perm=0,2,1,3) | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=2) | Mul After: MatMul MatMul | | Add Concat Add Concat | / | / Reshape Reshape | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=1) | Reshape (shape=[0, 0, -1]) )rr4r)rrrNr%r4rr rrRrSaxisrrUr>r?r<r^rV)r r&r'r(rrWrrFrKrGrHrIrErJr<rO) rrPr"rXconcatrYrZsln_b transpose_bnew_concat_nodes radjust_query_from_bnsh_to_bsdz;FusionMultiHeadAttentionMMDit.adjust_query_from_bnsh_to_bsdsBz++  C C C II   <4%)"{ v|   ! !4z++  +[ 9 F   <4!{/ V\\\RR 4/ V\\\RR 4/BB 4%*1- A$*1- A * LOU\!_5]1%/0,,X66       111=A=Q$_%9:!!/"8";V]1=MPV=VWWWr unsqueezec|j|j}|Ut|jdkrGt jd|j|jddzg|j ddg}nd}|j |Dt j |tj dgdg}|j||jt jd|jd|g|jddzg|j d }|j||j|j|j<|jd}||j|j<|S) Nr UnsqueezerrUr%)r>r?r<axesunsqueeze_axes_2)r< data_typedimsvalsr=)rgetr<r'r(rrFrKr rGr@ make_tensorrINT64rDrErHrIrJ)rreupdated_unsqueeze_outputnew_nodeinitializer_nameris rupdate_unsqueeze_axes_1_to_2z:FusionMultiHeadAttentionMMDit.update_unsqueeze_axes_1_to_2s#'#<#@#@#P#P # +9?##q((!+$?&-a07:;44[AA $6 :--.>??G'-'9-"-"3SS ((($ J../?AUVVV!+%OA.0@A&-a07:;44[AA    $ $X . . .:>:ND ( 7'/q'9 $8PD %in 5''raddr"ct|jdkrdS|j|gdgd|}|dSt |j}||d}||dgkrdS||d}||dgkrdS|j|gdgd|}|dS||d}||dgkrdS||d}||dgkrdS||d|djd<||d|djd<d S) a Update axes of Unsqueeze from [1] to [2] in the following pattern: Unsqueeze Unsqueeze (axes=[0]) (axes=[0]) | | Unsqueeze Unsqueeze ... (axes=[1]) ... (axes=[1]) | / | / Mul Mul | / Add Args: add (NodeProto): the Add node output_name_to_node (Dict[str, NodeProto]): mapping from output name to node Returns: bool: True if the pattern is matched and updated successfully, False otherwise. r%F)Mulrgrg)r r rNr r)rr rT)r'r(r r&rget_squeeze_or_unsqueeze_axesrs)rrtr"nodes_b fusion_utilsaxes_1axes_0nodes_as rupdate_unsqueeze_axesz3FusionMultiHeadAttentionMMDit.update_unsqueeze_axes(s& sy>>Q  5*..s4U4U4UW`W`W`buvv ?5"4:.. ;;GAJGG >Vs]]5;;GAJGG >Vs]]5*..s4U4U4UW`W`W`buvv ?5;;GAJGG >Vs]]5;;GAJGG >Vs]]5"?? KK "?? KK trc|j|gdgd}|dS|\}}}}}t|jdkrdS|j|ddgddg}|dS|\} } t j|d gd sdSt j| d gd sdSt j|d dsdS|||sdS|jd|jd<| jd| jd<tjd |j d| j dg|j dd zg|j d d} |j | |j |j| j<|j|j d| j d||j d|j ddzS)a3 Adjust graph to change query format from BNSH to BSD for Flux model. Note that the graph pattern is complex, and we only do a shallow match here. Before: | | Transpose(perm=0,2,1,3) Transpose(perm=0,2,1,3) | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=2) | Mul Mul | / Add | Mul After (Transpose nods are removed, and a Reshape is added): | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=1) | Mul Mul | / Add | Reshape (shape=[0, 0, -1]) )Addrvrr4r)rrrrrNr%r4rr rrRrSr^rrUr_rV)r r&r'r(rrWr}rrFrKrGrHrIrErJr<replace_input_of_all_nodesrO) rrPr"rXrt_mul_ar`rYrZrarbrcs r"adjust_flux_query_from_bnsh_to_bsdz@FusionMultiHeadAttentionMMDit.adjust_flux_query_from_bnsh_to_bsd]sBz++  Q Q Q OO   <426/VVUK v|   ! !4z++  +[ 9 F   <4!{/ V\\\RR 4/ V\\\RR 4/BB 4))#/BCC 4%*1- A$*1- A * LOU\!_5]1%/0,,X66       111=A=Q$_%9: --fmA.>@VWX@YZZZ!!#*Q-A1GHHHrct|j|gdgd}|dS|\}}}}tj|dgdsdS|||sdS|jd|jd<|jddz|jd<||jd|jddzS) a0 Adjust graph to change query format from BNSH to BSD for Flux model. Note that the graph pattern is complex, and we only do a shallow match here. Before: | Transpose(perm=0,2,1,3) | SimplifiedLayerNorm | Mul Mul | / Add | Mul After (Transpose is removed, and a Reshape is added): | SimplifiedLayerNorm | Mul Mul | / Add | Reshape (shape=[0, 0, -1]) )rrvr4r)rrrrNrRrSrrUrV)r r&rrWr}r(rKrO)rrPr"rXrtrrYrZs r)adjust_flux_single_query_from_bnsh_to_bsdzGFusionMultiHeadAttentionMMDit.adjust_flux_single_query_from_bnsh_to_bsds:z++  G G G LL   <4*.'VUK/ V\\\RR 4))#/BCC 4%*1- A 1 / 1 !!#*Q-A1GHHHrqc  tjd|g|dzg|jddgd}|j||j|j|j<| |dz|dzS)NrrUTranspose_BNSH_to_BSNH) name_prefixrS)r<rRrV) rrFr rGrHrIrErJr<rO)rrr" transpose_qs rtranspose_reshape_bnsh_to_bsdz;FusionMultiHeadAttentionMMDit.transpose_reshape_bnsh_to_bsds&  C [M,,[F^,__       ---9=9M$[%56!!!g+q6z:::rkvrK num_headsc|dksJ|||g}|g}tjd|||jd}d|_|jtjd|g|S)a~ Create a MultiHeadAttention node. Args: q (str): name of q k (str): name of k v (str): name of v output (str): output name of MHA num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. Returns: NodeProto: the node created. rrr=z com.microsoftr)rrFr rGdomain attributeextendmake_attribute) rrrrrKr mha_inputs mha_outputsmha_nodes rcreate_multihead_attention_nodez=FusionMultiHeadAttentionMMDit.create_multihead_attention_nodes,1}}}}AY h # ,,-ABB    *!!6#8i#P#P"QRRRrc|jdksJ|}|j|jdrdS|j|gdgd|}|dS|\}}}t j|dgdsdS|j|gdgd} | dS| \} } } } }}}}| jd}||jdkrdS|j| d d gd dg}|dS|\}}|jd}t j|dgd sdS|j|d dgd dg}|dS|djd| jdkrdS|jd }|j |dd |}|y|j |d d|}|dSt j|dgdsdS|j |d d |}|dSt j|dgdsdSn<|j |d d |}|dSt j|dgdsdS|r| ||n| ||d }|dkr!| |||du}|dkrdS|| | |}n| | |}|F|| |}|.|| |}||||}|||||jd|}|j||j|j|j<|j|||gd|_dS)Nrr)MatMulrr)rrrrrRrS)rrvSqrtDivrCastSliceShape)rrr rr rrrrvrr )rr rTr%rrr)r,r")r,)rrrrKrT)op_typer find_graph_outputrKmatch_child_pathrrWr&r( match_parentr0r5rdr\rrrrrHrIrErJr<nodes_to_remover prune_graph)rnodeinput_name_to_nodesr"softmaxr- matmul_s_v transpose_out reshape_outq_nodes matmul_qkrPsqrt_q_2div_qsqrt_q_shape_qq_bnshk_nodesmul_kr1r k_scale_nodesrconcat_v transpose_1 transpose_2rqueryrqs rfusez"FusionMultiHeadAttentionMMDit.fuses|y(((( : ' 'q(9 : :  F ++ 7779Q9Q9QSf   = F16. M;/ v|||TT  F*..  N N N $ $ $   ? FCJ@ 5(E61aQ W]1% % % F*..y5+:NQRTUPVWW ? F${  a / V\\\RR  F 44UVUOaQRVTT  F   !! $q(9 9 9 F  Q :**:xQdw*xx   *11+1J]2K"3KVV *11+1J]2K"3KVV   *11KQL_2K"3KVV   TD  x)< = = =##J0CQR#SS  >>11+?RT\dhThiiIA~~  66u>QRREE@@H[\\E =;;ECVWWE}FFuNabb=!>>vGZ[[E77%a( 8     ***6:6J$X]3 ##Z $LMMM r)r)__name__ __module__ __qualname____doc__r rrr+r0boolr5strrOr\rdrsdictr}rrrrr __classcell__)rs@rr r s|'i'''''' Z]B/ /im/ru////b#########4.HY.H`cfj`j.H.H.H.H`MX9MXVY\`V`MXMXMXMX^"(i"(C"("("("(H33cS\nI]3bf3333jRI RI[^ae[eRIRIRIRIh1Iy1Ibehlbl1I1I1I1If ;s ;CRVJ ; ; ; ;) ) )  )  )  ) ))))V       rr )loggingrnumpyrB fusion_baserryronnxrrrr onnx_modelr rloggerr rrrs $$$$$$============ 8  K  K  K  K  K  FK  K  K  K  K  r