tjaddlmZddlZddlmZddlZgdZeddZedd Z d#d$dZ d#d%dZ d&dZ d'd"Z dS)() annotationsN)TypeVar)fuse_conv_bn_evalfuse_conv_bn_weightsfuse_linear_bn_evalfuse_linear_bn_weightsConvTztorch.nn.modules.conv._ConvNd)boundLinearTztorch.nn.LinearFconvbn%torch.nn.modules.batchnorm._BatchNorm transposeboolreturnc .|js|jrtdtj|}|j|jtdt |j|j|j|j|j |j|j|\|_|_|S)a+Fuse a convolutional module and a BatchNorm module into a single, new convolutional module. Args: conv (torch.nn.modules.conv._ConvNd): A convolutional module. bn (torch.nn.modules.batchnorm._BatchNorm): A BatchNorm module. transpose (bool, optional): If True, transpose the convolutional weight. Defaults to False. Returns: torch.nn.modules.conv._ConvNd: The fused convolutional module. .. note:: Both ``conv`` and ``bn`` must be in eval mode, and ``bn`` must have its running buffers computed. Fusion only for eval!N3bn.running_mean and bn.running_var must not be None) trainingAssertionErrorcopydeepcopy running_mean running_varrweightbiaseps)r r r fused_convs S/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/nn/utils/fusion.pyrrs$ }6 64555t$$J "."8RSSS)=      * *&Jz conv_w torch.Tensorconv_btorch.Tensor | Nonebn_rmbn_rvbn_epsfloatbn_wbn_b-tuple[torch.nn.Parameter, torch.nn.Parameter]c|j}||jn|} |tj|}|tj|}|tj|}tj||z} |r!ddgdgt |jdz zz} n ddgdgt |jdz zz} ||| z| z|} ||z | z|z|z| } tj | |j tj | |j fS)aFuse convolutional module parameters and BatchNorm module parameters into new convolutional module parameters. Args: conv_w (torch.Tensor): Convolutional weight. conv_b (Optional[torch.Tensor]): Convolutional bias. bn_rm (torch.Tensor): BatchNorm running mean. bn_rv (torch.Tensor): BatchNorm running variance. bn_eps (float): BatchNorm epsilon. bn_w (Optional[torch.Tensor]): BatchNorm weight. bn_b (Optional[torch.Tensor]): BatchNorm bias. transpose (bool, optional): If True, transpose the conv weight. Defaults to False. Returns: Tuple[torch.nn.Parameter, torch.nn.Parameter]: Fused convolutional weight and bias. Ndtype) r1torch zeros_like ones_likersqrtlenshapereshapetonn Parameter requires_grad)r!r#r%r&r'r)r*rconv_weight_dtypeconv_bias_dtype bn_var_rsqrtr7 fused_conv_w fused_conv_bs rrr:se2 &,&8fll>OO ~!%(( |u%% |&&;uv~..L8B1#V\!2!2Q!677Q1#V\!2!2Q!677d\1::5AAAEEFLe^|3d:TAEEFL <)=>> <)=>> r linearc |js|jrtdtj|} |j|jkr*|jdkrtd|jd|j|j|jtdt|j |j |j|j|j |j |j \|_ |_ |S)aFuse a linear module and a BatchNorm module into a single, new linear module. Args: linear (torch.nn.Linear): A Linear module. bn (torch.nn.modules.batchnorm._BatchNorm): A BatchNorm module. Returns: torch.nn.Linear: The fused linear module. .. note:: Both ``linear`` and ``bn`` must be in eval mode, and ``bn`` must have its running buffers computed. rr-zaTo fuse, linear.out_features == bn.num_features or bn.num_features == 1, got linear.out_features=z and bn.num_features=Nr) rrrr out_features num_featuresrrrrrr)rBr fused_linears rrros 6"+64555=((L bo--"/Q2F2F c'-': c cQSQ` c c    "."8RSSS-C     ..*L* r linear_wlinear_bc|j}||jn|}|tj|}|tj||zz} || d|z} ||z | z|z|} tj| |jtj| |jfS)a2Fuse linear module parameters and BatchNorm module parameters into new linear module parameters. Args: linear_w (torch.Tensor): Linear weight. linear_b (Optional[torch.Tensor]): Linear bias. bn_rm (torch.Tensor): BatchNorm running mean. bn_rv (torch.Tensor): BatchNorm running variance. bn_eps (float): BatchNorm epsilon. bn_w (torch.Tensor): BatchNorm weight. bn_b (torch.Tensor): BatchNorm bias. Returns: Tuple[torch.nn.Parameter, torch.nn.Parameter]: Fused linear weight and bias. Nr.r0) r1r2r3r5 unsqueezer9r:r;r<) rGrHr%r&r'r)r*linear_weight_dtypelinear_bias_dtypebn_scalefused_wfused_bs rrrs.#.*2*>DW#E**ek%&.111H++B//229L2MMMG5 H,t377>O7PPG 8  gx'= > >@R@R'AA r )F)r r r rrrrr )r!r"r#r$r%r"r&r"r'r(r)r$r*r$rrrr+)rBr r rrr )rGr"rHr$r%r"r&r"r'r(r)r"r*r"rr+) __future__rrtypingrr2__all__r r rrrrr rrTs""""""      >??? ')#4 5 5 5 #####\22222j0000f""""""r