tjbdZddlmZddlZddlmZddlmZddlmZddl m Z m Z d d l m Z e jeZe d eGd deZGddejZGddejZGddejZGddejZe GddeZdgZdS)zPyTorch UnivNetModel model.) dataclassN)nn) ModelOutput)PreTrainedModel)auto_docstringlogging) UnivNetConfigz Output class for the [`UnivNetModel`], which includes the generated audio waveforms and the original unpadded lengths of those waveforms (so that the padding can be removed by [`UnivNetModel.batch_decode`]). ) custom_introcPeZdZUdZdZejdzed<dZejdzed<dS)UnivNetModelOutputa" waveforms (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Batched 1D (mono-channel) output audio waveforms. waveform_lengths (`torch.FloatTensor` of shape `(batch_size,)`): The batched length in samples of each unpadded waveform in `waveforms`. N waveformswaveform_lengths) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rj/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/univnet/modeling_univnet.pyrrsO+/Iu 4'...15e'$.55555rrcJeZdZdZdeffd ZdejfdZdZ dZ xZ S)#UnivNetKernelPredictorResidualBlockz Implementation of the residual block for the kernel predictor network inside each location variable convolution block (LVCBlock). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. configct|j|_|j|_|j|_|j|_|jdz dz}tj |j|_ tj |j|j|j|d|_ tj |j|j|j|d|_dS)Nr Tpaddingbias)super__init__model_in_channelschannelskernel_predictor_conv_size kernel_sizekernel_predictor_dropout dropout_probleaky_relu_sloperDropoutdropoutConv1dconv1conv2)selfrr! __class__s rr$z,UnivNetKernelPredictorResidualBlock.__init__;s 0 !<"; & 7#a'A-z$"344 Yt}dmT=MW^eijjj Yt}dmT=MW^eijjj r hidden_statesc"|}||}||}tj||j}||}tj||j}||zSN)r-r/r functional leaky_relur+r0)r1r3residuals rforwardz+UnivNetKernelPredictorResidualBlock.forwardKsy  ]33  =11  00@UVV  =11  00@UVV x''rctjj}ttjjdrtjjj}||j||jdSN weight_norm)rutilsr<hasattrparametrizationsr/r0r1r<s rapply_weight_normz5UnivNetKernelPredictorResidualBlock.apply_weight_normUs[h* 28,m < < @(3?K DJ DJrctj|jtj|jdSr5)rr=remove_weight_normr/r0r1s rrCz6UnivNetKernelPredictorResidualBlock.remove_weight_norm]s8 ##DJ/// ##DJ/////r) rrrrr r$rrr9rArC __classcell__r2s@rrr1skkkkkkk (U%6((((   0000000rrcXeZdZdZ d dededeffd Zdejfd Z d Z d Z xZ S) UnivNetKernelPredictora  Implementation of the kernel predictor network which supplies the kernel and bias for the location variable convolutional layers (LVCs) in each UnivNet LVCBlock. Based on the KernelPredictor implementation in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L7). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. conv_kernel_size (`int`, *optional*, defaults to 3): The kernel size for the location variable convolutional layer kernels (convolutional weight tensor). conv_layers (`int`, *optional*, defaults to 4): The number of location variable convolutional layers to output kernels and biases for. rrconv_kernel_size conv_layersctj|_djz|_||_||_|j|jz|jz|jz|_|j|jz|_j |_ j |_ j |_j|_j|_|jdz dz}t%j|j |j ddd|_t%jfdt-|jD|_t%j|j |j|j|d|_t%j|j |j|j|d|_dS)Nrr Tr c.g|]}tSr)r).0_rs r z3UnivNetKernelPredictor.__init__..s#'t't'tXY(KF(S(S't't'tr)r#r$model_hidden_channelsconv_in_channelsconv_out_channelsrJrKkernel_channels bias_channels num_mel_binsresnet_in_channels kernel_predictor_hidden_channelsresnet_hidden_channelsr'resnet_kernel_sizekernel_predictor_num_blocks num_blocksr+rr. input_conv ModuleListrange resblocks kernel_conv bias_conv)r1rrJrKr!r2s ` rr$zUnivNetKernelPredictor.__init__ss  & <!"V%A!A 0&  !D$: :T=R RUYUe e "3d6FF"("5&,&M#"("C < & 7*Q.14)D$;T=XZ[efmqrrr't't't't]bcgcr]s]s't't'tuu9  ')=t?V`gnr     ');T=T^elp   r spectrogramc|j\}}}||}tj||j}|jD] }||}||}||}| ||j |j |j |j |} | ||j |j |} | | fS)a Maps a conditioning log-mel spectrogram to a tensor of convolutional kernels and biases, for use in location variable convolutional layers. Note that the input spectrogram should have shape (batch_size, input_channels, seq_length). Args: spectrogram (`torch.FloatTensor` of shape `(batch_size, input_channels, seq_length)`): Tensor containing the log-mel spectrograms. Returns: tuple[`torch.FloatTensor, `torch.FloatTensor`]: tuple of tensors where the first element is the tensor of location variable convolution kernels of shape `(batch_size, self.conv_layers, self.conv_in_channels, self.conv_out_channels, self.conv_kernel_size, seq_length)` and the second element is the tensor of location variable convolution biases of shape `(batch_size, self.conv_layers. self.conv_out_channels, seq_length)`. )shaper^rr6r7r+rarbrcviewrKrSrTrJ contiguous) r1rd batch_sizerP seq_lengthr3resblockkernel_hidden_statesbias_hidden_stateskernelsbiasess rr9zUnivNetKernelPredictor.forwards "%0$5! Az 44  00@UVV  4 4H$H]33MM#// >>!^^M::'++     !  "  !     *,, $((     "    *,, rc.tjj}ttjjdrtjjj}||j|jD]}|||j||j dSr;) rr=r<r>r?r^rarArbrcr1r<layers rrAz(UnivNetKernelPredictor.apply_weight_normsh* 28,m < < @(3?K DO$$$^ & &E  # # % % % % D$%%% DN#####rctj|j|jD]}|tj|jtj|jdSr5)rr=rCr^rarbrcr1rrs rrCz)UnivNetKernelPredictor.remove_weight_normsw ##DO444^ ' 'E  $ $ & & & & ##D$4555 ##DN33333r)rrI rrrrr intr$rrr9rArCrErFs@rrHrHbs&!" $ $ $ $  $ $ $ $ $ $ L,5#4,,,,\ $ $ $4444444rrHc eZdZdZdededeffd ZddZ dd ej d ej d ej ded ef d Z dZ dZ xZ S)UnivNetLvcResidualBlocka Implementation of the location variable convolution (LVC) residual block for the UnivNet residual network. Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. kernel_size (`int`): The kernel size for the dilated 1D convolutional layer. dilation (`int`): The dilation for the dilated 1D convolutional layer. rr(dilationc"t|j|_||_||_|j|_|j|jdz zdz}tj|j|j|j||j|_ dS)Nr r)r!ry) r#r$rRhidden_channelsr(ryr+rr.conv)r1rr(ryr!r2s rr$z UnivNetLvcResidualBlock.__init__s %;&  & 7-4#3a#78A=I    ]     rc|}tj||j}||}tj||j}|||||}t j|ddd|jddft j |dd|jdddfz}||z}|SN)hop_size) rr6r7r+r|location_variable_convolutionrsigmoidr{tanh)r1r3kernelr"rr8s rr9zUnivNetLvcResidualBlock.forwards  00@UVV  -00  00@UVV ::=&RVai:jj  mAAA7M9M7Mqqq4P&QRRUZU_ !!!T133QQQ6 7V V  !=0 rr r3rr"rc|j\}}}|j\}}} } } || |zkrtd| |zd|d|t| dz dz z} tj|| | fdd}|d|d| zz|}||kr$tj|d|fdd}|d||}|d d d d d d d d d |f}|dd }|d | d}tj d ||} | d  d }| |z} | || d } | S) u Performs location-variable convolution operation on the input sequence (hidden_states) using the local convolution kernel. This was introduced in [LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation](https://huggingface.co/papers/2102.10815) by Zhen Zheng, Jianzong Wang, Ning Cheng, and Jing Xiao. Time: 414 μs ± 309 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each), test on NVIDIA V100. Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, in_channels, in_length)`): The input sequence of shape (batch, in_channels, in_length). kernel (`torch.FloatTensor` of shape `(batch_size, in_channels, out_channels, kernel_size, kernel_length)`): The local convolution kernel of shape (batch, in_channels, out_channels, kernel_size, kernel_length). bias (`torch.FloatTensor` of shape `(batch_size, out_channels, kernel_length)`): The bias for the local convolution of shape (batch, out_channels, kernel_length). dilation (`int`, *optional*, defaults to 1): The dilation of convolution. hop_size (`int`, *optional*, defaults to 256): The hop_size of the conditioning sequence. Returns: `torch.FloatTensor`: the output sequence after performing local convolution with shape (batch_size, out_channels, in_length). z#Dim 2 of `hidden_states` should be z ) but got zX. Please check `hidden_states` or `kernel` and `hop_size` to make sure they are correct.r rconstantrrNrIzbildsk,biokl->bolsd) rf ValueErrorrvrr6padunfold transposereinsum unsqueezerg)r1r3rr"ryrbatchrP in_length out_channelsr( kernel_lengthr!output_hidden_statess rrz5UnivNetLvcResidualBlock.location_variable_convolution s<,1q)=C\:q, ] 1 2 2]mh6N]]Zc]]]  S+/Q!6777 ))-'79KZYZ[[ %,,Q1w;0FQQ h  M--ma]JXYZZM%,,Q(CC %aaaAAAqqq)8)&;< %//155 %,,Q Q??  %|,A=RXYY~~b!!++B//3d:388 bQQ##rctjj}ttjjdrtjjj}||jdSr;)rr=r<r>r?r|r@s rrAz)UnivNetLvcResidualBlock.apply_weight_normMsIh* 28,m < < @(3?K DIrcNtj|jdSr5)rr=rCr|rDs rrCz*UnivNetLvcResidualBlock.remove_weight_normTs  ##DI.....rr})r r})rrrrr rvr$r9rrrrArCrErFs@rrxrxs            ,    * >$>$(>$!>$ >$  >$  >$>$>$>$@///////rrxcdeZdZdZ d dededeffd Zdejdejfd Z d Z d Z xZ S) UnivNetLvcBlocka# Implementation of the location variable convolution (LVC) residual block of the UnivNet residual block. Includes a `UnivNetKernelPredictor` inside to predict the kernels and biases of the LVC layers. Based on LVCBlock in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L98) Parameters: config (`UnivNetConfig`): Config for the `UnivNetModel` model. layer_id (`int`): An integer corresponding to the index of the current LVC resnet block layer. This should be between 0 and `len(config.resblock_stride_sizes) - 1)` inclusive. lvc_hop_size (`int`, *optional*, defaults to 256): The hop size for the location variable convolutional layers. r}rlayer_id lvc_hop_sizec ztj_j|_j|_j|_ |_ j _ tj _ tjjjdjzjjdzjdzzjdz_t#jj _tjfdt)j D_dS)Nr)strider!output_paddingcRg|]#}tjj|$Sr)rxr( dilations)rOirr1s rrQz,UnivNetLvcBlock.__init__..s1 r r rVW $VT-=t~a?P Q Q r r rr)r#r$rRr{resblock_kernel_sizesr(resblock_stride_sizesrresblock_dilation_sizesrcond_hop_lengthr+lenr]rConvTranspose1d convt_prerHkernel_predictorr_r`ra)r1rrrr2s`` rr$zUnivNetLvcBlock.__init__js& %;!7A28< 7A+ & 7dn--+    O;K1$t{Q6;?    !7vt?OQUQ` a a r r r r r[`aeap[q[q r r r  rr3rdc Ztj||j}||}||\}}t |jD]C\}}|dd|ddddddddf}|dd|ddddf}|||||j}D|Sr) rr6r7r+rr enumeraterar) r1r3rdrnrorrkrr"s rr9zUnivNetLvcBlock.forwards 00@UVV }55 // <<$T^44 a aKAxQQQ111aaaAAA-.F!!!Q111*%D$H]FD4K_```MMrc tjj}ttjjdrtjjj}||j|j|jD]}|dSr;) rr=r<r>r?rrrArarqs rrAz!UnivNetLvcBlock.apply_weight_normsh* 28,m < < @(3?K DN### //111^ & &E  # # % % % % & &rctj|j|j|jD]}|dSr5)rr=rCrrrarts rrCz"UnivNetLvcBlock.remove_weight_norms_ ##DN333 00222^ ' 'E  $ $ & & & & ' 'rrrurFs@rrrXs*            < U%6 UEV    &&&'''''''rrceZdZUeed<dZdZdeffd Ze dde j de j dzde j dzde j dzd e dzd e e j ezf d Zd Zd ZxZS) UnivNetModelrinput_featuresaudioc6ttj|_j|_t jjj dddd|_ tj }d}gj D]}||z} |t j fdt|D|_t jj dddd|_|dS)Nr rreflect)r(rr! padding_modec@g|]}t||S))rr)r)rOrr hop_lengthss rrQz)UnivNetModel.__init__..sG      !,Q   r)r!r)r#r$rr num_kernelsr+rr.r%rRconv_prerappendr_r`ra conv_post post_init)r1r num_layers hop_lengthrrr2s ` @rr$zUnivNetModel.__init__s<    v;<< & 7  $  ("     566   2 + +F#f,J   z * * * *      z**       6#?Aq_hiii rNnoise_sequence padding_mask generator return_dictreturnc ||n |jj}|dk}|s|d}|j\}} } |0|dk} | s|d}n1|| |jjf} t j| ||j|j }|jd} |dkr| dkr| |dd}n#| dkr|dkr| | dd}| |krtd| d|d|V|dkr|d}|jd}||krtd |d|d| d d}| d d}| |}|jD]}|||}tj||j}||}t j|}|d}d}|t j|d }|s||f}|St/|| S) a noise_sequence (`torch.FloatTensor`, *optional*): Tensor containing a noise sequence of standard Gaussian noise. Can be batched and of shape `(batch_size, sequence_length, config.model_in_channels)`, or un-batched and of shape (sequence_length, config.model_in_channels)`. If not supplied, will be randomly generated. padding_mask (`torch.BoolTensor`, *optional*): Mask indicating which parts of each sequence are padded. Mask values are selected in `[0, 1]`: - 1 for tokens that are **not masked** - 0 for tokens that are **masked** The mask can be batched and of shape `(batch_size, sequence_length)` or un-batched and of shape `(sequence_length,)`. generator (`torch.Generator`, *optional*): A [torch generator](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. return_dict: Whether to return a [`~utils.ModelOutput`] subclass instead of a plain tuple. Example: ```python >>> from transformers import UnivNetFeatureExtractor, UnivNetModel >>> from datasets import load_dataset, Audio >>> model = UnivNetModel.from_pretrained("dg845/univnet-dev") >>> feature_extractor = UnivNetFeatureExtractor.from_pretrained("dg845/univnet-dev") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> # Resample the audio to the feature extractor's sampling rate. >>> ds = ds.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> inputs = feature_extractor( ... ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt" ... ) >>> audio = model(**inputs).waveforms >>> list(audio.shape) [1, 140288] ``` Nrr)rdtypedevicer z&The batch size of `noise_sequence` is z+ and the batch size of `input_features` is z', but the two are expected to be equal.z$The batch size of `padding_mask` is r)dim)rr)rrrrrfr%rrandnrrrepeatrrrrarr6r7r+rrsqueezesumr)r1rrrrrkwargsspectrogram_batchedspectrogram_batch_sizespectrogram_lengthrPnoise_sequence_batchednoise_sequence_shapenoise_sequence_batch_sizepadding_mask_batch_sizer3rkwaveformroutputss rr9zUnivNetModel.forwardsb&1% ") =!/!9!9!! > >h9Rhh(>hhh   #!!Q&&+55a88 &2&8&; #&*@@@ l;Rll,Blll '00A66 '11!Q77 m44  D DH$H]NCCMM 00@UVV }55  =11 !((++   #$y1===  !12GN!-    rctjj}ttjjdrtjjj}||j|jD]}|||jdSr;) rr=r<r>r?rrarArrqs rrAzUnivNetModel.apply_weight_normQsh* 28,m < < @(3?K DM"""^ & &E  # # % % % % DN#####rctj|j|jD]}|tj|jdSr5)rr=rCrrarrts rrCzUnivNetModel.remove_weight_norm[s^ ##DM222^ ' 'E  $ $ & & & & ##DN33333r)NNNN)rrrr rmain_input_nameinput_modalitiesr$rrr Generatorbooltuplerr9rArCrErFs@rrrs! &O%}%%%%%%N4815,0#' y y )y )D0y '$. y ?T) y D[ y  u !$6 6y y y ^y v$$$4444444rr)r dataclassesrrrmodeling_outputsrmodeling_utilsrr=rr configuration_univnetr get_loggerrloggerrModulerrHrxrr__all__rrrrs"!!!!!!! ++++++------,,,,,,,,000000  H % %   6 6 6 6 6 6 6   6.0.0.0.0.0").0.0.0bu4u4u4u4u4RYu4u4u4p{/{/{/{/{/bi{/{/{/|M'M'M'M'M'biM'M'M'`v4v4v4v4v4?v4v4v4r  r