tjdZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z mZd d lmZejeZe eGd d e Ze eGd de Ze eGdde ZGddejZGddejZGddejZGddejZGddejZGddejZGddejZGdd ejZGd!d"ejZ e Gd#d$e Z!e d%&Gd'd(e!Z"d(d$gZ#dS))zPyTorch EnCodec model.N) dataclass)nn)initialization)PreTrainedAudioTokenizerBase) ModelOutputauto_docstringlogging) EncodecConfigcPeZdZUdZdZejdzed<dZej dzed<dS) EncodecOutputa\ audio_codes (`torch.LongTensor` of shape `(nb_frames, batch_size, nb_quantizers, frame_len)`, *optional*): Discrete code embeddings computed using `model.encode`. audio_values (`torch.FloatTensor` of shape `(batch_size, segment_length)`, *optional*): Decoded audio values, obtained using the decoder part of Encodec. N audio_codes audio_values) __name__ __module__ __qualname____doc__rtorch LongTensor__annotations__r FloatTensorj/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/encodec/modeling_encodec.pyrr&sN,0K!D(///-1L%#d*11111rrcdeZdZUdZdZejdzed<dZej dzed<dZ e dzed<dS)EncodecEncoderOutputa audio_codes (`torch.LongTensor` of shape `(nb_frames, batch_size, nb_quantizers, frame_len)`, *optional*): Discrete code embeddings computed using `model.encode`. audio_scales (list of length `nb_frames` of `torch.Tensor` of shape `(batch_size, 1)`, *optional*): Scaling factor for each `audio_codes` input. This is used to unscale each chunk of audio when decoding. last_frame_pad_length (`int`, *optional*): The length of the padding in the last frame, if any. This is used to ensure that the encoded frames can be outputted as a tensor. This value should be passed during decoding to ensure padding is removed from the encoded frames. Nr audio_scaleslast_frame_pad_length) rrrrrrrrrrrintrrrrr4sc  ,0K!D(///-1L%#d*111(,3:,,,,,rrc2eZdZUdZdZejdzed<dS)EncodecDecoderOutputz audio_values (`torch.FloatTensor` of shape `(batch_size, segment_length)`, *optional*): Decoded audio values, obtained using the decoder part of Encodec. Nr)rrrrrrrrrrrr"r"Gs6 .2L%#d*11111rr"c eZdZdZ ddededededef fd Zd ejd ejfd Ze dd ejde eefde de fdZ dZxZS) EncodecConv1dz;Conv1d with asymmetric or causal padding and normalization.r in_channels out_channels kernel_sizestridedilationc t|j|_|j|_|j|_|jdvrt d|j|dkr*|dkr$td|d|d|dtj ||||||_ tj j }ttj jd rtj jj }|jd kr||j |_ n%|jd krtjd||_|j jd }t'j|j jd t&j }|j jd }t'j|dz |zdzt&j }|d |d|d|d|d||z ddS)N weight_normtime_group_normIself.norm_type must be one of `"weight_norm"`, `"time_group_norm"`), got r zQEncodecConv1d has been initialized with stride > 1 and dilation > 1 (kernel_size=z stride=z , dilation=z).r)r,r-rdtyper(F) persistentr' padding_total)super__init__use_causal_convcausalpad_mode norm_type ValueErrorloggerwarningrConv1dconvutilsr,hasattrparametrizations GroupNormnormr'rtensorr(int64r)register_buffer) selfconfigr%r&r'r(r)r, __class__s rr5zEncodecConv1d.__init__Us ,  ) >!C C Cl\`\jll  A::(Q,, NNV!,VV6<VVIQVVV    Ik<fW_``` h* 28,m < < @(3?K >] * *# DI..DII ^0 0 0 Q 55DIi+A. di.q1EEE9%a(lK!Ox#?!#C5;WWW  Xv%@@@ ]KEJJJ _kF.BuUUUUUr hidden_statesreturnc|jd}||jz |jz|jz dz}t j|tjdz }||jz|jz|jz }||z S)zSee `pad_for_conv1d`.r )shaper'r3r(rceiltorE)rGrJlengthn_frames ideal_lengths r_get_extra_padding_for_conv1dz+EncodecConv1d._get_extra_padding_for_conv1d~s~ $R(T--0BBdkQTUU:h''**5;77!;$+-0@@4CUU f$$rzeropaddingsmodevaluecv|jd}|\}}|dkr"tj||||St ||}d}||kr*||z dz}tj|d|f}tj||||} | jd|z } | dd| fS)zTiny wrapper around torch.nn.functional.pad, just to allow for reflect padding on small input. If this is the case, we insert extra 0 padding to the right before the reflection happens. rMreflectrr .N)rNr functionalpadmax) rJrWrXrYrQ padding_left padding_rightmax_pad extra_padpaddedends r_pad1dzEncodecConv1d._pad1ds $R(&.# m 9  =$$]HdEJJ JlM22 W  &(1,IM--ma^LLM""=(D%HHl2*c4C4i  rc^||}|jr%|||j|f|j}n6|jdz}|j|z }|||||zf|j}||}|jdkr||}|S)N)rXr-)rTr7rer3r8r>r9rC)rGrJ extra_paddingr`r_s rforwardzEncodecConv1d.forwards::=II ;  KK 8JM7ZaeanKooMM!.!3M- =L KK mm.KLSWS`(M -00 >. . . IIm44Mr)r r )rUrV)rrrrr r5rTensorrT staticmethodtuplestrfloatreri __classcell__rIs@rr$r$Rs EEop'V'V#&'V69'VHK'VUX'Vhk'V'V'V'V'V'VR %| %  % % % %!!el!eCHo!S!bg!!!\!$rr$c <eZdZdZd dedededeffd ZdZxZS) EncodecConvTranspose1dzDConvTranspose1d with asymmetric or causal padding and normalization.r r%r&r'r(cft|j|_|j|_|j|_|jdvrt d|jtj|||||_ tj j }ttj j drtj j j }|jdkr||j |_ n%|jdkrtjd||_|js|jdkst ddSdS)Nr+r.r,r-r g?zB`trim_right_ratio` != 1.0 only makes sense for causal convolutions)r4r5r6r7trim_right_ratior9r:rConvTranspose1dr>r?r,r@rArBrC)rGrHr%r&r'r(r,rIs rr5zEncodecConvTranspose1d.__init__s. , & 7) >!C C Cl\`\jll &{L+vVV h* 28,m < < @(3?K  } , ,# DI..DII  !2 2 2 Q 55DI  ct4;;abb b c c;;rcV|jjd}|jjd}||z }||}|jdkr||}|jrt j||jz}n|dz}||z }|j d|z }|d||f}|S)Nrr-rgrM.) r>r'r(r9rCr7mathrOrtrN)rGrJr'r(r3r`r_rds rrizEncodecConvTranspose1d.forwardsi+A. !!$#f,  -00 >. . . IIm44M ; /!Imd6K&KLLMM*Q.M$}4 !"% 5%c<+;&;< r)r )rrrrr r5rirorps@rrrrrsNNccCcscQTc^acccccc0rrrc2eZdZdZdedeffd ZdZxZS) EncodecLSTMzz LSTM without worrying about the hidden state, nor the layout of the data. Expects input as convolutional layout. rH dimensioncttj|||j|_dSN)r4r5rLSTMnum_lstm_layerslstm)rGrHrzrIs rr5zEncodecLSTM.__init__s5 GIy&2HII rc|ddd}||d|z}|ddd}|S)Nrgrr )permuter)rGrJs rrizEncodecLSTM.forwardsQ%--aA66  -003mC %--aA66 r) rrrrr r r5rirorps@rryryslJ}JJJJJJJrrycBeZdZdZdededeeffd ZdZxZ S)EncodecResnetBlockz> Residual block from SEANet model as used by EnCodec. rHdim dilationsc ft|jdf}t|t|krt d||jz}g}t t||D][\}\}} |dkr|n|} |t|dz kr|n|} |tj gz }|t|| | || gz }\tj ||_ |j rt|||d|_dStj|_dS)Nr z7Number of kernel sizes should match number of dilationsrr/)r')r4r5residual_kernel_sizelenr:compress enumerateziprELUr$ ModuleListblockuse_conv_shortcutshortcutIdentity) rGrHrr kernel_sizeshiddenrir'r)in_chsout_chsrIs rr5zEncodecResnetBlock.__init__s; 3Q7 |  I . .VWW W'*3C i4P4P*Q*Q ^ ^ &A& XFFSSF#l"3"3a"777ccVG bfhhZ E mFFG[S[\\\] ]EE]5))  # *)&#sJJJDMMMKMMDMMMrc`|}|jD] }||}|||zSr|)rr)rGrJresiduallayers rrizEncodecResnetBlock.forwards@ Z 1 1E!E-00MM}}X&&66r) rrrrr r listr5rirorps@rrrso*}*3*49******(7777777rrc.eZdZdZdeffd ZdZxZS)EncodecEncoderz"SEANet encoder as used by EnCodec.rHc tt||j|j|jg}d}t |jD]|}||jz}t|j D]!}|t|||j |zdggz }"|tj gz }|t|||dz|dz|gz }|dz}}|t|||jzgz }|tj gz }|t|||jz|j|jgz }tj||_dS)Nr rgr'r()r4r5r$audio_channels num_filtersr'reversedupsampling_ratiosrangenum_residual_layersrdilation_growth_raterrry hidden_sizelast_kernel_sizerlayers)rGrHmodelscalingratio current_scalejrIs rr5zEncodecEncoder.__init__ sh vv'PRXRdeeff677  E#f&88M6566 j j,V]VE`bcEcefDghhii bfhhZ E mFM=1;LZ_bcZclqrrrs sE qLGG +fg0B&BCCDD "&(( -&2D(DfFXZ`ZqrrssmE** rc0|jD] }||}|Sr|rrGrJrs rrizEncodecEncoder.forward6*[ 1 1E!E-00MMrrrrrr r5rirorps@rrrsY,,+}++++++,rrc.eZdZdZdeffd ZdZxZS)EncodecDecoderz"SEANet decoder as used by EnCodec.rHc ttdt|jz}t ||j||jz|jg}|t|||jzgz }|jD]}||jz}|tj gz }|t|||dz|dz|gz }t|jD]$}|t||dz|j|zdfgz }%|dz}|tj gz }|t ||j|j|jgz }tj||_dS)Nrgrr )r4r5r rrr$rrr'ryrrrrrrrrrrrr)rGrHrrrrrrIs rr5zEncodecDecoder.__init__?s a3v788899vv'97VEW;WY_Ykllm +fg0B&BCCDD-  E#f&88M bfhhZ E &v}mq>P^cfg^gpuvvv E6566 o o,V]a5G&JeghJhjkIlmmnn MGG "&(( -(:F> MM-00 "INE#2#J/ rcPtj||j}|Sr|)rr\ embeddingrrGrrs rdecodezEncodecEuclideanCodebook.decode}s =**9djAAr) rrrrr r5rrrrorps@rrr^sw++ 9} 9 9 9 9 9 9rrc4eZdZdZdeffd ZdZdZxZS)EncodecVectorQuantizationzY Vector quantization implementation. Currently supports only euclidean distance. rHcptt||_dSr|)r4r5rcodebookrGrHrIs rr5z"EncodecVectorQuantization.__init__s, 088 rch|ddd}|j|}|SNrrgr )rrr)rGrJembed_ins rrz EncodecVectorQuantization.encodes3%--aA66 ='' 66rch|j|}|ddd}|Sr)rrrrs rrz EncodecVectorQuantization.decodes3='' 22##Aq!,,r) rrrrr r5rrrorps@rrrsl9}999999 rrceZdZdZdeffd Zd dedzdefdZd de j dedzde j fd Z d e j de j fd Z xZ S) EncodecResidualVectorQuantizerzResidual Vector Quantizer.rHctj|_j|_j|_t jfdtjD|_dS)Nc.g|]}tSr)r).0_rHs r z;EncodecResidualVectorQuantizer.__init__..s"$m$m$m1%>v%F%F$m$m$mr) r4r5r frame_ratenum_quantizersrrrrrs `rr5z'EncodecResidualVectorQuantizer.__init__sq #1 +$3m$m$m$m$mPUV\VkPlPl$m$m$mnn rN bandwidthrKc tj|j|jz}|j}|;|dkr5t t dtj|dz|z }|S)z:Return num_quantizers based on specified target bandwidth.NrVr i)rwlog2rrrr r^floor)rGrbw_per_qrs r get_num_quantizers_for_bandwidthz?EncodecResidualVectorQuantizer.get_num_quantizers_for_bandwidthsb9T/004?B,  Y__ Q 9t3Ch3N(O(O!P!PQQNr embeddingsc ||}|}g}|jd|D]F}||}||}||z }||Gt j|} | S)z Encode a given input tensor with the specified frame rate at the given bandwidth. The RVQ encode method sets the appropriate number of quantizers to use and returns indices for each quantizer. N)rrrrappendrstack) rGrrrr all_indicesrr quantized out_indicess rrz%EncodecResidualVectorQuantizer.encodes >>yII [.1 ( (Ell8,,G W--I)+H   w ' ' ' 'k+.. rcodesctjd|j}t|D],\}}|j|}||}||z}-|S)z7Decode the given codes to the quantized representation.rV)device)rrDrrrr)rGr quantized_outrrrrs rrz%EncodecResidualVectorQuantizer.decodesa S>>> #E** 6 6JAwKNE W--I)I5MMrr|)rrrrr r5rnr rrrjrrrorps@rrrs$$o}oooooo%$,RU%$,RWR^ ELU\rrc\eZdZUeed<dZdZejfdZ xZ S)EncodecPreTrainedModelrHencodec input_valuescFt|t|tjrpt j|j|jNtj |j |j |j dzz }t j|j| |dSdSt|tr|jj d}t#j|jjdt"j}|jjd}t#j|dz |zdzt"j}t j|j|t j|j |t j|j||z dSt|t0rzt j|jt#jdgt j|jt j|jt j|jdSdS)Nr)abr0r T)r4 _init_weights isinstancerr=initkaiming_normal_weightbiasrwsqrtgroupsr%r'uniform_r$r>rrDr(rEr)copy_r3rrrjzeros_rrr)rGmodulekr'r(r)rIs rrz$EncodecPreTrainedModel._init_weightss f%%% fbi ( ( *   / / /{&Ifmv/AFDVWXDY/YZ[[ fkaR1555555'& . . * +1!4K\&+"4Q"7u{KKKF{+A.H, a8'Ca'Gu{[[[K Jv}f - - - Jv); 7 7 7 Jv+[6-A B B B B B  8 9 9 * Jv}elD6&:&: ; ; ; K+ , , , K % % % K( ) ) ) ) )  * *r) rrrr rbase_model_prefixmain_input_namerno_gradrrorps@rrrsc !$OU]__****_*****rrz/ The EnCodec neural audio codec model. ) custom_introceZdZdeffd ZdejdedeejejdzffdZ ddejdejdzdedzd e dzdeejejdze fe zf d Z ed eejd e fd ZddejdejdzdejfdZ ddejdejdejdzd e dzde dzdeejejfezf dZe ddejdejdzdedzdejdzdejdzd e dzde dzdeejejfezfdZxZS) EncodecModelrHct|||_t||_t ||_t||_ttj |jj |_ d|j z|jj krtd|dS)Nrgz'The codebook_size must be a power of 2.)r4r5rHrencoderrdecoderr quantizerr rwrrbits_per_codebookr: post_initrs rr5zEncodecModel.__init__s     %f-- %f-- 7??!$TYt{/H%I%I!J!J d$ $ (A A AFGG G rrrrKNcd|jd}||jjz }|jj2|d|jjzkrt d|d|jjd}|jjrt j|dd|jdz }|d  dd  d z}||z }| dd}| |}|j ||}|d d}||fS) z Encodes the given input using the underlying VQVAE. If `config.normalize` is set to `True` the input is first normalized. The padding mask is required to compute the correct scale. rMNgh㈵>zDuration of frame (z) is longer than chunk r Trrg)rrg:0yE>r)rNrH sampling_ratechunk_length_s RuntimeError normalizerrrmeanrrrrr transpose) rGrrrQdurationscalemonorrs r _encode_framezEncodecModel._encode_frames0 #B'DK55 ; % 1h HbAb6b6brXrrVZVaVprrss s ;  &9\1d;;;l>PQR>SSDHHQKK$$T$::??AADHE'%/LJJr1%%E\\,// %%j)<<1%%e|r padding_mask return_dictc||n |jj}||jjd}||jjvr td|d|jjd|j\}}}|dks|dkrtd||jj}||}|} n |jj} |'tj| }n-| |jdd |jd }g} g} td|| D]x} |d | | |zf } | |d | | |zfz}| ||\}}| || |y| djd | d jd z }|dkr/tj| d d|fd }|| d <tj| } |s| | |fSt%| | |S) a Encodes the input audio waveform into discrete codes of shape `(nb_frames, batch_size, nb_quantizers, frame_len)`. - `nb_frames=1` if `self.config.chunk_length=None` (as the encoder is applied on the full audio), which is the case for the 24kHz model. Otherwise, `nb_frames=ceil(input_length/self.config.chunk_stride)`, which is the case for the 48kHz model. - `frame_len` is the length of each frame, which is equal to `ceil(input_length/self.config.hop_length)` if `self.config.chunk_length=None` (e.g., for the 24kHz model). Otherwise, if `self.config.chunk_length` is defined, `frame_len=self.config.chunk_length/self.config.hop_length`, e.g., the case for the 48kHz model with `frame_len=150`. Args: input_values (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Float values of the input audio waveform. padding_mask (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Padding mask used to pad the `input_values`. bandwidth (`float`, *optional*): The target bandwidth. Must be one of `config.target_bandwidths`. If `None`, uses the smallest possible bandwidth. bandwidth is represented as a thousandth of what it is, e.g. 6kbps bandwidth is represented as bandwidth == 6.0 Returns: EncodecEncoderOutput dict or a tuple containing: - audio_codes (`torch.LongTensor` of shape `(nb_frames, batch_size, nb_quantizers, frame_len)`, *optional*), - audio_scales (list of length `nb_frames` of `torch.Tensor` of shape `(batch_size, 1)`, *optional*), - last_frame_pad_length (`int`, *optional*). Nrz)This model doesn't support the bandwidth z. Select one of .r rgz1Number of audio channels must be 1 or 2, but got rM.)rY)rHr$target_bandwidthsr:rN chunk_length chunk_strider ones_likeboolrrr"rrr\r]rr)rGrr#rr$rchannels input_lengthr(r(encoded_framesscalesoffsetmaskframe encoded_framer r last_frames rrzEncodecModel.encode s`F&1%P>VWY>Z Z 1 $ $**>"+=CX?Yab*ccJ!+N2 ^44 C"F,AB B#NF!Q q(8uUUUVWXZVZ[3++---[F%HHH kI5I*IIV5III  E ;r?L Vf|333 4 4 4} }8MPU8U U 4 4 4 v 55 6 6 6&,:O O 6 6 6 f FF >>  q dWadddee eZrrr c|dd}|j|}||}|||dddz}|S)Nrr rM)rrrrr)rGrr routputss r _decode_framezEncodecModel._decode_frames`1%%^**511 ,,z**   2q! 4 44Grrrrrc||n |jj}|jj}|jt|dkrt dt||d}|dkr |dd| f}|||d}ng} t t||D]\\} \}} | t|dz kr|dkr |dd| f}||| } | | ]| | |jj pd}|3|j d|j dkr|dd|j df}|s|fSt|S)a Decodes the given frames into an output audio waveform. Note that the output might be a bit bigger than the input. In that case, any extra steps at the end can be trimmed. Args: audio_codes (`torch.LongTensor` of shape `(nb_frames, batch_size, nb_quantizers, frame_len)`, *optional*): Discrete code embeddings computed using `model.encode`. audio_scales (list of length `nb_frames` of `torch.Tensor` of shape `(batch_size, 1)`, *optional*): Scaling factor for each `audio_codes` input. padding_mask (`torch.Tensor` of shape `(channels, sequence_length)`): Padding mask used to pad the `input_values`. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. last_frame_pad_length (`int`, *optional*): Integer representing the length of the padding in the last frame, which is removed during decoding. Nr zExpected one frame, got r.rM) rHr$r(rr:rCrrrr@r)rNr") rGrrr#r$rr(r2rdecoded_framesrr r5s rrzEncodecModel.decodes6&1%E5K((1,,,1F1J1J!#'>)>(>'>">?E++E599%%f----33NDKD\Da`abbL  # (:2(>ASTVAW(W(W'-E|/A"/E-E(EFL # ? "#L111rc||n |jj}|'tj|}n-||jdd|jd}||td||td||||||d\}}}| |||||d}|s||fSt||S) a input_values (`torch.FloatTensor` of shape `(batch_size, channels, sequence_length)`, *optional*): Raw audio input converted to Float and padded to the appropriate length in order to be encoded using chunks of length self.chunk_length and a stride of `config.chunk_stride`. padding_mask (`torch.BoolTensor` of shape `(batch_size, channels, sequence_length)`, *optional*): Mask to avoid computing scaling factors on padding token indices (can we avoid computing conv on these+). Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. `padding_mask` should always be passed, unless the input was truncated or not padded. This is because in order to process tensors effectively, the input audio should be padded so that `input_length % stride = step` with `step = chunk_length-stride`. This ensures that all chunks are of the same shape bandwidth (`float`, *optional*): The target bandwidth. Must be one of `config.target_bandwidths`. If `None`, uses the smallest possible bandwidth. bandwidth is represented as a thousandth of what it is, e.g. 6kbps bandwidth is represented as `bandwidth == 6.0` audio_codes (`torch.LongTensor` of shape `(nb_frames, batch_size, nb_quantizers, frame_len)`, *optional*): Discrete code embeddings computed using `model.encode`. audio_scales (list of length `nb_frames` of `torch.Tensor` of shape `(batch_size, 1)`, *optional*): Scaling factor for each `audio_codes` input. return_dict (`bool`, *optional*): Whether to return outputs as a dict. last_frame_pad_length (`int`, *optional*): The length of the padding in the last frame, if any. This is used to ensure that the encoded frames can be outputted as a tensor. This value should be passed during decoding to ensure padding is removed from the encoded frames. Examples: ```python >>> from datasets import load_dataset >>> from transformers import AutoProcessor, EncodecModel >>> dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example") >>> audio_sample = dataset["train"]["audio"][0]["array"] >>> model_id = "facebook/encodec_24khz" >>> model = EncodecModel.from_pretrained(model_id) >>> processor = AutoProcessor.from_pretrained(model_id) >>> inputs = processor(raw_audio=audio_sample, return_tensors="pt") >>> outputs = model(**inputs) >>> audio_codes = outputs.audio_codes >>> audio_values = outputs.audio_values ```NrrMzBYou specified `audio_codes` but did not specify the `audio_scales`zBYou specified `audio_scales` but did not specify the `audio_codes`F)r$r)rr) rHr$rr*r+rrNr:rrr) rGrr#rrrr$rrs rrizEncodecModel.forwards%~&1%rOs !!!!!! &&&&&&:::::: 100000  H % %   2 2 2 2 2K 2 2  2 -----;-- -" 22222;22 2^^^^^BI^^^B66666RY666r     ")    77777777BRY>RYD!!!!!ry!!!H ()))))RY)))X*****9***: CQCQCQCQCQ)CQCQ  CQL 3 4r