tjfddlZddlmZddlmZddlZddlZddlm Z ddlm Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+e)j,e-Z.e(deGddeZ/Gdde j0Z1Gdde j0Z2GddeZ3GddeZ4Gdd eZ5Gd!d"e j0Z6Gd#d$e j0Z7 dTd&e j0d'ej8d(ej8d)ej8d*ej8dzd+e9dzd,e9d-e%e'fd.Z:Gd/d0e j0Z;Gd1d2e j0Z<Gd3d4eZ=Gd5d6e j0Z>Gd7d8e j0Z?Gd9d:eZ@Gd;de j0ZBe(Gd?d@e"ZC dUdAeDeEeEfdBe9dCeEd*ejFdzdDeEdEejGf dFZHeZIe(GdGdHeCZJe(dIGdJdKeCZKdLZLe(dMGdNdOeCZMe(dPGdQdReCZNgdSZOdS)VN)Callable) dataclass)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutput ModelOutputSequenceClassifierOutputWav2Vec2BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack)TransformersKwargsauto_docstringlogging)UniSpeechConfigzh Output type of [`UniSpeechForPreTrainingOutput`], with potential hidden states and attentions. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dS) UniSpeechForPreTrainingOutputa loss (*optional*, returned when model is in train mode, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentions)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__r r!r"r#tupler$n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/unispeech/modeling_unispeech.pyrr5s  &*D% d ")))15e'$.555;? 1D 8???6:5,t3:::59M5*+d299926Je'(4/66666r.rc$eZdZfdZdZxZS)UniSpeechSamePadLayerclt|dzdkrdnd|_dS)Nrr)super__init__num_pad_remove)selfnum_conv_pos_embeddings __class__s r/r5zUniSpeechSamePadLayer.__init__Ss: #:Q#>!#C#Caar.cJ|jdkr|ddddd|j f}|SNr)r6r7r#s r/forwardzUniSpeechSamePadLayer.forwardWs;   " ")!!!QQQ0F43F2F0F*FGMr.r%r&r'r5r= __classcell__r9s@r/r1r1RsLKKKKKr.r1c$eZdZfdZdZxZS) UniSpeechPositionalConvEmbeddingcttj|j|j|j|jdz|j|_tjj }ttjj drtjj j }trddl }|j|jjd5||jdd|_dddn #1swxYwYt|jdr-|jj jj}|jj jj}n|jj}|jj}|j|||j||n||jdd|_t-|j|_t0|j|_dS) Nr3) kernel_sizepaddinggroups weight_normr) modifier_rankweight)namedimparametrizations)r4r5nnConv1d hidden_sizer8num_conv_pos_embedding_groupsconvutilsrGhasattrrLr deepspeedzeroGatheredParametersrI original0 original1weight_gweight_vregister_external_parameterr1rErfeat_extract_activation activation)r7configrGrTrYrZr9s r/r5z)UniSpeechPositionalConvEmbedding.__init__^s I    62a77     h* 28,m < < @(3?K % ' ' E    22493CST2UU I I'K aHHH  I I I I I I I I I I I I I I Ity"455 .95<F95<F9-9- N 6 6tX F F F N 6 6tX F F F F# DIH!DDDDI,V-KLL  !?@sC??DDc|dd}||}||}||}|dd}|S)Nrr3) transposerQrEr]r<s r/r=z(UniSpeechPositionalConvEmbedding.forwardse%//155  -00  ]33  66 %//155 r.r>r@s@r/rBrB]sMAAAAABr.rBc&eZdZdfd ZdZxZS)UniSpeechNoLayerNormConvLayerrcZt|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t|j |_ dS)NrrrDstridebias)r4r5conv_dim in_conv_dim out_conv_dimrMrN conv_kernel conv_stride conv_biasrQrr\r]r7r^layer_idr9s r/r5z&UniSpeechNoLayerNormConvLayer.__init__s r@s@r/rbrbsR A A A A A Ar.rbc&eZdZdfd ZdZxZS)UniSpeechLayerNormConvLayerrct|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t j |jd|_ t|j|_dS)NrrrdT)elementwise_affine)r4r5rgrhrirMrNrjrkrlrQ LayerNorm layer_normrr\r]rms r/r5z$UniSpeechLayerNormConvLayer.__init__s r@s@r/rsrssR A A A A A Ar.rsc&eZdZdfd ZdZxZS)UniSpeechGroupNormConvLayerrct|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t|j |_ t j|j|jd|_dS)NrrrdT) num_groups num_channelsaffine)r4r5rgrhrirMrNrjrkrlrQrr\r] GroupNormrwrms r/r5z$UniSpeechGroupNormConvLayer.__init__s r@s@r/r|r|sRrrrrrr r.r|c.eZdZdZfdZdZdZxZS)UniSpeechFeatureEncoderz.Construct the features from raw audio waveformctjdkr7tdgfdt jdz Dz}nDjdkr!fdt jD}nt djd tj||_ d |_ d |_ dS) Ngrouprrnc8g|]}t|dzS)rr)rb.0ir^s r/ z4UniSpeechFeatureEncoder.__init__..s@OOO.fq1uEEEOOOr.rlayerc2g|]}t|S)r)rsrs r/rz4UniSpeechFeatureEncoder.__init__..s4DE+FQ???r.z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) r4r5feat_extract_normr|rangenum_feat_extract_layers ValueErrorrM ModuleList conv_layersgradient_checkpointing_requires_grad)r7r^rr9s ` r/r5z UniSpeechFeatureEncoder.__init__s   #w . .6vJJJKOOOOv=ABBOOOKK % 0 0INvOmInInKKt1Ittt =55&+#"r.cP|D] }d|_ d|_dSNF) parameters requires_gradrr7params r/_freeze_parametersz*UniSpeechFeatureEncoder._freeze_parameterss4__&& ( (E"'E  #r.cr|dddf}|jr|jrd|_|jD] }||}|S)NT)rtrainingrr)r7 input_valuesr# conv_layers r/r=zUniSpeechFeatureEncoder.forwards[$QQQW-    /4= /*.M '* 6 6J&J}55MMr.)r%r&r'r(r5rr=r?r@s@r/rrs\88#####($$$       r.rc$eZdZfdZdZxZS)UniSpeechFeatureProjectionc.ttj|jd|j|_tj|jd|j|_ tj |j |_ dS)Nrzeps) r4r5rMrvrglayer_norm_epsrwLinearrO projectionDropoutfeat_proj_dropoutdropoutr7r^r9s r/r5z#UniSpeechFeatureProjection.__init__sn ,vr':@UVVV)FOB$79KLLz&":;; r.c||}||}||}||fSrp)rwrr)r7r#norm_hidden_statess r/r=z"UniSpeechFeatureProjection.forwardsC!__];;(:;;  ]33 000r.r>r@s@r/rrsG<<<<< 1111111r.rmodulequerykeyvalueattention_maskscalingrkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nrzr3rrK)prr) sizer)matmulr`rM functionalsoftmaxrr contiguous) rrrrrrrr attn_weights attn_outputs r/eager_attention_forwardr s**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r.ceZdZdZ ddededed ed ed ed edzffd Z dde j de j dzde j dzdedzde e de e j e j dze e j dzff dZxZS)UniSpeechAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderrf is_causalr^c t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)rf)r4r5rrrhead_dimr^rrrrrMrk_projv_projq_projout_proj) r7rrrrrfrr^r9s r/r5zUniSpeechAttention.__init__)s "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r.r#key_value_statesroutput_attentionsrreturnc |du}|jdd}g|d|jR}|||dd} |r|n|} g| jddd|jR} || | dd} || | dd} tj|j j t}||| | | |f|j sdn|j |j|d|\}}|jg|dR}||}||dfS)z#Input shape: Batch x Time x ChannelNrzrr3r)rrr)shaperrviewr`rrr get_interfacer^_attn_implementationrrrrreshaperr)r7r#rrrris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs r/r=zUniSpeechAttention.forwardHs.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT -?R))]B^)#2#.BBDMBB[[0055h??II!QOO {{>2277AAKKAqQQ (?(M K ,.E) ) %8$7      % $}>CC$,L/ % %  % % ! \*k);;;;;;FFHH mmK00 L$..r.)rFTFN)NNF)r%r&r'r(intfloatboolrr5r)Tensorrr r,r=r?r@s@r/rr&sJGG  )-CCCC C  C  CC $&CCCCCCD15.2). 0/0/|0/ ,-0/ t+ 0/  $; 0/-.0/ u|U\D0% 2E2LL M0/0/0/0/0/0/0/0/r.rc$eZdZfdZdZxZS)UniSpeechFeedForwardcttj|j|_tj|j|j|_ t|j trt|j |_n |j |_tj|j|j|_tj|j|_dSrp)r4r5rMractivation_dropoutintermediate_dropoutrrOintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r/r5zUniSpeechFeedForward.__init__|s $&Jv/H$I$I!"$)F,>@X"Y"Y f' - - 9'-f.?'@D $ $'-'8D $If&>@RSS j)>??r.c||}||}||}||}||}|Srp)rrrrrr<s r/r=zUniSpeechFeedForward.forwardsg// >> 00?? 11-@@ ))-88 ++M:: r.r>r@s@r/rr{sL @ @ @ @ @r.rc&eZdZfdZddZxZS)UniSpeechEncoderLayerctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_dS)NFrrrrr^r)r4r5rrOnum_attention_headsattention_dropout attentionrMrrrrvrrwr feed_forwardfinal_layer_normrs r/r5zUniSpeechEncoderLayer.__init__s +(0,    z&"788 ,v'9v?TUUU088 " V-?VEZ [ [ [r.NFc|}||||\}}}||}||z}||}|||z}||}|f}|r||fz }|SNrr)rrrwrrr7r#rr attn_residualr_outputss r/r=zUniSpeechEncoderLayer.forwards% )- .L]*8* * & |Q ]33 % 5  66 %(9(9-(H(HH --m<<  "  '   &Gr.rr>r@s@r/rrsQ \ \ \ \ \r.rc ^eZdZfdZ d dejdejdzdeded ef d ZxZ S) UniSpeechEncoderct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_dS)Nrc.g|]}tSr-)rrrr^s r/rz-UniSpeechEncoder.__init__..s"$l$l$lq%:6%B%B$l$l$lr.Fr4r5r^rBpos_conv_embedrMrvrOrrwrrrrrnum_hidden_layerslayersrrs `r/r5zUniSpeechEncoder.__init__s  >vFF,v'9v?TUUUz&"788 m$l$l$l$lERXRjLkLk$l$l$lmm &+###r.NFTr#rroutput_hidden_states return_dictc|rdnd}|rdnd}|;|ddd|jd}d||<t|j||}||} || |jz}||}| |}tpt|} |j D]a} |r||fz}tjg} |jo| |jjk} | r| r| |||}|d}| rd}|r ||dfz}b|r||fz}|st#d |||fDSt%||| S) Nr-rzrr3rr^ inputs_embedsrrNNc3K|]}||V dSrpr-rvs r/ z+UniSpeechEncoder.forward..(mmq_`_l_l_l_l_lmmr.last_hidden_stater#r$) unsqueezerepeatrr r^r todevicerwrr r r r)randr layerdropr,rr7r#rrrrall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r/r=zUniSpeechEncoder.forwards #7@BBD$5?bb4  %$2$<$(>}?S(T(TT  66  ]33 022R6LT6R6R [ P PE# I$58H$H!#(*R.. !]Z/BT[EZ/ZN! 1[ 1 %!.Te!!! !.a 0  - ,   P&9]1=MBB gii  $.$/BB r.r#c||}||}||}||}|Srp)r2r3r5r6r<s r/r=z!UniSpeechAttnAdapterLayer.forwardsL -00  m44  M22  m44 r.)r%r&r'r5r)r*r=r?r@s@r/r-r-s[ C C C C CU%6r.r-cReZdZfdZ ddejdejdzdefdZxZS) $UniSpeechEncoderLayerStableLayerNormctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_t#|ddt%||_dSd|_dS)NFrrr/)r4r5rrOrrrrMrrrrvrrwrrrgetattrr- adapter_layerrs r/r5z-UniSpeechEncoderLayerStableLayerNorm.__init__s +(0,    z&"788 ,v'9v?TUUU088 " V-?VEZ [ [ [ 6-t 4 4 @!:6!B!BD   !%D   r.NFr#rrcJ|}||}||||\}}}||}||z}||||z}|j|||z}|f}|r||fz }|Sr)rwrrrrr<rs r/r=z,UniSpeechEncoderLayerStableLayerNorm.forward,s &  66 )- .L]*8* * & |Q ]33 % 5 %(9(9$:O:OP]:^:^(_(__   ))D,>,>},M,MMM "  '   &Gr.r) r%r&r'r5r)rrr=r?r@s@r/r9r9s~&&&&&,/3"' | t+ r.r9c.eZdZfdZ ddZxZS)UniSpeechEncoderStableLayerNormct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_dS)Nrc.g|]}tSr-)r9r s r/rz.Ns" c c ca 1& 9 9 c c cr.Fr rs `r/r5z(UniSpeechEncoderStableLayerNorm.__init__Gs  >vFF,v'9v?TUUUz&"788 m c c c c5IaCbCb c c c   ',###r.NFTc|rdnd}|rdnd}|;|ddd|jd}d||<t|j||}||} || z}||}tpt|} |j D]a} |r||fz}tj g} |j o| |jj k} | r| r| |||}|d}| rd}|r ||dfz}b||}|r||fz}|std |||fDSt!||| S) Nr-rzrr3rrrrc3K|]}||V dSrpr-rs r/rz:UniSpeechEncoderStableLayerNorm.forward..rr.r)rrrr r^r rr r r r)rrr rwr,rr!s r/r=z'UniSpeechEncoderStableLayerNorm.forwardRs#7@BBD$5?bb4  %$2$<$r@s@r/r?r?FsZ , , , , ," = = = = = = = = r.r?c>eZdZdZfdZedZdZxZS)UniSpeechGumbelVectorQuantizerz Vector quantization using gumbel softmax. See `[CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX](https://huggingface.co/papers/1611.01144) for more information. ct|j|_|j|_|j|jzdkr td|jd|jdtj tj d|j|jz|j|jz|_ tj |jd|j|jz|_d|_dS)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenationrrzr3)r4r5num_codevector_groupsr~num_codevectors_per_groupnum_varscodevector_dimrrM Parameterr)r* codevectorsrrg weight_proj temperaturers r/r5z'UniSpeechGumbelVectorQuantizer.__init__s  68  4? 2a 7 7Y&*?YY59_YYY  <  a4=!@&BW[_[jBj k k  9V_R%8$/DM:YZZr.c|d}tjtjtj||d }|S)Nrrrz)meanr)expsumxlogy)probsmarginal_probs perplexitys r/_compute_perplexityz2UniSpeechGumbelVectorQuantizer._compute_perplexitysU**Y %+nn*U*U[] ^ ^ ^^__ccee r.c|j\}}}||}|||z|jzd}|jrt j||j d |}tj |||z|jdd}| |}n|d}|j|jd|ddd}|||z|jd}| |}|||zd}|d|jz} | ||z|j|jd} | d||d} | |fS)NrzT)tauhardrrg?ry)rrMrr~rrMrgumbel_softmaxrrNtype_asr)rrWargmax new_zerosscatter_rrLrIrR) r7r# batch_sizesequence_lengthrOcodevector_probscodevector_soft_distrVcodevector_idxcodevectors_per_grouprLs r/r=z&UniSpeechGumbelVectorQuantizer.forwards3@3F0 O[((77 %**:+G$/+Y[]^^ = D!};;##%%4+;$ <  gm$$  $)="":#?RTUU[[]]ce$$$ 112FGGJJ+11b199N6}6 8KLUUN''A..    044Z/5QSWSbdfgg 112BCCJ+00o1MrRR 0 : :2 > >AQ Q+00o1Mt`d`moqrr !oob))..z?BOO J&&r.) r%r&r'r(r5 staticmethodrWr=r?r@s@r/rErEsl (\ #'#'#'#'#'#'#'r.rEceZdZUeed<dZdZdZdZdZ dZ dZ e j fdZde jezfdZd ed e jfd ZxZS) UniSpeechPreTrainedModelr^ unispeechraudioTc Lt|t|trZt j|jjddt j|jj t j |j dSt|trwt j|j jddtjd|j jd|j jzz zt j|j j ddSt|t&rgtjd|jjz }t j |jj| |t j |jj | |dSt|t,jrnt j|j|j Ptj|j|j|jdzz }t j |j | |dSdSdS)zInitialize the weightsrr)rPstdrr3)abN)r4 _init_weightsrrEinitnormal_rMrIzeros_rfuniform_rLrBrQmathsqrtrD in_channels constant_rr in_featuresrMrNkaiming_normal_rF)r7rkr9s r/roz&UniSpeechPreTrainedModel._init_weightss f%%% f< = = 6 L+2! D D D D K*/ 0 0 0 M&, - - - - -  @ A A 6 L " !v{'>q'AFKD['["\]]]     N6;+Q / / / / /  : ; ; 6 !f/;;< > > > M&+0QB! < < < < < <  * * 6   / / /{&Ifmv/AFDVWXDY/YZ[[ fkaR1555555  6 6'&r. input_lengthsczd}t|jj|jjD]\}}||||}|S)zH Computes the output length of the convolutional layers c<tj||z |ddzS)Nfloor) rounding_moder)r)div) input_lengthrDres r/_conv_out_lengthzSUniSpeechPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths&9\K7wWWWZ[[ [r.)zipr^rjrk)r7r{rrDres r/ _get_feat_extract_output_lengthsz9UniSpeechPreTrainedModel._get_feat_extract_output_lengthss\  \ \ \ $'t{'> @W#X#X Q Q K,,]KPPMMr.feature_vector_lengthrc|ddddf}||tj}|jd}tj||f|j|j}d|tj |jd|j|dz f<| dgd dg }|S)Nrzrr)dtyperr)r) cumsumrrr)longrzerosrrarangeflipr)r7rrnon_padded_lengthsoutput_lengthsr`s r/"_get_feature_vector_attention_maskz;UniSpeechPreTrainedModel._get_feature_vector_attention_mask s,22r2::111b5A>>?QRRUUV[V`aa#)!,  . /~7KTbTi   uv ^%9!%<^EZ[[[]kno]opq',,bT2299"==BBB4HHMMOOr.)r%r&r'rr+base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr)no_gradro LongTensorrrrr?r@s@r/rhrhs #$O&*#NU]__6666_64 e>NQT>T      ]b]m        r.rhr mask_prob mask_length min_masksrc@|\}dkrtdkrtdddtjdfd}|9|dnfd t|D}tj |ft }g} |} | d kr|S|D]} || } tj tj | dz z | d } t| d krdz }n| d }tj| tj| | z tj |zg} | | tj| } tj| dddddf|| f} | || z} tj ddddf}tj||| f|| z}| |z} | dz krdz | | dz k<tj|| dd|S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rnum_masked_spanepsilonrrrras r/compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spanCs~i,6DwNOOoy99 [ (? : :-.Vs 9 9 9!o 9 9 9r.rrF)replace)rnprandomritemdetachrRtolistrrrchoicerlen concatenateonesint32appendarray broadcast_torrput_along_axis)rrrrrr`rr{ spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrrspec_aug_mask_idxdummy_mask_idxoffsetsrras `` ` @@r/_compute_mask_indicesrsP0#(JQABBB_$$ :^i : :'6 : : :   innQ$$&&G         $  % ##B''..000 9 9 9 9uZ'8'8 9 9 9Hj/:$GGGM11/BBa%55 11,??I,, IlkAo6 7 7RW-    ! !Q & &-q0NN.q1NN (;o(MUWU] ^ ^ ^ao o p   !!"34444"455111aaa:&5H+(V,33J@SVa@abbi $$T4]3Gog 4G'UVV^^'+5G,g5/A"555GVYZGZ-!0CCDm%7B??? r.ceZdZdeffd Z ddejdejdzdejdzfdZe ddej dzdej dzdejdzd e dzd e dzd e dzd e e zfd ZxZS)UniSpeechModelr^ct|||_t||_t ||_|jdks |jdkrBtj tj |j |_|jrt#||_nt'||_|dSNr)r4r5r^rfeature_extractorrfeature_projectionmask_time_probmask_feature_probrMrKr)rrOrsmasked_spec_embeddo_stable_layer_normr?encoderr post_initrs r/r5zUniSpeechModel.__init__s     !8!@!@"}G[chcm n n n /3/E/H/HI\/]/]M+ , ; (1 , , ,#8[)+7 K;+< $$$ $)<0D]Mainis#t#t#t #74#@#G#GO]_#`#` 23M. /r.rrrrrc ||n |jj}||n |jj}||n |jj}||}|dd}|!||jd|}||\} }| | ||} | | ||||} | d} |s| |f| ddzSt| || j | j S)a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nrr3)rrrrrrr)rextract_featuresr#r$)r^rrrrr`rrrrrUniSpeechBaseModelOutputr#r$) r7rrrrrrrrr#encoder_outputss r/r=zUniSpeechModel.forwardsW 2C1N--TXT_Tq$8$D $+Jj &1%~1  ,, )/!5# '  (*  K!#34qrr7JJ J'+-)7&1     r.rNNNNN)r%r&r'rr5r)r*rrrrrr,rr=r?r@s@r/rrs7(7;26 ,,(,!,t3,(4/ ,,,,\/36:)-,0#'3 3 lT)3  t+3 !,t3 3  $; 3 #Tk 3 D[3  ) )3 3 3 ^3 3 3 3 3 r.rzZ UniSpeech Model with a vector-quantization module and ctc loss for pre-training. ceZdZdeffd ZdefdZdZe dde j de j d e j defd Z e dd e j d zd e j d zded zded zded zdeezf dZxZS)UniSpeechForPreTrainingr^ct|t||_t j|j|_t||_ t j |j |j |_ t j |j |j|_t j |j|j|_t j|j|_|dSrp)r4r5rrirMrfeat_quantizer_dropoutdropout_featuresrE quantizerrrJproj_codevector_dim project_qrO project_hidnum_ctc_classesctc_proj final_dropoutrrrs r/r5z UniSpeechForPreTraining.__init__s    '// " 6+H I I7??6#8&:TUU9V%?ASTT &"4f6LMM z&"677  r.rNc||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rrN)r7rNs r/set_gumbel_temperaturez.UniSpeechForPreTraining.set_gumbel_temperature$s&1"""r.cB|jjdSz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. Nrirrr7s r/freeze_feature_encoderz.UniSpeechForPreTraining.freeze_feature_encoder*! (;;=====r.rtarget_featuresnegative_featurespredicted_featuresctj||gd}tj||d}||}||z }|S)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rrrz)r)catcosine_similarityrr\)rrrrNlogitss r/compute_contrastive_logitsz2UniSpeechForPreTraining.compute_contrastive_logits1sq )_6G$HaPPP();)A)A)C)C_EZEZE\E\bdeee00+% r.Nrrrrrrc N||n |jj}||||||}|d}||d} || \} } || |jjj} | | } tj | d| d |jj} | dd} tj| |j} | dd} | d} || d| | dz}||}||}d}|s#|||| | f|ddzS|| | f|ddzSt/||| | |j|jS) a Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, UniSpeechForPreTraining >>> feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> model = UniSpeechForPreTraining.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> # TODO: Add full pretraining example ```Nrrrrzrr3)rr r!r"r#r$)r^rrirrrrrIrrr)emptyrfill_ replace_probr` bernoullirrr masked_fillrrrr#r$)r7rrrrrrrtransformer_featuresrquantized_featuresr"prob_replace_matrixsampled_replace_matrixrrs r/r=zUniSpeechForPreTraining.forwardEsW,&1%,A,A$.BWB],^,^__!--.@AA#k*>*C*CA*F*FH\HaHabcHdHdeekk K $  2;;AqAA!&1D!E!E!J!J!L!L!O!OPdPk!l!l!7!A!A!Q!G!G!7!A!A"!E!E%112H#NN  * *,B+BC H H  f%%v&& c24FH]^ahijikikalll(*<>STW^_`_a_aWbb b,1'9"7!/)     r.)r)NNNN)r%r&r'rr5rrrrfr)r*rrrrr,rr=r?r@s@r/rrsa       1#1111 >>>  * ,"- \&/3)-,0#' E E lT)E  t+E  $; E #Tk E D[ E  . .E E E ^E E E E E r.rr3zq UniSpeech Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ceZdZddedzffd ZdZdZdZe dde j dzde j dzd e dzd e dzd e dzd e j dzd e e zfdZxZS)UniSpeechForCTCN target_langct|t||_t j|j|_||_|j td|j dt|dr|j r|jn|j}t j||j |_|dS)a3 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`UniSpeechForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `UniSpeechForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r4r5rrirMrrrr vocab_sizerr9rSroutput_hidden_sizerOrlm_headr)r7r^rrr9s r/r5zUniSpeechForCTC.__init__s    '//z&"677 &   $HHHH *1)G)G vFL^ vF % %djdv y!3V5FGG  r.c TttjdkrdS|j}|)t |jddt d|d|2t |jddtddS|| |ddSdS) a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNr/zCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load) rr)rrr;r^rloggerinfo load_adapter)r7rrs r/ tie_weightszUniSpeechForCTC.tie_weightss 6 7 75<;O;O O O F &  "wt{||jjkrt d|jj||||||}|d} || } || } d} |Z||ntj |tj }| | d tj } |dk} | d}|| }tj| dtjdd}tjjd 5tj||| ||jj|jj|jj } dddn #1swxYwY|s| f|t6dz}| | f|zn|St9| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrrrz)rKrrF)enabled)blank reduction zero_infinityrrr#r$)r^rrrrrirrr) ones_likerrrRr masked_selectrMr log_softmaxfloat32r`backendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr#r$)r7rrrrrrrrr#rrr{ labels_masktarget_lengthsflattened_targets log_probsoutputs r/r=zUniSpeechForCTC.forwards$&1%>>(((/3)-,0#'&*E E lT)E  t+E  $; E #Tk E D[ E  t#E   E E E ^E E E E E r.rz UniSpeech Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZe d dejdzdejdzde dzde dzd e dzd ejdzd e e zfd Z xZ S)"UniSpeechForSequenceClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_tj|j|j|_|dS)Nrz`Sequence classification does not support the use of UniSpeech adapters (config.add_adapter=True)r)r4r5rSrrrrir use_weighted_layer_sumrMrKr)r layer_weightsrrOclassifier_proj_size projector num_labels classifierr)r7r^ num_layersr9s r/r5z+UniSpeechForSequenceClassification.__init__+s    6= ) ) f.@ r (//-1  ( S!#ej.D.Dz.Q!R!RD 6#5v7RSS)F$?ARSS r.cB|jjdSrrrs r/rz9UniSpeechForSequenceClassification.freeze_feature_encoder<rr.cL|jD] }d|_ dSrrrs r/rz4UniSpeechForSequenceClassification.freeze_base_modelCrr.Nrrrrrrrc d||n |jj}|jjrdn|}||||||}|jjrx|t} t j| d} tj |j d} | | dddz d} n|d} | | } || d} n|| jd|} | ddd| jd} d | | <| d| d ddz } || }d}|Kt)}|| d|jj| d}|s|f|tdz}||f|zn|St-|||j|j S) a  input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`UniSpeechProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrrzrr3rr)r^rr-rir$r)stackrMrrr.rrRr0rPrrrrr2rr1rr#r$)r7rrrrrrrrr# norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctr)s r/r=z*UniSpeechForSequenceClassification.forwardKsW0&1%->r1a-H-HHMMRSMTTMM#AJM}55  !)..1.55MMBB=CVWXCY[ijjL"."8"8"<"<"C"CAq-J]^_J`"a"a 25M.. /)--!-44|7G7GA7G7N7N7S7STVXY7Z7ZZM//  '))H8FKKDK,BCCV[[QS__UUD FY)F)G)G!HHF)-)9TGf$$v E'!/)     r.r)r%r&r'r5rrrr)rrr,rr=r?r@s@r/r+r+$s">>>(((/3)-,0#'&*C C lT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C C ^C C C C C r.r+)rrr+rrhrr;)Prtcollections.abcr dataclassesrnumpyrr)torch.nnrMrrrp activationsrintegrations.deepspeedr integrations.fsdpr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrrmodeling_utilsrrrprocessing_utilsrrRrrrconfiguration_unispeechr get_loggerr%rrModuler1rBrbrsr|rrrrrrrrrr-r9r?rErhr,rrndarrayrrrrr$rr+__all__r-r.r/rPsf* $$$$$$!!!!!! %%%%%%&&&&&&!!!!!!@@@@@@777777666666BBBBBB999999srrrrrrrrr&&&&&&@@@@@@@@@@444444  H % %  77777K77   7.BI*****ry***Z$>*"<6"<0&&&&&bi&&&R 1 1 1 1 1 1 1 1*!%% I% <% % < % L4' % T\ %%'(%%%%8R/R/R/R/R/R/R/R/j290!!!!!6!!!HE E E E E ryE E E P 2++++++E+++\I I I I I biI I I XC'C'C'C'C'RYC'C'C'LAAAAAAAAP/3 tt c?ttt$t+ t  t Z ttttn3t t t t t -t t t n w w w w w 6w w  w t!" K K K K K .K K  K \ e e e e e )Ae e  e P   r.