"S2S3\5r?"S4S5\ Rd5r@"S6S7\ Rd5rA"S8S9\5rB"S:S;\ Rd5rC"S<S=\ Rd5rD\'"S>S?\#55rESTS@\F\G\G4SA\;SB\GS)\\ RSC\GSD\ R4 SEjjrJ\ rK\'"SFSG\E55rL\'"SHS9"SISJ\E55rMSKrN\'"SLS9"SMSN\E55rO\'"SOS9"SPSQ\E55rP/SRQrQg)UN) dataclass)CallableOptionalUnion)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutput ModelOutputSequenceClassifierOutputWav2Vec2BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringis_torch_flex_attn_availablelogging)UniSpeechConfig)make_flex_block_causal_maskzh Output type of [`UniSpeechForPreTrainingOutput`], with potential hidden states and attentions. ) custom_introc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\\R\ S'Sr\\\R\ S 'S rg) UniSpeechForPreTrainingOutput:a loss (*optional*, returned when model is in train mode, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . (classification) loss. projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentions)__name__ __module__ __qualname____firstlineno____doc__r"rtorch FloatTensor__annotations__r#r$r%r&tupler'__static_attributes__r(h/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/unispeech/modeling_unispeech.pyr r :s )-D(5$$ %,48hu0018>B):): ;B9=8E$5$56=8[TU]5 US-S:XaSUlgSUlg)Nrr)super__init__num_pad_remove)selfnum_conv_pos_embeddings __class__s r4r;UniSpeechSamePadLayer.__init__Xs) #:Q#>!#Car3cXURS:aUSS2SS2SUR*24nU$Nrr<r=r&s r4forwardUniSpeechSamePadLayer.forward\s6    ")!Q0F43F3F2F0F*FGMr3rCr)r*r+r,r;rEr2 __classcell__r?s@r4r6r6WsKr3r6c.^\rSrSrU4SjrSrSrU=r$) UniSpeechPositionalConvEmbeddingbc>[TU]5 [R"URURUR UR S-UR S9Ul[RRn[[RRS5(a$[RRRn[5(Ga%SSK nURRURR SS9 U"URSSS9UlSSS5 [URS5(aUURRR R"nURRR R$nO,URR&nURR(nURR+X5 URR+X5 OU"URSSS9Ul[-UR 5Ul[0UR2Ulg!,(df  GN,=f) Nr9) kernel_sizepaddinggroups weight_normr) modifier_rankweight)namedimparametrizations)r:r;nnConv1d hidden_sizer>num_conv_pos_embedding_groupsconvutilsrQhasattrrVr deepspeedzeroGatheredParametersrS original0 original1weight_gweight_vregister_external_parameterr6rOr feat_extract_activation activation)r=configrQr^rcrdr?s r4r;)UniSpeechPositionalConvEmbedding.__init__cs II      6622a777   hh** 288,,m < <((33??K % ' ' 224993C3CST2U' aH Vtyy"4559955<<FF9955<<FF99--99-- NN 6 6t F NN 6 6t F#DIIH!DDI,V-K-KL  !?!?@VUs I IcURSS5nURU5nURU5nURU5nURSS5nU$)Nrr9) transposer[rOrgrDs r4rE(UniSpeechPositionalConvEmbedding.forwardsV%//15  -0  ]3  6 %//15 r3)rgr[rOrGrIs@r4rKrKbsABr3rKc2^\rSrSrSU4SjjrSrSrU=r$)UniSpeechNoLayerNormConvLayercb>[TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl g)NrrrNstridebias)r:r;conv_dim in_conv_dim out_conv_dimrWrX conv_kernel conv_stride conv_biasr[r rfrgr=rhlayer_idr?s r4r;&UniSpeechNoLayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [ R"URSS9Ul [URUlg)NrrrqT)elementwise_affine)r:r;rtrurvrWrXrwrxryr[ LayerNorm layer_normr rfrgrzs r4r;$UniSpeechLayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl [ R"URURSS9Ulg)NrrrqT) num_groups num_channelsaffine)r:r;rtrurvrWrXrwrxryr[r rfrg GroupNormrrzs r4r;$UniSpeechGroupNormConvLayer.__init__s [TU]5 URS:Xa?[USS9/[ UR S- 5Vs/sHn[ XS-S9PM sn-nOUURS:Xa,[ UR 5Vs/sH n[XS9PM nnO[SURS35e[R"U5Ul SUl S Ul gs snfs snf) Ngroupr)r{rlayerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)r:r;feat_extract_normrrangenum_feat_extract_layersrnr ValueErrorrW ModuleList conv_layersgradient_checkpointing_requires_grad)r=rhirr?s r4r; UniSpeechFeatureEncoder.__init__s   # #w .6vJKv==ABOBA.f1uEBOK % % 0INvOmOmInInA+F?In K01I1I0JJst ==5&+#"O s CC#cNUR5H nSUlM SUlgNF) parameters requires_gradrr=params r4_freeze_parameters*UniSpeechFeatureEncoder._freeze_parameterss#__&E"'E '#r3cUSS2S4nUR(aUR(aSUlURH nU"U5nM U$)NT)rtrainingrr)r= input_valuesr& conv_layers r4rEUniSpeechFeatureEncoder.forwardsK$QW-    4==*.M '**J&}5M+r3)rrr) r)r*r+r,r-r;rrEr2rHrIs@r4rrs8#($  r3rc.^\rSrSrU4SjrSrSrU=r$)UniSpeechFeatureProjectionc4>[TU]5 [R"URSUR S9Ul[R"URSUR5Ul [R"UR5Ul g)Nreps) r:r;rWrrtlayer_norm_epsrLinearrY projectionDropoutfeat_proj_dropoutdropoutr=rhr?s r4r;#UniSpeechFeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r3cnURU5nURU5nURU5nX4$r~)rrr)r=r&norm_hidden_statess r4rE"UniSpeechFeatureProjection.forwards7!__];(:;  ]3 00r3)rrrrGrIs@r4rrs< 11r3rmodulequerykeyvalueattention_maskscalingr head_maskc UcURS5S-n[R"XRSS55U-n UbX-n [R R U SS9n UbXRSSSS5-n [R RXURS9n [R"X5n U RSS5R5n X4$)Nrr9rrUr)pr) sizer.matmulrkrW functionalsoftmaxviewrr contiguous) rrrrrrrrkwargs attn_weights attn_outputs r4eager_attention_forwardrs**R.D(<<}}Q':;gEL!#4 ==((2(>L#nnQAq&AA ==((6??([L,,|3K''1-88:K  $$r3cZ^\rSrSrSrSS\S\S\S\S\S\S \\ 4U4S jjjr SS \ RS \\ RS \\ RS\\ RS\\S\ \S\\ R\\ R\\\ R44SjjrSrU=r$)UniSpeechAttentioni-z=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr is_decoderrs is_causalrhc>[TU]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l [R"XUS9Ul [R"XUS9Ul[R"XUS9Ul[R"XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)rs)r:r;rrrhead_dimrhrrrrrWrk_projv_projq_projout_proj) r=rrrrrsrrhr?s r4r;UniSpeechAttention.__init__0s "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB r3r&key_value_statesrlayer_head_maskoutput_attentionsrreturnc USLnURSSupU(aURSOU n XSUR4n XSUR4n URU5R"U 6R SS5n U(aUOUnUR U5R"U 6R SS5nUR U5R"U 6R SS5n[nURRS:wa[URRnU"UU UUU4UR(dSO URURUUS.UD6unnURXS5R5nUR!U5nUUS4$)z#Input shape: Batch x Time x ChannelNrrr9eager)rrrr)shaperrrrkrrrrh_attn_implementationrrrrreshaperr)r=r&rrrrris_cross_attentionbsztgt_lensrc_len q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrs r4rEUniSpeechAttention.forwardOs.T9%**3B/ /A"((+wr4==9 DMM:{{=166 FPPQRTUV -?)][[055~FPPQRTUV {{>277HRRSTVWX (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,LL/% %  % ! \"))#;FFH mmK0 L$..r3) rhrrrrrrrrrrr)rFTFN)NNNF)r)r*r+r,r-intfloatboolrrr;r.Tensorrrr1rEr2rHrIs@r4rr-s G  ,0CCC C  C  CC)CCD481526,1 3/||3/#5<<03/!. 3/ "%,,/ 3/ $D> 3/-.3/ u||Xell3XeELL>Q5RR S3/3/r3rc.^\rSrSrU4SjrSrSrU=r$)UniSpeechFeedForwardic>[TU]5 [R"UR5Ul[R "URUR5Ul [UR[5(a[URUlOURUl[R "URUR5Ul[R"UR 5Ulgr~)r:r;rWractivation_dropoutintermediate_dropoutrrYintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r4r;UniSpeechFeedForward.__init__s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' - -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r3cURU5nURU5nURU5nURU5nUR U5nU$r~)r r rrrrDs r4rEUniSpeechFeedForward.forwardsX// > 00? 11-@ ))-8 ++M: r3)r r rrrrGrIs@r4rrs @r3rc2^\rSrSrU4SjrSSjrSrU=r$)UniSpeechEncoderLayeric>[TU]5 [URURUR SUS9Ul[R"UR5Ul [R"URURS9Ul [U5Ul[R"URURS9Ulg)NFrrrrrhr)r:r;rrYnum_attention_headsattention_dropout attentionrWrrrrrrr feed_forwardfinal_layer_normrs r4r;UniSpeechEncoderLayer.__init__s +((00,,  zz&"7"78 ,,v'9'9v?T?TU08 " V-?-?VEZEZ [r3cUnURXUS9upnURU5nXA-nURU5nXRU5-nUR U5nU4nU(aXu4- nU$Nrr)rrrrrr=r&rr attn_residualr_outputss r4rEUniSpeechEncoderLayer.forwards% )- L]*8* & Q ]3 %5  6 %(9(9-(HH --m<  "   &Gr3)rrrrrrrGrIs@r4rrs \r3rc ^\rSrSrU4SjrS S\R S\\RS\ S\ S\ 4 S jjr S\ \RS4S \R4S jr S r U=r$)UniSpeechEncoderic>[TU]5 Xl[U5Ul[ R "URURS9Ul [ R"UR5Ul [ R"[UR5Vs/sHn[!U5PM sn5UlSUlgs snfNrF)r:r;rhrKpos_conv_embedrWrrYrrrrrrrnum_hidden_layersrlayersrr=rhr#r?s r4r;UniSpeechEncoder.__init__s  >vF,,v'9'9v?T?TUzz&"7"78 mmERXRjRjLk$lLkq%:6%BLk$lm &+#%m CNr&rroutput_hidden_states return_dictcU(aSOSnU(aSOSnUb4URS5RSSURS5nSX)'URUU5nUR U5n X-nUR U5nUR U5n[5=(d [U5n URHn U(aXa4-n[R"/5n UR=(a XRR:n U (aU (a U "XUS9nUSnU (aSnU(dM|UWS4-nM U(aXa4-nU(d[SXU455$[!UUUS 9$) Nr(rrr9rr NNc3,# UH ocMUv M g7fr~r(.0vs r4 +UniSpeechEncoder.forward..m$[q$[ last_hidden_stater&r') unsqueezerepeatr_update_full_maskr*rrr r r,r.randrrh layerdropr1rr=r&rrr0r1all_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r4rEUniSpeechEncoder.forwards#7BD$5b4  %$2$<$[TU]5 URUlURUl[ R"UR 5Ul[ R"UR UR5Ul [ R"5Ul [ R"URUR 5Ul g)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)r:r;adapter_attn_dim input_dimrY hidden_dimrWrnormrlinear_1ReLUact_fnlinear_2rs r4r;"UniSpeechAttnAdapterLayer.__init__s 00 ,,LL1  $//4>>B ggi  $..$//B r3r&cURU5nURU5nURU5nURU5nU$r~)rarbrdrerDs r4rE!UniSpeechAttnAdapterLayer.forward-s@ -0  m4  M2  m4 r3)rdr`r_rbrera) r)r*r+r,r;r.r/rEr2rHrIs@r4r\r\s  CU%6%6r3r\ct^\rSrSrU4SjrSS\R S\\R S\4Sjjr Sr U=r $) $UniSpeechEncoderLayerStableLayerNormi7c>[TU]5 [URURUR SUS9Ul[R"UR5Ul [R"URURS9Ul [U5Ul[R"URURS9Ul[#USS5b[%U5UlgSUlg)NFrrr^)r:r;rrYrrrrWrrrrrrrrrgetattrr\ adapter_layerrs r4r;-UniSpeechEncoderLayerStableLayerNorm.__init__8s +((00,,  zz&"7"78 ,,v'9'9v?T?TU08 " V-?-?VEZEZ [ 6-t 4 @!:6!BD !%D r3r&rrcUnURU5nURXUS9upnURU5nXA-nXRUR U55-nUR bXR U5-nU4nU(aXu4- nU$r)rrrrrrmr!s r4rE,UniSpeechEncoderLayerStableLayerNorm.forwardKs &  6 )- L]*8* & Q ]3 %5 %(9(9$:O:OP]:^(__    )),>,>},MMM "   &Gr3)rmrrrrrr) r)r*r+r,r;r.rrrrEr2rHrIs@r4rjrj7sC&,26"' ||!. r3rjc~^\rSrSrU4SjrS SjrS\\RS4S\R4Sjr Sr U=r $) UniSpeechEncoderStableLayerNormiec>[TU]5 Xl[U5Ul[ R "URURS9Ul [ R"UR5Ul [ R"[UR5Vs/sHn[!U5PM sn5UlSUlgs snfr))r:r;rhrKr*rWrrYrrrrrrrr+rjr,rr-s r4r;(UniSpeechEncoderStableLayerNorm.__init__fs  >vF,,v'9'9v?T?TUzz&"7"78 mmCHIaIaCb cCba 1& 9Cb c  ',# dr/NcU(aSOSnU(aSOSnUb4URS5RSSURS5nSX)'URUU5nUR U5n X-nUR U5n[ 5=(d [U5n URHn U(aXa4-n[R"/5n UR=(a XRR:n U (aU (a U "XUS9nUSnU (aSnU(dM|UWS4-nM URU5nU(aXa4-nU(d[SXU455$[!UUUS 9$) Nr(rrr9rr r3c3,# UH ocMUv M g7fr~r(r5s r4r8:UniSpeechEncoderStableLayerNorm.forward..r:r;r<)r>r?rr@r*rr r r,r.rArrhrBrr1rrCs r4rE'UniSpeechEncoderStableLayerNorm.forwardqs#7BD$5b4  %$2$<$[TU]5 URUlURUlUR UR-S:wa&[SUR SURS35e[R"[R"SURUR -UR UR-55Ul [R"URSURUR -5UlSUlg)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenationrrr9)r:r;num_codevector_groupsrnum_codevectors_per_groupnum_varscodevector_dimrrW Parameterr.r/ codevectorsrrt weight_proj temperaturers r4r;'UniSpeechGumbelVectorQuantizer.__init__s  6688  4?? 2a 7)&*?*?)@A559__4EEWY  <<   a4==!@&BWBW[_[j[jBj k 99V__R%8$//DMM:YZr3c URSS9n[R"[R"U[R"US-5-SS9*5R5nU$)NrrgHz>r)meanr.expsumlog)probsmarginal_probs perplexitys r4_compute_perplexity2UniSpeechGumbelVectorQuantizer._compute_perplexitysR*YY .599^VZEZ;[*[ac ddeiik r3cURup#nURU5nURX#-UR-S5nUR(a[ R RUR5URSS9RU5n[R"URX#-URS5R5SS9nURU5nOyURSS9nUR"UR6R!SURSS5S5nURX#-URS5nURU5nURX#-S5nUR#S5UR$-n U RX#-URUR&S5n U R)S5RX#S5n X4$)NrT)tauhardrr?r)rrrrrrWrgumbel_softmaxrrtype_asr.rrargmax new_zerosscatter_r>rrr) r=r& batch_sizesequence_lengthrYcodevector_probscodevector_soft_distrcodevector_idxcodevectors_per_grouprs r4rE&UniSpeechGumbelVectorQuantizer.forwards3@3F3F0 [((7 %**:+G$//+Y[]^ ==!}};;##%4+;+;$ < gm$  $)=="":#?RTU[[]ce$ 112FGJ+11b19N,66 8K8KLUUN''A.   044Z5QSWSbSbdfg 112BCJ+001MrR 0 : :2 >AQAQ Q+001Mt`d`m`moqr !oob)..zBO &&r3)rrrrr) r)r*r+r,r-r; staticmethodrrEr2rHrIs@r4r|r|s+ ( #'#'r3r|c\rSrSr%\\S'SrSrSrSr Sr Sr Sr S\ \R\44SjrS \S \R4S jrS rg )UniSpeechPreTrainedModeli rh unispeechrTc [U[5(aURRRR SSS9 URR RR5 [RRUR5 g[U[5(a[RR URRSS[R"SURR SURR"-- 5-S9 [RR%URR S5 g[U[&5(a[R"SUR(R*- 5n[RRUR(RU*US9 [RRUR(R U*US9 g[U[R,5(akURRR SUR.R0S9 UR b%UR RR5 gg[U[R2[R445(aJUR RR5 URRR7S5 g[U[R85(a[RR;UR5 UR bh[R"UR<UR"UR S-- 5n[RRUR U*US9 ggg) zInitialize the weightsrr)rstdrr9)abNr)r r|rrSdatanormal_rszero_rWinituniform_rrKr[mathsqrtrN in_channels constant_rr in_featuresrrhinitializer_rangerrfill_rXkaiming_normal_rP)r=rks r4 _init_weights&UniSpeechPreTrainedModel._init_weightss f< = =    % % * * 2 2 2 C    # # ( ( . . 0 GG  V// 0  @ A A GGOO "" !v{{'>'>q'AFKKD[D['["\]]   GG  fkk.. 2  : ; ; !f//;;;._conv_out_length<s 99\7wWZ[[ [r3)ziprhrwrx)r=rrrNrrs r4 _get_feat_extract_output_lengths9UniSpeechPreTrainedModel._get_feat_extract_output_lengths7sG  \ $'t{{'>'> @W@W#X K,]PM$Yr3feature_vector_lengthrcURSS9SS2S4nURU5R[R5nUR Sn[R "XQ4URURS9nSU[R"UR SURS9US- 4'URS/5RS5RS/5R5nU$)Nrrr)rTdevicer)r) cumsumrtor.longrzerosrTrarangeflipr)r=rrnon_padded_lengthsoutput_lengthsrs r4"_get_feature_vector_attention_mask;UniSpeechPreTrainedModel._get_feature_vector_attention_maskFs,22r2:1b5A>>?QRUUV[V`V`a#))!,   /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr3r(N)r)r*r+r,rr0base_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrrr. LongTensorrrrr2r(r3r4rr sg #$O&*#N9B eEDTDTVYDY>Z   ]b]m]m r3rr mask_prob mask_length min_masksrc ^^^^^UunmTS:a [S5eTT:a[STSTS35e[RRS5R 5mUUUUU4SjnUb-UR 5R S5R5O[U5Vs/sHnTPM snn[R"UT4[S 9n /n U"T5n U S :XaU $UHn U"U 5n [RR[R"U TS- - 5U S S 9n[U5S :XaTS- nOUS n[R"U[R"X- [R S 9U-/5nU R#U5 M [R$"U 5n [R&"U SS2SS2S4X[T45n U R)X[T-5n [R"T5SSSS24n[R&"UX[T45R)X[T-5nU U-n U R+5TS- :a TS- XTS- :'[R,"XSS5 U $s snf) a2 Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `c>[TU-T- T-5n[UT5nUT-T:aTT-nUTS- - U:a[UTS- - S5nU$)z;Given input length, compute how many spans should be maskedrr)rmax)rnum_masked_spanepsilonrrrrs r4compute_num_masked_span6_compute_mask_indices..compute_num_masked_span|soi,6DwNOoy9 [ (? :-^\rSrSrS\4U4SjjrSS\RS\\RS\\R4Sjjr \ SS\\RS\\RS\\RS \\ S \\ S \\ S \\\44S jj5rSrU=r$)UniSpeechModelirhc>[TU]U5 Xl[U5Ul[ U5UlURS:dURS:aG[R"[R"UR5R55UlUR (a[#U5UlO['U5UlUR)5 g)Nr)r:r;rhrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrWrr.rrYrmasked_spec_embeddo_stable_layer_normrrencoderr' post_initrs r4r;UniSpeechModel.__init__s   !8!@"G[G[chcmcm n /3/E/E/H/HI\I\/]M , ;; ( (1 ,#8)++77 KK;;++<< $ $)<<0DMaMainisis#t #74#@#G#GO]_#` 23M /r3rrr0r1rcUbUOURRnUbUOURRnUbUOURRnUR U5nUR SS5nUbUR URSU5nURU5upURXUS9nURUUUUUS9n U SnU(d X4U SS-$[UUU RU RS9$)a mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nrr9)rrrrr0r1r)r=extract_featuresr&r')rhrr0use_return_dictrrkrrrrrUniSpeechBaseModelOutputr&r') r=rrrrr0r1rr&encoder_outputss r4rEUniSpeechModel.forwards72C1N-TXT_T_TqTq$8$D $++JjJj &1%59 ,((,$E$5$56,!!1!12 ,\269=,0/3&*2 u||,2 !.2 $E$5$56 2 $D> 2 'tn 2 d^2  u.. /2 2 r3rzZ UniSpeech Model with a vector-quantization module and ctc loss for pre-training. c8^\rSrSrS\4U4SjjrS\4SjrSrSr \ SS\ RS \ RS \ RS\4S jj5r \SS \\ R S \\ R S\\S\\S\\S\\\44 Sjj5rSrU=r$)UniSpeechForPreTrainingiGrhc8>[TU]U5 [U5Ul[R "UR 5Ul[U5Ul [R"URUR5Ul [R"URUR5Ul[R"URUR 5Ul[R "UR$5UlUR)5 gr~)r:r;rrrWrfeat_quantizer_dropoutdropout_featuresr| quantizerrrproj_codevector_dim project_qrY project_hidnum_ctc_classesctc_proj final_dropoutrrrs r4r; UniSpeechForPreTraining.__init__Ms  '/ " 6+H+H I7?6#8#8&:T:TU99V%?%?ASAST &"4"4f6L6LM zz&"6"67  r3rc$XRlg)zR Set the Gumbel softmax temperature to a given value. Only necessary for training N)rr)r=rs r4set_gumbel_temperature.UniSpeechForPreTraining.set_gumbel_temperature\s&1"r3cZ[R"S[5 UR5 gz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nwarningswarn FutureWarningfreeze_feature_encoderr=s r4freeze_feature_extractor0UniSpeechForPreTraining.freeze_feature_extractorb'   Q  ##%r3cLURRR5 g Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. Nrrrr2s r4r1.UniSpeechForPreTraining.freeze_feature_encodern ((;;=r3target_featuresnegative_featurespredicted_featuresc[R"X/SS9n[R"UR5UR5SS9nUR U5nXC- nU$)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rrr)r.catcosine_similarityrr)r<r=r>rlogitss r4compute_contrastive_logits2UniSpeechForPreTraining.compute_contrastive_logitsus\ ))_$HaP(();)A)A)C_EZEZE\bde0% r3rrrr0r1rc UbUOURRnURUUUUUS9nUSnURUS5nUR U5upUR U R UR RR55n URU 5n [R"URS5URS55RURR5n U RSS5n [R "U 5R#5R UR$5n U RSS5n U R'S5n UR)U S5U R)U )S5-n UR+U 5n UR-U 5n SnU(dUb XX4USS-$XyU 4USS-$[/UUU U UR0UR2S9$) a[ Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, UniSpeechForPreTraining >>> feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> model = UniSpeechForPreTraining.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> # TODO: Add full pretraining example ```Nrrrrrr9)r"r#r$r%r&r')rhrrrrr!rrSrTr"r.emptyrr replace_probrk bernoullirrr> masked_fillrr$r r&r')r=rrrr0r1r$transformer_featuresrquantized_featuresr%prob_replace_matrixsampled_replace_matrixrBr"s r4rEUniSpeechForPreTraining.forwards*&1%,A,A$..BWBWB]B],^_!--.@A#kk*>*C*CA*FH\HaHabcHdekk KK $ $ 2;;AqA!&1D!E!J!J!L!O!OPdPkPk!l!7!A!A!Q!G!7!A!A"!E%112H#N  * *,B+BC H  f%v&4F^ahijikalll(>STW^_`_aWbb b,1'9"7!//))   r3)r$rrr"r!rr)r)NNNN)r)r*r+r,rr;rr(r3r1rr.r/rCrrrrrr1r rEr2rHrIs@r4rrGs  1#1 &>  ** ,,"-- &26,0/3&* D u||,D !.D $D> D 'tn D d^ D  u33 4D D r3rr9zq UniSpeech Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). c^\rSrSrSS\\4U4SjjjrSrSrSr Sr \ SS\\ RS \\ RS \\S \\S \\S \\ RS\\\44Sjj5rSrU=r$)UniSpeechForCTCi target_langc>[TU]U5 [U5Ul[R "UR 5UlX lURc[SURS35e[US5(aUR(a URO URn[R "X1R5UlUR%5 g)a  target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`UniSpeechForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `UniSpeechForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r:r;rrrWrr%rrQ vocab_sizerr?r]rSoutput_hidden_sizerYrlm_headr)r=rhrQrUr?s r4r;UniSpeechForCTC.__init__s  '/zz&"6"67 &    $00@AHH *1)G)GFL^L^F % %djdvdv yy!35F5FG  r3cURnUb'[URSS5c[SUS35eUc.[URSS5b[R S5 gUbUR USS9 gg)a This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. Nr^zCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load)rQrlrhrloggerinfo load_adapter)r=rQs r4 tie_weightsUniSpeechForCTC.tie_weightss&&  "wt{{(26,0/3&*)-D u||,D !.D $D> D 'tn D d^ D &D  un$ %D D r3rPz UniSpeech Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c^\rSrSrU4SjrSrSrSr\SS\ \ RS\ \ RS\ \ S \ \ S \ \ S \ \ RS \ \\44S jj5rSrU=r$)"UniSpeechForSequenceClassificationioc">[TU]U5 [US5(aUR(a [ S5e[ U5UlURS-nUR(a2[R"[R"U5U- 5Ul [R"URUR 5Ul[R"UR UR$5UlUR)5 g)NrSz`Sequence classification does not support the use of UniSpeech adapters (config.add_adapter=True)r)r:r;r]rSrrrr+use_weighted_layer_sumrWrr.r layer_weightsrrYclassifier_proj_size projector num_labels classifierr)r=rh num_layersr?s r4r;+UniSpeechForSequenceClassification.__init__vs  6= ) )f.@.@r (/--1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r3cZ[R"S[5 UR5 gr+r-r2s r4r3;UniSpeechForSequenceClassification.freeze_feature_extractorr5r3cLURRR5 gr7r9r2s r4r19UniSpeechForSequenceClassification.freeze_feature_encoderr;r3cTURR5H nSUlM grdrers r4rf4UniSpeechForSequenceClassification.freeze_base_modelrhr3rrrr0r1rirc0UbUOURRnURR(aSOUnURUUUUUS9nURR(aiU[n[ R "USS9n[RRURSS9n XRSSS5-RSS9nOUSnURU5nUcURSS9n OURUR SU5n U R#S5R%SSUR S5n S X)'URSS9U RSS9RSS5- n UR'U 5n SnUbF[)5nU"U RSURR*5URS55nU(dU 4U[S-nUbU4U-$U$[-UU UR.UR0S 9$) a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`UniSpeechProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrrrr9rro)rhrrrr{r.stackrWrrrrrrrrrr>r?rrrrr&r')r=rrrr0r1rir$r& norm_weights pooled_output padding_maskexpand_padding_maskrBr"loss_fctrs r4rE*UniSpeechForSequenceClassification.forwards.&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r3)rrrrr)r)r*r+r,r;r3r1rfrrr.rrrr1rrEr2rHrIs@r4rros" &>(26,0/3&*)-B u||,B !.B $D> B 'tn B d^ B &B  u.. /B B r3r)rPrrrr)NrNrB)Rrr. dataclassesrtypingrrrnumpyrr.torch.nnrWr activationsr integrations.deepspeedr integrations.fsdpr modeling_attn_mask_utilsr r modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrprocessing_utilsrr\rrrconfiguration_unispeechrintegrations.flex_attentionr get_loggerr)rZr Moduler6rKrnrrrrrrrrrrr'r\rjrrr|rr1rrndarrayrrrrr{rPr__all__r(r3r4rsH, !,, %!@7gB9G&JJ4 !!J   H %  :K:  :.BII*ryy*Z$>*"<6"<0&bii&R 1 1* $(,% II% <<% % << % U\\* % e_ %% %%