tjAfdZddlZddlZddlmZddlmZddlZddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5dZ6dZ7e3j8e9Z:dZ;e/deGdde-Z< djde=e>e>fde?de>d e j@dzd!e>d"ejAf d#ZBdkd$e=d%e>d&ejAdzfd'ZCGd(d)eZDGd*d+eZEGd,d-eZFGd.d/e jGZHGd0d1e jGZIGd2d3e jGZJGd4d5e jGZK dld7e jGd8e jLd9e jLd:e jLd e jLdzd;e?dzdZMGd?d@e jGZNGdAdBe jGZOGdCdDeZPGdEdFeZQGdGdHe jGZRGdIdJe jGZSGdKdLe jGZTGdMdNe jGZUGdOdPe jGZVGdQdRe jGZWe/GdSdTe(ZXe/GdUdVeXZYe/dWGdXdYeXZZe/dZGd[d\eXZ[e/d]Gd^d_eXZ\e/Gd`daeXZ]Gdbdce jGZ^Gdddee jGZ_e/dfGdgdheXZ`gdiZadS)mzPyTorch Wav2Vec2 model.N)Callable) dataclass) load_file)nn)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack) ModelOutputTransformersKwargsauto_docstring cached_filecheck_torch_load_is_safeis_peft_availablelogging)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensorsza Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dZejdzed <dZejdzed <dS) Wav2Vec2ForPreTrainingOutputa loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://huggingface.co/papers/2006.11477). diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____doc__r'torch FloatTensor__annotations__r(r)r*r+tupler,r-r.l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyr&r&Bs$&*D% d ")))15e'$.555;? 1D 8???6:5,t3:::59M5*+d299926Je'(4/66615e'$.555/3NE%,33333r8r&shape mask_prob mask_lengthattention_mask min_masksreturnc@|\}dkrtdkrtdddtjdfd}|9|dnfd t|D}tj |ft }g} |} | d kr|S|D]} || } tj tj | dz z | d } t| d krdz }n| d }tj| tj| | z tj |zg} | | tj| } tj| dddddf|| f} | || z} tj ddddf}tj||| f|| z}| |z} | dz krdz | | dz k<tj|| dd|S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. r!z&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedr!r)intmax) input_lengthnum_masked_spanepsilonr<r;r>sequence_lengths r9compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spans~i,6DwNOOoy99 [ (? : :-z)_compute_mask_indices..s 9 9 9!o 9 9 9r8dtyperF)replace) ValueErrornprandomranditemdetachsumtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaperDput_along_axis)r:r;r<r=r> batch_sizerI input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrErFspec_aug_mask_idxdummy_mask_idxoffsetsrGrHs `` ` @@r9_compute_mask_indicesrpesP0#(JQABBB_$$ :^i : :'6 : : :   innQ$$&&G         $  % ##B''..000 9 9 9 9uZ'8'8 9 9 9Hj/:$GGGM11/BBa%55 11,??I,, IlkAo6 7 7RW-    ! !Q & &-q0NN.q1NN (;o(MUWU] ^ ^ ^ao o p   !!"34444"455111aaa:&5H+(V,33J@SVa@abbi $$T4]3Gog 4G'UVV^^'+5G,g5/A"555GVYZGZ-!0CCDm%7B??? r8features_shape num_negativesmask_time_indicesc|\}}tj|}tj|||ftj}||t ntj|t }t|D]}||dz }|||} tj tj|dzdddf|dz|f} tj d||dz|f} | | | kxxdz cc<| | ||||<||xx||zz cc<|S)z> Sample `num_negatives` vectors from feature vectors. )r:rPNrOr!r)size) rSr^r[rbastyper\rarZrXrerTrandint) rqrrrsrhrHsequence_length_rangesampled_negative_indices batch_idxhighmapped_masked_indicesfeature_indicessampled_indicess r9_sample_negative_indicesrs#1JIo66 "xz?M.Zbdbjkkk+<*G  &&&RWUckoMpMpMp:&& K K  +//11A5 56G 6R S/")D1H*=*=aaag*FPQS`Habb)++At4!8]:S+TT?:;;;q@;;;MbbqLr +,=i,HI !+++y?/JJ++++ ##r8c&eZdZdfd ZdZxZS)Wav2Vec2NoLayerNormConvLayerrcZt|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t|j |_ dS)Nrr! kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r9rz%Wav2Vec2NoLayerNormConvLayer.__init__s !#C#Caar8cJ|jdkr|ddddd|j f}|SNr)rrs r9rzWav2Vec2SamePadLayer.forwardxs;   " ")!!!QQQ0F43F2F0F*FGMr8rrs@r9rrssLKKKKKr8rc.eZdZdZfdZdZdZxZS)Wav2Vec2FeatureEncoderz.Construct the features from raw audio waveformctjdkr7tdgfdt jdz Dz}nDjdkr!fdt jD}nt djd tj||_ d |_ d |_ dS) Ngrouprrc8g|]}t|dzS)r!r)rrLirs r9rNz3Wav2Vec2FeatureEncoder.__init__..s>NNNIJ,Va!eDDDNNNr8r!layerc2g|]}t|S)r)rrs r9rNz3Wav2Vec2FeatureEncoder.__init__..s4CD*6A>>>r8z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rrfeat_extract_normrrZnum_feat_extract_layersrRr ModuleList conv_layersgradient_checkpointing_requires_grad)rrrrs ` r9rzWav2Vec2FeatureEncoder.__init__s   #w . .5fqIIIJNNNNNSTZTruvTvNwNwNNNKK % 0 0HMfNlHmHmKKt1Ittt =55&+#"r8cP|D] }d|_ d|_dSNF) parameters requires_gradrrparams r9_freeze_parametersz)Wav2Vec2FeatureEncoder._freeze_parameterss4__&& ( (E"'E  #r8cr|dddf}|jr|jrd|_|jD] }||}|S)NT)rtrainingrr)r input_valuesr+ conv_layers r9rzWav2Vec2FeatureEncoder.forwards[$QQQW-    /4= /*.M '* 6 6J&J}55MMr8)r/r0r1r2rrrrrs@r9rr~s\88#####&$$$       r8rc$eZdZfdZdZxZS)Wav2Vec2FeatureProjectionc.ttj|jd|j|_tj|jd|j|_ tj |j |_ dS)NrJeps) rrrrrlayer_norm_epsrLinearr projectionDropoutfeat_proj_dropoutdropoutrrrs r9rz"Wav2Vec2FeatureProjection.__init__sn ,vr':@UVVV)FOB$79KLLz&":;; r8c||}||}||}||fSr)rrr)rr+norm_hidden_statess r9rz!Wav2Vec2FeatureProjection.forwardsC!__];;(:;;  ]33 000r8rrs@r9rrsG<<<<< 1111111r8rmodulequerykeyvaluescalingrkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)NrJr#rr)prr!) rur3matmulrr functionalsoftmaxrr contiguous) rrrrr=rrr attn_weights attn_outputs r9eager_attention_forwardr s**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r8ceZdZdZ ddededed ed ed ed edzffd Z dde j de j dzde j dzdedzde e de e j e j dze e j dzff dZxZS)Wav2Vec2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderr is_causalrc t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r)rrrrrhead_dimrrRrrrrrk_projv_projq_projout_proj) rrrrrrrrrs r9rzWav2Vec2Attention.__init__s "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r8r+key_value_statesr=output_attentionsrr?c |du}|jdd}g|d|jR}|||dd} |r|n|} g| jddd|jR} || | dd} || | dd} tj|j j t}||| | | |f|j sdn|j |j|d|\}}|jg|dR}||}||dfS)z#Input shape: Batch x Time x ChannelNrJr!r#r)rrr)r:rrviewrrrr get_interfacer_attn_implementationr rrrrfrr)rr+rr=rris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacer r s r9rzWav2Vec2Attention.forwards.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT -?R))]B^)#2#.BBDMBB[[0055h??II!QOO {{>2277AAKKAqQQ (?(M K ,.E) ) %8$7      % $}>CC$,L/ % %  % % ! \*k);;;;;;FFHH mmK00 L$..r8)rFTFN)NNF)r/r0r1r2rCfloatr\r"rr3Tensorrrr6rrrs@r9r r sJGG  (,CCCC C  C  CC%CCCCCCD15.2). 0/0/|0/ ,-0/ t+ 0/  $; 0/-.0/ u|U\D0% 2E2LL M0/0/0/0/0/0/0/0/r8r c$eZdZfdZdZxZS)Wav2Vec2FeedForwardcttj|j|_tj|j|j|_ t|j trt|j |_n |j |_tj|j|j|_tj|j|_dSr)rrrractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r9rzWav2Vec2FeedForward.__init__(s $&Jv/H$I$I!"$)F,>@X"Y"Y f' - - 9'-f.?'@D $ $'-'8D $If&>@RSS j)>??r8c||}||}||}||}||}|Sr)r/r3r-r4r6rs r9rzWav2Vec2FeedForward.forward5sg// >> 00?? 11-@@ ))-88 ++M:: r8rrs@r9r*r*'sL @ @ @ @ @r8r*c&eZdZfdZddZxZS)Wav2Vec2EncoderLayerctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_dS)NFrrrrrr)rrr rnum_attention_headsattention_dropout attentionrrr5rrrrr* feed_forwardfinal_layer_normrs r9rzWav2Vec2EncoderLayer.__init__@s *(0,    z&"788 ,v'9v?TUUU/77 " V-?VEZ [ [ [r8NFc|}||||\}}}||}||z}||}|||z}||}|f}|r||fz }|SNr=r)r>rrr?r@rr+r=r attn_residualr rMoutputss r9rzWav2Vec2EncoderLayer.forwardOs% )- .L]*8* * & |Q ]33 % 5  66 %(9(9-(H(HH --m<<  "  '   &Gr8rrrs@r9r9r9?sQ \ \ \ \ \r8r9cReZdZfdZ ddejdejdzdefdZxZS) #Wav2Vec2EncoderLayerStableLayerNormctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_t#|ddt%||_dSd|_dS)NFr;radapter_attn_dim)rrr rr<r=r>rrr5rrrrr*r?r@getattrWav2Vec2AttnAdapterLayer adapter_layerrs r9rz,Wav2Vec2EncoderLayerStableLayerNorm.__init__ds *(0,    z&"788 ,v'9v?TUUU/77 " V-?VEZ [ [ [ 6-t 4 4 @!9&!A!AD   !%D   r8NFr+r=rcJ|}||}||||\}}}||}||z}||||z}|j|||z}|f}|r||fz }|SrB)rr>rr?r@rMrDs r9rz+Wav2Vec2EncoderLayerStableLayerNorm.forwardws &  66 )- .L]*8* * & |Q ]33 % 5 %(9(9$:O:OP]:^:^(_(__   ))D,>,>},M,MMM "  '   &Gr8r) r/r0r1rr3r(r\rrrs@r9rHrHcs~&&&&&,/3"' | t+ r8rHc ^eZdZfdZ d dejdejdzdeded ef d ZxZ S) Wav2Vec2Encoderct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_dS)Nrc.g|]}tSr7)r9rLrMrs r9rNz,Wav2Vec2Encoder.__init__..s"$k$k$ka%9&%A%A$k$k$kr8Frrrrpos_conv_embedrrrrrrr5rrrZnum_hidden_layerslayersrrs `r9rzWav2Vec2Encoder.__init__s  =fEE,v'9v?TUUUz&"788 m$k$k$k$k5QWQiKjKj$k$k$kll &+###r8NFTr+r=routput_hidden_states return_dictc|rdnd}|rdnd}|;|ddd|jd}d||<t|j||}||} || |jz}||}| |}tpt|} |j D]a} |r||fz}tjg} |jo| |jjk} | r| r| |||}|d}| rd}|r ||dfz}b|r||fz}|st#d |||fDSt%||| S) Nr7rJr!r#rr inputs_embedsr=rCNNc3K|]}||V dSrr7rLvs r9 z*Wav2Vec2Encoder.forward..(mmq_`_l_l_l_l_lmmr8last_hidden_stater+r,) unsqueezerepeatr:r rrUtodevicerrr r rWr3rUr layerdropr6rrr+r=rrXrYall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r9rzWav2Vec2Encoder.forwards #7@BBD$5?bb4  %$2$<$(>}?S(T(TT  66  ]33 022R6LT6R6R [ P PE# I$58H$H!#(*R.. !]Z/BT[EZ/ZN! 1[ 1 %!.Te!!! !.a 0  - ,   P&9]1=M.s" b b bQ 0 8 8 b b br8FrTrs `r9rz'Wav2Vec2EncoderStableLayerNorm.__init__s  =fEE,v'9v?TUUUz&"788 m b b b b%H`BaBa b b b   ',###r8NFTc|rdnd}|rdnd}|;|ddd|jd}d||<t|j||}||} || z}||}tpt|} |j D]a} |r||fz}tj g} |j o| |jj k} | r| r| |||}|d}| rd}|r ||dfz}b||}|r||fz}|std |||fDSt!||| S) Nr7rJr!r#rr[rCr]c3K|]}||V dSrr7r_s r9raz9Wav2Vec2EncoderStableLayerNorm.forward..rbr8rc)rerfr:r rrUrr r rWr3rUrrirr6rrjs r9rz&Wav2Vec2EncoderStableLayerNorm.forwards#7@BBD$5?bb4  %$2$<$ >AQ Q+00o1Mt`d`moqrr !oob))..z?BOO J&&r8r) r/r0r1r2r staticmethodrrrrs@r9r|r|%sv (   \ #'#'#'#'#'#'#'#'r8r|c$eZdZfdZdZxZS)Wav2Vec2AdapterctjjkrCt jjj|_t jj|_ndx|_|_t j fdtj D|_ j |_ dS)Nc36K|]}tVdSr)Wav2Vec2AdapterLayerrSs r9raz+Wav2Vec2Adapter.__init__..|s,#k#kQ$8$@$@#k#k#k#k#k#kr8)rroutput_hidden_sizerrrprojrproj_layer_normrrZnum_adapter_layersrWrirs `r9rzWav2Vec2Adapter.__init__rs   $(: : : &"4f6OPPDI#%<0I#J#JD /3 3DI,m#k#k#k#k%PVPiJjJj#k#k#kkk )r8cX|j1|j*||}||}|dd}|jD]=}tj}|jr ||jkr ||}>|dd}|Sr)rrrrWrSrTrri)rr+rlayerdrop_probs r9rzWav2Vec2Adapter.forwards 9 T%9%E IIm44M 00??M%//155 [ 5 5EY--//N= 5^dn%D%D %m 4 4 %//155 r8rrs@r9rrqsG * * * * *r8rc$eZdZfdZdZxZS)rcttj|jd|jz|j|jd|_dS)Nr#r!)rr)rrrrradapter_kernel_sizeadapter_striderrs r9rzWav2Vec2AdapterLayer.__init__sU I  % ) )  &(     r8cr||}tj|d}|S)Nr!r)rrrglurs r9rzWav2Vec2AdapterLayer.forwards3 -00  ))-Q)?? r8rrs@r9rrsG     r8rc4eZdZfdZdejfdZxZS)rLctt|j|_|j|_t j|j|_t j |j|j|_ t j |_ t j |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rrrJ input_dimr hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rs r9rz!Wav2Vec2AttnAdapterLayer.__init__s 0 ,L11  $/4>BB gii  $.$/BB r8r+c||}||}||}||}|Sr)rrrrrs r9rz Wav2Vec2AttnAdapterLayer.forwardsL -00  m44  M22  m44 r8)r/r0r1rr3r4rrrs@r9rLrLs[ C C C C CU%6r8rLceZdZUeed<dZdZdZdZdZ dZ dZ e j fdZdde jezd edzfd Z dd ed e jfd ZdZdZddefdZxZS)Wav2Vec2PreTrainedModelrwav2vec2raudioTc t|t|tr4|j|jdSt|trZtj |j j ddtj |j j tj|jdSt|t rwtj |jj ddt%jd|jjd|jjzz ztj|jj ddSt|t.rgt%jd|jjz }tj|jj | |tj|jj | |dSt|t4jrntj|j |j Pt%j|j|j|jdzz }tj|j | |dSdSdS)zInitialize the weightsrr!)rstdrr#)abN)r _init_weightsr0Wav2Vec2ForPreTraining project_hidreset_parameters project_qr|initnormal_rrzeros_runiform_rrrmathsqrtr in_channels constant_rr in_featuresrrkaiming_normal_r)rrkrs r9rz%Wav2Vec2PreTrainedModel._init_weightss: f%%% f4 5 5 6   / / 1 1 1   - - / / / / /  = > > 6 L+2! D D D D K*/ 0 0 0 M&, - - - - -  ? @ @ 6 L " !v{'>q'AFKD['["\]]]     N6;+Q / / / / /  9 : : 6 !f/;;< > > > M&+0QB! < < < < < <  * * 6   / / /{&Ifmv/AFDVWXDY/YZ[[ fkaR1555555  6 6'&r8Nri add_adapterc| |jjn|}d}t|jj|jjD]\}}||||}|r3t |jjD]}||d|jj}|S)zH Computes the output length of the convolutional layers Nc<tj||z |ddzS)Nfloor) rounding_moder!)r3divrErrs r9_conv_out_lengthzRWav2Vec2PreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths&9\K7wWWWZ[[ [r8r!)rrziprrrZrr)rrirrrrrMs r9 _get_feat_extract_output_lengthsz8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengthss 2=1Ddk--+  \ \ \ $'t{'> @W#X#X Q Q K,,]KPPMM  _4;9:: _ _ 0 04;C] ^ ^ r8feature_vector_lengthr=c |ddddf}|||}|tj}|jd}tj||f|j|j}d|tj |jd|j|dz f<| dgd dg }|S)NrJrrr)rPrhr!)rh) cumsumrrgr3longr:r[rPrhr^flipr\)rrr=rnon_padded_lengthsoutput_lengthsrhs r9"_get_feature_vector_attention_maskz:Wav2Vec2PreTrainedModel._get_feature_vector_attention_masks ,22r2::111b5A>>?Q_j>kk'**5:66#)!,  . /~7KTbTi   uv ^%9!%<^EZ[[[]kno]opq',,bT2299"==BBB4HHMMOOr8c|jjt|jdi}|D]N\}}t |t r4|D]\}}||d||g< Ot |tr9|j D]\}}||dd|g< |S)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`..lm_head) rrJrRr named_modulesr0rLnamed_parametersjoinWav2Vec2ForCTCr)radapter_weightsrr param_namers r9 _get_adaptersz%Wav2Vec2PreTrainedModel._get_adapterss ; ' /vvvww w ..00 J JLD&&":;; J)/)@)@)B)BJJ%JDIOCHHdJ-?$@$@AA dN + + E#|<<>> E E e?D)T): ; ;<<r8c|D],}t|tr||-t|tr||jdSdS)zc (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesr0rLrrr)rrs r9init_adapter_layersz+Wav2Vec2PreTrainedModel.init_adapter_layerss} llnn + +F&":;; +""6*** dN + + -   t| , , , , , - -r8 target_langc x|jjtd|d||jkr"|s td|ddS|dd}|dd}|d d}|d d}|d d}|d d} |d d} |jj} d} | dur{t |} t| | ||||| |}t|} n9#t$r| rYn*t$r| rtd| d| d| dYnwxYw| t |} t| | ||||| |}tt!j|dd} n;#t$rt$rt$rtd| d| d| dwxYw|t'| t'z }t't'| z }t+|dkr)td|dd|dt+|dkr)td|dd|d| djd}||jjkr=t3j|jj||j|j|_||j_fd| D} | | d||_dS)a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to r cache_dirforce_downloadFproxieslocal_files_onlytokenrevisionuse_safetensors)filenamerrrrrrzCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named cpuT) map_location weights_onlyrzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightrhrPcNi|]!\}}|||"Sr7)rg)rLrr`rs r9 z8Wav2Vec2PreTrainedModel.load_adapter..s0QQQdaaoa011QQQr8)strict)!rrJrRrloggerwarningpop _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileOSError ExceptionWAV2VEC2_ADAPTER_PT_FILErr3loadrsetkeysr_rr: vocab_sizerrrrhrPritemsload_state_dict)rr force_loadrrrrrrrrmodel_path_or_id state_dictfilepath weight_pathunexpected_keys missing_keystarget_vocab_sizers @r9 load_adapterz$Wav2Vec2PreTrainedModel.load_adapter+sx ; ' /r rrrss s $* * *: * NNOOOO P P P FJJ{D11 $4e<<**Y--!::&8%@@ 7D))::j$// **%6==;4  % ' '188EEH )$%#1#%5%'    ,K88    "    "!J5EJJ=MJJ?GJJJ   /66{CCH" )$%#1#%5%'    )***"Z!&!%          F1AFF9IFF;CFFF ,,..joo//0037K7K7M7M3N3NN?//1122S9J9J5K5KK   ! # #tKttW[W`W`apWqWqtttuu u    " "nKnnTXT]T]^jTkTknnnoo o''78>qA  6 6 69 .0A$+]a]gDL&7DK "RQQQj>N>N>P>PQQQ  Z666's$&D<< E2 $E21E2NQT>Tcgjncn,Y]%(:?:J( - - -m'm'm'm'm'm'm'm'm'm'r8rceZdZdeffd ZdZ ddejdejdzdejdzfdZ e dd ej dzdej dzdejdzd e dzd e dzd e dzd e ezfdZxZS) Wav2Vec2Modelrct|||_t||_t ||_|jdks |jdkrBtj tj |j |_|jrt#||_nt'||_|jrt+|nd|_|dSNr)rrrrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrr3r(rrmasked_spec_embeddo_stable_layer_normrvencoderrPrradapter post_initrs r9rzWav2Vec2Model.__init__s     !7!?!?";F"C"C  3 & &&*BS*H*H%'\%,v?Q2R2R2[2[2]2]%^%^D "  & 39&AADLL*622DL282DNv...$  r8c8|jdSz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)r,rrs r9freeze_feature_encoderz$Wav2Vec2Model.freeze_feature_encoders 1133333r8Nr+rsr=ct|jdds|S|\}}}|#|j|j||<n|jjdkr|jr|t||f|jj|jj ||jj }tj ||j tj}|j|j||<|jjdkr|jrt||f|jj|jj|jj}tj ||j tj}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)r;r<r=r>r)r;r<r>rJ)rKrrur0rgrPr.rrpmask_time_lengthmask_time_min_masksr3rtrhr\r/mask_feature_lengthmask_feature_min_masksr)rr+rsr=rhrHrmask_feature_indicess r9_mask_hidden_statesz!Wav2Vec2Model._mask_hidden_statesst{$8$?? ! 4A3E3E3G3G0 O[  (/3/E/H/HI\/]/]M+ , , [ '! + + + 5_-+4 K8-+9 !!! !& ->}G[chcm n n n /3/E/H/HI\/]/]M+ , ; (1 , , ,#8[)+7 K;+< $$$ $)<0D]Mainis#t#t#t #74#@#G#GO]_#`#` 23M. /r8rrrXrYr?c :||n |jj}||n |jj}||n |jj}||}|dd}|#||jd|d}||\} }| | ||} | | ||||} | d} |j | | } |s| |f| ddzSt| || j | j S) a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nr!r#Fr)rsr=r=rrXrYr)rdextract_featuresr+r,)rrrXrYr,rrr:r-r@r2r3rr+r,) rrr=rsrrXrYrrCr+encoder_outputss r9rzWav2Vec2Model.forward&s| 2C1N--TXT_Tq$8$D $+Jj &1%uEN+/*A*ABR*S*S' '00 ->~1  ,, )/!5# '  (* < # LL77M K!#34qrr7JJ J&+-)7&1     r8r]NNNNN)r/r0r1r"rr8r3r4r'r@rr(r\r6rrrrs@r9r)r)sF~(4447;26 ,,(,!,t3,(4/ ,,,,\/36:)-,0#'8 8 lT)8  t+8 !,t3 8  $; 8 #Tk 8 D[8  ( (8 8 8 ^8 8 8 8 8 r8r)z? Wav2Vec2 Model with a quantizer and `VQ` head on top. ceZdZdeffd ZdefdZdZe dde j de j d e j de fd Z e dd e jd zd e jd zde jd zde jd zded zded zded zdeezfdZxZS)rrct|t||_t j|j|_t||_ t j |j |j |_ t j |j|j |_|dSr)rrr)rrrfeat_quantizer_dropoutdropout_featuresr| quantizerrrproj_codevector_dimrrrr4rs r9rzWav2Vec2ForPreTraining.__init__hs    %f-- " 6+H I I6v>>9V%79STT6#8&:TUU r8rc||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rJr)rrs r9set_gumbel_temperaturez-Wav2Vec2ForPreTraining.set_gumbel_temperatureus&1"""r8cB|jjdSr6rr,rr7s r9r8z-Wav2Vec2ForPreTraining.freeze_feature_encoder{! '::<<<<>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r=rrXrsrYrr!Fr)rsrJr#rz-infirX) reduction)r'r(r)r*r+r,r-r.)$rrYrgr3r\rrrIrr:rJrrrPrrpermuterYcontrastive_logits_temperatureallanyr'rrfrurrr cross_entropyrr~rXdiversity_loss_weightr&r+r,)rrr=rsryrrXrYrrFtransformer_featuresrCquantized_featuresr*r'r-r.rhrHrnegative_quantized_featuresrX neg_is_postargetnum_codevectorss r9rzWav2Vec2ForPreTraining.forwardsF&1%uEN59NN 0A5C5 5 110224>3H3NOO!^^,>??3777. # /7I7O 4J +=*A*A"k*R*R(--//44R88+ '+F*J*JOR++gaAq!! ( 44"47++$ : F-0KKPPQSTTJ~~ 7).vqrr :&%%a++33B AGGF,11333t;FFq!LLTTVVF!}::6<<>>6]b:cc "kCdkFggO.1FF/Y]n]r]r]t]ttN$dk&G.&XXD c24FH]^ahijikikalll(*<>STW^_`_a_aWbb b+1'9"7!/)-)    r8)rQ)NNNNNN)r/r0r1r"rrCrMr8rr3r4r'rYrr( BoolTensorr\r6r&rrrs@r9rrbs ~      1#1111 === ! * ,"- \(/359<@)-,0#'] ] lT)]  t+] !+d2 ] #("2T"9 ]  $; ] #Tk] D[]  - -] ] ] ^] ] ] ] ] r8rzp Wav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ceZdZddedzffd ZdZdZdZe dde j dzde j dzd e dzd e dzd e dzd e j dzd e e zfdZxZS)rNrct|t||_t j|j|_||_|j td|j dt|dr|j r|jn|j}t j||j |_|dS)a2 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Wav2Vec2ForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)rrr)rrr final_dropoutrrrrRrrrrrrrr4)rrrrrs r9rzWav2Vec2ForCTC.__init__>s    %f-- z&"677 &   $HHHH *1)G)G vFL^ vF % %djdv y!3V5FGG  r8c TttjdkrdS|j}|)t |jddt d|d|2t |jddtddS|| |ddSdS) a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNrJzCannot pass `target_lang`: rz)By default `target_lang` is set to 'eng'.T)r) rr3rhrrKrrRrinfor)rrrs r9 tie_weightszWav2Vec2ForCTC.tie_weights[s 6 7 75<;O;O O O F &  "wt{||jjkrt d|jj||||||}|d} || } || } d} |Z||ntj |tj }| | d tj } |dk} | d}|| }tj| dtjdd}tjjd 5tj||| ||jj|jj|jj } dddn #1swxYwY|s| f|t6dz}| | f|zn|St9| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rBrrOrJ)rrPr!F)enabled)blankr[ zero_infinityr'rXr+r,)rrYrDrrRrrrr3 ones_likerrrXrg masked_selectrr log_softmaxfloat32rbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr+r,)rrr=rrXrYrvrrFr+rXr'ri labels_masktarget_lengthsflattened_targets log_probsoutputs r9rzWav2Vec2ForCTC.forwards$&1% 1` a classification loss is computed (Cross-Entropy). NTrBr!rrJrr#rr{)rrYrrrr3stackrrrrrrXrrrr:rerfrrrrr+r,)rrr=rrXrYrvrrFr+ norm_weights pooled_output padding_maskexpand_padding_maskrXr'loss_fctrs r9rz)Wav2Vec2ForSequenceClassification.forwardsW0&1%->r1a-H-HHMMRSMTTMM#AJM}55  !)..1.55MMBB=CVWXCY[ijjL"."8"8"<"<"C"CAq-J]^_J`"a"a 25M.. /)--!-44|7G7GA7G7N7N7S7STVXY7Z7ZZM//  '))H8FKKDK,BCCV[[QS__UUD FY)F)G)G!HHF)-)9TGf$$v E'!/)     r8rE)r/r0r1rr8rtrr3r(r\r6rrrrs@r9rrs"===(((/3)-,0#'&*C C lT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C C ^C C C C C r8rceZdZfdZdZdZe d dejdzdejdzdejdzde dzd e dzd e dzd e e zfd Z xZ S)#Wav2Vec2ForAudioFrameClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_|j|_|dS)NrzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r!)rrrrrRr)rrVrrrr3rarrrrrr4rs r9rz,Wav2Vec2ForAudioFrameClassification.__init__;s    6= ) ) f.@ t &f-- -1  ( S!#ej.D.Dz.Q!R!RD )F$68IJJ + r8cB|jjdSr6rOr7s r9r8z:Wav2Vec2ForAudioFrameClassification.freeze_feature_encoderKrPr8cL|jD] }d|_ dSrrrsrs r9rtz5Wav2Vec2ForAudioFrameClassification.freeze_base_modelRrur8Nrr=rvrrXrYr?c ||n |jj}|jjrdn|}||||||}|jjrx|t} t j| d} tj |j d} | | dddz d} n|d} | | } d} |`t} | | d|jt j| d|jd} |s| f|tdz}|St#| | |j|j S) rNTrBr!rrJr)axisr{)rrYrrrr3rrrrrrrXrrrrrr+r,)rrr=rvrrXrYrrFr+rrXr'rrs r9rz+Wav2Vec2ForAudioFrameClassification.forwardZs0&1%->r1a-H-HHMMRSMTTMM#AJM//  '))H8FKKDO<t>t>tuuD Y)F)G)G!HHFM$!/)     r8rE)r/r0r1rr8rtrr3r(r\r6rrrrs@r9rr9s ===(((/3&*)-,0#': : lT):  t+:  t# :  $; : #Tk : D[:  & &: : : ^: : : : : r8rc&eZdZdfd ZdZxZS) AMSoftmaxLoss>@皙?ct||_||_||_t jtj||d|_ t j |_ dS)NT)r) rrscalemarginrrrr3randnrrr')rrrrrrs r9rzAMSoftmaxLoss.__init__se   $l5;y*#E#EUYZZZ ')) r8c|}tj|jd}tj|d}t j||}||jz }tj||j }|j t j | ||z}| ||}|S)Nrrr!)rrr normalizerr3mmrone_hotrrrr\r') rr+rvr cos_thetapsionehotrXr's r9rzAMSoftmaxLoss.forwards!!((!(<< // 1/EE H]F33 $+%&&vt??ek&++--iHHHyy(( r8)rrrrs@r9rrsL******       r8rcDeZdZdfd ZdejdejfdZxZS) TDNNLayerrct|dkr|j|dz n |j||_|j||_|j||_|j||_tj |j|jz|j|_ tj |_ dS)Nrr!)rrtdnn_dimrr tdnn_kernelr tdnn_dilationdilationrrkernelrrrs r9rzTDNNLayer.__init__s .s#QQQy++QQQr8rJr#)rrr)rrVrrrr3rarrrrrrZr_rtdnnxvector_output_dimr,rrr objectiver4)rrr tdnn_layersrs ` r9rzWav2Vec2ForXVector.__init__s'    %f-- -1  ( S!#ej.D.Dz.Q!R!RD 6#5vq7IJJQQQQU3v;O;O5P5PQQQ M+.. !#6?2+>+BFD]!^!^)F$=v?XYY&v'@&BSTT r8cB|jjdSr6rOr7s r9r8z)Wav2Vec2ForXVector.freeze_feature_encoderrPr8cL|jD] }d|_ dSrrrsrs r9rtz$Wav2Vec2ForXVector.freeze_base_modelrur8ricDd}|jjD]}|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nr!r7rs r9rzEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengths!;.69A= =r8r!)rr)rrirrs r9_get_tdnn_output_lengthsz+Wav2Vec2ForXVector._get_tdnn_output_lengthssE  > > >  ;2 L LK,,]KKKMMr8Nrr=rrXrYrvr?c >||n |jj}|jjrdn|}||||||}|jjrx|t} t j| d} tj |j d} | | dddz d} n|d} | | } |jD] } | | } |-| d} | d} n|| d}||}g} g} t'|D]k\}}| | |d|fd| | |d|fdlt j| } t j| } t j| | gd}||}||}d}||||}|s||f|tdz}||f|zn|St3||||j|jS) rNTrBr!rrJr)r'rX embeddingsr+r,)rrYrrrr3rrrrrrrXrrrrrr enumeratercrVr,rrrr+r,)rrr=rrXrYrvrrFr+r tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsrXr'rs r9rzWav2Vec2ForXVector.forwards0&1%->r1a-H-HHMMRSMTTMM#AJM}55 ) 6 6J&J}55MM  !)..1.55M(,,,33LL*.*O*OP^PbPbghPbPiPi*j*j '"&"?"?@["\"\ ML&':;; J J 6$$]1gvg:%>%C%C%C%J%JKKK##M!WfW*$=$A$Aa$A$H$HIIII!K 66M ;|44L!I}l&CLLL 223DEE!233  >>&&11D F/07;X;Y;Y3ZZF)-)9TGf$$v E(!/)     r8rE)r/r0r1rr8rtr3r'rCrrr(r\r6rrrrs@r9rrs) &===((( e6F6L    /3)-,0#'&*P P lT)P  t+P  $; P #Tk P D[ P  t#P   P P P ^P P P P P r8r)rrrrrr)rrrr+)br2rrcollections.abcr dataclassesrnumpyrSr3safetensors.torchrr rtorch.nnrr r activationsr integrations.deepspeedr integrations.fsdpr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrrprocessing_utilsrrrrrrrrr configuration_wav2vec2r"rr  get_loggerr/rrr&r6rCr'r'ndarrayrprrrrModulerrrrr(r r r*r9rHrPrvr|rrrLrr)rrrrrrr__all__r7r8r9rsY  $$$$$$!!!!!! 999999%%%%%%&&&&&&!!!!!!@@@@@@777777666666BBBBBB999999srrrrrrrrr&&&&&&322222,5  H % %!"  44444;44   4B/3 tt c?ttt$t+ t  t Z ttttn$$U$3$[][ehl[l$$$$D#=*!;6!;0*****bi***Z29%%%%%RY%%%P 1 1 1 1 1 1 1 1,!%% I% <% % < % L4' % T\ %%'(%%%%8R/R/R/R/R/ R/R/R/j")0!!!!!5!!!H+++++*D+++\E E E E E biE E E PI I I I I RYI I I XI'I'I'I'I'BII'I'I'Xbi>29$ry2\'\'\'\'\'o\'\'\'~C C C C C +C C C L N N N N N 4N N  N b K K K K K ,K K  K \ e e e e e (?e e  e P[ [ [ [ [ *A[ [ [ |BI. @ C C C C C 0C C  C L   r8