tjddlmZddlZddlZddlmZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&e$j'e(Z)Gddej*Z+Gddej*Z,GddeZ-GddeZ.GddeZ/Gddej*Z0Gddej*Z1 dJd!ej*d"ej2d#ej2d$ej2d%ej2dzd&e3dzd'e3d(e e"fd)Z4Gd*d+ej*Z5Gd,d-ej*Z6Gd.d/eZ7Gd0d1ej*Z8Gd2d3ej*Z9Gd4d5eZ:Gd6d7ej*Z;e#Gd8d9eZ< dKd:e=e>e>fd;e3dd%ej?dzd=e>d>ej@f d?ZAe#Gd@dAe<ZBdZCe#dBCGdDdEe<ZDe#dFCGdGdHe<ZEgdIZFdS)L)CallableN)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack)TransformersKwargsauto_docstringlogging) HubertConfigc$eZdZfdZdZxZS)HubertPositionalConvEmbeddingc:ttj|j|j|j|jdz|j|_d|_t|ddr tj |j|_nWtj j }ttj jdrtj jj }trddl}|j|jjd5||jdd |_dddn #1swxYwYt|jd r-|jjjj}|jjjj}n|jj}|jj}|j|||j||n||jdd |_t3|j|_t6|j|_dS) N) kernel_sizepaddinggroupsconv_pos_batch_normF weight_normr modifier_rankweight)namedimparametrizations)super__init__nnConv1d hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsconv batch_normgetattr BatchNorm1dutilsr!hasattrr'r deepspeedzeroGatheredParametersr$ original0 original1weight_gweight_vregister_external_parameterHubertSamePadLayerrrfeat_extract_activation activation)selfconfigr!r5r:r; __class__s h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/hubert/modeling_hubert.pyr)z&HubertPositionalConvEmbedding.__init__.s1 I    62a77      60% 8 8 I nV-?@@DOO(.Krx0-@@ D h7C )++ I    ^66ty7GWX6YYMM + DIH! L L LDIMMMMMMMMMMMMMMM49&8992#y9@JH#y9@JHH#y1H#y1H::4JJJ::4JJJJ'K aHHH )&*HII  !?@sD77D;>D;c|dd}|j||}||}||}||}|dd}|S)Nrr) transposer0r/rr?r@ hidden_statess rCforwardz%HubertPositionalConvEmbedding.forwardSs~%//155 ? & OOM::M -00  ]33  66 %//155 __name__ __module__ __qualname__r)rH __classcell__rBs@rCrr-sM#A#A#A#A#AJ       rIrc$eZdZfdZdZxZS)r=clt|dzdkrdnd|_dS)Nrrr)r(r)num_pad_remove)r@r-rBs rCr)zHubertSamePadLayer.__init__`s: #:Q#>!#C#CaarIcJ|jdkr|ddddd|j f}|SNr)rRrFs rCrHzHubertSamePadLayer.forwardds;   " ")!!!QQQ0F43F2F0F*FGMrIrJrOs@rCr=r=_sLKKKKKrIr=c&eZdZdfd ZdZxZS)HubertNoLayerNormConvLayerrcZt|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t|j |_ dS)Nrrrstridebias)r(r)conv_dim in_conv_dim out_conv_dimr*r+ conv_kernel conv_stride conv_biasr/rr>r?r@rAlayer_idrBs rCr)z#HubertNoLayerNormConvLayer.__init__ks r?ras rCr)z!HubertLayerNormConvLayer.__init__s r? GroupNormrkras rCr)z!HubertGroupNormConvLayer.__init__s z1HubertFeatureEncoder.__init__..s>LLLGH*6AEBBBLLLrIrlayerc2g|]}t|S)r{)rgr}s rCrz1HubertFeatureEncoder.__init__..s'wwwA3FQGGGwwwrIz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) r(r)feat_extract_normrprangenum_feat_extract_layers ValueErrorr* ModuleList conv_layersgradient_checkpointing_requires_grad)r@rArrBs ` rCr)zHubertFeatureEncoder.__init__s   #w . .3FQGGGHLLLLLQRXRpstRtLuLuLLLKK % 0 0wwwwQVW]WuQvQvwwwKKt1Ittt =55&+#"rIcP|D] }d|_ d|_dSNF) parameters requires_gradrr@params rC_freeze_parametersz'HubertFeatureEncoder._freeze_parameterss4__&& ( (E"'E  #rIcr|dddf}|jr|jrd|_|jD] }||}|S)NT)rtrainingrr)r@ input_valuesrG conv_layers rCrHzHubertFeatureEncoder.forwards[$QQQW-    /4= /*.M '* 6 6J&J}55MMrI)rKrLrM__doc__r)rrHrNrOs@rCrxrxs\88#####"$$$       rIrxc$eZdZfdZdZxZS)HubertFeatureProjectioncTt|j|_|jr+tj|jd|j|_tj|jd|j |_ tj |j |_ dS)Nrneps)r(r)feat_proj_layer_normr*rjr[layer_norm_epsrkLinearr, projectionDropoutfeat_proj_dropoutdropoutr@rArBs rCr)z HubertFeatureProjection.__init__s $*$?!  $ [ l6?2+>FDYZZZDO)FOB$79KLLz&":;; rIc|jr||}||}||}|Srd)rrkrrrFs rCrHzHubertFeatureProjection.forwardsF  $ ; OOM::M 66  ]33 rIrJrOs@rCrrsG<<<<<rIrmodulequerykeyvalueattention_maskscalingrkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nrnrrr&)prr) sizetorchmatmulrEr* functionalsoftmaxrr contiguous) rrrrrrrr attn_weights attn_outputs rCeager_attention_forwardrs**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$rIceZdZdZ ddededed ed ed ed edzffd Z dde j de j dzde j dzdedzde e de e j e j dze e j dzff dZxZS)HubertAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderrZ is_causalrAc t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)rZ)r(r)rrrhead_dimrArrrrr*rk_projv_projq_projout_proj) r@rrrrrZrrArBs rCr)zHubertAttention.__init__ s "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB rIrGkey_value_statesroutput_attentionsrreturnc |du}|jdd}g|d|jR}|||dd} |r|n|} g| jddd|jR} || | dd} || | dd} tj|j j t}||| | | |f|j sdn|j |j|d|\}}|jg|dR}||}||dfS)z#Input shape: Batch x Time x ChannelNrnrrr)rrr)shaperrviewrErrr get_interfacerA_attn_implementationrrrrreshaperr)r@rGrrrris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs rCrHzHubertAttention.forward(s.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT -?R))]B^)#2#.BBDMBB[[0055h??II!QOO {{>2277AAKKAqQQ (?(M K ,.E) ) %8$7      % $}>CC$,L/ % %  % % ! \*k);;;;;;FFHH mmK00 L$..rI)rFTFN)NNF)rKrLrMrintfloatboolrr)rTensorrr tuplerHrNrOs@rCrrsJGG  &*CCCC C  C  CCt#CCCCCCD15.2). 0/0/|0/ ,-0/ t+ 0/  $; 0/-.0/ u|U\D0% 2E2LL M0/0/0/0/0/0/0/0/rIrc$eZdZfdZdZxZS)HubertFeedForwardcttj|j|_tj|j|j|_ t|j trt|j |_n |j |_tj|j|j|_tj|j|_dSrd)r(r)r*ractivation_dropoutintermediate_dropoutrr,intermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrs rCr)zHubertFeedForward.__init__\s $&Jv/H$I$I!"$)F,>@X"Y"Y f' - - 9'-f.?'@D $ $'-'8D $If&>@RSS j)>??rIc||}||}||}||}||}|Srd)rrrrrrFs rCrHzHubertFeedForward.forwardisg// >> 00?? 11-@@ ))-88 ++M:: rIrJrOs@rCrr[sL @ @ @ @ @rIrc&eZdZfdZddZxZS)HubertEncoderLayerctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_dS)NFrrrrrAr)r(r)rr,num_attention_headsattention_dropout attentionr*rrrrjrrkr feed_forwardfinal_layer_normrs rCr)zHubertEncoderLayer.__init__ts ((0,    z&"788 ,v'9v?TUUU-f55 " V-?VEZ [ [ [rINFc|}||||\}}}||}||z}||}|||z}||}|f}|r||fz }|SNrr)rrrkrrr@rGrr attn_residualr_outputss rCrHzHubertEncoderLayer.forwards% )- .L]*8* * & |Q ]33 % 5  66 %(9(9-(H(HH --m<<  "  '   &GrIrrJrOs@rCrrssQ \ \ \ \ \rIrc ^eZdZfdZ d dejdejdzdeded ef d ZxZ S) HubertEncoderct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_dS)Nrc.g|]}tS)rr~rrAs rCrz*HubertEncoder.__init__..s"$i$i$iA%7%?%?$i$i$irIFr(r)rArpos_conv_embedr*rjr,rrkrrrrrnum_hidden_layerslayersrrs `rCr)zHubertEncoder.__init__s  ;FCC,v'9v?TUUUz&"788 m$i$i$i$ivOgIhIh$i$i$ijj &+###rINFTrGrroutput_hidden_states return_dictc|rdnd}|rdnd}|;|ddd|jd}d||<t|j||}||} || |jz}||}| |}tpt|} |j D]a} |r||fz}tjg} |jo| |jjk} | r| r| |||}|d}| rd}|r ||dfz}b|r||fz}|st#d |||fDSt%||| S) NrrnrrrrA inputs_embedsrrNNc3K|]}||V dSrdrr~vs rC z(HubertEncoder.forward..(mmq_`_l_l_l_l_lmmrIlast_hidden_staterG attentions) unsqueezerepeatrr rArtodevicerkrrr rrrandr layerdroprr r@rGrrrrall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss rCrHzHubertEncoder.forwards #7@BBD$5?bb4  %$2$<$(>}?S(T(TT  66  ]33 022R6LT6R6R [ P PE# I$58H$H!#(*R.. !]Z/BT[EZ/ZN! 1[ 1 %!.Te!!! !.a 0  - ,   P&9]1=MBB gii  $.$/BB rIrGc||}||}||}||}|Srd)r+r,r.r/rFs rCrHzHubertAttnAdapterLayer.forwardsL -00  m44  M22  m44 rI)rKrLrMr)r FloatTensorrHrNrOs@rCr&r&s[ C C C C CU%6rIr&cReZdZfdZ ddejdejdzdefdZxZS) !HubertEncoderLayerStableLayerNormctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_t#|ddt%||_dSd|_dS)NFrrr()r(r)rr,rrrr*rrrrjrrkrrrr1r& adapter_layerrs rCr)z*HubertEncoderLayerStableLayerNorm.__init__s ((0,    z&"788 ,v'9v?TUUU-f55 " V-?VEZ [ [ [ 6-t 4 4 @!7!?!?D   !%D   rINFrGrrcJ|}||}||||\}}}||}||z}||||z}|j|||z}|f}|r||fz }|Sr)rkrrrrr5rs rCrHz)HubertEncoderLayerStableLayerNorm.forward s &  66 )- .L]*8* * & |Q ]33 % 5 %(9(9$:O:OP]:^:^(_(__   ))D,>,>},M,MMM "  '   &GrIr) rKrLrMr)rrrrHrNrOs@rCr3r3s~&&&&&,/3"' | t+ rIr3c.eZdZfdZ ddZxZS)HubertEncoderStableLayerNormct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_dS)Nrc.g|]}tSr)r3rs rCrz9HubertEncoderStableLayerNorm.__init__...s" ` ` `1 .v 6 6 ` ` `rIFrrs `rCr)z%HubertEncoderStableLayerNorm.__init__'s  ;FCC,v'9v?TUUUz&"788 m ` ` ` `fF^@_@_ ` ` `   ',###rINFTc|rdnd}|rdnd}|;|ddd|jd}d||<t|j||}||} || z}||}tpt|} |j D]a} |r||fz}tj g} |j o| |jj k} | r| r| |||}|d}| rd}|r ||dfz}b||}|r||fz}|std |||fDSt!||| S) Nrrnrrrr rr c3K|]}||V dSrdrr s rCrz7HubertEncoderStableLayerNorm.forward..jrrIr)rrrr rArrrr rrrrrrkrr rs rCrHz$HubertEncoderStableLayerNorm.forward2s#7@BBD$5?bb4  %$2$<$>CC+DDD input_lengthsczd}t|jj|jjD]\}}||||}|S)zH Computes the output length of the convolutional layers c<tj||z |ddzS)Nfloor) rounding_moder)rdiv) input_lengthrrYs rC_conv_out_lengthzPHubertPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths&9\K7wWWWZ[[ [rI)ziprAr^r_)r@rMrTrrYs rC _get_feat_extract_output_lengthsz6HubertPreTrainedModel._get_feat_extract_output_lengthss\  \ \ \ $'t{'> @W#X#X Q Q K,,]KPPMMrIfeature_vector_lengthrc||dtj}|jd}tj||f|j|j}d|tj |jd|j|dz f<| dg d dg }|S)Nrnr)dtyperr)r) rVsumrrlongrzerosrYrarangeflipcumsumr)r@rWroutput_lengths batch_sizes rC"_get_feature_vector_attention_maskz8HubertPreTrainedModel._get_feature_vector_attention_masks>>~?Q?QRT?U?UVVYYZ_Zdee#)!,  . /~7KTbTi   uv ^%9!%<^EZ[[[]kno]opq',,bT2299"==BBB4HHMMOOrI)rKrLrMr__annotations__base_model_prefixmain_input_nameinput_modalities_no_split_modulessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrno_gradrE LongTensorrrVrbrNrOs@rCr>r>rs  $O-/CD&*#NU]__````_`2 e>NQT>T      ]b]m        rIr>r mask_prob mask_length min_masksrc@|\}dkrtdkrtdddtjdfd}|9|dnfd t|D}tj |ft }g} |} | d kr|S|D]} || } tj tj | dz z | d } t| d krdz }n| d }tj| tj| | z tj |zg} | | tj| } tj| dddddf|| f} | || z} tj ddddf}tj||| f|| z}| |z} | dz krdz | | dz k<tj|| dd|S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rSnum_masked_spanepsilonrornrpsequence_lengths rCcompute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spans~i,6DwNOOoy99 [ (? : :-.s 9 9 9!o 9 9 9rIrYrF)replace)rnprandomritemdetachrZtolistrr\rchoicer]len concatenateonesint32appendarray broadcast_torrtput_along_axis)rrnrorrprarxrM spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrSruspec_aug_mask_idxdummy_mask_idxoffsetsrvrws `` ` @@rC_compute_mask_indicesrsP0#(JQABBB_$$ :^i : :'6 : : :   innQ$$&&G         $  % ##B''..000 9 9 9 9uZ'8'8 9 9 9Hj/:$GGGM11/BBa%55 11,??I,, IlkAo6 7 7RW-    ! !Q & &-q0NN.q1NN (;o(MUWU] ^ ^ ^ao o p   !!"34444"455111aaa:&5H+(V,33J@SVa@abbi $$T4]3Gog 4G'UVV^^'+5G,g5/A"555GVYZGZ-!0CCDm%7B??? rIceZdZdeffd Z ddejdejdzdejdzfdZe ddej dzdej dzdejdzd e dzd e dzd e dzd e e zfd ZxZS)rIrAct|||_t||_t ||_|jdks |jdkrBtj tj |j |_|jrt#||_nt'||_|dSNr)r(r)rArxfeature_extractorrfeature_projectionmask_time_probmask_feature_probr* Parameterrrr,rJrCdo_stable_layer_normr8encoderr post_initrs rCr)zHubertModel.__init__-s     !5f!=!="9&"A"A  3 & &&*BS*H*H%'\%,v?Q2R2R2[2[2]2]%^%^D "  & 17??DLL(00DL rINrGmask_time_indicesrct|jdds|S|\}}}|#|j|j||<n|jjdkr|jr|t||f|jj|jj ||jj }tj ||j tj}|j|j||<|jjdkr|jrt||f|jj|jj|jj}tj ||j tj}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)rnrorrp)rrY)rnrorprn)r1rArrCrrYrrrmask_time_lengthmask_time_min_masksrr$rrrmask_feature_lengthmask_feature_min_masksexpand)r@rGrrrarwr,mask_feature_indicess rC_mask_hidden_stateszHubertModel._mask_hidden_states?st{$8$?? ! 4A3E3E3G3G0 O[  (/3/E/H/HI\/]/]M+ , , [ '! + + + 5_-+4 K8-+9 !!! !& ->}G[chcm n n n /3/E/H/HI\/]/]M+ , ; (1 , , ,#8[)+7 K;+< $$$ $)<0D]Mainis#t#t#t #74#@#G#GO]_#`#` 23M. /rIrrrrrc ||n |jj}||n |jj}||n |jj}||}|dd}|!||jd|}||} | | |} | | ||||} | d} |s| f| ddzSt| | j | j S)a1 mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Example: ```python >>> from transformers import AutoProcessor, HubertModel >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft") >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft") >>> def map_to_array(example): ... example["speech"] = example["audio"]["array"] ... return example >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.map(map_to_array) >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values # Batch size 1 >>> hidden_states = model(input_values).last_hidden_state ```Nrr)rrrrrrr)rArrrrrErbrrrrr rGr) r@rrrrrrrextract_featuresrGencoder_outputss rCrHzHubertModel.forwardms@J2C1N--TXT_Tq$8$D $+Jj &1%.safetensors or adapter..bin. Only relevant when using an instance of [`HubertForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `HubertForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r(r)rIr?r*r final_dropoutrr vocab_sizerrBr4routput_hidden_sizer,rlm_headr)r@rArrrBs rCr)zHubertForCTC.__init__s    !&)) z&"677 &   $HHHH *1)G)G vFL^ vF % %djdv y!3V5FGG  rIc TttjdkrdS|j}|)t |jddt d|d|2t |jddtddS|| |ddSdS) a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNr(zCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load) rrrrr1rArloggerinfo load_adapter)r@rrs rC tie_weightszHubertForCTC.tie_weightss 6 7 75<;O;O O O F &  "wt{||jjkrt d|jj||||||}|d} || } || } d} |Z||ntj |tj }| | d tj } |dk} | d}|| }tj| dtjdd}tjjd 5tj||| ||jj|jj|jj } dddn #1swxYwY|s| f|t6dz}| | f|zn|St9| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrrzrn)r&rYrF)enabled)blank reduction zero_infinitylosslogitsrGr)rArrtrrr?rrr ones_liker[rVrZr masked_selectr*r log_softmaxfloat32rEbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrrGr)r@rrrrrrrrrGrrrM labels_masktarget_lengthsflattened_targets log_probsoutputs rCrHzHubertForCTC.forwards$&1% 1` a classification loss is computed (Cross-Entropy). NTrrrrnrrrr)rArrr?rrstackr*rrrDrrZrmeanrbrrrrrrrrGr)r@rrrrrrrrrG norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrs rCrHz'HubertForSequenceClassification.forwardssW0&1%->r1a-H-HHMMRSMTTMM#AJM}55  !)..1.55MMBB=CVWXCY[ijjL"."8"8"<"<"C"CAq-J]^_J`"a"a 25M.. /)--!-44|7G7GA7G7N7N7S7STVXY7Z7ZZM//  '))H8FKKDK,BCCV[[QS__UUD FY)F)G)G!HHF)-)9TGf$$v E'!/)     rIr)rKrLrMr)rrrrrrrrrHrNrOs@rCrKrKLs";;;(((/3)-,0#'&*C C lT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C C ^C C C C C rIrK)rrKrIr>rrT)Gcollections.abcrnumpyr|rtorch.nnr*rrrF activationsrintegrations.deepspeedrintegrations.fsdpr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr rrmodeling_utilsrrrprocessing_utilsrr3rrrconfiguration_hubertr get_loggerrKrModulerr=rVrgrprxrrrrrrrrr&r3r8r>rrrmndarrayrrIrrrK__all__rrIrCrs$*%$$$$$ %%%%%%&&&&&&!!!!!!@@@@@@777777666666BBBBBB999999YYYYYYYYYYrrrrrrrrrr&&&&&&@@@@@@@@@@......  H % %/////BI///d!;*9690#####29###Lbi0!%% I% <% % < % L4' % T\ %%'(%%%%8R/R/R/R/R/biR/R/R/j 0!!!!!3!!!HE E E E E BIE E E PRY2+++++(B+++\I I I I I 29I I I X>>>>>O>>>J/3 tt c?ttt$t+ t  t Z ttttnG G G G G 'G G G T!" K K K K K (K K  K \ e e e e e &;e e  e P f e erI