tjddlmZddlmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z mZmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZmZm Z ddl!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5ddl6m7Z7m8Z8m9Z9ddl:m;Z;mZ?e*deGddeZ@e*deGdd eZAGd!d"ejBZCGd#d$ejBZDGd%d&e4ZEGd'd(e3ZFGd)d*e7ZGGd+d,eZHe*Gd-d.e%ZIGd/d0eIZJGd1d2e8ZKGd3d4e;ZLe*d5Gd6d7eIeZMgd8ZNdS)9)Callable) dataclassN)strict)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)PreTrainedConfig)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPast)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)RopeParameters)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs) GlmAttentionGlmRotaryEmbeddingapply_rotary_pos_emb)LlamaDecoderLayer LlamaModeleager_attention_forward) WhisperModelshift_tokens_rightzUsefulSensors/moonshine-tiny) checkpointceZdZUdZdZdgZdddddZd Zee d <d Z ee d <d Z ee d<dZ ee d<dZ ee d<dZee d<dZee d<dZedze d<dZedze d<dZedze d<dZee d<dZee d<dZee d<dZee d<dZee d<dZee d <dZeezdze d!<dZee d"<d#Z ee d$<d%Z!eeze d&<dZ"edze d'<d(Z#ee$ezdze d)<dZ%edze d*<dZ&ee d+<fd,Z'xZ(S)-MoonshineConfiga encoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `encoder_num_key_value_heads=encoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `encoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. decoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `decoder_num_key_value_heads=decoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `decoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `decoder_num_attention_heads`. pad_head_dim_to_multiple_of (`int`, *optional*): Pad head dimension in encoder and decoder to the next multiple of this value. Necessary for using certain optimized attention implementations. encoder_hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder. decoder_hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. Example: ```python >>> from transformers import MoonshineModel, MoonshineConfig >>> # Initializing a Moonshine style configuration >>> configuration = MoonshineConfig().from_pretrained("UsefulSensors/moonshine-tiny") >>> # Initializing a model from the configuration >>> model = MoonshineModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` moonshinepast_key_valuesdecoder_num_key_value_headsdecoder_num_attention_headsdecoder_num_hidden_layersdecoder_hidden_act)num_key_value_headsnum_attention_headsnum_hidden_layers hidden_acti vocab_sizei  hidden_sizeiintermediate_sizeencoder_num_hidden_layersencoder_num_attention_headsNencoder_num_key_value_headspad_head_dim_to_multiple_ofgeluencoder_hidden_actsiluimax_position_embeddingsg{Gz?initializer_rangedecoder_start_token_idT use_cacherope_parametersis_encoder_decoderFattention_biasattention_dropout bos_token_idr" eos_token_id pad_token_idtie_word_embeddingsc |j |j|_|j |j|_|ddt jdi|dS)Npartial_rotary_factorg?)r?r>r0r1 setdefaultsuper __post_init__)selfkwargs __class__s m/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/moonshine/modular_moonshine.pyrWzMoonshineConfig.__post_init__se  + 3/3/OD ,  + 3/3/OD ,13777'''''''))__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inference attribute_mapr8int__annotations__r9r:r<r2r>r1r?r0r@rBstrr3rDrEfloatrGrHboolrIrdictrJrKrMrNrOlistrPrQrW __classcell__rZs@r[r-r-3s.& & PJ#4"5<<8* MJK!s!!!%&s&&&%&s&&&'(((('((((.2t222.2t222.2t222$$$$$$$$#&S&&&#u###"#C###It48O^d*T1888#### ND   %(us{((( L#*   +,L#S /D(,,,#L#*### $$$$(((((((((r\r-z Extends [~modeling_outputs.BaseModelOutput] to include the output attention mask since sequence length is not preserved in the model's forward. ) custom_introc2eZdZUdZdZejdzed<dS)MoonshineEncoderModelOutputa attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Nattention_mask)r]r^r_r`rptorchTensorrerTr\r[roros5+/NEL4'.....r\rocBeZdZfdZdejdejfdZxZS)MoonshineEncoderMLPc t||_t||_t j|j|j|_ t j|j|j|_ dSN rV__init__configr activation_fnnnLinearr9r:fc1fc2rXryr7rZs r[rxzMoonshineEncoderMLP.__init__sc  #J/9V/1IJJ9V5v7IJJr\ hidden_statesreturnc||}||}||}|Srv)r}rzr~)rXrs r[forwardzMoonshineEncoderMLP.forwards=// **=99 // r\r]r^r_rxrqrrrrkrls@r[rtrtscKKKKKU\elr\rtcBeZdZfdZdejdejfdZxZS)MoonshineDecoderMLPct||_t||_t j|j|jdz|_ t j|j|j|_ dS)Nr"rwrs r[rxzMoonshineDecoderMLP.__init__sh  #J/9V/1IA1MNN9V5v7IJJr\rrc||}|dd\}}|||z}||}|S)Nr")dim)r}chunkrzr~)rXrgates r[rzMoonshineDecoderMLP.forwards_// +11!1<< t**400=@ // r\rrls@r[rrscKKKKKU\elr\rceZdZdS)MoonshineRotaryEmbeddingN)r]r^r_rTr\r[rrsDr\rceZdZdededededef fd Z ddejd e ejejfdzd ejdzd e dzd ejdzd e e de ejejdze ejdzffdZ xZS)MoonshineAttentionry layer_idx is_causalr5r4cV|||dt||||_t |d|j|jz|_|jj 0|jj }||j|zdz |zz}||jz |_ dSd|_ dS)N)r5r4head_dimrFr) updaterVrxrgetattrr9r5rryr@head_dim_padding) rXryrrr5r4target_multipletarget_head_dimrZs r[rxzMoonshineAttention.__init__s  .AZmnnooo +++" F4F&Jd4dee  ; 2 >"kEO-$-/2QTU2UZi1ijO$3dm$CD ! ! !$%D ! ! !r\Nrposition_embeddingsrpr/key_value_statesrYrc N|jdd\}}|||||jj|jdd} |du} |?|j|j } | rd|j|j <|j }n|j }||n|} | r3|r1| r/|j |j j } |j |j j}n|| |d|jj|jdd} || |d|jj|jdd}| r!||| ||j \} }| s;|\}}t%| | ||\} } ||| ||j \} }t'j|jjt,}|jo |duo|dk}|jdkrt2jj| d|jf} t2jj| d|jf} t2jj|d|jf}||| | ||f|jsdn|j|j|d|\}}|jdkr|dd|j f}| ||d!}|"|}||fS) NrrFr"TrrL)dropoutscalingr.)#shapeq_projviewryr4r transpose is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesk_projv_projrr%r get_interface_attn_implementationr(rrrqr{ functionalpadtrainingrMrreshape contiguouso_proj)rXrrrpr/rrYbszq_len query_statesis_cross_attentionrcurrent_states key_states value_statescossinattention_interfacer attn_output attn_weightss r[rzMoonshineAttention.forwardsm#("- U KK & & + +C 8WY]Yf g g q qrsuv w w .T9  &(377GGJ! G=A*4>:"1"G"1"F.>-I))}  l/ lj l(/?DJ*1$.AHLL N++c2t{> NN1a   N++c2t{> NN1a  " lo&A+:+A+A*l\`\j+k+k( L! l*HC';L*VY[^'_'_ $L**+:+A+A*l\`\j+k+k( L(?(M K ,.E) ) NK~'=K%!)  1 $ $ 8.22EO) t+ O)  O)  ,- O)-.O) u|U\D0% 2E2LL MO)O)O)O)O)O)O)O)r\rc(eZdZdedeffd ZxZS)MoonshineEncoderLayerryrcFt||t||d|j|j|_t ||j|_tj |j d|_ tj |j d|_ dS)NFryrrr5r4bias)rVrxrr>r? self_attnrtrBmlpr{ LayerNormr9input_layernormpost_attention_layernormrXryrrZs r[rxzMoonshineEncoderLayer.__init__*s ++++ & B & B    'vv/HII!|F,>UKKK(* V5Ge(T(T(T%%%r\)r]r^r_r-rdrxrkrls@r[rr)sW U U3 U U U U U U U U U Ur\rceZdZddededzffd Z ddejdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzde ejejfdzde ejejfdzde e de eje ejejfdzffdZxZS)MoonshineDecoderLayerNryrct|j|_t||d|j|j|_t||d|j|j|_t||j |_ tj |jd|_ tj |jd|_tj |jd|_dS)NTrFr)rVrxr9rr5r4r encoder_attnrr7rr{rrrfinal_layernormrs r[rxzMoonshineDecoderLayer.__init__;s !-+ & : & :    / & : & :    'vv/@AA!|F,>UKKK(* V5Ge(T(T(T%!|F,>UKKKr\Frrpencoder_hidden_statesencoder_attention_mask position_idsencoder_position_idsr/rHrencoder_position_embeddingsrYrc D|} ||}|jd|||||| d| \}} | |z}|9|} ||}||||||\}} | |z}|} ||}||}| |z}|S)N)rrprr/rHr)rrrpr/rHrT)rrrrrr)rXrrprrrrr/rHrrrYresidual_s r[rzMoonshineDecoderLayer.forwardSs!,,];; )4> ')%+ 3      q!=0 ,$H 99-HHM#00+!65 /# 1   M1%}4M ,,];; //  =0 r\rv) NNNNNNFNN)r]r^r_r-rdrxrqrr LongTensorr rhrrr FloatTensorrrkrls@r[rr:sLLL3:LLLLLL6/3596:048<(,!&HLPT,,|, t+, %|d2 , !& t 3 , &- ,$.5,,$;,#5<#=>E,&+5<+E%F%M,+,, u %(95;L(L"MPT"TT U,,,,,,,,r\rcTeZdZUeed<dZdZdZdZddgZ dZ dZ dZ de jfd Zd S) MoonshinePreTrainedModelrymodel input_valuesaudioTrr input_lengthsct|dz dz dz}t|dz dz dz}t|dz dz dz}|S)zH Computes the output length of the convolutional layers @rFrr")rd)rXroutput_conv1_lengthoutput_conv2_lengthoutput_conv3_lengths r[ _get_feat_extract_output_lengthsz9MoonshinePreTrainedModel._get_feat_extract_output_lengthssc"=3#6""N######r\rc eZdZdZdZeedZdeffd Z de j fdZ de j fd Z ee ddejd ejd zd eedeezfd ZxZS)MoonshineEncoderz Transformer encoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoonshineEncoderLayer`] Args: config: MoonshineConfig r) attentionsrryc`t|_j}t jd|ddd|_t j|d|zdd |_t jd|z|dd |_t j d|d |_ t j fd tj D|_t j|d |_t#|_d|_|dS)NrFrrF) kernel_sizestriderr"rr)rrgh㈵>) num_groups num_channelsepsc0g|]}t|SrT)r.0idxrys r[ z-MoonshineEncoder.__init__..s$ c c cC "63 / / c c cr\rry)rVrxryr9r{Conv1dconv1conv2conv3 GroupNorm groupnorm ModuleListranger<rr layer_normr rotary_embgradient_checkpointing post_init)rXry embed_dimrZs ` r[rxzMoonshineEncoder.__init__s     & Yq)ReTTT Yy!i-QqQQQ Yq9}iQqQQQ PTUUUm c c c c5Aa;b;b c c c   ,yu===2&AAA&+# r\rc|jSrvrrXs r[get_input_embeddingsz%MoonshineEncoder.get_input_embeddingss zr\valuec||_dSrvr)rXrs r[set_input_embeddingsz%MoonshineEncoder.set_input_embeddingss  r\NrprYc |d}tj||}||}tj||}tj||}| ddd}d}|;| |j d}d}|ddd|fdd|f}|}t|j |||}tjd|j d|j d}||| } |jD]} | |f||| d |}||}t)|||nd S) a. Args: input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `input_values`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) rFrr"Nri.ry inputs_embedsrprdevicer)rprr)last_hidden_staterp) unsqueezer{rtanhrrrArrpermuterrrryrqarangerr rr rord) rXrrprYroutput_attention_maskmask_lendownsample_striderr encoder_layers r[rzMoonshineEncoder.forwards.$--a00  **4::l+C+CDD }55  **4::m+D+DEE  **4::m+D+DEE %--aA66 !%  %<<^=QRT=UVVH * +C1D1D3D1D,DEc9H9nUN$2 !2;')"/    |A}':1'=mFZ[[[eefghh "oom,oWW![  M)M-)$7   MM 66 *+:O:[044666ae    r\rv)r]r^r_r`rrr_can_record_outputsr-rxr{Modulerrrr!rqrrrrrrrrrkrls@r[rrs%O(. $bi") /3< < '<  t+< +, < ( ( < < < _ < < < < < r\rcDeZdZdZeeddeeedddZdeffd Z e e dde j d zd e jd zd e j d zd ed zd e jd zded zde jd zde jd zdeedeezfdZxZS)MoonshineDecoder input_idsrFr)index layer_namer)rrcross_attentionsrycttjjd|_tjfdtjD|_ dS)NFrc0g|]}t|SrT)rrs r[rz-MoonshineDecoder.__init__.. s$$s$s$sC%:63%G%G$s$s$sr\) rVrxr{rr9normrrr6rrXryrZs `r[rxzMoonshineDecoder.__init__ sl    L!3%@@@ m$s$s$s$sSXY_YqSrSr$s$s$stt r\Nrprr/rrHrrrYrc |du|duz rtd|||}|r8|6tt|jt|j}|V||nd} t j|jd|j | z}| d}t|j||||} t|j|||}|} | | | } |jD]}|| | |f||||| d | } || } t!| |r|nd S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `encoder_hidden_states`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embedsrrrFr)ryrrpr/rrr)rrr/rHr)rr/) ValueError embed_tokensr r ryget_seq_lengthrqrrrrrrr rr.r)rXr(rprr/rrHrrrYpast_seen_tokens causal_maskrr decoder_layers r[rzMoonshineDecoder.forwards0  -t"; < [YZZ Z   --i88M  v01,dk2R2R2RT`hlhsTtTtTtuuO  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    ";;'1"7 " " " & "oom,oWW![  M)M% (>) /#$7    MM -00 8+/8BOOd    r\)NNNNNNNN)r]r^r_rr rrr$r-rxrr!rqrrrr rrhrrrrrrkrls@r[r'r's!O$n%7q[YYY.*N+=QSabbb uuuuuuu  .2.204(,26!%:>6:G G #d*G  t+G &- G  G (4/ G $;G  %047G !& t 3G +,G  ( (G G G _ G G G G G r\r'c4eZdZdZee ddejdzdejdzdejdzdejdzde e ejdzde dzd e ejdzd e ejdzd e dzd e e d efdZdS)MoonshineModelc td)NzNot needed for Moonshine)AttributeErrorrs r[_mask_input_featuresz#MoonshineModel._mask_input_features\s7888r\Nrrpdecoder_input_idsdecoder_attention_maskencoder_outputsr/decoder_inputs_embedsdecoder_position_idsrHrYrc ||j|fd|i| }|jd|||j|j|||| d| } t | j| j| j| j| j|j|j|jS)a  input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, MoonshineModel >>> from datasets import load_dataset >>> model = MoonshineModel.from_pretrained("UsefulSensors/moonshine-tiny") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_values, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 288] ``` Nrp)r(rprrr/rrrH)rr/decoder_hidden_statesdecoder_attentionsr+encoder_last_hidden_staterencoder_attentionsrT) encoderdecoderrrprr/rrr+) rXrrpr<r=r>r/r?r@rHrYdecoder_outputss r[rzMoonshineModel.forward_sZ  "/;t|L/r/rYg/rkq/r/rOEQT\ F '1"1"C#2#A+/- F F  F F "-?+;"1"?.9,=&5&G"1"?.9    r\) NNNNNNNNN)r]r^r_r;rrrqrrrr rhrrrrrTr\r[r8r8[sI999262659:>BF6:AE?C!%C C '$.C (4/C !+d2 C !& 04 7 C uU%6784? C -t3C  %U%67$>C $E$45<C $;C +,C  C C C ^C C C r\r8zj The Moonshine Model with a language modeling head. Can be used for automatic speech recognition. ceZdZddiZdeffd ZdZdZdej fdZ e e dd e jd zd e jd zd e jd zd e jd zdeee jd zded zdee jd zdee jd zded zde jd zdeedefdZxZS)!MoonshineForConditionalGenerationzproj_out.weightz!model.decoder.embed_tokens.weightryct|t||_t j|j|jd|_| dS)NFr) rVrxr8rr{r|r9r8proj_outr r/s r[rxz*MoonshineForConditionalGeneration.__init__s`    #F++  &"4f6GeTTT  r\c|jSrvrLrs r[get_output_embeddingsz7MoonshineForConditionalGeneration.get_output_embeddingss }r\c||_dSrvrN)rXnew_embeddingss r[set_output_embeddingsz7MoonshineForConditionalGeneration.set_output_embeddingss & r\rc4|jSrv)rrrs r[rz6MoonshineForConditionalGeneration.get_input_embeddingssz..000r\Nrrpr<r=r>r/r?r@rHlabelsrYc | )|'|%t| |jj|jj}|j|f|||||||| d| } || j} d}| Kt}|| d|jj | d}t|| | j | j | j | j| j| j| j S)a0 input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoProcessor, MoonshineForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-tiny") >>> model = MoonshineForConditionalGeneration.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> generated_ids = model.generate(input_values, max_new_tokens=100) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```N)rpr<r>r=r/r?r@rHr) losslogitsr/rBrCr+rDrrE)r*ryrPrGrrLrrrr8rr/rBrCr+rDrrE)rXrrpr<r=r>r/r?r@rHrTrYoutputsrWrVloss_fcts r[rz)MoonshineForConditionalGeneration.forwards d   (-B-J$6DK4dk6X%%!'1dj  ' )/+#9+"7!5 ' '  ' ' w899  '))H8FNN2t{/EFFWYHZHZ[[D#3")"?&9$5&-&G")"?&9    r\) NNNNNNNNNN)r]r^r__tied_weights_keysr-rxrOrRr{r%rrrrqrrrr rhrrrrrkrls@r[rJrJs ,-PQ'''1bi1111262659:>BF6:AE?C!%*.S S '$.S (4/S !+d2 S !& 04 7 S uU%6784? S -t3S  %U%67$>S $E$45<S $;S  4'S +,S  S S S ^S S S S S r\rJ)r-r8rrJ)Ocollections.abcr dataclassesrrqtorch.nnr{huggingface_hub.dataclassesrr activationsr cache_utilsr r r configuration_utilsr generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingr r!glm.modeling_glmr#r$r%llama.modeling_llamar&r'r(whisper.modeling_whisperr)r* get_loggerr]loggerr-ror%rtrrrrrrrr'r8rJ__all__rTr\r[rssE%$$$$$!!!!!! ......%%%%%%!!!!!!CCCCCCCCCC333333))))))JJJJJJJJBBBBBB999999211111FFFFFFFF&&&&&&RRRRRRRRRRRR777777EEEEEEEEUUUUUUUUUUYYYYYYYYYYGGGGGGGG  H % %9:::S(S(S(S(S(&S(S(;:S(l   / / / / // / /   /     ")        ")         1   e)e)e)e)e)e)e)e)PUUUUU-UUU"EEEEE6EEEP########0d d d d d /d d d NV V V V V zV V V rI I I I I \I I I X i i i i i (@/i i  i X   r\