#Q1o " o5M&NN ==((2(>L#nnQAq&AA ==((6??([L,,|3K''1-88:K  $$c`^\rSrSrSrSS\S\S\S\S\S\S \\S \\ 4U4S jjjr S \ RS \S\4Sjr SS\ RS\\ RS\\ RS\S\\ R\\ R\\\ R44 SjjrSrU=r$)VoxtralAttentionJz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr# is_decoderbias is_causal layer_idxconfigc `>[T U]5 XlX lX0lX-UlXlUR U-UR:wa[SURSUS35eUR S-UlX@l X`l Uc4U(a-[RSURRS35 Xpl[ R""XSS9Ul[ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ulg) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r'zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrD)super__init__rArBr#head_dimrG ValueErrorr"rCrElogger warning_once __class____name__rFrLineark_projv_projq_projout_proj) selfrArBr#rCrDrErFrGrPs r;rKVoxtralAttention.__init__Ms "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"     *4>>+B+B*CD,,  #ii 5A ii 4@ ii 4@  )TB r=tensorseq_lenbszcURX2URUR5RSS5R 5$)Nrr)r6rBrLr1r7)rWrYrZr[s r;_shapeVoxtralAttention._shapeus5{{3GQQRSUVWbbddr= hidden_statesr!layer_head_maskoutput_attentionsreturnc PUR5upgnURURU5UR-Xv5n URUR U5SU5n URUR U5SU5n [ n URRS:wa[URRn U "UU U U U4UR(dSO URSUUS.UD6upU RXgS5R5n URU 5n X4$)z#Input shape: Batch x Time x Channelr&eager?)r#r"rar$)r.r]rUr"rSrTr<rG_attn_implementationrr-r#reshaper7rV)rWr_r!r`rar8r[tgt_len_ query_states key_states value_statesattention_interfacer:r9s r;forwardVoxtralAttention.forwardxs(,,.a{{4;;}#= #Lg[ [[]!;RE {{4;;}#=r3G (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,/% %  % ! "))#;FFH mmK0 ((r=) rGr#rArLrErCrSrFrBrVrUr"rT)reFTFNN)NNF)rQ __module__ __qualname____firstlineno____doc__intfloatboolrrrKr/Tensorr]tuplero__static_attributes__ __classcell__rPs@r;r?r?Js*G  #'*.&C&C&C &C  &C  &C&CC=&C'&C&CPeU\\eCece 2626"' ))||))!.))"%,,/ ))  )) u||Xell3XeELL>Q5RR S))))r=r?c ^\rSrSrS\4U4SjjrS S\RS\RS\RS\S\R4 S jjr S r U=r $) VoxtralEncoderLayerrGcj>[TU]5 URUl[ URUR UR US9Ul[R"UR5Ul URUl [URUlURUl[R "URUR"5Ul[R "UR"UR5Ul[R"UR5Ulg)N)rArBr#rG)rJrKd_modelrAr?encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr#ractivation_function activation_fnactivation_dropoutrRencoder_ffn_dimfc1fc2final_layer_normrWrGrPs r;rKVoxtralEncoderLayer.__init__s )nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r=r_r!r`rarbcUnURU5nURUUUUS9up[RR XRUR S9nXQ-nUnUR U5nURURU55n[RR XRUR S9nURU5n[RR XRUR S9nXQ-nUR[R:XaC[R"UR5RS- n[R "X*US9nX4$)aW Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r_r!r`rar+i)minmax)rrrr4r#r-rrrrrdtyper/float16finforclamp)rWr_r!r`raresidualr9 clamp_values r;roVoxtralEncoderLayer.forwardsC$!11-@ &*nn')+/ '5' #  --m||VZVcVc-d  0  --m< **488M+BC  --m?V?Vaeanan-o /  --m||VZVcVc-d  0   %-- /++m&9&9:>>EK!KK <[YM**r=) rrr#rArrrrr)F) rQrqrrrsrrKr/rxrwrorzr{r|s@r;r~r~s^=}=.#( )+||)+ )+ )+  )+  )+)+r=r~cN\rSrSr%\\S'SrSrSrSr Sr Sr Sr Sr SrSrSrSrg) VoxtralPreTrainedModelrGmodelTNpast_key_valuesc[URS5(aURROURRRn[ U[ R [ R45(aWURRRSUS9 URb%URRR5 gg[ U[ R5(aJURRRS5 URRR5 g[ U[ R5(adURRRSUS9 UR b2URRUR R5 ggg)Ninitializer_rangere)meanstdrf)hasattrrGr audio_config isinstancerrRConv1dweightdatanormal_rDzero_rfill_ Embedding padding_idx)rWrrs r; _init_weights$VoxtralPreTrainedModel._init_weightss? t{{$788 KK ) )));; fryy"))4 5 5 MM   & &CS & 9{{&   &&('  - - MM   $ $S ) KK   " " $  - - MM   & &CS & 9!!- ""6#5#56<<>..r=)rQrqrrrsr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraphrrzrr=r;rrsH &*#"3N "&!?r=rz: The Voxtral encoder, which is a Whisper encoder. ) custom_introc^\rSrSr%Sr\\S'SrS/r\ \ S.r S\4U4Sjjr Sr S \R4S jrS \R4S jr\SS \\4Sjj5rS\R.4SjrSrU=r$)VoxtralEncoderiz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`VoxtralEncoderLayer`]. Args: config: VoxtralEncoderConfig rGinput_featuresr~) attentionsr_c>[TU]U5 URUlURUlUR nUR UlURUlURUl UR(a[R"U5OSUl [R"UR USSS9Ul[R"X"SSSS9Ul[R$"URU5UlUR&R)S5 [R*"[-UR.5Vs/sHn[1U5PM sn5Ul[R4"UR 5Ul[R8"SSS9UlSUlUR?5 gs snf) Nrfrr) kernel_sizepaddingr)rstriderF)r) rJrKr#encoder_layerdrop layerdropr num_mel_bins pad_token_idrmax_source_positionsscale_embeddingmathsqrt embed_scalerrconv1conv2rembed_positionsrequires_grad_ ModuleListrangeencoder_layersr~layersr layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rWrGrArjrPs r;rKVoxtralEncoder.__init__sH  ~~ 11NN "//!..$*$?$?!393I3I499Y/sYYt00)TUV YYy1VWX !||D,E,EyQ ++E2mm%PVPePeJf$gJfQ%8%@Jf$gh ,,v~~6,,q3&+# %hs GcNUR5H nSUlM SUlg)NF) parameters requires_grad_requires_grad)rWparams r;_freeze_parameters!VoxtralEncoder._freeze_parameters8s#__&E"'E '#r=rbcUR$NrrWs r;get_input_embeddings#VoxtralEncoder.get_input_embeddings=s zzr=r cXlgrrrWr s r;set_input_embeddings#VoxtralEncoder.set_input_embeddings@s r=r8c URRURRS-URRS-nUR SU:wa"[ SUSUR SSUS35eURURRRURRRS9n[RRURU55n[RRUR U55nURSSS 5nURRnXV-RUR5n[RR!XpR UR"S 9n[%UR&5HupU "UUS S 9n U SnM UR)U5n[+US 9$)a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rr&z:Qwen2Audio expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .)rdevicerrr+N)r!r`)last_hidden_state)rGrrrrr3rMtorrrrr4gelupermuterr#r- enumeraterrr ) rWrr!r8expected_seq_length inputs_embeds embed_posr_idx encoder_layer layer_outputss r;roVoxtralEncoder.forwardCs&#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :LM`Laamn|oCoCDFoGnHHuvIuJJKL (**1B1B1H1HQUQ[Q[QbQbQiQi*j **4::n+EF  **4::m+DE %--aA6 ((// &266}7J7JK  --m||VZVcVc-d "+DKK"8 C)- $M *!,M #9 6 +  r= input_lengthsc4US- S-S-nUS- S-S-nX4$)zc Computes the output length of the convolutional layers and the output length of the audio encoder rrr)rWroutput_lengthss r; _get_feat_extract_output_lengths/VoxtralEncoder._get_feat_extract_output_lengthsts5'*q014 '!+1A5,,r=)rrrrr#rrrrrrrrrr)rQrqrrrsrtrrmain_input_namerr?r~_can_record_outputsrKrrModulerrrrrror/ LongTensorrrzr{r|s@r;rrs  ! &O./&, 34$ bii"))- +, - - `-e>N>N--r=rc6^\rSrSrS\4U4SjjrSrSrU=r$)VoxtralMultiModalProjectori}rGc^>[TU]5 [R"URR UR RSS9Ul[URUl [R"UR RUR RSS9Ul g)NFrI) rJrKrrRrintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rs r;rK#VoxtralMultiModalProjector.__init__~sz  &"5"5"G"GI[I[IgIgnst &556 &"4"4"@"@&BTBTB`B`glm r=clURU5nURU5nURU5nU$r)r r r)rWaudio_featuresr_s r;ro"VoxtralMultiModalProjector.forwards2 n5 /  m4 r=)r r r) rQrqrrrsrrKrorzr{r|s@r;rr}sn}n r=rzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. c^\rSrSrS/rSS0rSS/S/40rS/rU4SjrS r S r S r S r S r SrS\R 4Sjr\\SS\\R*S\\R S\\R,S\\R*S\\S\\R S\\R*S\\S\\R*S\\\R,4S\\S\4Sjj55rU4SjrSr U=r!$) VoxtralForConditionalGenerationizlm_head.weightlm_head colwise_repr_logitsrc.>[TU]U5 URRUl[R "UR 5Ul[R "UR5Ul [U5Ul UR5 gr) rJrKr  vocab_sizer from_configr audio_towerrlanguage_modelrmulti_modal_projectorrrs r;rK(VoxtralForConditionalGeneration.__init__sn   ,,77$001D1DE2>>v?Q?QR%?%G" r=c6URR5$r)rrrs r;r4VoxtralForConditionalGeneration.get_input_embeddingss""7799r=c:URRU5 gr)rrrs r;r4VoxtralForConditionalGeneration.set_input_embeddingss 007r=c6URR5$r)rget_output_embeddingsrs r;r$5VoxtralForConditionalGeneration.get_output_embeddingss""88::r=c:URRU5 gr)rset_output_embeddings)rWnew_embeddingss r;r'5VoxtralForConditionalGeneration.set_output_embeddingss 11.Ar=c:URRU5 gr)r set_decoder)rWdecoders r;r++VoxtralForConditionalGeneration.set_decoders ''0r=c6URR5$r)r get_decoderrs r;r/+VoxtralForConditionalGeneration.get_decoders""..00r=rcURU5nURnURSURRR 5nUR U5nU$)a( This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector. Args: input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] Returns: `torch.FloatTensor`: The audio embeddings. r&)rrrhrGrrr)rWr audio_outputsaudio_hidden_states audio_embedss r;get_audio_embeds0VoxtralForConditionalGeneration.get_audio_embedssZ((8 +==199"dkk>V>V>h>hi112EF r= input_idsr! position_idsrrlabels use_cachecache_positionlogits_to_keepr8rbc UcUR5"U5nUb-URU5n XRR:Hn XU 'UR"SUUUUUUU U S.U D6nU$)a Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```)r!r8rrr9r:r;r<r)rr5rGaudio_token_idr)rWr7rr!r8rrr9r:r;r<r8r4audio_token_maskoutputss r;ro'VoxtralForConditionalGeneration.forwardsb   557 BM  %00@L )KK,F,FF .:* ++/+>+> , )%+')) ,  , r=c>URSS5nURS5n[TU] "U0UD6nUb USS:XaX5S'U$)Nrr;r)popgetrJprepare_inputs_for_generation)rWargsr8rr; model_inputsrPs r;rE=VoxtralForConditionalGeneration.prepare_inputs_for_generationsZ $4d;$45wD!!1!12Dc5<</0D+,D DDL  r=r)rrr)NreN)3rtypingrrrr/r activationsr cache_utilsr generationr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrautorrconfiguration_voxtralrr get_loggerrQrNrrxrvr<r?r~rrrr__all__rr=r;r\s|, ,, ! )9``F&RR/2F   H % $(,% II% <<% % << % U\\* % e_ %% %%