r functionalgelupermuteembed_positionsdropoutr@ enumeratelayers layer_normr ) selfinput_featuresrAr9expected_seq_length inputs_embeds embed_posr8idx encoder_layer layer_outputss r&forwardVoxtralEncoder.forwardCs&#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :LM`Laamn|oCoCDFoGnHHuvIuJJKL (**1B1B1H1HQUQ[Q[QbQbQiQi*j **4::n+EF  **4::m+DE %--aA6 ((// &266}7J7JK  --m||VZVcVc-d "+DKK"8 C)- $M *!,M #9 6 +  r%rN) r r!r"r#rr)_can_record_outputsrr rr]r$rr%r&r5r58s;', - +, - - r%r5c6^\rSrSrS\4U4SjjrSrSrU=r$)VoxtralMultiModalProjectortrDc^>[TU]5 [R"URR UR RSS9Ul[URUl [R"UR RUR RSS9Ul g)NF)bias) super__init__rLinear audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rUrD __class__s r&rg#VoxtralMultiModalProjector.__init__usz  &"5"5"G"GI[I[IgIgnst &556 &"4"4"@"@&BTBTB`B`glm r%clURU5nURU5nURU5nU$r_)rmrorp)rUaudio_featuresr8s r&r]"VoxtralMultiModalProjector.forward{s2 n5 /  m4 r%)rormrp) r r!r"r#rrgr]r$ __classcell__rrs@r&rbrbtsn}n r%rbzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. c^\rSrSrS/rSS0rSS/S/40rS/rU4SjrS r S r S r S r S r SrS\R 4Sjr\\SS\\R*S\\R S\\R,S\\R*S\\S\\R S\\R*S\\S\\R*S\\\R,4S\\S\4Sjj55rU4SjrSr U=r!$) VoxtralForConditionalGenerationzlm_head.weightlm_head colwise_repr8logitsrPc.>[TU]U5 URRUl[R "UR 5Ul[R "UR5Ul [U5Ul UR5 gr_) rfrgrk vocab_sizer from_configri audio_towerrlanguage_modelrbmulti_modal_projector post_initrqs r&rg(VoxtralForConditionalGeneration.__init__sn   ,,77$001D1DE2>>v?Q?QR%?%G" r%c6URR5$r_)rget_input_embeddingsrUs r&r4VoxtralForConditionalGeneration.get_input_embeddingss""7799r%c:URRU5 gr_)rset_input_embeddings)rUvalues r&r4VoxtralForConditionalGeneration.set_input_embeddingss 007r%c6URR5$r_)rget_output_embeddingsrs r&r5VoxtralForConditionalGeneration.get_output_embeddingss""88::r%c:URRU5 gr_)rset_output_embeddings)rUnew_embeddingss r&r5VoxtralForConditionalGeneration.set_output_embeddingss 11.Ar%c:URRU5 gr_)r set_decoder)rUdecoders r&r+VoxtralForConditionalGeneration.set_decoders ''0r%c6URR5$r_)r get_decoderrs r&r+VoxtralForConditionalGeneration.get_decoders""..00r%rVcURU5nURnURSURRR 5nUR U5nU$)a( This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector. Args: input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] Returns: `torch.FloatTensor`: The audio embeddings. r;)rrCreshaperDrirjr)rUrV audio_outputsaudio_hidden_states audio_embedss r&get_audio_embeds0VoxtralForConditionalGeneration.get_audio_embedssZ((8 +==199"dkk>V>V>h>hi112EF r% input_idsrA position_idspast_key_valuesrXlabels use_cachecache_positionlogits_to_keepr9returnc UcUR5"U5nUb-URU5n XRR:Hn XU 'UR"SUUUUUUU U S.U D6nU$)a Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```)rArrrXrrrrr)rrrDaudio_token_idr)rUrrVrArrrXrrrrr9raudio_token_maskoutputss r&r]'VoxtralForConditionalGeneration.forwardsb   557 BM  %00@L )KK,F,FF .:* ++/+>+> , )%+')) ,  , r%c>URSS5nURS5n[TU] "U0UD6nUb USS:XaX5S'U$)NrVrr)popgetrfprepare_inputs_for_generation)rUargsr9rVr model_inputsrrs r&r=VoxtralForConditionalGeneration.prepare_inputs_for_generationsZ $4d;$45wD!!1!12Dc5<</0D+,D DDL  r%rz)r,r5rz))typingrrrr activationsr cache_utilsr generationr modeling_outputsr r r processing_utilsr utilsrrr utils.genericrautorr qwen2_audio.modeling_qwen2_audiorrrrconfiguration_voxtralrrr)r,r5Modulerbrz__all__rr%r&rs # ! )``&II/2 1 *  0 6 4 &4  4 n   K&