tj7ddlmZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z dd l mZdd lmZmZmZmZdd lmZdd lmZd dlmZmZd dlmZmZmZmZd dl m!Z!m"Z"ddl#m$Z$ej%e&Z'Gdde!Z(Gdde"Z)GddeZ*eGddeZ+edeGdde Z,edGd d!eZ-Gd"d#eZ.ed$Gd%d&eZ/ed'Gd(d)eZ0gd*Z1dS)+) dataclassN)nn)ACT2FN)Cache)create_bidirectional_mask)BaseModelOutputWithPooling ModelOutput)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs)Qwen2AudioEncoderQwen2AudioPreTrainedModel)VoxtralForConditionalGeneration VoxtralModelVoxtralModelOutputWithPastVoxtralMultiModalProjector)WhisperAttentionWhisperEncoderLayer)AudioFlamingo3ConfigceZdZdS)AudioFlamingo3AttentionN__name__ __module__ __qualname__w/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/audioflamingo3/modular_audioflamingo3.pyrr.Dr$rceZdZdS)AudioFlamingo3EncoderLayerNrr#r$r%r(r(2r&r$r(ceZdZdZdS)AudioFlamingo3PreTrainedModelTN)r r!r"_supports_attention_backendr#r$r%r*r*6s"&r$r*ceZdZdS)!AudioFlamingo3ModelOutputWithPastNrr#r$r%r-r-:sDr$r-zZ Base class for AudioFlamingo3 causal language model (or autoregressive) outputs.  custom_introceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZejdzed<dS) $AudioFlamingo3CausalLMOutputWithPastaV loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. audio_hidden_states (`torch.FloatTensor`, *optional*): Hidden states of the audio encoder after projection. Nlosslogitspast_key_values hidden_states attentionsaudio_hidden_states)r r!r"__doc__r2torch FloatTensor__annotations__r3r4rr5tupler6r7r#r$r%r1r1?s  &*D% d ")))'+FE  $+++$(OUT\(((59M5*+d299926Je'(4/66648*T188888r$r1zT The audio model from AudioFlamingo3 without any head or projection on top. c teZdZdZeedZee dde j de j dzde e zfdZ dS) AudioFlamingo3EncoderzY AudioFlamingo3 encoder: Whisper encoder, average pool (time/2), then LayerNorm. )r5r6Ninput_featuresinput_features_maskreturnc |jddz dzdz}|d}|dz dzdz}tj||j|dddfk}t j||}t j| |}| ddd}||j j z}t j ||j |j}t|j||}|jD]4} |jotjg|jk} | s | ||}5| ddd}|| ddd}||}t-| S) ap Args: input_features (`torch.FloatTensor` of shape `(batch_size, feature_size, sequence_length)`): Log-Mel features extracted from raw audio. Use the processor/feature extractor to compute and pad these features from waveform input. input_features_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. rrdeviceNr)ptraining)config inputs_embedsattention_mask)last_hidden_state)shapesumr9arangerEr functionalgeluconv1conv2permuteembed_positionsweightdropoutrGrrHlayersrand layerdrop avg_pooler layer_normr ) selfr?r@kwargsseq_leninput_features_lengthsrIr5rJlayerdrops r%forwardzAudioFlamingo3Encoder.forwardhs("'+a/A59!4!8!8!;PQQQTjklklklnrkrTss **4::n+E+EFF  **4::m+D+DEE %--aA66 &(<(CC  --mt|VZVc-dd 2;'.   [ E EE=DUZ^^dn%DD E %m^ D D &--aA66  66>>q!QGG  66 )+    r$N)r r!r"r8r(r_can_record_outputsrrr9Tensorr<r rbr#r$r%r>r>Ys  4-  483 3  3 #\D03 + + 3 3 3 _ 3 3 3 r$r>c(eZdZdZdeffd ZxZS)!AudioFlamingo3MultiModalProjectorz Audio adaptor (small MLP) that projects AudioFlamingo3Encoder features to the LLM embedding space so they can replace `` tokens. rHcJttj|jj|jj|j|_t|j |_ tj|jj|jj|j|_ dS)N)bias) super__init__rLinear audio_config hidden_size text_configprojector_biaslinear_1rprojector_hidden_actactlinear_2r\rH __class__s r%rkz*AudioFlamingo3MultiModalProjector.__init__s    +V-?-KRXRg    &56   *F,>,JQWQf    r$)r r!r"r8rrk __classcell__rvs@r%rgrgsO  3          r$rgz The AudioFlamingo3 model (fine-tuned Whisper encoder, multi-modal projector, Qwen2 language model), without a language modeling head. creZdZdZdZdZfdZeedde j de j de e deezfd Zee dd e jdzde j dzde j dzd e j dzd e jdzd edzde j dzdedzde e fdZxZS)AudioFlamingo3ModelNcJt|dSrc)rjrkrus r%rkzAudioFlamingo3Model.__init__s!      r$zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.r.r?r@r]rAc |j|f|dd|}||j}|dt j}|j|\}}t j|j d|j dddf|dddfk} || |j |_ |S)a1 input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padded feature indices. T)r@ return_dictrCrrDN) audio_towermulti_modal_projectorrKrMtor9long _get_feat_extract_output_lengthsrNrLrE pooler_output) r\r?r@r] audio_output audio_embeds input_lengths_ post_lengths valid_masks r%get_audio_featuresz&AudioFlamingo3Model.get_audio_featuress"(t'  0CQU  Y_  11,2PQQ ,//3366uzBB *KKMZZ<\,"4Q"7 @STTTUY[\[\[\U\]`lmnmnmnptmt`uu %1*-- @S2T2T%U "r$ input_idsrJ position_idsr4rI use_cachec |||}d} |e|c|||dj} |||| } || | |j}|jd|||||d| } t| j | j | j | j | S)z input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. NT)r})rIaudio_features)rIrJrr4r)rKr4r5r6r7r#) get_input_embeddingsrrget_placeholder_maskmasked_scatterrrElanguage_modelr-rKr4r5r6) r\rr?r@rJrr4rIrr]rspecial_audio_maskoutputss r%rbzAudioFlamingo3Model.forwards $  7D5577 BBM  %)*?22>CVdh2iiwL"&!:!:|";"" *889K\__]j]qMrMrssM%$% ')%+      1%7#3!/) ,     r$)NNNNNNNN)r r!r"_tp_plan_pp_plan_keep_in_fp32_modules_strictrkrr r9r:rer r r<r r LongTensorrboolrbrwrxs@r%rzrzsHH#' !!!!!^w)#\+,  + + 4.23737.204(,26!%, , #d*, )D0, #\D0 ,  t+ , &- , , (4/, $;, +,, , , ^, , , , , r$rzz The AudioFlamingo3 model which consists of a fine-tuned Whisper encoder, a multi-modal projector and a Qwen2 language model. cJeZdZdZfdZdZee ddej dzdej dzdej dzdej dzd ej dzd e dzd ej dzd ej dzd e dzdeej zdeedeezfdZddde ffdZxZS)&AudioFlamingo3ForConditionalGenerationNct|t||_|dSrc)rjrkrzmodel post_initrus r%rkz/AudioFlamingo3ForConditionalGeneration.__init__s@    (00  r$c *|jj||fi|Src)rr)r\r?r@r]s r%rz9AudioFlamingo3ForConditionalGeneration.get_audio_featuress#,tz,^=P[[TZ[[[r$rrr?r@rJrr4rIlabelsrlogits_to_keepr]rAc j|jd|||||||| d| } | j} t| trt | dn| }|| dd|ddf}d}| |jd|||jjj d| }t||| j | j | j | jS)a input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Example: ```python >>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor >>> model_id = "nvidia/audio-flamingo-3-hf" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") ```)rr?r@rJrr4rIrN)r3r vocab_size)r2r3r4r5r6r7r#)rrK isinstanceintslicelm_head loss_functionrHrorr1r4r5r6r7)r\rr?r@rJrr4rIrrrr]rr5 slice_indicesr3r2s r%rbz.AudioFlamingo3ForConditionalGeneration.forwards>$*  ) 3)%+'       1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD4#3!/) ' ;     r$F)is_first_iterationrc|dd}|dd}tj|i|}|s|dds|||d<|||d<|S)Nr?r@rF)poprjprepare_inputs_for_generationget)r\rargsr]r?r@ model_inputsrvs r%rzDAudioFlamingo3ForConditionalGeneration.prepare_inputs_for_generation\s$4d;;$jj)>EErz)2 dataclassesrr9r activationsr cache_utilsr masking_utilsrmodeling_outputsr r processing_utilsr utilsr r rr utils.genericrutils.output_capturingr qwen2_audio.modeling_qwen2_audiorrvoxtral.modeling_voxtralrrrrwhisper.modeling_whisperrrconfiguration_audioflamingo3r get_loggerr loggerrr(r*r-r1r>rgrzr__all__r#r$r%rs "!!!!! !!!!!! 666666GGGGGGGG&&&&&&RRRRRRRRRRRR777777555555 MLLLLLLL>>>>>>  H % %     .        !4   '''''$='''      (B      99999;99   9( ? ? ? ? ? -? ?  ? D     (B   " T T T T T ,T T  T n UUUUU-LUU  Up   r$