tjmhddlZddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z dd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+e!j,e-Z. d7dej/dej0dej0dej0dej0dzde1dzde1fdZ2Gdd ej/Z3Gd!d"eZ4eGd#d$eZ5eGd%d&eZ6ed'(eGd)d*eZ7ed+(Gd,d-e5Z8Gd.d/ej/Z9ed0(Gd1d2e5Z:ed3(Gd4d5e5e Z;gd6Z>L=((6?([[L,|U33K''1--88::K  $$ceZdZdZ ddededed ed ed ed edzd edzffd Z dde j de j dzde dzde j dzdede e dee j e j dzee j dzffdZxZS)AudioFlamingo3Attentionz=Multi-headed attention from 'Attention Is All You Need' paperr FTN embed_dim num_headsr' is_decoderbias is_causal layer_idxconfigc pt||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ |*|r(t d|j jd||_t!j||d|_t!j||||_t!j||||_t!j||||_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r*zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrA)super__init__r>r?r'head_dimrD ValueErrorr&r@rBlogger warning_once __class____name__rCrLineark_projv_projq_projout_proj) selfr>r?r'r@rArBrCrDrMs r9rHz AudioFlamingo3Attention.__init__MsW "" !Y.  MI %$. 8 83dn33%.333 }d* $"      ,4>+B,,,    #i 95AAA i 94@@@ i 94@@@  )YTBBB r; hidden_stateskey_value_statespast_key_valuesr%output_attentionsr6returnc |du}|jdd}g|d|jR} |||jz| dd} |Tt|tr?|j |j } |rd|j |j <|j }n|j }||n|} |r3|r1| r/|j|j j} |j|j j}n|dd|j|jf}|| |dd} || |dd}||| ||j \} }t+j|jjt2}||| | ||f|jsdn|jd|d |\}}|jg|dR}||}||fS) z#Input shape: Batch x Time x ChannelNr)rrTrr ?)r'r&rX)shaperIrRr&viewr2r5 isinstancer is_updatedgetrCcross_attention_cacheself_attention_cachelayerskeysvaluesr?rPrQupdater get_interfacerD_attn_implementationr:r.r'reshaperS)rTrUrVrWr%rXr6is_cross_attention input_shape hidden_shape query_statesr_current_states key_states value_stateskv_shapeattention_interfacer8r7s r9forwardzAudioFlamingo3Attention.forwardus.T9#)#2#. 88b8$-88  M22T\AGG UU__`acdeepprr   &:oGZ+[+[ &(377GGJ! G=A*4>:"1"G"1"F.>-I))}  l/ lj l(/?DJ*1$.AHLL $ADNDMJH^4499(CCMMaQRSS^^``J;;~66;;HEEOOPQSTUU``bbL*+:+A+A*l\`\j+k+k( L(?(M K ,.E) ) %8$7      % $}>CC$,/ % %  % % ! \*k);;;;;;FFHH mmK00 L((r;)r FTFNN)NNNF)rN __module__ __qualname____doc__intfloatboolrrHr0Tensorrrr tuplers __classcell__rMs@r9r=r=JsyGG   $.2&C&C&C&C &C  &C  &C&C:&C%t+&C&C&C&C&C&CV15(,.2"' H)H)|H) ,-H) H)  t+ H)  H)-.H) u|U\D0% 2E2LL MH)H)H)H)H)H)H)H)r;r=cfeZdZdeffd ZdejdejdeedejfdZ xZ S)AudioFlamingo3EncoderLayerrDct|j|_t |j|j|j||_tj |j|_ |j |_ t|j |_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)r>r?r'rD)rGrHd_modelr>r=encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr'ractivation_function activation_fnactivation_dropoutrOencoder_ffn_dimfc1fc2final_layer_normrTrDrMs r9rHz#AudioFlamingo3EncoderLayer.__init__s 0n4,     %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r;rUr%r6rYc |}||}|jd||d|\}}tj||j|j}||z}|}||}|||}tj||j |j}| |}tj||j|j}||z}|j tj kr9tj|j jdz }tj|| |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rUr%r,i)minmax)rrrr3r'r.rrrrrdtyper0float16finforclamp)rTrUr%r6residual_ clamp_values r9rsz"AudioFlamingo3EncoderLayer.forwards[!11-@@ )4> ')     q  --mt|VZVc-dd  =0  --m<< **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0  %- / /+m&9::>EK!K K<[YYYMr;) rNrtrurrHr0rzrrrsr|r}s@r9rrs=3======$"|" "+, "  """"""""r;rc>eZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dS)AudioFlamingo3PreTrainedModelrDmodel)audiotextTr=rWN) rNrtrur__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_attention_backendrr;r9rrsS    (&*#23#4"5N"&r;rc2eZdZUdZdZejdzed<dS)!AudioFlamingo3ModelOutputWithPastzg audio_hidden_states (`torch.FloatTensor`, *optional*): Projected audio hidden states. Naudio_hidden_states)rNrtrurvrr0 FloatTensorrrr;r9rrs7 59*T188888r;rzZ Base class for AudioFlamingo3 causal language model (or autoregressive) outputs.  custom_introceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZejdzed<dS) $AudioFlamingo3CausalLMOutputWithPastaV loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. audio_hidden_states (`torch.FloatTensor`, *optional*): Hidden states of the audio encoder after projection. NlosslogitsrWrU attentionsr)rNrtrurvrr0rrrrWrrUr{rrrr;r9rrs  &*D% d ")))'+FE  $+++$(OUT\(((59M5*+d299926Je'(4/66648*T188888r;rzT The audio model from AudioFlamingo3 without any head or projection on top. c eZdZUdZeed<dZdZdgZe e dZ deffd Z dZ d ejfd Zd ejfd Zee ddejdejd zd eezfdZdejfdZxZS)AudioFlamingo3EncoderzY AudioFlamingo3 encoder: Whisper encoder, average pool (time/2), then LayerNorm. rDinput_featuresrr)rUrctj|_j|_j}j|_j|_jrtj |nd|_ tj |j|dd|_tj ||ddd|_tj|j||_|jdtjfdt)jD|_tjj|_tjdd |_d|_|dS) Nr[rr) kernel_sizepaddingr)rstriderFc.g|]}tSr)r).0rrDs r9 z2AudioFlamingo3Encoder.__init__..Ns"$n$n$nA%?%G%G$n$n$nr;)r)rGrHr'encoder_layerdrop layerdropr num_mel_binsmax_source_positionsscale_embeddingmathsqrt embed_scalerConv1dconv1conv2 Embeddingembed_positionsrequires_grad_ ModuleListrangeencoder_layersrcr layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rTrDr>rMs ` r9rHzAudioFlamingo3Encoder.__init__>sN    ~ 1N "/$*$?!393IR49Y///sYt0)TUVVV Yy)1VWXXX !|D,EyQQ ++E222m$n$n$n$nQVW]WlQmQm$n$n$noo ,v~66,q333&+# r;cP|D] }d|_ d|_dS)NF) parameters requires_grad_requires_grad)rTparams r9_freeze_parametersz(AudioFlamingo3Encoder._freeze_parametersWs4__&& ( (E"'E  #r;rYc|jSNr)rTs r9get_input_embeddingsz*AudioFlamingo3Encoder.get_input_embeddings\s zr;r$c||_dSrr)rTr$s r9set_input_embeddingsz*AudioFlamingo3Encoder.set_input_embeddings_s  r;Ninput_features_maskc |jddz dzdz}|d}|dz dzdz}tj||j|dddfk}t j||}t j| |}| ddd}||j j z}t j ||j |j}t|j||}|jD]4} |jotjg|jk} | s | ||}5| ddd}|| ddd}||}t-| S) ap Args: input_features (`torch.FloatTensor` of shape `(batch_size, feature_size, sequence_length)`): Log-Mel features extracted from raw audio. Use the processor/feature extractor to compute and pad these features from waveform input. input_features_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. r)rrdeviceNrr,)rD inputs_embedsr%)last_hidden_state)r\sumr0arangerrr3gelurrpermuterweightr'r.r rDrcrandrrrr) rTrrr6seq_leninput_features_lengthsrrUr%layerdrops r9rszAudioFlamingo3Encoder.forwardbs("'+a/A59!4!8!8!;PQQQTjklklklnrkrTss **4::n+E+EFF  **4::m+D+DEE %--aA66 &(<(CC  --mt|VZVc-dd 2;'.   [ E EE=DUZ^^dn%DD E %m^ D D &--aA66  66>>q!QGG  66 )+    r; input_lengthsc6|dz dzdz}|dz dzdz}||fS)zs Computes the output length of the convolutional layers and the output length of the audio encoder rrr)rTroutput_lengthss r9 _get_feat_extract_output_lengthsz6AudioFlamingo3Encoder._get_feat_extract_output_lengthss7'*q014 '!+1A5n,,r;r)rNrtrurvrrmain_input_namerrrr=_can_record_outputsrHrrModulerrrrr0rzr{rrs LongTensorrr|r}s@r9rr)sP  ('''&O564- :2$$$ bi") 483 3  3 #\D03 + + 3 3 3 _ 3 l-e>N--------r;rc.eZdZdZdeffd ZdZxZS)!AudioFlamingo3MultiModalProjectorz Audio adaptor (small MLP) that projects AudioFlamingo3Encoder features to the LLM embedding space so they can replace `` tokens. rDcJttj|jj|jj|j|_t|j |_ tj|jj|jj|j|_ dS)NrF) rGrHrrO audio_config hidden_size text_configprojector_biaslinear_1rprojector_hidden_actactlinear_2rs r9rHz*AudioFlamingo3MultiModalProjector.__init__s    +V-?-KRXRg    &56   *F,>,JQWQf    r;c||}||}||}|Sr)rrr)rTaudio_featuresrUs r9rsz)AudioFlamingo3MultiModalProjector.forwards; n55 //  m44 r;)rNrtrurvrrHrsr|r}s@r9rrs^  3      r;rz The AudioFlamingo3 model (fine-tuned Whisper encoder, multi-modal projector, Qwen2 language model), without a language modeling head. ceZdZdZdZdZfdZeedde j de j de e deezfd Zd e jd e j d e j fd Zee dd e jdzde j dzde j dzde j dzde jdzdedzd e j dzdedzde e deezfdZxZS)AudioFlamingo3ModelNct|tj|j|_tj|j|_t||_ | dSr) rGrHr from_configr audio_towerrlanguage_modelrmulti_modal_projectorrrs r9rHzAudioFlamingo3Model.__init__sm    $01DEE'3F4FGG%Fv%N%N" r;zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.rrrr6rYc |j|f|dd|}||j}|dt j}|j|\}}t j|j d|j dddf|dddfk} || |j |_ |S)a1 input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padded feature indices. T)r return_dictr)rrN) rr rrtor0longrrr\r pooler_output) rTrrr6 audio_output audio_embedsrr post_lengths valid_masks r9get_audio_featuresz&AudioFlamingo3Model.get_audio_featuress"(t'  0CQU  Y_  11,2PQQ ,//3366uzBB *KKMZZ<\,"4Q"7 @STTTUY[\[\[\U\]`lmnmnmnptmt`uu %1*-- @S2T2T%U "r; input_idsrrc>|e||tj|jjtj|jk}|d}n||jjk}|}|j d}| d | |j}t|||kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)rrr)rz6Audio features and audio tokens do not match, tokens: z , features: )rr0tensorrDaudio_token_idrrallrr\ unsqueeze expand_asr rnumel)rTrrrspecial_audio_maskn_audio_tokensn_audio_featuress r9get_placeholder_maskz(AudioFlamingo3Model.get_placeholder_masks  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11)/2/99"==GG VVYYZgZnoo , - 3 3 5 59M9M9O9O O s^ s saq s s   "!r;r% position_idsrW use_cachec |||}d} |e|c|||dj} |||| } || | |j}|jd|||||d| } t| j | j | j | j | S)z input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. NT)r )rr)rr%r!rWr")rrWrUrrr) rrrr masked_scatterr rr rrrWrUr) rTrrrr%r!rWrr"r6rroutputss r9rszAudioFlamingo3Model.forwards $  7D5577 BBM  %)*?22>CVdh2iiwL"&!:!:|";"" *889K\__]j]qMrMrssM%$% ')%+      1%7#3!/) ,     r;)NNNNNNNN)rNrtru_tp_plan_pp_plan_keep_in_fp32_modules_strictrHrrr0rrzrrr{rrrr rryrrsr|r}s@r9rrsHH#' ^w)#\+,  + + 4")":?:K"]b]n""""0.23737.204(,26!%, , #d*, )D0, #\D0 ,  t+ , &- , , (4/, $;, +,,  2 2, , , ^, , , , , r;rz The AudioFlamingo3 model which consists of a fine-tuned Whisper encoder, a multi-modal projector and a Qwen2 language model. cPeZdZdgZdZfdZdZee dde j dzde j dzde j dzd e j dzd e j dzd e dzd e j dzd e j dzdedzdee j zdeedeezfdZdddeffdZxZS)&AudioFlamingo3ForConditionalGenerationrNct|t||_t j|jj|jjd|_ | dS)NFrF) rGrHrrrrOrr vocab_sizelm_headrrs r9rHz/AudioFlamingo3ForConditionalGeneration.__init__<se    (00 y!3!?ASA^ejkkk  r;c *|jj||fi|Sr)rr)rTrrr6s r9rz9AudioFlamingo3ForConditionalGeneration.get_audio_featuresBs#,tz,^=P[[TZ[[[r;rrrrr%r!rWrlabelsr"logits_to_keepr6rYc j|jd|||||||| d| } | j} t| trt | dn| }|| dd|ddf}d}| |jd|||jjj d| }t||| j | j | j | jS)a input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Example: ```python >>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor >>> model_id = "nvidia/audio-flamingo-3-hf" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") ```)rrrr%r!rWrr"N)rr/r,)rrrWrUrrr)rrr^rwslicer- loss_functionrDrr,rrWrUrr)rTrrrr%r!rWrr/r"r0r6r%rU slice_indicesrrs r9rsz.AudioFlamingo3ForConditionalGeneration.forwardEs>$*  ) 3)%+'       1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD4#3!/) ' ;     r;F)is_first_iterationr5c|dd}|dd}tj|i|}|s|dds|||d<|||d<|S)Nrrr"F)poprGprepare_inputs_for_generationr`)rTr5argsr6rr model_inputsrMs r9r8zDAudioFlamingo3ForConditionalGeneration.prepare_inputs_for_generations$4d;;$jj)>EErNs, $$$$$$!!!!!! !!!!!!55555555))))))666666BBBBBB999999``````````FFFFFFFF&&&&&&jjjjjjjjjjjjjj777777555555[[[[[[[[  H % %!%% I% <% % < % L4' % T\ %%%%%4s)s)s)s)s)bis)s)s)l55555!;555p ' ' ' ' 'O ' ' ' 99999(?99 9  99999;99   9( r-r-r-r-r-9r-r-  r-j . p p p p p 7p p  p f WWWWW-JOWW  Wt   r;