tj` ddlZddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZmZmZdd lmZmZdd lmZdd lmZmZmZmZmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&ej'e(Z) d3dej*dej+dej+dej+dej+dzde,dzde,fdZ-Gddej*Z.Gdd eZ/eGd!d"eZ0ed#$Gd%d&e0Z1Gd'd(ej*Z2ed)$eGd*d+eZ3ed,$Gd-d.e0Z4ed/$Gd0d1e0e Z5gd2Z6dS)4N)Callable) dataclass)nn)ACT2FN)Cache)GenerationMixin)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)merge_with_config_defaults)capture_outputs) AutoModel) VoxtralConfigVoxtralEncoderConfigmodulequerykeyvalueattention_maskscalingdropoutc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nrr)dimptrainingr) sizetorchmatmul transposer functionalsoftmaxr$r+ contiguous) rrr r!r"r#r$kwargs attn_weights attn_outputs j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/voxtral/modeling_voxtral.pyeager_attention_forwardr7.s**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$ceZdZdZ ddededed ed ed ed edzd edzffd Zde j dedefdZ dde j de j dzdede e j e j dze e j dzffdZ xZS)VoxtralAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr$ is_decoderbias is_causal layer_idxconfigc pt||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ |*|r(t d|j jd||_t!j||d|_t!j||||_t!j||||_t!j||||_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r'zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr>)super__init__r;r<r$head_dimrA ValueErrorr#r=r?logger warning_once __class____name__r@rLineark_projv_projq_projout_proj) selfr;r<r$r=r>r?r@rArJs r6rEzVoxtralAttention.__init__KsW "" !Y.  MI %$. 8 83dn33%.333 }d* $"      ,4>+B,,,    #i 95AAA i 94@@@ i 94@@@  )YTBBB r8tensorseq_lenbszc||||j|jddS)Nrr)viewr<rFr/r2)rQrRrSrTs r6_shapezVoxtralAttention._shapess<{{3GGQQRSUVWWbbdddr8 hidden_statesr"output_attentionsreturnc N|\}}}||||jz||}|||d|} |||d|} t j|jj t} | ||| | |f|j sdn|j d|d|\} } | ||d} || } | | fS)z#Input shape: Batch x Time x Channelr&r?)r$r#rY)r,rWrOr#rMrNr get_interfacerA_attn_implementationr7r+r$reshaper2rP)rQrXr"rYr3rTtgt_len_ query_states key_states value_statesattention_interfacer5r4s r6forwardzVoxtralAttention.forwardvs;(,,..Wa{{4;;}#=#= #LgWZ[[ [[]!;!;REE {{4;;}#=#=r3GG (?(M K ,.E) ) %8$7      % $}>CC$,/ % %  % % ! \"))#w;;FFHH mmK00 L((r8)rFTFNNNF)rK __module__ __qualname____doc__intfloatboolrrEr-TensorrWtuplerf __classcell__rJs@r6r:r:HsfGG   $'+&C&C&C&C &C  &C  &C&C:&C$&C&C&C&C&C&CPeU\eCeceeee /3"' ')')|') t+') ') u|U\D0% 2E2LL M ')')')')')')')')r8r:cfeZdZdeffd ZdejdejdeedejfdZ xZ S)VoxtralEncoderLayerrAct|j|_t |j|j|j||_tj |j|_ |j |_ t|j |_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)r;r<r$rA)rDrEd_modelr;r:encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr$ractivation_function activation_fnactivation_dropoutrLencoder_ffn_dimfc1fc2final_layer_normrQrArJs r6rEzVoxtralEncoderLayer.__init__s )n4,     %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r8rXr"r3rZc |}||}|jd||d|\}}tj||j|j}||z}|}||}|||}tj||j |j}| |}tj||j|j}||z}|j tj kr9tj|j jdz }tj|| |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rXr"r)i)minmax)rzrxrr0r$r+rr|rr}rdtyper-float16finforclamp)rQrXr"r3residualra clamp_values r6rfzVoxtralEncoderLayer.forwards[!11-@@ )4> ')     q  --mt|VZVc-dd  =0  --m<< **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0  %- / /+m&9::>EK!K K<[YYYMr8) rKrhrirrEr-rnrrrfrprqs@r6rsrss=}======$"|" "+, "  """"""""r8rscHeZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ dZdZdS)VoxtralPreTrainedModelrAmodel)audiotextTNpast_key_values)rKrhrir__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraphrr8r6rrsb (&*##4"5N "&!r8rz: The Voxtral encoder, which is a Whisper encoder.  custom_introceZdZUdZeed<dZdZdgZe e dZ deffd Z dZ d ejfd Zd ejfd Zee ddeed eezfdZdejfdZxZS)VoxtralEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`VoxtralEncoderLayer`]. Args: config: VoxtralEncoderConfig rAinput_featuresrrs) attentionsrXctj|_j|_j}j|_j|_jrtj |nd|_ tj |j|dd|_tj ||ddd|_tj|j||_|jdtjfdt)jD|_tjj|_tjdd |_d|_|dS) Nr\rr) kernel_sizepaddingr)rstriderFc.g|]}tSr)rs).0rarAs r6 z+VoxtralEncoder.__init__..s"$g$g$gQ%8%@%@$g$g$gr8)r)rDrEr$encoder_layerdrop layerdropru num_mel_binsmax_source_positionsscale_embeddingmathsqrt embed_scalerConv1dconv1conv2 Embeddingembed_positionsrequires_grad_ ModuleListrangeencoder_layerslayersry layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rQrAr;rJs ` r6rEzVoxtralEncoder.__init__sM    ~ 1N "/$*$?!393IR49Y///sYt0)TUVVV Yy)1VWXXX !|D,EyQQ ++E222m$g$g$g$g%PVPeJfJf$g$g$ghh ,v~66,q333&+# r8cP|D] }d|_ d|_dSrg) parameters requires_grad_requires_grad)rQparams r6_freeze_parametersz!VoxtralEncoder._freeze_parameterss4__&& ( (E"'E  #r8rZc|jSNr)rQs r6get_input_embeddingsz#VoxtralEncoder.get_input_embeddingss zr8r!c||_dSrr)rQr!s r6set_input_embeddingsz#VoxtralEncoder.set_input_embeddings!s  r8Nr3c r|jj|jjdz|jjdz}|jd|kr$t d|d|jdd|d||jjj |jjj }tj ||}tj ||}|ddd }|jj}||z|j }tj ||j|j }t%|jD]\}} | || }||}t+| S) a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rr&z7Voxtral expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .rdevicerrr))r")last_hidden_state)rArrrrshaperGtoweightrrrr0gelupermuterr$r+ enumeraterrr ) rQrr"r3expected_seq_length inputs_embeds embed_posrXidx encoder_layers r6rfzVoxtralEncoder.forward$s(#k>ARSTAUUX\XbXijkXll   #': : :IJ]IIkykAClDIIsFIII (**1B1HQUQ[QbQi*jj **4::n+E+EFF  **4::m+D+DEE %--aA66 (/ &266}7JKK  --mt|VZVc-dd "+DK"8"8   C)M-MM  66 )+    r8 input_lengthsc6|dz dzdz}|dz dzdz}||fS)zs Computes the output length of the convolutional layers and the output length of the audio encoder rrr)rQroutput_lengthss r6 _get_feat_extract_output_lengthsz/VoxtralEncoder._get_feat_extract_output_lengthsTs7'*q014 '!+1A5n,,r8r)rKrhrirjrrmain_input_namerrr:rs_can_record_outputsrErrModulerrrrrrror rfr- LongTensorrrprqs@r6rrsC  !   &O./&, 32$$$ bi") + + +, + + + + + + _ + \-e>N--------r8rc*eZdZdeffd ZdZxZS)VoxtralMultiModalProjectorrAc6ttj|jj|jjd|_t|j |_ tj|jj|jjd|_ dSNFrC) rDrErrL audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rs r6rEz#VoxtralMultiModalProjector.__init__^sv  &"5"GI[Ignsttt &56 &"4"@&BTB`glmmm r8c||}||}||}|Sr)rrr)rQaudio_featuresrXs r6rfz"VoxtralMultiModalProjector.forwardds; n55 //  m44 r8)rKrhrirrErfrprqs@r6rr]sZn}nnnnnn r8rzL Base class for Voxtral outputs, with hidden states and attentions. c2eZdZUdZdZejdzed<dS)VoxtralModelOutputWithPastzg audio_hidden_states (`torch.FloatTensor`, *optional*): Projected audio hidden states. Naudio_hidden_states)rKrhrirjrr- FloatTensorrrr8r6rrks7 59*T188888r8rz The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model, without a language modeling head. c~eZdZfdZeeddejdee de e zfdZ dej d ejd ejfd Zee ddej d zdejd zd ejd zdej d zded zd ejd zded zdee de ezfdZxZS) VoxtralModelct|tj|j|_tj|j|_t||_ | dSr) rDrEr from_configr audio_towerrlanguage_modelrmulti_modal_projectorrrs r6rEzVoxtralModel.__init__sm    $01DEE'3F4FGG%?%G%G" r8zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.rrr3rZc |j|fddi|}|j}|d|jjj}||}||_|S)aa input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] return_dictTr&)rrr_rArrr pooler_output)rQrr3 audio_outputsr audio_embedss r6get_audio_featureszVoxtralModel.get_audio_featuressl)(TTTTVTT +=199"dk>V>hii112EFF &2 #r8 input_idsrrc>|e||tj|jjtj|jk}|d}n||jjk}|}|j d}| d | |j}t|||kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. Nrr&rz6Audio features and audio tokens do not match, tokens: z , features: )rr-rRrAaudio_token_idlongrallsumr unsqueeze expand_asrrnumel)rQrrrspecial_audio_maskn_audio_tokensn_audio_featuress r6get_placeholder_maskz!VoxtralModel.get_placeholder_masks  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11)/2/99"==GG VVYYZgZnoo , - 3 3 5 59M9M9O9O O s^ s saq s s   "!r8Nr" position_idsr use_cachec |||}d} |d|b||dj} |||| } || | |j}|jd|||||d|} t| j | j | j | j | S)NT)r)rr)r"rrrr)rrrXrrr) rrrrmasked_scatterrrrrrrrXr) rQrrr"rrrrr3rroutputss r6rfzVoxtralModel.forwards  7D5577 BBM  %)*?22>t2TTbL"&!:!:|";"" *889K\__]j]qMrMrssM+>4+>, )%+' , ,  , , *%7#3!/) ,     r8)NNNNNNN)rKrhrirErrr-rrrror rrrrnrrmrrfrprqs@r6rrzs^w#/;ABT;U + +&")":?:K"]b]n""""0.237.204(,26!%' ' #d*' )D0'  t+ ' &- '  ' (4/' $;' +,'  + +' ' ' ^' ' ' ' ' r8rzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model. c*eZdZdgZfdZdZee ddej dzdej dzdej dzd ej dzd e dzd ej dzd ej dzd e dzdeej zdeedeezfdZfdZxZS)VoxtralForConditionalGenerationrct|t||_t j|jj|jjd|_ | dSr) rDrErrrrLrr vocab_sizelm_headrrs r6rEz(VoxtralForConditionalGeneration.__init__se    !&)) y!3!?ASA^ejkkk  r8c&|jj|i|Sr)rr)rQargsr3s r6rz2VoxtralForConditionalGeneration.get_audio_featuress,tz,d=f===r8Nrrrr"rrrlabelsrlogits_to_keepr3rZc \|jd|||||||d| } | j} t| trt | dn| } || dd| ddf}d}| |jd|||jjj d| }t||| j | j | j S)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```)rrr"rrrrN)logitsrr)lossr"rrXrr)rr isinstancerkslicer loss_functionrArrr rrXr)rQrrr"rrrrrr r3rrX slice_indicesr"r#s r6rfz'VoxtralForConditionalGeneration.forwards`$*  ))%+'       1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD&#3!/)     r8c|dd}|dd}tj|i|}|s|dds||d<|S)Nris_first_iterationFrT)popgetrDprepare_inputs_for_generation)rQrr3rr) model_inputsrJs r6r,z=VoxtralForConditionalGeneration.prepare_inputs_for_generationAsw $4d;;#ZZ(>>.237.204(,26*.!%-.I I #d*I )D0I  t+ I &- I  I (4/I  4'I $;I el*I +,I  ' 'I I I ^I V         r8r)rrrr)Nr)7rcollections.abcr dataclassesrr-r activationsr cache_utilsr generationr modeling_layersr modeling_outputsr r r modeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrutils.output_capturingrautorconfiguration_voxtralrr get_loggerrKrHrrnrlr7r:rsrrrrrr__all__rr8r6r?s, $$$$$$!!!!!! !!!!!! ))))))999999kkkkkkkkkkFFFFFFFF&&&&&&jjjjjjjjjjjjjj777777555555FFFFFFFF  H % %!%% I% <% % < % L4' % T\ %%%%%4U)U)U)U)U)ryU)U)U)p555554555p " " " " "_ " " " m-m-m-m-m-+m-m-  m-`          99999!899   9 ` ` ` ` ` )` `  ` F eeeee&