tjGhddlZddlmZmZmZmZmZerddlZerddlZ erddl m Z ddl m Z mZmZddlmZddlmZmZmZmZddlmZmZdd lmZejeZGd d ed ZGdded Z eGddeZ!dgZ"dS)N)auto_docstringis_mistral_common_availableis_soundfile_availableis_torch_availablelogging)TranscriptionRequest) AudioInput load_audio_asmake_list_of_audio) BatchFeature) AudioKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)_get_template_variablesc$eZdZUdZedzed<dS)VoxtralAudioKwargsz max_source_positions (`int`, *optional*, defaults to `3000`): Maximum number of positions per chunk when splitting mel spectrogram features along the time dimension. Nmax_source_positions)__name__ __module__ __qualname____doc__int__annotations__l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/voxtral/processing_voxtral.pyrr's. *$$$$$rrF)totalc>eZdZUeed<ddiddddddd ddd d Zd S) VoxtralProcessorKwargs audio_kwargspaddingTi>FiSi ) sampling_rater% truncationpad_to_multiple_ofrpt)return_tensors return_dicttokenize) text_kwargsr$ common_kwargsN)rrrrr _defaultsrrr r#r#0sd$$$$ t #"($(   #  IIIrr#ceZdZfdZdZ ddeeeefeeeeefzdedzdeedzdeeeefdzd ed ed ed ed edzdedededzdefdZ e dde e zee zee zdzde efdZ ddeeezezdedeeedzzdzdedzdeeezdzde ef dZxZS) VoxtralProcessorcd|_||j|_t||dS)N)audio_token_idconvert_ids_to_tokens audio_tokensuper__init__)selffeature_extractor tokenizer __class__s r r8zVoxtralProcessor.__init__GsG !$::4;NOO *I66666rc g}|D]`}|j|fi|}|d|jjd|}||ddat j|S)aX Handles specific logic of Voxtral expected input features: audio arrays should be padded to next multiple of 480000 (duration is a multiple of 30s), see VoxtralProcessorKwargs' default audio_kwargs. Then mel input features are extracted and stacked along batch dimension, splitting into chunks of max_source_positions. input_featuresr)r:reshape feature_sizeappend transposetorchcat)r9audiorkwargsinput_features_list audio_array audio_inputsr>s r _retrieve_input_featuresz)VoxtralProcessor._retrieve_input_featuresQs !  G GK141+HHHHL**:;CC&3R9MN & &~'?'?1'E'E F F F Fy,---rNF conversation chat_templatetools documentsadd_generation_promptcontinue_final_messagereturn_assistant_tokens_maskr,r*r+load_audio_from_videoprocessor_kwargsreturnc  |r"|rtd|rtdt|ttfr=t|dttfst |ddrd}|}nd}|g}| pi} t |fd| D}|rtd|} | r| | d <|j tfi| }|d }|d }| d d } | d krt|j j d|d }d |d <|jj|fi|}| ddru| ddr_|dd }t#|}|'|d}|j||fi||d<t'|| S|s|dS|S)a This method applies the model's chat completion template given a conversation. It relies on MistralCommonBackend's [`~MistralCommonBackend.apply_chat_template`] to prepare input ids to the model and on WhisperFeatureExtractor's [`~WhisperFeatureExtractor.__call__`] to prepare input features to the model. Note that audio is padded to the nearest 30-second multiple prior to mel feature extraction. A `conversation` is a list of messages, where each message is a dictionary with a `role` and a `content` field. For Voxtral, `role` can be `"user"` or `"assistant"`. The `content` field can be a string or a list of dictionaries with a `type` field. See example below. ```python from huggingface_hub import hf_hub_download from transformers.audio_utils import load_audio_as audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3" audio_path = hf_hub_download(repo_id="hf-internal-testing/dummy-audio-samples", filename="bcn_weather.mp3", repo_type="dataset") audio_base64 = load_audio_as(audio_path, return_format="base64", force_mono=True) # audio + text conversation = [ { "role": "user", "content": [ {"type": "audio", "url": audio_url}, {"type": "audio", "path": audio_path}, {"type": "audio", "base64": audio_base64}, {"type": "text", "text": "How many audio do you hear?"}, ], }, ] processor = VoxtralProcessor.from_pretrained("mistralai/Voxtral-Mini-3B-2507") inputs = processor.apply_chat_template(conversation) ``` Args: conversation (`Union[list[Dict, [str, str]], list[list[dict[str, str]]]]`): The conversation to format. a continue_final_message and add_generation_prompt are not compatible. Use continue_final_message when you want the model to continue the final message, and add_generation_prompt when you want to add a header that will prompt it to start a new assistant message instead.zKcontinue_final_message is not compatible with return_assistant_tokens_mask.rcontentTFc$i|] \}}|v || Srr).0kvtemplate_kwargss r z8VoxtralProcessor.apply_chat_template..s*'d'd'dA1TcKcKc1KcKcKcrz^Kwargs passed to `processor.__call__` have to be in `processor_kwargs` dict, not in `**kwargs`r*r-r$Nr)% only supports `return_tensors='pt'`.r,r+rGrr>data tensor_type) ValueError isinstancelisttuplehasattrritemsloggerwarning _merge_kwargsr#getr<rr;apply_chat_templatepopdictrLr )r9rMrNrOrPrQrRrSr,r*r+rTrUrH is_batched conversationsprocessor_kwargs_from_kwargs output_kwargsr-r$tokenizer_kwargsencoded_instruct_inputsrGrarr]s @r rmz$VoxtralProcessor.apply_chat_templatebsp " p$  c, p !nooo lT5M 2 2 + |Au 6 6 +:A,q/S\:]:] +J(MMJ)NM ,1r1-@@'d'd'd'd'd'd'd$ ' < NNp    <   @1? - .** "    $M2 $^4 $)94@@ T ! ! 7^^^__ _(7-1)*"D$."D]"g"gVf"g"g ??:u - - K}e44 K/33GTBB344$+7+;+;JJJJ .*1- -&&ra Method to prepare text to be fed as input to the model. This method forwards the `text` arguments to MistralCommonBackend's [`~MistralCommonBackend.__call__`] to encode the text. Please refer to the docstring of the above methods for more information. This method does not support audio. To prepare the audio, please use: 1. `apply_chat_template` [`~VoxtralProcessor.apply_chat_template`] method. 2. `apply_transcription_request` [`~VoxtralProcessor.apply_transcription_request`] method. ) custom_introtextrHc <t|tr|g}tfd|Drtjdjt fi|}j|fi|d}t||d ddS)Nc3*K|] }j|vVdSN)r6)rZtr9s r z,VoxtralProcessor.__call__..s+33t1$333333rz is present in the provided text which is not supported by VoxtralProcessor. Please use the `apply_chat_template` method instead.r-r*r`) rdstranyrcr6rkr#r;r rl)r9rwrHrsouts` r __call__zVoxtralProcessor.__call__s dC  6D 3333d333 3 3 #ggg +*+ALLVLL dnTBB]=%ABB- 2N2R2RScei2j2jkkkkrrGmodel_idlanguager&formatc R|jtfi|}|d}|d} t|t} t d|D} | p| } | rO%t d| ddn(| dkrtdd | dd | d|d d } |d d }| d d }| d d }|dd}|dkrt|j j d| rt|ddg}n]| rfd|D}nKt|}|tdt|tr|gt|z}t|t|kr0tdt|dt|dtstdg}t!||D]\}}t#j}|jdkr|d}t+j||| d||d|||}t|}t|tr|g|z}n|dg|z}t||kr#tdt|d |dg}g}g}t!||D]\}}|||d!}t3j|}|jj|}||j||j|d"|j D|rY| rW|j|fd#d i|}tC|}| d$} |j"|| fi| |d%<tG||&S|S)'a This method applies the model's transcription request template given a language and audio. It relies on MistralCommonBackend and WhisperFeatureExtractor to prepare input ids and input features to the model. ```python from transformers import VoxtralProcessor model_id = "mistralai/Voxtral-Mini-3B-2507" processor = VoxtralProcessor.from_pretrained(model_id) language = "en" audio = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama.mp3" # set the language is already know for better accuracy inputs = processor.apply_transcription_request(language=language, audio=audio, model_id=model_id) # but you can also let the model detect the language automatically inputs = processor.apply_transcription_request(audio=audio, model_id=model_id) ``` Args: audio (`str`, `list[str]`, `np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): The audio or batch of audio to be prepared. If provided as a string, it should correspond to the path or url of the audio file. model_id (`str`: The hub model id of the model to use for transcription. language (`str`, `list[Union[str, None]]`, *optional*): The language or languages of the audio. If not provided or None, automatic language detection will be used for all audio. If provided as a string (a language code in the [ISO 639-1 alpha-2 format](https://en.wikipedia.org/wiki/ISO_639-1) e.g. `"en"`), it will be applied uniformly to all audio. If provided as a list of strings/ None values, e.g. `["en", None, "fr"]`, will be applied to each audio individually with a one-to-one mapping, with a None value indicating automatic language detection for that audio. sampling_rate (`int`, *optional*): The sampling rate of the audio. Necessary if it is provided as `np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`. Used to avoid silent errors when passing audio that is not in the expected sampling rate. format (`str`, `list[str]`, *optional*): The format of the audio, necessary if is provided as `np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`. r-r$c3@K|]}t|tVdSrz)rdr}rZels r r|z?VoxtralProcessor.apply_transcription_request..*s,AARZC00AAAAAArNzUYou've provided audio without specifying the sampling rate. It will be assumed to be r&z$, which can result in silent errors.z The sampling rate of the audio (z5) does not match the sampling rate of the processor (zD). Please provide resampled the audio to the expected sampling rate.r+Fr,r*r)r_bufferT return_format force_monor&c6g|]}t|ddS)rTr)r )rZrr&s r z@VoxtralProcessor.apply_transcription_request..Gs8ln bTYfgggrzH`format` must be provided when passing audio arrays to VoxtralProcessor.z,When passed as a list of audio, the length (z#) must match the number of format ()zHPlease install `soundfile` to encode audio arrays with VoxtralProcessor.r@)axis) sampleraterrz0When passed as a list of languages, the length (z") must match the number of audio ()modelfilercg|] }|j Sr)rJrs r rz@VoxtralProcessor.apply_transcription_request..s a a aB a a aradd_special_tokensrr>r`)$rkr#rdr}allri warning_oncercrnr<rr r lenr ImportErrorzipioBytesIOndimmeansfwriteseekrCr from_openair;encode_transcriptiontokensrwextendaudiosrorLr )!r9rGrrr&rrHrsr-r$is_stris_list_of_stris_list_of_audior+r,_r* audio_buffersarrayfrn_audio input_idstexts audio_arraysaudio_el language_elopenai_transcription_requesttranscription_requesttokenized_transcription_requestencodingrars! ` r apply_transcription_requestz,VoxtralProcessor.apply_transcription_requestsp\+* "    $M2 $^4 E3''AA5AAAAA & 8.9  $##plxzImJppp,"??? _}__|HIX|Y___%_5 "oomU;; ??:u55   ]E 2 2   Z / /$)94@@ T ! ! 7^^^__ _ ! ""5TanooopEE  "rwEE'u--E~ !klll&#&& / CJJ.5zzS[[(( A3u::AAruv|r}r}AAA*++ n!"lmmmMv.. - -q:??!JJAJ..E<3PYZ[[[[ A$$V,,,,!Ee** h $ $ ( zG+HH  v'H x==G # #~3x==~~t{~~~   %(%9%9 c c !Hk! ',, ( %9$DEa$b$b !.2n.F.[.[\q.r.r +   <C D D D LL8= > > >    a a:Y:` a a a b b b b  K K)4>'," H~~(4'7'78N'O'O$)F)F "6**:F**%&$>JJJJ r) NNNFFFFNFFN)NNN)rrrr8rLreror}boolrmrrrrr#rr rr __classcell__)r<s@r r1r1Es|77777...(%)#'15&+',-2%)!&+(,s's'4S>*T$tCH~2F-GGs'Tzs'DzD s' S#X'$. s' $ s'!%s''+s's'd s's' $s'+s' s's's's'j^   l++d9o=EV@WWZ^^l/0lll  l.37$()- aaT#Y+aaS4Z((4/ a Tz a d3i$& a/0aaaaaaaarr1)#rutilsrrrrrrE soundfiler-mistral_common.protocol.transcription.requestr audio_utilsr r r feature_extraction_utilsr processing_utilsrrrrtokenization_utils_baserrutils.chat_template_utilsr get_loggerrrirr#r1__all__rrr rs uuuuuuuuuuuuuuLLL  SRRRRRRHHHHHHHHHH444444UUUUUUUUUUUUCCCCCCCC@@@@@@  H % %%%%%%E%%%%-U*OOOOO~OOOd  r