tjddlmZddlmZmZmZddlZddlZddl m Z ddl m Z ddl mZmZmZmZdd lmZdd lmZerdd lmZdd lmZdd lmZejeZer ddlZddl m!Z!dZ"ddZ#dZ$GddeZ%dS)) defaultdict) TYPE_CHECKINGAnyUnionN)GenerationConfig)PreTrainedTokenizer)is_torch_availableis_torchaudio_availableis_torchcodec_availablelogging) ffmpeg_read) ChunkPipeline)BeamSearchDecoderCTC)SequenceFeatureExtractor)PreTrainedModel)(MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMEScg}|D]\}}}tt||z}tt||z |z}tt||z |z}|||f}|||S)z Rescales the stride values from audio space to tokens/logits space. (160_000, 16_000, 16_000) -> (2000, 200, 200) for instance. )introundappend)strideratio new_stridesinput_nleftrighttoken_n new_strides q/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/pipelines/automatic_speech_recognition.pyrescale_strider")sK &''ueGeO,,--5'12233E%'/G34455tU+ :&&&& c#`K|jd}||z |z }td||D]}||z} ||| } || |jdd} || |} |dkrdn|} | |k} | rdn|}| jd}|| |f}| jd| kr | |d| V| rdSdS)NrptT sampling_ratereturn_tensorsreturn_attention_maskdtype)is_lastr)shaperanger'to)inputsfeature_extractor chunk_len stride_left stride_rightr+ inputs_lenstepchunk_start_idx chunk_end_idxchunk processed _stride_leftr, _stride_rightrs r! chunk_iterr==saJ { "\ 1D J55')3 }45%% +9"&     ! 5 11I+q00qqk :-$6, KN \=9 ;q>L ( (%EE9E E E E   EE 'r#c Ffd|ddD}|ddD]}fd|dD}d}d}tdt|dzD]l}|dz }tjtj|| dtj|d|k} | |z |z} | dkr | |kr|}| }m|||dtj|S)Nc&g|] }|jv |Sall_special_ids.0tok_id tokenizers r! z1_find_longest_common_sequence..^s&iii6yOhAhAhAhAhAhr#rrc&g|] }|jv |Sr@rArCs r!rGz1_find_longest_common_sequence..`s'lll6FR[RkDkDkDkDkDkr#gg@)tolistr.lennpsumarrayextend) sequencesrFsequencenew_seq new_sequenceindexmax_iepsmatchesmatchings ` r!_find_longest_common_sequencerYWs?jiiiYq\!_%;%;%=%=iiiHQRR= . .llllWQZ->->-@-@lll q#l++a/00  Ag+CfRXhrssm44bqbAQ8R8RRSSG{S(H{{x$ UVV,---- 8H  r#c XeZdZdZdZdZdZdZdZe ddZ ddd d e d e fdzd e dzd e de fdzde edfdzf fd Zdejeze zezdedeee efffd Z ddZedZddZd dZ d!dedzfdZxZS)""AutomaticSpeechRecognitionPipelineao Pipeline that aims at extracting spoken text contained within some audio. The input can be either a raw waveform or a audio file. In case of the audio file, ffmpeg should be installed for to support multiple audio formats Unless the model you're using explicitly sets these generation parameters in its configuration files (`generation_config.json`), the following default values will be used: - max_new_tokens: 256 - num_beams: 5 Example: ```python >>> from transformers import pipeline >>> transcriber = pipeline(model="openai/whisper-base") >>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac") {'text': ' He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered flour-fatten sauce.'} ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) Arguments: model ([`PreTrainedModel`]): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from [`PreTrainedModel`]. feature_extractor ([`SequenceFeatureExtractor`], *optional*): The feature extractor that will be used by the pipeline to encode waveform for the model. tokenizer ([`PreTrainedTokenizer`], *optional*): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from [`PreTrainedTokenizer`]. decoder (`pyctcdecode.BeamSearchDecoderCTC`, *optional*): [PyCTCDecode's BeamSearchDecoderCTC](https://github.com/kensho-technologies/pyctcdecode/blob/2fd33dc37c4111417e08d89ccd23d28e9b308d19/pyctcdecode/decoder.py#L180) can be passed for language model boosted decoding. See [`Wav2Vec2ProcessorWithLM`] for more information. device (Union[`int`, `torch.device`], *optional*): Device ordinal for CPU/GPU supports. Setting this to `None` will leverage CPU, a positive will run the model on the associated CUDA device id. TF)max_new_tokens num_beamsNmodelrr1rrFdecoderrdevicez torch.devicec |jjdkrd|_nU|jjt jvrd|_n.|jjdvrd|_n|||_d|_nd|_tj |||fd|i|dS) Nwhisperseq2seq_whisperseq2seq) parakeet_tdt parakeet_rnntnemotron_asr_streamingnemotron3_5_asrtdt ctc_with_lmctcrb) config model_typetype __class____name__rvaluesrasuper__init__)selfr`r1rFrarbkwargsrqs r!ruz+AutomaticSpeechRecognitionPipeline.__init__s < "i / /)DII _ %)Q)X)Z)Z Z Z!DII \ $(v v vDII  "DL%DIIDI +<VVVVvVVVVVr#r0rwreturnc 8tj|fi|S)ae Transcribe the audio sequence(s) given as inputs to text. See the [`AutomaticSpeechRecognitionPipeline`] documentation for more information. Args: inputs (`np.ndarray` or `bytes` or `str` or `dict`): The inputs is either : - `str` that is either the filename of a local audio file, or a public URL address to download the audio file. The file will be read at the correct sampling rate to get the waveform using *ffmpeg*. This requires *ffmpeg* to be installed on the system. - `bytes` it is supposed to be the content of an audio file and is interpreted by *ffmpeg* in the same way. - (`np.ndarray` of shape (n, ) of type `np.float32` or `np.float64`) Raw audio at the correct sampling rate (no further check will be done) - `dict` form can be used to pass raw audio sampled at arbitrary `sampling_rate` and let this pipeline do the resampling. The dict must be in the format `{"sampling_rate": int, "raw": np.array}` with optionally a `"stride": (left: int, right: int)` than can ask the pipeline to treat the first `left` samples and last `right` samples to be ignored in decoding (but used at inference to provide more context to the model). Only use `stride` with CTC models. return_timestamps (*optional*, `str` or `bool`): Only available for pure CTC models (Wav2Vec2, HuBERT, etc) and the Whisper model. Not available for other sequence-to-sequence models. For CTC models, timestamps can take one of two formats: - `"char"`: the pipeline will return timestamps along the text for every character in the text. For instance, if you get `[{"text": "h", "timestamp": (0.5, 0.6)}, {"text": "i", "timestamp": (0.7, 0.9)}]`, then it means the model predicts that the letter "h" was spoken after `0.5` and before `0.6` seconds. - `"word"`: the pipeline will return timestamps along the text for every word in the text. For instance, if you get `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp": (1.0, 1.5)}]`, then it means the model predicts that the word "hi" was spoken after `0.5` and before `0.9` seconds. For the Whisper model, timestamps can take one of two formats: - `"word"`: same as above for word-level CTC timestamps. Word-level timestamps are predicted through the *dynamic-time warping (DTW)* algorithm, an approximation to word-level timestamps by inspecting the cross-attention weights. - `True`: the pipeline will return timestamps along the text for *segments* of words in the text. For instance, if you get `[{"text": " Hi there!", "timestamp": (0.5, 1.5)}]`, then it means the model predicts that the segment "Hi there!" was spoken after `0.5` and before `1.5` seconds. Note that a segment of text refers to a sequence of one or more words, rather than individual words as with word-level timestamps. generate_kwargs (`dict`, *optional*): The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a complete overview of generate, check the [following guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). Return: `Dict`: A dictionary with the following keys: - **text** (`str`): The recognized text. - **chunks** (*optional(, `list[Dict]`) When using `return_timestamps`, the `chunks` will become a list containing all the various text chunks identified by the model, *e.g.* `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp": (1.0, 1.5)}]`. The original full text can roughly be recovered by doing `"".join(chunk["text"] for chunk in output["chunks"])`. )rt__call__)rvr0rwrqs r!rzz+AutomaticSpeechRecognitionPipeline.__call__s%r uww11&111r#c pi}i} i} |<|jdvr.|s,d} |jdkr| dz } t| ||d<|||d<d|vr(| |d| |t |dd |j| d<t |d d|j| d <|j| d <||| d <|$|jdkrtd || d <|| d <t|dr#t|j dr|p |j j }||jdkr|rtd|jdkr|dkrtd|jdkr|dvrtd|jdkr|dkrtd|| d<|| d<|| | fS)N)rfreaUsing `chunk_length_s` is very experimental with seq2seq models. The results will not necessarily be entirely accurate and will have caveats. More information: https://github.com/huggingface/transformers/pull/20104. Ignore this warning with pipeline(..., ignore_warning=True).rez To use Whisper for long-form transcription, use rather the model's `generate` method directly as the model relies on it's own chunking mechanism (cf. Whisper original paper, section 3.8. Long-form Transcription).chunk_length_sstride_length_sgenerate_kwargsassistant_modelassistant_tokenizerrFdecoder_kwargsz)Only Whisper can return language for now.return_languagegeneration_configreturn_timestampsrfzEWe cannot return_timestamps yet on non-CTC models apart from Whisper!rlwordzRCTC with LM can only predict word level timestamps, set `return_timestamps='word'`rm)charrzCTC can either predict character level timestamps, or word level timestamps. Set `return_timestamps='char'` or `return_timestamps='word'` as required.rzWhisper cannot return `char` timestamps, only word level or segment level timestamps. Use `return_timestamps='word'` or `return_timestamps=True` respectively.) rploggerwarningupdatepopgetattrrrFr ValueErrorhasattrrr) rvr|r}ignore_warningrrrr~preprocess_paramsforward_paramspostprocess_params type_warnings r!_sanitize_parametersz7AutomaticSpeechRecognitionPipeline._sanitize_parameterss  %y:::>:- 9 111 4L |,,,2@ . /  &3B / 0  / /  ! !/"5"56G"H"H I I Io... 4*D 1 1 =040DN, - 4. 5 5 A*..N; '484LN0 1  %3A / 0  &y--- !LMMM4C 0 10?N, - 4, - - ^'$:PRe2f2f ^ 1 ]T5K5]   (yI%%*;% !hiiiyM)).?6.I.I !uvvvyE!!&7?O&O&O `y---2Cv2M2M _3DN. /6G 2 3 .2DDDr#ct|jjdd}|jjjdkr||jjz}|S)zSample stride per output.inputs_to_logits_ratiorlasr_ctc)rr`rnror1 hop_length)rvalign_tos r! _align_toz,AutomaticSpeechRecognitionPipeline._align_toKsE 4:,.FJJ :  ': 5 5 .9 9Hr#rc# Kt|tr|ds|drtj|dj}nd|vr|d} n$td ||pt |jd d }|r-|jd kr"t ||d <|d kr d|d<d|d<d|vr |j|d<t|jdr |jj nd} | | d|i|}|r|jd krd|d<|jj d'i|} |d kr?|jd kr4d| vr| d| dd} nFd| dD} | d| d} n(t| trd| vr d| di} nd| i} |jd kr||| d<|rt| trd| vr| d}|r|dr|ddd}t|tr|dn|}| d|j}t|dr[t|j}|D] }||vrt%j|g| d<n!nH|jdvr|jj ||jj d|i} |jd'i| }|j}|jdkrd |i} nd|d!"i} |Md#|jz }t|t.rt1|g|d| d<nt1||| d<n|jd$krc|jj ||jj i} d|vr|d| d<|jj d'i| }d|ji} ntd%|jd&|}d|i| |S)(Nattention_maskr num_framesr,z0num_frames must be used only when stride is Nonerfreinput_features input_valueszhSeq2Seq speech recognition model requires either a `input_features` or `input_values` key, but only has rFrerTreturn_token_timestampsreturn_segmentsrmain_input_namer0segmentsrOtoken_timestamps)tokensrcJg|] }tjd|D!S)cg|] }|d S)rr@)rDsegments r!rGzJAutomaticSpeechRecognitionPipeline._forward...s"["["[77+=#>"["["[r#)rcat)rD segment_lists r!rGz?AutomaticSpeechRecognitionPipeline._forward..sA(((( "["[l"["["[\\(((r#rrresult lang_to_idlang_idrmrlrllogitsdimrrkzUnsupported model type .r@)rrrpkeysrrboolrr`rgeneraterrrsetrrsrIrtensorrargmaxrtupler"rO)rv model_inputsrrr~rrrr,r0rroutrrrfull_seq gen_configlang_idstoken_idoutputsrrrs r!_forwardz+AutomaticSpeechRecognitionPipeline._forwardsV%))*:DAA!!(D11!%%lD99 ""9--  *"8OPP P 96 6 6 <//%))*:;;<//%)).99 bLXL]L]L_L_bb !2 pWT=SUhjo5p5p   >TY2C%C%C7;9=O$56#/997;7M 34((($&,K%8N^__CCFD)) )kV.C.C !45(y---%$*CM"*z&$'?'?*JRXDXDX &j1H *HQK *!)!Q!9:DVT:R:R#^6+#6#6X^%4%8%89LdNd%e%e ":|<<*'*:+@+G+G+I+I'J'JH,4OO,=,=**#+x#7#75:\8*5M5MC N$)E$8Y0 0 0 *L,<,> **,,[[400F!di3I&I&I'-$u "E/aaago.   u % % % % # %di9,,1+t~NNEE Y+ + +!3@4:CTCiiN 2@M' L Lv%%;A(;K8I{L.I=0K M1L'0+|'KF8$  ]+]]F F**"("3";;==A--&-&7&7&:&:G%,%6%6q%9%9%<%<6(CSCY%<%Z%Z +09j&BR5SY[+\+\+\x(!^77"3 /- 8ND((N;Q777EMM!$$E 9 % %%!#-DL-eFF~FFE8A;D  k %Qx{ 8Dkk4D4<NND,^h#i#ijjjj Y+ + +"&)u"4 8(iiDWi[hiiD  t/$.//BX\`m "%.."n>>wHrssG  (2P!P!PF~H ] ]^,x7/==L)H4.<< t,='>eUY][[\\\\!'HX D!!# # #F JJx & & & JJx & & & JJy$ ' ' ' JJx & & & JJ)4 0 0 0 JJy$ ' ' '  # #1a"""" #22E22r#)NNNN)NNNNNN)rN)FN)NNN)rr __module__ __qualname____doc___pipeline_calls_generate_load_processor_load_image_processor_load_feature_extractor_load_tokenizerr_default_generation_configrrr rrurKrrrrrrzrpropertyrrrr! __classcell__)rqs@r!r[r[ps''R $O!"O!1!1"""LP04=A48 WW W!!;S!@ADHW'- W -s23d: W c>)*T1 WWWWWW292rzE1C7$>92#92RVW[\_ad\dWeRf929292929292zPEPEPEPEd  X H:H:H:H:Ts4s4s4s4ljnn3n3-1D[n3n3n3n3n3n3n3n3r#r[)N)& collectionsrtypingrrrrrrK generationrtokenization_pythonr utilsr r r r audio_utilsrbaser pyctcdecoder!feature_extraction_sequence_utilsrmodeling_utilsr get_loggerrrrrmodels.auto.modeling_autorr"r=rYr[r@r#r!r9s$#####,,,,,,,,,, ))))))555555aaaaaaaaaaaa$$$$$$1000000LLLLLL000000  H % %ULLLTTTTTT(42V 3V 3V 3V 3V 3V 3V 3V 3V 3V 3r#