tjZ.dZddlZddlZddlZddlmZddlZddlm Z m Z m Z m Z m Z ddlmZddlmZddlmZd d lmZmZejeZd d d ddZiddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1id2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSidTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtduidvdwdxdydzd{d|d}d~dddddddddddddddddddddddddiddddddddddddddddddddddddddddddddddddddddddddddddddɜZidʄeDdd&d(dddddBdBdddd͜ ZddgZGdЄdeZ dde e!de"dzde"de"fd؄Z#ddلZ$ddۜd܄Z%dd݄Z$dބZ& dde e!de"dzde"de"fd߄Z#de e!fdZ'de e!fdZ(dZ)dgZ*dS)z!Tokenization classes for Whisper.N) lru_cache) AddedToken Tokenizerdecoderspre_tokenizers processors)BPE)TokenizersBackend)logging)BasicTextNormalizerEnglishTextNormalizerz vocab.jsonztokenizer.jsonz merges.txtznormalizer.json) vocab_filetokenizer_file merges_filenormalizer_fileenenglishzhchinesedegermanesspanishrurussiankokoreanfrfrenchjajapanesept portuguesetrturkishplpolishcacatalannldutchararabicsvswedishititalianid indonesianhihindififinnishvi vietnamesehehebrewuk ukrainianelgreekmsmalaycsczechroromaniandadanishhu hungariantatamilno norwegianththaiururduhrcroatianbg bulgarianlt lithuanianlalatinmimaoriml malayalamcywelshskslovaktetelugufapersianlvlatvianbnbengalisrserbianaz azerbaijanisl slovenianknkannadaetestonianmk macedonianbrbretoneubasqueis icelandichyarmeniannenepalimn mongolianbsbosniankkkazakhsqalbanianswswahiliglgalicianmrmarathipapunjabisisinhalakmkhmersnshonayoyorubasosomaliaf afrikaansococcitankageorgianbe belarusiantgtajiksdsindhigugujaratiamamharicyiyiddishlolaouzuzbekfofaroesehtzhaitian creolepspashtotkturkmennnnynorskmtmaltesesanskrit luxembourgishmyanmartibetantagalogmalagasyassamesetatarhawaiianlingalahausabashkirjavanese sundanese cantonese)salbmybotlmgastthawlnhabajwsuyueci|]\}}|| Sr).0codelanguages n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/whisper/tokenization_whisper.py rs>>>.$x>>>rr) burmese valencianflemishhaitian letzeburgeschpushtopanjabi moldavianmoldovan sinhalese castilianmandarin translate transcribecneZdZdZeZddgZeZ d6de e e e fzdzffd Z d7d e ffd Z d8d Zed9dZd:defdZdZ d;dededzdedededededed e ffd Zdddddededed e ffdZdZed:dZdd'ee d(ee dzd)ed ee ffd* Z!d?d,Z"d-Z#d@d/e fd0Z$d1ee d2e d3e fd4Z%ed5Z&xZ'S)AWhisperTokenizerag Construct a "fast" Whisper tokenizer (backed by HuggingFace's *tokenizers* library). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. normalizer_file (`str`, *optional*): Path to the normalizer_file file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The beginning of sequence token. The `decoder_start_token_id` is used to set the first token as `"<|startoftranscript|>"` when generating. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (Whisper tokenizer detect beginning of words by the preceding space). language (`str`, *optional*): The language of the transcription text. The corresponding language id token is appended to the start of the sequence for multilingual speech recognition and speech translation tasks, e.g. for Spanish the token `"<|es|>"` is appended to the start of sequence. This should be used for multilingual fine-tuning only. task (`str`, *optional*): Task identifier to append at the start of sequence (if any). This should be used for mulitlingual fine-tuning, with `"transcribe"` for speech recognition and `"translate"` for speech translation. predict_timestamps (`bool`, *optional*, defaults to `False`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. input_idsattention_maskN <|endoftext|>Fvocabc pt|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}||ni|_||ng|_t t |j|jdddd|_tj ||j_ tj |j_ tjd ||||||| | d| |Ct|d 5} t!j| |_dddn #1swxYwYnd|_t'jd |_||_| |_| |_|dS) NFT)lstriprstrip normalizedspecial)rmergesdropoutcontinuing_subword_prefixend_of_word_suffixfuse_unk)add_prefix_space) unk_token bos_token eos_tokenrrrtaskpredict_timestampsutf-8encodingz<\|(\d+\.\d+)\|>r) isinstancestrr_vocab_mergesrr _tokenizerr ByteLevel pre_tokenizerrdecodersuper__init__openjsonloadenglish_spelling_normalizerrecompile timestamp_patrrrset_prefix_tokens)selfrrrrrrrrrrkwargs vocab_handle __class__s rrzWhisperTokenizer.__init__sM )S)) JyuX\ ] ] ] ] )S)) JyuX\ ] ] ] ] )S)) JyuX\ ] ] ] ]  %0eeb !'!3vv # k|*,#%       )7(@Rb(c(c(c%"*"4"6"6 -+1    &o888 KL379\3J3J0 K K K K K K K K K K K K K K K04D ,Z(;<<   "4      s;E!!E%(E%{Gz?returnc|jddz}gg}d}d}d} t|D]\} } | |krt| |z |z} | |krA| dko|| dz |ko|| dz |k } | r |||zz }n| }|| z }|dd}|} | }|d| |zdd |g|d| g}|D]{}t |t r||-|r7|t || f|d |d |S) z Timestamp tokens are above the special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". r Nz<|z.2fz|>skip_special_tokensr) all_special_ids enumeratefloatappendr r rdecodejoin)r token_idsr*time_precision segment_sizetimestamp_beginoutputscur_max_timestampprev_segments_lenpenultimate_timestampitoken timestamplast_was_single_endingdecoded_outputssr s r_decode_with_timestampsz(WhisperTokenizer._decode_with_timestampss.r2Q6$ #!),, * *HAu''!5?#:n"LMM 000-.!V.!!a%(O;c !a%@PTc@c9*./)^l-JJ)),A))-BB)")#2#,(9%$-!KY1B%BKKKKLLLr"""" ""5)))) + +A!S!! +&&q)))) +&&uww~~aM`~'a'abbbb&&r****ww'''rc8g}dtt|vr8t|dr(t|jr|}t j|}|jddkr't|jdkrtd|j ddz}||k}t j |dd|ddzddz}|jddkr| dkrgSt j |dddz|vr6t j |t j |dddz}t j |dd}d} d} |D]} ||| } t| dkr| d|z } | d|z }| | kr2|dko||dz |ko||dz |k }|r| |z } n| | z } |} || } || }||}| || |z| |zz||z| |zzfd | }|S) a Compute offsets for a given tokenized input Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. segment_size (`int`, *optional*, defaults to 1500): The number of features in the input mel spectrogram. torchcpurr z)Can only process a single input at a timer%Nr'textr;)r typehasattrcallablerBnparrayshapelen ValueErrorr+wheresumr.item_preprocess_token_ids_decode_filter_timestamp_ids)rr1r2r3offsetsr4timestamp_tokens consecutive last_slicer6r7 current_slice sliced_tokensstart_timestamp_positionend_timestamp_positionis_single_endingrDs r_compute_offsetsz!WhisperTokenizer._compute_offsetsHs c$y//** * * 50I0I *hW`WdNeNe *! IHY'' ?1  ! !c)/&:&:Q&>&>HII I.r2Q6$7h/47G7KKLLQORSS  Q 1 $ $)9)=)=)?)?1)D)DI X& ' ' *2 . 2+ E E)K:J1K1KA1Nr1RUV1VWWKX.//215 ( ' 'M%j&>?M=!!A%%+8+;+@+@+B+B_+T()6r):)?)?)A)AO)S&+.???'1Q(!*q.1_DuS]`aSaIbfuIu<$(?)\9)))->>)$:!!% : := I I ||M2211$77 $4~EHY\jHjj2^CFWZhFhh&'JJrc`|fdtdDS)a Compute the timestamp token ids for a given precision and save to least-recently used (LRU) cache. Args: time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. c g|] }d|zz S)z<|%.2f|>r)rr9r2s r z2WhisperTokenizer.timestamp_ids..s$*h*h*hSTJ!n:L,M*h*h*hri)convert_tokens_to_idsrange)rr2s `r timestamp_idszWhisperTokenizer.timestamp_idss8))*h*h*h*hX]^fXgXg*h*h*hiiirr*c|rA|d}|d}||||}|S)a Pre-process the token ids for decoding by removing the prompt tokens ids and timestamp token ids. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Typically, obtained using the `__call__` method of the tokenizer. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens from the token ids. If `True`, the prompt token ids will be removed. <|startofprev|><|startoftranscript|>)r` _strip_prompt)rr1r*prompt_token_iddecoder_start_token_ids rrPz&WhisperTokenizer._preprocess_token_idssT  _"889JKKO%)%?%?@W%X%X "**9oG]^^Irc8tj|jd|S)Nr)rsubr)rrDs rrRz&WhisperTokenizer._filter_timestamp_idssvd("d333rclean_up_tokenization_spacesoutput_offsetsr2decode_with_timestamps normalizebasic_normalizeremove_diacriticsc \||} tj| f||||| d| } |r| ||} n9t | t rfd| D} n| } |r||} | | dS| S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Will remove the previous tokens (pre-prompt) if present. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). output_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output the offsets of the tokens. This should only be set if the model predicted timestamps. If there are previous tokens (pre-prompt) to decode, they will only appear in the decoded text if they contain timestamp tokens. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. decode_with_timestamps (`bool`, *optional*, defaults to `False`): Whether or not to decode with timestamps included in the raw text. normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the English text normalizer to the decoded text. Only applicable when the target text is in English. Otherwise, the basic text normalizer should be applied. basic_normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the Basic text normalizer to the decoded text. Applicable to multilingual target text. remove_diacritics (`bool`, *optional*, defaults to `False`): Whether or not to remove diacritics when applying the Basic text normalizer. Removing diacritics may destroy information in the decoded text, hence it should be used with caution. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. r))r*rkrnrorp)r2r*c:g|]}|Sr)rR)rtrs rr_z+WhisperTokenizer.decode..s'DDD!22155DDDr)r2)rDrS)rPrr/r?r listrRr\)rr1r*rkrlr2rmrnrorpr filtered_idsrDrSr s` rr/zWhisperTokenizer.decodes d11  32  uww~  3)E+/      " 8//^Qd0DD $%% 8DDDDtDDD11$77  6++In+UUG W55 5 r)rnrorpctj|i|}|r||}|S|r|||}|S|S)Nrp)rrQrnro) rrnrorpargsrrD clean_textr s rrQzWhisperTokenizer._decodesjuww///  --J   --dFW-XXJ Krc@t|j}||S)z Normalize a given string using the `EnglishTextNormalizer` class, which performs commons transformation on english text. )rr)rrD normalizers rrnzWhisperTokenizer.normalizes$ +4+KLL z$rc8t|}||S)z Normalize a given string using the `BasicTextNormalizer` class, which performs commons transformation on multilingual text. rw)r)rDrpr{s rroz WhisperTokenizer.basic_normalizes% );LMMM z$rsave_directoryfilename_prefixc tj|s td|ddStj||r|dzndt dz}tj||r|dzndt dz}tj||r|dzndt dz}t|dd 5}|tj |j d d d dzdddn #1swxYwYt|dd 5}|d| d|j Ddddn #1swxYwY|j]t|dd 5}|tj |jd d d dzdddn #1swxYwY|||fS)NzVocabulary path (z) should be a directory-rrrrwrr r'TF)indent sort_keys ensure_ascii z#version: 0.2 c3FK|]}d|dzVdS) rN)r0)r merge_pairs r z3WhisperTokenizer.save_vocabulary..5s3YYjchhz22T9YYYYYYr)ospathisdirloggererrorr0VOCAB_FILES_NAMESrwriterdumpsr  writelinesrr)rr}r~r merge_filerfwriters rsave_vocabularyz WhisperTokenizer.save_vocabulary!sw}}^,,  LLT^TTT U U U FW\\ oM_s222QbcoQpp  W\\ oM_s222QbcpQqq  ',, oM_s222QbctQuu  *cG 4 4 4 b GGDJt{1SXYYY\`` a a a b b b b b b b b b b b b b b b*cG 4 4 4 Z LL* + + +   YYDLYYY Y Y Y Z Z Z Z Z Z Z Z Z Z Z Z Z Z Z  + 7osW=== Jt?UYhmnnnquu                J88s614D11D58D5:FFF84G88G<?G<rrrc x||n|j|_||n|j|_||n|j|_|j}||}|j}|j}dd|D}tj |d|d|d|d||fgt|||j _ dS) a Override the prefix tokens appended to the start of the label sequence. This method can be used standalone to update the prefix tokens as required when fine-tuning. Example: ```python >>> # instantiate the tokenizer and set the prefix token to Spanish >>> tokenizer = WhisperTokenizerFast.from_pretrained("openai/whisper-tiny", language="spanish") >>> # now switch the prefix token from Spanish to French >>> tokenizer.set_prefix_tokens(language="french") ``` Args: language (`str`, *optional*, defaults to `None`): The language of the transcription text. task (`str`, *optional*, defaults to `None`): Task identifier to append at the start of sequence (if any). predict_timestamps (`bool`, *optional*, defaults to `None`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. Nrcg|]}|dS):0rrr:s rr_z6WhisperTokenizer.set_prefix_tokens..]s#G#G#GUuLLL#G#G#Grz $A:0 rz $A:0 $B:1 z:1)singlepairspecial_tokens) rrr prefix_tokensconvert_ids_to_tokensr eos_token_idr0rTemplateProcessingzipbackend_tokenizerpost_processor) rrrrprefix_token_idsprefixeseosrprefix_templates rrz"WhisperTokenizer.set_prefix_tokens?s,%-$8dm  ,DD$) 8J8V"4"4\`\s---.>??n( ((#G#Gh#G#G#GHH0:0M%44S444#77777l#X/001 1 1 ---rc 6|d}|d}|d}|d}tt}|j|j|_|jt vrt |j}n|jt vr|j}nt|jdk}td|jd|r&tt n%tt d|j -|j tvrtd |j d t|g}|j.| |d z||z|j "| |j d kr|n||js| ||S) Nrez <|translate|>z<|transcribe|><|notimestamps|>r'zUnsupported language: z. Language should be one of: .zUnsupported task: z. Task should be in: r r)r`tuple LANGUAGESkeysrlowerTO_LANGUAGE_CODEvaluesrKrLrtrTASK_IDSr.indexr) r bos_token_idtranslate_token_idtranscribe_token_idnotimestamps_token_idlangs language_idis_language_code bos_sequences rrzWhisperTokenizer.prefix_tokensgs112IJJ !77HH"889IJJ $ : :;M N Ninn&&'' = $ M//11DM} 000.t}= "2"9"9";";;;"m #&t}#5#5#:  rT]rr;Kn-4466777QUVfVkVkVmVmQnQnrrr 9 y(( !`di!`!`V^!`!`aaa$~ = $    q 05;;{3K3K K L L L 9   tyL7P7P 3 3Vh i i i& 7    5 6 6 6rcX||j|z|jgzS|j|z|z|jgzS)z=Build model inputs from a sequence by appending eos_token_id.)rr)r token_ids_0 token_ids_1s r build_inputs_with_special_tokensz1WhisperTokenizer.build_inputs_with_special_tokenss?  % 3t7H6II I!K/+=AR@SSSrrralready_has_special_tokensc|r$t||dSdgt|jz}dg}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr Nr)rget_special_tokens_maskrKr)rrrr prefix_ones suffix_onesr s rrz(WhisperTokenizer.get_special_tokens_masks& & 7722'[]a3 cC 2333 c  1#K(8(8"89KG GqcC $4$445!s;?O?O9OPS^^^rTc|||| |jdd}dt|D}|S)N)rrrr c"g|] \}}|dz|f S)r r)rrankr:s rr_z;WhisperTokenizer.get_decoder_prompt_ids..s%\\\KD%tax/\\\r)rrr,)rrr no_timestamps forced_tokensforced_decoder_idss rget_decoder_prompt_idsz'WhisperTokenizer.get_decoder_prompt_idssV D8TaPabbb *122. \\9]C[C[\\\!!rc*t|||||S)N)return_timestampsreturn_languager2) _decode_asr)r model_outputsrrr2s rrzWhisperTokenizer._decode_asrs(  /+)     rrHrDc2dd|zd}|ddd}tfd|Dd}|(|}td |d || |dS) z`Converts prompt text to IDs that can be passed to [`~WhisperForConditionalGeneration.generate`].rdrF)add_special_tokensrr Nc3>K|]}|jdk|VdS)rN)r+)rxrs rrz2WhisperTokenizer.get_prompt_ids..s6 \ \qqDDXYZD[?[?[?[?[?[?[ \ \rzJEncountered text in the prompt corresponding to disallowed special token: r) tensor_type)stripnextrrLconvert_to_tensors)rrDreturn_tensorsbatch_encodingprompt_text_idsspecial_token_idr:s` rget_prompt_idszWhisperTokenizer.get_prompt_idss/tzz||1CX]^^^)5abb9 \ \ \ \O \ \ \^bcc  '../?@@Erjorrrss s))n)EEEk**rr1rgrhct|ts||}|s|S|d|k}|r#||vr|||dSgS|S)Nr)r rt_convert_to_listr)rr1rgrh has_prompts rrfzWhisperTokenizer._strip_prompts)T** 9--i88I  q\_4  %22 1G!H!H!J!JKK rct|dr&|}t|tjr|}|S)Nnumpy)rFrBrr rHndarraytolist)r1s rrz!WhisperTokenizer._convert_to_listsZ 9g & & 0! --//I i , , +!((**Ir) NNNrrrFNNF)Fr!r")r!r")r!)F)FNFr!FFFFN)NNN)NF)NNT)rH)(__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr modelr dictintrr?r\rrbboolrPrRr-r/rQrn staticmethodrorrrpropertyrtrrrrrrrfr __classcell__)r s@rrrs@$$L*$&67 E.2!!!  F!F!T#s(^#d*F!F!F!F!F!F!TW[.(.( .(.(.(.(.(.(bBBBBHjjjYjD&444%*48$ $', %"'PP"P'+Tk P  P  P!%PPP P PPPPPPf(-egl    $ ?C `d              \ 99c9C$J9Z_`cZd9999>gk& & d & 14t& X\_cXc& & & & PtCyXDTTQUVYQZTTTTpu__9_379t3C_hl_ c______>""""    + +3 + + + +tCy3`c$\rr "'“¡¿([{-"'.。,，!！?？:：”)]}、tokensrprepend_punctuationsappend_punctuationsc||j}|d}|dvrt||\}}}nt||\}}}t||||||||fSz Groups tokens by word. Returns a tuple containing a list of strings with the words, and a list of `token_id` sequences with the tokens making up each word. Nr>rrQrrr#rr_split_tokens_on_unicode_split_tokens_on_spaces_merge_punctuations tokenizerrrrrwords word_tokens token_indicess r_combine_tokens_into_wordsr}%QQQ,DYPV,W,W){MM,CIv,V,V){M{M;OQdeee +} ,,rc |d}t|}g}r dg}t|ddD]\}d}||ddf}t|} td|| zD]} | dz } td|| z t ||| z| z } t j|| } td| |z t | | }t j||t| tkrtdr.tfdt| D}nt j| k}|| z | z}|dkr||kr|}| |f}!|\} }| zdz}|zdz}| |d|||d}t|}r0| d|dz|d| ||Stdkr| ||fS|gfS)Nrr r&@iThere is a bug within whisper `decode_asr` function, please report it. Dropping to prevent bad inference.c3|K|]6\}}||kr%|zdz|zk2dV7dSr Nr ridxelem left_startleft_token_timestamp_sequenceright right_startseq_idxtoken_timestamp_sequencess rrz0_find_longest_common_sequence..V~!Tc **9*s:JK4Wq[A+PSBSTUU UUUU rr' rKr,ramaxminrHrI RuntimeErrorrNextend sequencesr left_sequence left_lengthtotal_sequencetotal_token_timestamp_sequenceright_sequencemax_ max_indices right_lengthr9eps left_stopleft right_stopmatchesmatchingleft_mid right_midr rrrrs ` @@@@@r_find_longest_common_sequencer+ aLMm$$KN ,(A!(D%)+&#,Yqrr]#;#;[_[_"KA6 <>** q+ 455( O( OAg+C Q a00JK|)Ca)GHHI8M*Y*>?@@Da[11K\1--JH^K ,BCDDE4yyCJJ&&") 0%.t__&//{S(H{{x$)9k:N ;F8Y Z *q0+-!3 mIXI6777&yzz2 -(( $ _ * 1 12OPYQYPY2Z [ [ [,EgPQk,RS\S]S],^ )-((( ( $%%))&--.KLLL===r!!rr")r3c $ /d//fd}g}|}d} |ddz} g} g} d} d}t|j}|d}|d}t|D]\}}|d d }||||}|d kr |d d }d}| }d}d}d}d |vrL|d \}}}| |z} ||z }|r||z | z}|r*t |D]}|| kr||| z |z|krn|}g}g} t|D]\}!}||vr||g}"|"dd}"t |"}#|#s/ri|#/krc|sa| |t| }$||$}%|%|d<| |g} g}|}|#|d<|#/|| krjt|| z |z}&|&|kr7|!dko||!dz | ko||!dz | k }'|'r |||zz }n|}||z }|}|&}|| z |z| z|z}(t|(d}(|r ||krd} A| s| r ||krd} O|dd  |(|dd <j|(|dd kr~|(|dd<| ||d kr| | t| | \}$})||$}%|%|d<|d krt||$|)/||d<| |g} g}g} g} |}+| ||d krf|!d krtd| zd}*nt||!dz | zd}*t||!| zd}+| |*|+fd |vr| ||z z } |r2| ||d kr| | td| Ds|}g} g}g} g} | rz|rt dt| | \}$})||$}%|%|d<|d krt||$|)/||d<| |dd|D},|s|r~|D]I}|s|dnt)|d|d<|s|dJ|d kr'g}-|D]}|-|dd|-i}.nd|i}.ni}.|,|.fS)z Internal method meant to only be used by asr pipeline. Handles all the little quirks specific to whisper to handle the various options not allowed in other seq2seq models NcddgddS)Nr)rr;rDr) last_languagesr new_chunkz_decode_asr..new_chunks)t bQQQrr&rr Frdrerrwordtoken_timestampsstrider'r(rDrTr;rc3K|]}|VdSrr)rps rrz_decode_asr..Qs"11Aa111111rzWhisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.rc3&K|] }|dV dS)rDNr)rchunks rrz_decode_asr..ks&::%f ::::::rchunks)r`setr+r,rrfreversedr/rgetr.r+r-round_collate_word_timestampsanyrwarningr0poprr)0rrrrr2r3r0r8r7 time_offsetr4previous_tokensprevious_token_timestampsskipright_stride_startr+rgrhchunk_idoutputr1r2last_timestampfirst_timestampr6r7r8 chunk_len stride_left stride_rightr:current_tokenscurrent_token_timestampsr9rDrresolved_tokens resolved_textr;r<timeresolved_token_timestamps start_timeend_time full_text new_chunksoptionalr/s0 @rrrs$MRRRRRF IKKEK556HIIAMOO " D)344O556GHHO&<<=TUU%m44l*l*&8$Q'..00 ++IH^__  & &%&89!<CCEE  )  # v  39(3C 0I{L ; &K!*\!9   Q"-">"P /%i00 / /E//+6!&!8N JM_ _ _!E).#% "),,l Ll LHAu '' ''00AbDz$==..'% ,])B)BK\)B'..~>>>*G*X*X(1(8(8(I(I (5f  e,,,+-)+ ) (0E*%$,MM/))"5?#:n"LMM 000-.!V.!!a%(O;c !a%@PTc@c9*.C)^l-JJ)),A))-BB)(9%$-!/>AKORccT1~~!+,e~&=&=  DD$,o$,%/2I2I DD;'*2,0E+&q))u[1!44404k*1-'..~>>>,665<<=UVVVEb+-FFFB)B)2(8(8(I(I (5f ,66-E )? 2 2 2 F**)001IJJJ1111111 *IKKE ON(* %') $   NN]    6S 66 6 22"((99 %f  & &5?,E}VeE'N  e::6:::::IO & &E$ ? +&&&&%*5+=%>%>k"" & *%%%  & &J 2 2!!%.1111 *-HH &)HH h rc |d}t|}g}r dg}t|ddD]\}d}||ddf}t|} td|| zD]} | dz } td|| z t ||| z| z } t j|| } td| |z t | | }t j||t| tkrtdr.tfdt| D}nt j| k}|| z | z}|dkr||kr|}| |f}!|\} }| zdz}|zdz}| |d|||d}t|}r0| d|dz|d| ||Stdkr| ||fS|gfS)Nrr r&rrc3|K|]6\}}||kr%|zdz|zk2dV7dSr rr s rrz0_find_longest_common_sequence..rrr'rrs ` @@@@@rr+r+r,rc~ t|||\}}}|rd|ini fdt||D}|S)Nrcpg|]2\}}||dd|ddfd3S)rr%r rCr)rr1indicesoptional_language_fieldr2s rr_z,_collate_word_timestamps..sj D' *71:6q9;KGTVK;XYZ;[\  & r)rr) rrr2rrr_rtimingsr]s ` @rr=r=sq8FHUUE1m8GOz844R ! 66 G Nrc||j}|d}|dvrt||\}}}nt||\}}}t||||||||fSrrrs rrr rrcJ||d}d}g}g}g}g}g}d} t|D]\} } || || ||d} || vsK| | |zt |ks"|| | |z|krU|| ||||g}g}| t | z } |||fS)zlCombine tokens into words by splitting at any position where the tokens are decoded as valid unicode points.T)rmu�r)r/r,r.rrK) rr decoded_fullreplacement_charrrrrMcurrent_indicesunicode_offset token_idxr:decodeds rrr#sQ##F4#HHL EKMNON%f--++ 5e$$$y)))"">$"OO G + + .> ? ??3|CTCTTTNW]];K-L-LLMQaaa LL ! ! !   ~ . . .   1 1 1N O c'll *N +} ,,rc"t||\}}}g}g}g}t|||D]\}} } | d|jk} |d} |dv} | s| s| st |dkr@|||| || |d|z|d<|d| |d| |||fS)zLCombine tokens into words by splitting at whitespace and punctuation tokens.rrz !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~r%)rrr startswithrrKr.r)rrsubwordssubword_tokens_listsubword_indices_listrrrsubwordsubword_tokenssubword_indicesr with_space punctuations rrrCs;:RS\^d:e:e7H!#7 EKM47BUWk4l4l 6 60 #y'=='',, mmoo)MM  6j 6K 63u::?? LL ! ! !   ~ . . .   1 1 1 1b G+E"I O " "> 2 2 2 "  $ $_ 5 5 5 5 +} ,,rc0t|dz }t|dz }|dkr||drh|||vrL||||z||<||||z||<||||z||<d||<g||<g||<n|}|dz}|dkd}d}|t|kr||ds\|||vrR||xx||z cc<||xx||z cc<||xx||z cc<d||<g||<g||<n|}|dz }|t|kd|D|dd<d|D|dd<d |D|dd<dS) z1Merges punctuation tokens with neighboring words.r'r rrrcg|]}||Srr)rr1s rr_z'_merge_punctuations..|s///$////rNcg|]}||Srrrs rr_z'_merge_punctuations..}s4445e4444rcg|]}||Srr)rr s rr_z'_merge_punctuations..~s000#C0#000r)rKrirendswith)rrr\ prependedappendedr9js rrr[s E QA E QA q&& 8  s # # a(8(8I(E(EQx%(*E!Hq F1I-F1I gaj0GAJE!HF1IGAJJA Q q&& A A c%jj..Qx  %% %(h*>*> !HHHa HHH 1III "III AJJJ'!* $JJJE!HF1IGAJJA Q c%jj..0////E!!!H44F444F111I00000GAAAJJJr)Nrrr)+rrrr functoolsrrrH tokenizersrrrrrtokenizers.modelsr tokenization_utils_tokenizersr utilsr english_normalizerrr get_loggerrrrritemsrrrrtrr rr+rr=rrr__all__rrrrs&(' RRRRRRRRRRRRRR!!!!!!>>>>>>JJJJJJJJ  H % %&( e )e )e  (e  ) e   ) e   ( e  (e  *e  ,e  )e  (e  )e  'e  (e  )e   )!e " ,#e e $ '%e & )'e ( ,)e * (+e , +-e . '/e 0 '1e 2 '3e 4 *5e 6 (7e 8 +9e : ';e < +=e > &?e @ &Ae B *Ce D +Ee e e F ,Ge H 'Ie J 'Ke L +Me N 'Oe P (Qe R (Se T )Ue V )We X )Ye Z )[e \ -]e ^ +_e ` )ae b *ce d ,ee f (ge e e h (ie j +ke l *me n (oe p +qe r )se t (ue v *we x )ye z *{e | )}e ~ )e @ )Ae B 'Ce D 'Ee F (Ge H (Ie e e J +Ke L )Me N *Oe P ,Qe R 'Se T (Ue V *We X )Ye Z )[e \ %]e ^ '_e ` )ae b ce d (ee f )ge h )ie j )ke e l               Ie e e  P>>IOO,=,=>>>  &M M M M M (M M M f 0A -- I-Dj- -  ----4u"u"u"u"posyyyyyxu"u"u"u"p   & 0A -- I-Dj- -  ----4-S ----@-tCy----0#1#1#1L  r