tj$dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z e rddl Z e rddl Ze jeZddiZd ZGd d eZd gZdS) zTokenization class for VITS.N)Any)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availablelogging vocab_filez vocab.jsonc`tjd}||}|du}|S)Nz [^\x00-\x7F])recompilesearch) input_stringnon_roman_patternmatch has_non_romans h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_charactersr$s7 ?33  $ $\ 2 2E%M c eZdZdZeZddgZ d dfd Zed Z d Z d Z dZ dde dededzd ee ee efffdZde d ee fdZdee d e fdZdZdZdde de dzd ee dzfdZxZS) VitsTokenizera Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. input_idsattention_maskNTFreturnc ft|d5} tj| |_dddn #1swxYwYd|jD|_||_||_||_||_ ||_ tj d|||||||dd| dS)Nutf-8encodingci|]\}}|| Sr!).0kvs r z*VitsTokenizer.__init__..Vs>>>A1>>>rnone) pad_token unk_tokenlanguage add_blank normalize phonemize is_uromanspecial_tokens_patternr!) openjsonloadencoderitemsdecoderr)r*r+r,r-super__init__) selfr r'r(r)r*r+r,r-kwargs vocab_handle __class__s rr6zVitsTokenizer.__init__Gs*w / / / 3<9\22DL 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3?>););)=)=>>>   """" #)  s 9==c*t|jSN)lenr2)r7s r vocab_sizezVitsTokenizer.vocab_sizejs4<   rc|fdtjD}|j|S)Nc<i|]}||Sr!)convert_ids_to_tokens)r"ir7s rr%z+VitsTokenizer.get_vocab..os)RRRa++A..RRRr)ranger>updateadded_tokens_encoder)r7vocabs` r get_vocabzVitsTokenizer.get_vocabns@RRRR5;Q;QRRR T./// rct|jt|jz}d}d}|t |krwd}|D];}|||t |z|kr||z }|t |z }d}n<|s"|||z }|dz }|t |kw|S)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr2keysrEr=lower)r7rall_vocabulary filtered_textrB found_matchwords rnormalize_textzVitsTokenizer.normalize_textssdl//1122T$:S:X:X:Z:Z5[5[[ #l####K&  AD M 12d::!T)MTNA"&KE ;  a!6!6!8!88 Q#l####rcH|jdkr|dd}|S)z4Special treatment of characters in certain languagesronuțuţ)r)replace)r7texts r_preprocess_charzVitsTokenizer._preprocess_chars' =E ! !<<d++D rrVis_split_into_wordsr+c ||nj}|r|}|}t|rXjrQt st dn(tj }| |}j rNtstdtj |ddddd}tjdd |}nG|rEd t%t'fd |}||fS) a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `tuple[str, dict[str, Any]]`: The prepared text and the unused kwargs. NaCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usespeakT)r)backendstrippreserve_punctuation with_stressz\s+ rIc|jvSr<)r2)charr7s rz8VitsTokenizer.prepare_for_tokenization..sTT\=Qr)r+rRrWrr-rloggerwarningurUromanromanize_stringr,r ImportError phonemizerr subjoinrKfilterr\)r7rVrXr+r8rOuromans` rprepare_for_tokenizationz&VitsTokenizer.prepare_for_tokenizationse4"+!6IIDN  -&&t,,D--d33 #M 2 2 Ft~ F&(( Fy & 6 6} E E > l*,, k!"ijjj&0  %) MF63 >>MM  lGGD0Q0Q0Q0QS`)a)a$b$bcciikkMf$$rct|}|jr6|dgt|dzdzz}||ddd<|}|S)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rrJN)rKr*_convert_id_to_tokenr=)r7rVtokens intersperseds r _tokenizezVitsTokenizer._tokenizes[d > " 55a889S[[1_q=PQL!'LA !F rrrcv|jrt|dkr |ddd}d|S)NrJrprI)r*r=rk)r7rrs rconvert_tokens_to_stringz&VitsTokenizer.convert_tokens_to_strings9 > "c&kkAooADqD\Fwwvrc<||jvr |j|S|jS)z0Converts a token (str) in an id using the vocab.)r2 unk_token_id)r7tokens r_convert_token_to_idz"VitsTokenizer._convert_token_to_ids% DL <& &  rc6|j|S)z=Converts an index (integer) in a token (str) using the vocab.)r4get)r7indexs rrqz"VitsTokenizer._convert_id_to_tokens|&&&rsave_directoryfilename_prefixc tj|s td|ddStj||r|dzndt dz}t|dd5}|tj |j d d d d zdddn #1swxYwY|fS)NzVocabulary path (z) should be a directory-rIr wrrrpTF)indent sort_keys ensure_ascii ) ospathisdirrcerrorrkVOCAB_FILES_NAMESr/writer0dumpsr2)r7r~rr fs rsave_vocabularyzVitsTokenizer.save_vocabularys!w}}^,,  LLT^TTT U U U FW\\ oM_s222QbcoQpp  *cG 4 4 4 c GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c}s4CC C )rrNTTTF)rN)FNr<)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr6propertyr>rGrRrWstrbooltupledictrrnrKrtrvrzrqr __classcell__)r:s@rrr.s**$&67 !  ! ! ! ! ! ! F!!X! *VZ?%?%?%.2?%GKd{?% sDcN" #?%?%?%?%B c d3i    tCyS !!! '''  c C$J Z_`cZdgkZk        rr)rr0rr typingrtokenization_pythonrutilsrrrrirmre get_loggerrrcrrr__all__r!rrrs#" 666666JJJJJJJJJJ  H % %!<0GGGGG'GGGT  r