"`): The begin of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. normalize (`bool`, *optional*, defaults to `False`): Whether to convert numeric quantities in the text to their spelt-out english counterparts. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). input_idsattention_maskNsp_model_kwargsreturnc >Uc0OUUlXlX`lSUl[R "S0URD6UlUR RU5 [T U]$"SUUUUUURS.UD6 g)N) bos_token eos_token unk_token pad_token normalizer) rr r _normalizerspmSentencePieceProcessorsp_modelLoadsuper__init__) selfr rrrrrrkwargs __class__s j/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr#SpeechT5Tokenizer.__init__Qs&5%URRU[S9$)zPTake as input a string and return a list of strings (tokens) for words/sub-words)out_type)r encodestr)r$r.s r' _tokenizeSpeechT5Tokenizer._tokenizes}}##D3#77r)c8URRU5$)z0Converts a token (str) in an id using the vocab.)r piece_to_id)r$tokens r'_convert_token_to_id&SpeechT5Tokenizer._convert_token_to_ids}}((//r)c<URRU5nU$)z=Converts an index (integer) in a token (str) using the vocab.)r IdToPiece)r$indexrXs r'_convert_id_to_token&SpeechT5Tokenizer._convert_id_to_tokens ''. r)c"/nSnSnUHWnXPR;a2U(dUS- nX0RRU5U-- nSn/nMDURU5 SnMY X0RRU5- nUR 5$)z:Converts a sequence of tokens (string) in a single string.Fr+T)all_special_tokensr decodeappendstrip)r$tokenscurrent_sub_tokens out_stringprev_is_specialrXs r'convert_tokens_to_string*SpeechT5Tokenizer.convert_tokens_to_strings E///&#%Jmm223EFNN "&%'""))%0"' mm**+=>> !!r)cJUcXR/-$X-UR/-$)z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r$ token_ids_0 token_ids_1s r' build_inputs_with_special_tokens2SpeechT5Tokenizer.build_inputs_with_special_tokenss1  "3"3!44 4(D,=,=+>>>r)rnroalready_has_special_tokensc>U(a[TU]XSS9$S/nUcS/[U5-U-$S/[U5-S/[U5--U-$)NT)rnrorrr r)r"get_special_tokens_masklen)r$rnrorr suffix_onesr&s r'rt)SpeechT5Tokenizer.get_special_tokens_maskss &72']a3 c  C#k**k9 9c+&&A3[1A+AB[PPr)save_directoryfilename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RRUR5[RRU5:waG[RRUR5(a[URU5 U4$[RRUR5(dC[US5nURR5nURU5 SSS5 U4$U4$!,(df  U4$=f)NzVocabulary path (z) should be a directory-rar wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfileropenr serialized_model_protowrite)r$rxryout_vocab_fileficontent_spiece_models r'save_vocabulary!SpeechT5Tokenizer.save_vocabularys,ww}}^,, LL,^,<zzzFN)Fr3)NF)"__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesrdictrSrr#r0propertyr6r-setterrCrIrNlistrTrYr^rjintrpboolrttupler__static_attributes__ __classcell__)r&s@r'rr"s](T*$&67 48 "$sCx.1    :..  !!  ,8c8d3i80 "&?QUVYQZ?sx Q9 Q3;DI3F Qko Q c Q Q!c!HSM!]bcf]g!!r)r)rr}shutilrtypingrrr rtokenization_utilsrutilsrutils.import_utilsr number_normalizerr get_loggerrrrr__all__rr)r'rsp'  5*6   H %!#34 %&y!+y!'y!x  r)