"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. sp_model_kwargs (`dict[str, Any]`, `Optional`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. use_default_system_prompt (`bool`, *optional*, defaults to `False`): Whether or not the default system prompt for Gemma should be used. spaces_between_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add spaces between special tokens. input_idsattention_masksp_model_kwargsc  >Uc0OUUl[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUnXlXplXlXl[R"S0URD6Ul URRU5 [T U]4"SUUUUUUUU U U S. U D6 g)NFT) normalizedspecial) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrclean_up_tokenization_spacesuse_default_system_promptspaces_between_special_tokens)r isinstancestrrr rrr spmSentencePieceProcessorsp_modelLoadsuper__init__)selfr rrrrrrrrr r!kwargs __class__s d/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/gemma/tokenization_gemma.pyr*GemmaTokenizer.__init__^s&5%[TU]"U40UD6$)zE Args: text: TextInput Simply calls PreTrainedTokenizer's method )r)tokenize)r+rKr,r-s r.rNGemmaTokenizer.tokenizes w///r0c >URRU[S9$)zf Args: text: TextInput Returns a tokenized string. The Gemma tokenizer never adds a prefix space. )out_type)r'encoder$)r+rKr,s r. _tokenizeGemmaTokenizer._tokenizes }}##D3#77r0c8URRU5$)z0Converts a token (str) in an id using the vocab.)r' piece_to_id)r+tokens r._convert_token_to_id#GemmaTokenizer._convert_token_to_ids}}((//r0c<URRU5nU$)z=Converts an index (integer) in a token (str) using the vocab.)r' IdToPiece)r+indexrWs r._convert_id_to_token#GemmaTokenizer._convert_id_to_tokens ''. r0c/nSnUHGnX@R;a$X0RRU5U-- n/nM6URU5 MI X0RRU5- nU$)z:Converts a sequence of tokens (string) in a single string.)_added_tokens_encoderr'decodeappend)r+tokenscurrent_sub_tokens out_stringrWs r.convert_tokens_to_string'GemmaTokenizer.convert_tokens_to_stringst E222mm223EFNN %'""))%0  mm**+=>> r0filename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RRUR5[RRU5:waG[RRUR5(a[URU5 U4$[RRUR5(dC[US5nURR5nURU5 SSS5 U4$U4$!,(df  U4$=f)z Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directoryN-r`r wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfileropenr'r5write)r+save_directoryriout_vocab_fileficontent_spiece_models r.save_vocabularyGemmaTokenizer.save_vocabularys.ww}}^,, LL,^,<U(a[TU]XSS9$UR(aS/O/nUR(aS/O/nUcUS/[ U5--U-$US/[ U5--U-U-S/[ U5--U-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr)r)get_special_tokens_maskrrlen)r+rrrrrr-s r.r&GemmaTokenizer.get_special_tokens_masks$ &72']a3 #00sb "00sb  A3[)9#9:\I I sS%% '  sS%%  '    r0cUR(a UR/O/nUR(a UR/O/nS/[ X1-U-5-nUbUS/[ X2-U-5-- nU$)aM Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`list[int]`): List of ids. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rr)rrrrrrs r.$create_token_type_ids_from_sequences3GemmaTokenizer.create_token_type_ids_from_sequencessv./3.@.@))*b .2.@.@))*b s<5 DEE  " qcC :\ IJJ JF r0 token_idsskip_special_tokensr!c (/n/nUHnU(aXpR;aMXpR;a]U(a*URURR U55 URURUR 5 /nMURU5 M U(a*URURR U55 U(aSR U5nOSR U5nUR[S5$)N r`) all_special_ids_added_tokens_decoderrcr'rbcontentrrreplaceSPIECE_UNDERLINE)r+rrr!r, sub_textscurrent_sub_textidss r._decodeGemmaTokenizer._decode1s C"s.B.B'B000#$$T]]%9%9:J%KL  !;!;C!@!H!HI#%  '',    T]]112BC D (+I *I  !1377r0)rrr'rr r ) zzzzNTFFFFr)NF)FF)"__name__ __module__ __qualname____firstlineno____doc__rsvocab_files_namesmodel_input_namesrdictr$rr*r7r=propertyrArIlistrNrSrXr]rgtupler|rintboolrrr__static_attributes__ __classcell__)r-s@r.rr+s,\*$&67 48%*"'&+( "$sCx.1( ( T C .. 0[0tCy080 !x}!X]^aXb!6 sx# 9# 3;DI3F# ko# c# # LJN93;DI3F cH%*.3 898"8(, 8 88r0r)rmshutilrtypingrrrrr%tokenization_utilsrr utilsr utils.import_utilsr tokenization_utils_baser get_loggerrrprsrr__all__r"r0r.rsy, //A*4   H %!#45 %&`8(`8'`8F  r0