When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. mask_token (`str`, *optional*, defaults to `"[MASK]"`): The MASK token, useful for masked language modeling. cls_token (`str`, *optional*, defaults to `"[CLS]"`): The CLS token (reserved in the vocab, but not actually used). sep_token (`str`, *optional*, defaults to `"[SEP]"`): The separator token, which is used when building a sequence from two sequences. input_idsattention_maskreturnc ,>[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUnSUlUUUUUUS.Ul[ UR5Ul[T U] "SUUUUUUUS.UD6 g)NF)lstriprstrip)rr) pad_token bos_token eos_token mask_token cls_token sep_tokenmodel_max_length) isinstancestrr_utf_vocab_size_added_tokens_decoderlen_num_special_tokenssuper__init__) selfrrrrrrrkwargs __class__s l/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/perceiver/tokenization_perceiver.pyr%PerceiverTokenizer.__init__;s-JTT]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir KUV`beKfKfZ 5Glv IST]_bIcIcJyuEir IST]_bIcIcJyuEir # 6 "$'t'A'A#B   !-  c0n[UR5Hn[U5nX R-X'M UR UR 5 U$N)ranger chrr#updateadded_tokens_encoder)r&vocabitokens r) get_vocabPerceiverTokenizer.get_vocabdsNt++,AFE777EL-  T../ r+cUR$r-)r )r&s r) vocab_sizePerceiverTokenizer.vocab_sizels###r+ token_ids_0 token_ids_1already_has_special_tokensc>U(a[TU]XSS9$UcS/S/[U5--S/-$S/S/[U5--S/-S/[U5--S/-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r:r;r<rr)r$get_special_tokens_maskr")r&r:r;r<r(s r)r>*PerceiverTokenizer.get_special_tokens_maskps$ &72']a3   3!s;///1#5 5sqcC ,,-3sS=M7MNRSQTTTr+cUcUR/U-UR/-$UR/U-UR/-U-UR/-$)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks. A sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) cls_token_id sep_token_id)r&r:r;s r) build_inputs_with_special_tokens3PerceiverTokenizer.build_inputs_with_special_tokenssb&  %%&48I8I7JJ J%%&48I8I7JJ[X\`\m\m[nn nr+textcbURS5Vs/sHn[U5PM nnU$s snf)zPTake as input a string and return a list of strings (tokens) for words/sub-wordsutf-8)encoder/)r&rEr3tokenss r) _tokenizePerceiverTokenizer._tokenizes/"&++g"67"6Q#a&"67 8s,cp[U5S:waURnU$[U5UR-nU$)z0Converts a token (str) in an id using the vocab.r)r" unk_token_idordr#)r&r4token_ids r)_convert_token_to_id'PerceiverTokenizer._convert_token_to_ids: u:?((H5zD$<$<rCrJrPrTr^tuplerc__static_attributes__ __classcell__)r(s@r)r r s;<%&67'  ' ' R4S>$$sxU9U3;DI3FUkoU cUU:JNo9o3;DI3Fo co0cd3i  cHSM]bcf]gr+r N) ritypingrtokenization_utilsrrutilsr get_loggerreloggerr __all__rr+r)rys?(A   H %k,k\ r+