h"dZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZd dlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(erddl)m*Z*ndZ*ejVe,Z-ee.e/e e.e e.ffgdderdndffderdnderdndffdderdndffdderdndffdderdndffd derd!ndffd"derdndffd#d$erd%nderd&ndffd'd(derdndffd)erd*nddffd+d,d-erd.nderd/ndffd0d1erd2ndffd3d4derd5ndffd6d7d8derdndffd9d:erd;ndffdderd5ndffd?d@erdAndffdBderdndffdCdDerdEnderdFndffdGdHerdnderdndffdIderdndffdJd@erdAndffdKderdndffdLderdndffdMdNerdOnderdPndffdQdRerdSndffdTderd!ndffdUderd!ndffdVderdndffdWdXerdYndffdZd[erd\ndffd]erd^nderd_ndffd`daderd5ndffdbdcddd@erdAndffded:erd;ndffdfdgerdhndffdierdjnderdkndffdlerdnderdndffdmerdnderdndffdnerdnderdndffdoerdnderdndffdpdqerdnderdndffdrdserdtndffdudverdwndffdxdyerdzndffd{d:erd;ndffd|derdndffd}derdndffd~derdndffderdnddffdderd:nderd;ndffdderd5ndffdderdndffderdnddffddderd;ndffdderdndffddderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffdderdndffdderd5ndffdderd5ndffdderd5ndffdderd5ndffdderd5ndffderdnddffdd:erd;ndffdd:erd;ndffdd:erd;ndffdderdndffddderd5ndffdd:erd;ndffddddddderdndffdderdndffdderd5ndffdderdndffddd@erdAndffdderdndffdderdndffdderdndffdd:erd;ndffdd:erd;ndffddXerdYndffderdnderdndffdderdndffderdnderdndffdderdnderdndffdderd5ndffdderdndffdderdndffdderdndffdderdndffdderdndffdderdndffderdnderdndffderdnderdndffderdnderdndffdderdndffdderdndffdderdndffdderdndffdderdndffderdnderdndffddderdndffderdnddffdderdndffdderdndffderdnddffderdnderdndffderdnderdndffdd@erdAndffdderdndffdderdndffdderd:nderd;ndffde rdn erdnder e sdndffde rdn erdnder e sdndffdderdndffderdnddffdderdndffdderdndffdderd5ndffdderd5ndffdderd5ndffdderdndffdderdndffdd@erdAndffderdnderd ndffd derdndffd derdndffd d erdndffddderd5ndffdderdndffderdnderdndffderdnderdndffderdnderdndffdderdndffdderdndffdderd;ndffdderdndffdderdndffdderdndffdderdndffd d:erd;ndffd!derdndffd"derdndffd#derdndffd$d%erd1nderd2ndffd&erd1nderd2ndffd'd(erdnderdndffd)dRerdSndffd*derdndffd+derdndffd,d-derdndffd.de rdn erd5ndffd/erd0nddffd1d2derdndffd3dXerdYndffd4dXerdYndffd5dXerdYndffd6dXerdYndffd7dXerdYndffd8dXerdYndffd9dXerdYndffd:dXerdYndffd;dXerdYndffddXerdYndffd?d@dAerdBndffdCerdnderdndffdDerdEnderdFndffdGerdHnderdIndffdJdKerdLndffdMd@erdAndffdNd@erdAndffdOdPdQerdRndffdSderdndffdTerdUnderdVndffdWerdUnderdVndffdXerdnderdndffdYerdZnddffd[erdnderdndffd\derd5ndffd]erd^nddffd_d`erdanddffdbdcdderdendffdfderdndffdgd:erd;ndffdherdnderdndffdierdnderdndffdjerdnderdndffdkdldmdnderdndffdoerdpnderdqndffdrerdnderdndffdsderdndffdtderdndffduderdndffdvderdndffdwdxe rdn erdnder e sdndffdydzd{d|d}d~erdndffdderdndffderdnderdndffdderdnddffderdnderdndffderdnderdndffderdnderdndffdderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffZ0e"e$e0Z1e$jdDcic]\}}|| c}}Z3de.de e4edffdZ5 dde e.ejle.fde e e.ejle.fde7de e7de e8e.e.fde e e7e.fde e.de7de.de8e.effdZ9GddZ:ddgZ;ycc}}w(zAuto Tokenizer class.N) OrderedDict)AnyOptionalUnion)is_mistral_common_available)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code)load_gguf_checkpoint)PreTrainedTokenizer)TOKENIZER_CONFIG_FILE) cached_fileextract_commit_hashis_g2p_en_availableis_sentencepiece_availableis_tokenizers_availablelogging)EncoderDecoderConfig)_LazyAutoMapping)CONFIG_MAPPING_NAMES AutoConfigconfig_class_to_model_typemodel_type_to_module_name!replace_list_option_in_docstrings)PreTrainedTokenizerFastaimv2 CLIPTokenizerCLIPTokenizerFastalbertAlbertTokenizerAlbertTokenizerFastalign BertTokenizerBertTokenizerFastarceeLlamaTokenizerLlamaTokenizerFastaria aya_visionCohereTokenizerFastbark)bart) BartTokenizerBartTokenizerFastbarthezBarthezTokenizerBarthezTokenizerFast)bartpho)BartphoTokenizerNbertzbert-generationBertGenerationTokenizer)z bert-japanese)BertJapaneseTokenizerN)bertweet)BertweetTokenizerNbig_birdBigBirdTokenizerBigBirdTokenizerFastbigbird_pegasusPegasusTokenizerPegasusTokenizerFast)biogpt)BioGptTokenizerNbitnetr) blenderbot)BlenderbotTokenizerBlenderbotTokenizerFast)zblenderbot-small)BlenderbotSmallTokenizerNblipzblip-2 GPT2TokenizerGPT2TokenizerFastbloomBloomTokenizerFastblt bridgetowerRobertaTokenizerRobertaTokenizerFastbros)byt5) ByT5TokenizerN camembertCamembertTokenizerCamembertTokenizerFast)canine)CanineTokenizerN chameleon chinese_clipclapclipclipseg)clvp) ClvpTokenizerN code_llamaCodeLlamaTokenizerCodeLlamaTokenizerFastcodegenCodeGenTokenizerCodeGenTokenizerFastcoherecohere2colpalicolqwen2Qwen2TokenizerQwen2TokenizerFastconvbertConvBertTokenizerConvBertTokenizerFastcpm CpmTokenizerCpmTokenizerFast)cpmant)CpmAntTokenizerNcsm)ctrl) CTRLTokenizerN)zdata2vec-audioWav2Vec2CTCTokenizerNz data2vec-textdbrxdebertaDebertaTokenizerDebertaTokenizerFastz deberta-v2DebertaV2TokenizerDebertaV2TokenizerFast deepseek_v2 deepseek_v3 deepseek_vldeepseek_vl_hybrid)dia) DiaTokenizerN diffllama distilbertDistilBertTokenizerDistilBertTokenizerFastdprDPRQuestionEncoderTokenizerDPRQuestionEncoderTokenizerFastelectraElectraTokenizerElectraTokenizerFastemu3ernieernie4_5 ernie4_5_moeernie_mErnieMTokenizer)esm) EsmTokenizerNexaone4falcon falcon_mambaGPTNeoXTokenizerFastfastspeech2_conformerFastSpeech2ConformerTokenizer)flaubert)FlaubertTokenizerN flex_olmofnet FNetTokenizerFNetTokenizerFast)fsmt) FSMTTokenizerNfunnelFunnelTokenizerFunnelTokenizerFastgemmaGemmaTokenizerGemmaTokenizerFastgemma2gemma3 gemma3_textgemma3n gemma3n_textgitglmglm4glm4_moeglm4v glm4v_moezgpt-sw3GPTSw3Tokenizergpt2 gpt_bigcodegpt_neogpt_neox)gpt_neox_japanese)GPTNeoXJapaneseTokenizerNgpt_ossgptj)zgptsan-japanese)GPTSanJapaneseTokenizerN)graniterJN) granitemoer)granitemoehybridr)granitemoesharedrzgrounding-dinogroupvitheliumherbertHerbertTokenizerHerbertTokenizerFast)hubertrxibertideficsidefics2idefics3 instructblipinstructblipvideointernvljambajanusjetmoe)jukebox)JukeboxTokenizerNzkosmos-2XLMRobertaTokenizerXLMRobertaTokenizerFastz kosmos-2.5layoutlmLayoutLMTokenizerLayoutLMTokenizerFast layoutlmv2LayoutLMv2TokenizerLayoutLMv2TokenizerFast layoutlmv3LayoutLMv3TokenizerLayoutLMv3TokenizerFast layoutxlmLayoutXLMTokenizerLayoutXLMTokenizerFastled LEDTokenizerLEDTokenizerFastliltllamallama4 llama4_textllava llava_nextllava_next_videollava_onevision longformerLongformerTokenizerLongformerTokenizerFastlongt5 T5TokenizerT5TokenizerFast)luke) LukeTokenizerNlxmertLxmertTokenizerLxmertTokenizerFastm2m_100M2M100Tokenizermambamamba2marianMarianTokenizermbartMBartTokenizerMBartTokenizerFastmbart50MBart50TokenizerMBart50TokenizerFastmegaz megatron-bert metaclip_2)zmgp-str)MgpstrTokenizerNminimaxmistralMistralCommonTokenizermixtralmllamamlukeMLukeTokenizerzmm-grounding-dino mobilebertMobileBertTokenizerMobileBertTokenizerFast modernbert moonshinemoshimpnetMPNetTokenizerMPNetTokenizerFastmptmramt5 MT5TokenizerMT5TokenizerFastmusicgenmusicgen_melodymvp MvpTokenizerMvpTokenizerFast)myt5) MyT5TokenizerNnemotronnezhanllb NllbTokenizerNllbTokenizerFastznllb-moe nystromformerolmoolmo2olmo3olmoez omdet-turbo oneformerz openai-gptOpenAIGPTTokenizerOpenAIGPTTokenizerFastoptowlv2owlvit paligemma)parakeet)ParakeetCTCTokenizerNpegasus pegasus_x) perceiver)PerceiverTokenizerN persimmonphiphi3phimoe)phobert)PhobertTokenizerN pix2structpixtralplbartPLBartTokenizer) prophetnet)ProphetNetTokenizerNqdqbertqwen2 qwen2_5_omni qwen2_5_vl qwen2_audio qwen2_moeqwen2_vlqwen3 qwen3_moe qwen3_nextqwen3_omni_moeqwen3_vl qwen3_vl_moe)rag) RagTokenizerNrealmRealmTokenizerRealmTokenizerFastrecurrent_gemmareformerReformerTokenizerReformerTokenizerFastrembertRemBertTokenizerRemBertTokenizerFast retribertRetriBertTokenizerRetriBertTokenizerFastrobertazroberta-prelayernorm)roc_bert)RoCBertTokenizerNroformerRoFormerTokenizerRoFormerTokenizerFastrwkv seamless_m4tSeamlessM4TTokenizerSeamlessM4TTokenizerFastseamless_m4t_v2 shieldgemma2siglipSiglipTokenizersiglip2smollm3speech_to_textSpeech2TextTokenizer)speech_to_text_2)Speech2Text2TokenizerNspeecht5SpeechT5Tokenizer)splinter)SplinterTokenizerSplinterTokenizerFast squeezebertSqueezeBertTokenizerSqueezeBertTokenizerFaststablelm starcoder2switch_transformerst5t5gemma)tapas)TapasTokenizerN)tapex)TapexTokenizerN)z transfo-xl)TransfoXLTokenizerNtvpudop UdopTokenizerUdopTokenizerFastumt5 video_llavaviltvipllava visual_bert)vits) VitsTokenizerNvoxtral)wav2vec2rx)z wav2vec2-bertrx)zwav2vec2-conformerrx)wav2vec2_phoneme)Wav2Vec2PhonemeCTCTokenizerNwhisperWhisperTokenizerWhisperTokenizerFastxclipxglm XGLMTokenizerXGLMTokenizerFast)xlm) XLMTokenizerNzxlm-prophetnetXLMProphetNetTokenizerz xlm-robertazxlm-roberta-xlxlnetXLNetTokenizerXLNetTokenizerFastxlstmxmodyosozambazamba2 class_namereturnc|dk(rtStjD]\\}}||vs t|}|dvr|dk(rt j dd}nt j d|d} t ||cStjjD]}|D]}t |dd|k(s|ccS!t j d}t||r t ||Sy#t$rYwxYw) Nr)r r ministralr z.tokenization_mistral_common transformers.ztransformers.models__name__) rTOKENIZER_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorTOKENIZER_MAPPING_extra_contentvalueshasattr)r module_name tokenizersmodule tokenizer main_modules i/var/www/html/aiagenthome/venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.pytokenizer_class_from_namers ..&&#:#@#@#B Z  #3K@KAAjTlFl"001OQ_`"001[M1BDYZ vz22$C(66==? #Iy*d3zA  $@)).9K{J'{J// "  s0 C-- C98C9pretrained_model_name_or_path cache_dirforce_downloadresume_downloadproxiestokenrevisionlocal_files_only subfolderc  | jdd} | )tjdt| t d| }| j d} t |t||||||||ddd| } | tjdiSt| | } t| d 5} tj| }ddd| d<|S#1swYxYw) a Loads the tokenizer configuration from a pretrained model tokenizer configuration. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. subfolder (`str`, *optional*, defaults to `""`): In case the tokenizer config is located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Passing `token=True` is required when you want to use a private model. Returns: `dict`: The configuration of the tokenizer. Examples: ```python # Download configuration from huggingface.co and cache. tokenizer_config = get_tokenizer_config("google-bert/bert-base-uncased") # This model does not have a tokenizer config so the result will be an empty dict. tokenizer_config = get_tokenizer_config("FacebookAI/xlm-roberta-base") # Save a pretrained tokenizer locally and you can reload its config from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") tokenizer.save_pretrained("tokenizer-test") tokenizer_config = get_tokenizer_config("tokenizer-test") ```use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`. _commit_hashF) rrrrrrrr _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errorsrz\Could not locate the tokenizer configuration file, will try to use the model config instead.zutf-8)encoding)popwarningswarn FutureWarning ValueErrorgetrrloggerinforopenjsonload)rrrrrrrrrkwargsr commit_hashresolved_config_filereaderresults rget_tokenizer_configr5sRZZ 0$7N!  A    uv v**^,K&%%'))..305  # rs %&:KHK "W 56" 6(F> M 6 5s CC cNeZdZdZdZeeedZe ddZ y) AutoTokenizera This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [`AutoTokenizer.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). ctd)Nz}AutoTokenizer is designed to be instantiated using the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfs r__init__zAutoTokenizer.__init__s _  c B |jdd}|;tjdt|j d t d||d<|jdd}d|d<|jd d}|jd d}|jd d}|j d } |d} t j |d} | ,t d |ddjdt Dd| \} } |r#| t| } ntjd| t| } | t d| d| j|g|i|St|fi|}d|vr|d|d<|j d}d}d|vr4t|dttfr|d}n|dj dd}|t|t sM| r3t#|| fi|}t%|dd}t'j(d(i|}nt'j|fd |i|}|j*}t-|drd|j.vr|j.d}|du}t1|t2vxs%|duxrt|duxst|dzdu}|r<|r |d|d}n|d}d|vr|j5dd}nd}t7|||||}|rI|rGt9|fi|} |jdd}| j;| j|g|d |i|S|[d} |r!|j=ds|d}t|} | |}t|} | t dd | j|g|i|St|t>rzt1|j@t1|jBurDtjd!|jBjDd"|j@jDd#|jB}tGt1|jH}|Tt2t1|\}}|r|s||j|g|i|S||j|g|i|St d$t d%|jDd&djd't2Dd))a] Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary. The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. - A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes) inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__()` method. config ([`PretrainedConfig`], *optional*) The configuration object used to determine the tokenizer class to instantiate. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. use_fast (`bool`, *optional*, defaults to `True`): Use a [fast Rust-based tokenizer](https://huggingface.co/docs/tokenizers/index) if it is supported for a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer is returned instead. tokenizer_type (`str`, *optional*): Tokenizer type to be loaded. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details. Examples: ```python >>> from transformers import AutoTokenizer >>> # Download vocabulary from huggingface.co and cache. >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> # Download vocabulary from huggingface.co (user-uploaded) and cache. >>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased") >>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) >>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/") >>> # Download vocabulary from huggingface.co and define model-specific arguments >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base", add_prefix_space=True) ```rNrrrconfigT _from_autouse_fasttokenizer_typetrust_remote_code gguf_filezPassed `tokenizer_type` z3 does not exist. `tokenizer_type` should be one of z, c3 K|]}|ywN.0cs r z0AutoTokenizer.from_pretrained..s D,Cq,Cs rzt`use_fast` is set to `True` but the tokenizer class does not have a fast version. Falling back to the slow version.zTokenizer class z is not currently imported.rtokenizer_classauto_maprF)return_tensorsFastrrz-- code_revisionz- does not exist or is not currently imported.z The encoder model config class: z3 is different from the decoder model config class: z. It is not recommended to use the `AutoTokenizer.from_pretrained()` method in this case. Please use the encoder and decoder specific tokenizer classes.zzThis tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.z!Unrecognized configuration class z8 to build an AutoTokenizer. Model type should be one of c34K|]}|jywr)rrs rrz0AutoTokenizer.from_pretrained..s4[IZAQZZIZsr)%rrrrrrrjoinrrwarningfrom_pretrainedr isinstancetuplelistr rr r for_modelrrrtypersplitr r register_for_auto_classendswithrdecoderencoder __class__rr)clsrinputsrrrrrrrrtokenizer_class_tupletokenizer_class_nametokenizer_fast_class_nametokenizer_configconfig_tokenizer_classtokenizer_auto_map gguf_path config_dicthas_remote_codehas_local_code class_ref upstream_repo_tokenizer_class_candidate model_typetokenizer_class_pytokenizer_class_fasts rrzAutoTokenizer.from_pretrainedsZ $4d;  % MME zz'". l-F7OHd+#|::j$/$4d;"JJ':DAJJ{+   %"O$;$?$?PT$U !$, .~.>>qyy D,C DDEQH ?T ; ";,8&?@Y&ZONN=&"; "!1!5!56G!H! ) )*:6 F%5j%A"%5j%A%E%EoW[%\" " )f&67 +,I9 _X^ _I"6yQV"WX`"aK'11@K@F'775IZ^dF&,%;%; "vz*&///Q%+___%E",D8f):: "$ . )*@AMZ,-Cf-LMUYY  .q1=.q1 .q1 y ) 5a 8 $ 9!#@.Racp!  0;IGdohnoO ?D1A  3 3 52?22-06J[_e $ /"O 6 ? ? G/E.Fd,K)";?22 ^^F/V 0E0EF  !7Hf7V 4  4#5G5O;+;;[o^dohnoo$: /0@0@/AB++/994[IZ4[+[*\\] _  rNc| | td|t|tr td|t|tr td|=|;t|tr+|j|k7rtd|jd|d|t j vrt |\}}||}||}t j|||f|y) a Register a new tokenizer in this mapping. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. slow_tokenizer_class ([`PretrainedTokenizer`], *optional*): The slow tokenizer to register. fast_tokenizer_class ([`PretrainedTokenizerFast`], *optional*): The fast tokenizer to register. NzKYou need to pass either a `slow_tokenizer_class` or a `fast_tokenizer_classz:You passed a fast tokenizer in the `slow_tokenizer_class`.z:You passed a slow tokenizer in the `fast_tokenizer_class`.zThe fast tokenizer class you are passing has a `slow_tokenizer_class` attribute that is not consistent with the slow tokenizer class you passed (fast tokenizer has z and you passed z!. Fix one of those so they match!)exist_ok)r issubclassrr slow_tokenizer_classrrregister) config_classr'fast_tokenizer_classr% existing_slow existing_fasts rr(zAutoTokenizer.registers ',@,Hjk k  + ;OQh0iYZ Z  + ;OQd0eYZ Z ! ,$0/1HI$99=QQ['<<==MNbMcd!!  ,;; ;+<\+J (M=#+'4$#+'4$""<2FH\1]hp"qr)NNF) r __module__ __qualname____doc__r classmethodrrr staticmethodr(rrrrrsH &'>?` @` D)r)rrrr)NFNNNNF)rJs# #''G3\?5<3*B"   H %P+c5# 1M+N&NOS  '>'@#d  S  %?%A!t)@)B%  S  ?;R;T$7Z^_`S  #=T=V%9\`ab!S " " ;?S @ 2AS D &@&B"*A*C&  CS P /KbKd1GjnopQS R .SS T D7N7P3VZ[\US V KWS X AYS Z /:Q:S#6Y]^_[S \ O$@ d(?(A$t  yS F /BYB[+>aefgGS J "*A*C&  IS X '>'@#d  WS f '>'@#d  eS r *sS v (B(D$$,C,E(4  uS B 'CZC\)?bfghCS D D3J3L/RVWXES F T4K4M0SWXYGS H %?V?X';^bcdIS J &@W@Y(<_cdeKS L )F]F_+BeijkMS P "<">D&=&?"T  OS \ .]S ^ 4K4M0SWXY_S ` *aS b ;cS d -I`Ib/EhlmneS f /:Q:S#6Y]^_gS h 'CZC\)?bfghiS l (B(D$$,C,E(4  kS z $>$@ d(?(A$t  yS H $>$@ d(?(A$t  GS V $>$@ d(?(A$t  US d !$>$@ d(?(A$t  cS p (qS t $>$@ d(?(A$t  sS @ -LcLe/HkopqAS D -5L5N1TX  CS P 'CZC\)?bfghQS R /:Q:S#6Y]^_SS T ?;R;T$7Z^_`US V d4K4M0SWXYWS X $8O8Q 4W[\]YS Z *D*F&DRVWX[S \ (]S ` #:#<$'>'@#d  _S l D7N7P3VZ[\mS n $:Q:S 6Y]^_oS r $0C0E ,4QU V qS x 2yS z t4K4M0SWXY{S | /:Q:S#6Y]^_}S ~ *S @ %@W@Y'<_cdeAS D $>$@ d(?(A$t  CS R $>$@ d(?(A$t  QS ` $>$@ d(?(A$t  _S n $>$@ d(?(A$t  mS | $>$@ d(?(A$t  {S J $>$@ d(?(A$t  IS V 9P9R"5X\]^WS X 4K4M0SWXYYS Z $5L5N1TXYZ[S \ d9P9R5X\]^]S ^ 46M6O2UYZ[_S ` t:Q:S6Y]^_aS b *D*F&DRVWXcS d /:Q:S#6Y]^_eS f AXAZ*=`defgS h _=T=V&9\`abiS j d6M6O2UYZ[kS l BmS n T8O8Q4W[\]oS p /:Q:S#6Y]^_qS r ?sS t -uS v 0wS x 6yS z 6{S | OD[D]-@cghi}S ~ o>U>W':]abcS @ D7N7P3VZ[\AS B 'CZC\)?bfghCS D 3ES F %AXAZ'=`defGS H T3J3L/RVWXIS J &@W@Y(<_cdeKS L &@W@Y(<_cdeMS N /BYB[+>aefgOS P G^G`0CfjklQS R &@W@Y(<_cdeSS V $>$@ d(?(A$t  US b 41H1J-PTUVcS f $>$@ d(?(A$t  eS r 0sS v )C)E%4-D-F)D  uS B ;R;T7Z^_`C S D )F]F_+BeijkE S F -LcLe/HkopqG S H -LcLe/HkopqI S J +I`Ib-EhlmnK S L 7N7P!3VZ[\M S N 'F]F_)BeijkO S R $>$@ d(?(A$t  Q S ` $>$@ d(?(A$t  _ S n $>$@ d(?(A$t  m S z #=T=V%9\`ab{ S | (BYB[*>aefg} S ~ .H_Ha0Dgklm S @ -G^G`/CfjklA S B -LcLe/HkopqC S F !;!= 4%<%>!D  E S R *S S T %@W@Y'<_cdeU S V *D*F&DRVWXW S X 43J3L/RVWXY S Z D4K4M0SWXY[ S \ )C)E%4QUVW] S ` $>$@ d(?(A$t  _ S n &@&B"*A*C&  m S z $@W@Y&<_cde{ S | ?CZC\,?bfgh} S @ %-D-F)D   S L /M S P #=#?T'>'@#d  O S ^ /0)*D*F&D(?(AJeJg$mq   ] S p /0)*D*F&D(?(AJeJg$mq   o S @ $>U>W&:]abcA S B 'A'C#tTUC S D G^G`0CfjklE S F -LcLe/HkopqG S H ;R;T7Z^_`I S J t:Q:S6Y]^_K S L 46M6O2UYZ[M S N #=T=V%9\`abO S P 1H1J-PTUVQ S R #?V?X%;^bcdS S V "<">D&=&?"T  U S b m:Q:S%6Y]^_c S d ]AXAZ,=`defe S f 7N7P!3VZ[\g S h *i S j d9P9R5X\]^k S l ?;R;T$7Z^_`m S p #=#?T'>'@#d  o S ~ #=#?T'>'@#d  } S L %?%A!t)@)B%  K S X $2I2K.QUVWY S Z 43J3L/RVWX[ S \ 40G0I,tTU] S ^ 43J3L/RVWX_ S b  5L5N1TX Y a S h ?V?X(;^bcdi S l  !?V?X#;^b c k S r 9P9R"5X\]^s S t ?;R;T$7Z^_`u S v O$@ d(?(A$t  gS t #?V?X%;^bcduS v "U>W&:]abcyS z 0{S |  aefggS h )CZC\+?bfghiS l  (?(A$t  kS x &@W@Y(<_cdeyS |  (?(A$t  {S J  (?(A$t  IS X  (?(A$t  WS d ,F]F_.BeijkeS f &@W@Y(<_cdegS h *D[D],@cghiiS j (kS l #=T=V%9\`abmS p $>$@ d(?(A$t  oS ~ 'A'C#+B+D'$  }S L &@&B"*A*C&  KS X +I`Ib-EhlmnYS Z 'CZC\)?bfgh[S ^ # ;R;T!7Z^ _ ]S d 1eS f )F]F_+BeijkgS h $2I2K.QUVWiS l *D*F&D.E.G*T  kS z *D*F&D.E.G*T  yS H $>$@ d(?(A$t  GS T )C)E%4QUVWUS X $>$@ d(?(A$t  WS d T8O8Q4W[\]eS f 6P6R2X\^bcdgS h >iS j -G-I)tUYZ[kS l EmS p  #CZC\%?bf g oS v d6M6O2UYZ[wS x @W@Y)<_cdeyS | "!;!= 4%<%>!D  {S J !;!= 4%<%>!D  IS X $>$@ d(?(A$t  WS d ,eS f ,gS h 5iS j 9P9R"5X\]^kS n #=#?T'>'@#d  mS | !;!= 4%<%>!D  {S H )CZC\+?bfghIS J /:Q:S#6Y]^_KS L &@W@Y(<_cdeMS N AXAZ*=`defOS P *QS T /0)*D*F&D(?(AJeJg$mq   SS d 5eS f :gS h ?iS j DkS l 'CZC\)?bfghmS n ?;R;T$7Z^_`oS r #=#?T'>'@#d  qS ~ (S @ 8R8T4Z^`defAS D )C)E%4-D-F)D  CS R )C)E%4-D-F)D  QS ` $>$@ d(?(A$t  _S l 43J3L/RVWXmS p )C)E%4-D-F)D  oS ~ %?%A!t)@)B%  }S L $>$@ d(?(A$t  KS Z $>$@ d(?(A$t  YS U n%%9;RS#=#7#=#=#?@#?41a!Q$#?@#%S 42H@9= &*(,(,""l#(bkk#.>)>#?lc2;;s#3345lld^ l d38n % l E$)$ % lsmlll #s(^l^\r\r~  0_ AsB6 AE