tj, UdZddlmZmZddlmZejeZdZ dZ dZ dZ dZ d Zd Ze d e d e d ede dediZeeefed<deDZeeefed<GddeZdgZdS)z Tokenization classes for CANINE.) AddedTokenPreTrainedTokenizer)loggingiiiiiiz[CLS]z[SEP]z[BOS]z[MASK]z[PAD]z [RESERVED]SPECIAL_CODEPOINTSci|]\}}|| Sr ).0 codepointnames l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/canine/tokenization_canine.py r4s-p-p-p/)TdI-p-p-pSPECIAL_CODEPOINTS_BY_NAMEc eZdZdZgdZeeeeeeeeeeee ddffd Z e de fdZ dZd edeefd Zd ede fd Zd e defdZdZxZS)CanineTokenizera Construct a CANINE tokenizer (i.e. a character splitter). It turns text into a sequence of characters, and then converts each character into its Unicode code point. [`CanineTokenizer`] inherits from [`PreTrainedTokenizer`]. Refer to superclass [`PreTrainedTokenizer`] for usage examples and documentation concerning parameters. Args: model_max_length (`int`, *optional*, defaults to 2048): The maximum sentence length the model accepts. ) input_idsattention_masktoken_type_idsFic  8t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}i|_tD]\} } | |j| <d|jD|_t|_t|j|_ tj d||||||||dddd | dS) NF)lstriprstripTci|]\}}|| Sr r )r r r s r rz,CanineTokenizer.__init__..bs+; ; ; /iIt; ; ; r all_zeroscls_sep) bos_token eos_token sep_token cls_token pad_token mask_tokenadd_prefix_spacemodel_max_lengthtoken_type_ids_pattern%token_type_ids_include_special_tokensspecial_tokens_patternr ) isinstancestrr_special_codepointsritems_special_codepoint_stringsUNICODE_VOCAB_SIZE_unicode_vocab_sizelen_num_special_tokenssuper__init__) selfrrrrr r!r"r#kwargsr r __class__s r r1zCanineTokenizer.__init__GsJTT]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir KUU_adJeJeuZ 4FFFFku 46 17799 7 7OIt-6D $T * *; ; 373K3Q3Q3S3S; ; ; '$6 #&t'?#@#@  !--#.26#,  rreturnc|jS)N)r-)r2s r vocab_sizezCanineTokenizer.vocab_sizexs ''rcvdt|jD}||j|S)Nc.i|]}t||Sr )chr)r is r rz-CanineTokenizer.get_vocab..}s ;;;qQ;;;r)ranger7updateadded_tokens_encoder)r2vocabs r get_vocabzCanineTokenizer.get_vocab|s9;;E$/$:$:;;; T./// rtextc t|S)z5Tokenize a string (i.e. perform character splitting).)list)r2rAs r _tokenizezCanineTokenizer._tokenizesDzzrtokencd t|S#t$rtd|dwxYw)zaConverts a token (i.e. a Unicode character) in an id (i.e. its integer Unicode code point value).zinvalid token: '')ord TypeError ValueError)r2rEs r _convert_token_to_idz$CanineTokenizer._convert_token_to_idsH :u::  : : :888899 9 :s/indexc |tvr t|St|S#t$rtd|wxYw)z Converts a Unicode code point (integer) in a token (str). In case it's a special code point, convert to human-readable format. z invalid id: )rr:rIrJ)r2rLs r _convert_id_to_tokenz$CanineTokenizer._convert_id_to_tokens\  5***)%00u::  5 5 53E3344 4 5s ''Ac,d|S)N)join)r2tokenss r convert_tokens_to_stringz(CanineTokenizer.convert_tokens_to_stringswwvr)__name__ __module__ __qualname____doc__model_input_namesr:CLSSEPPADMASKr1propertyintr7r@r(rCrDrKrNrS __classcell__)r4s@r rr7sP  JII#c((#c((#c((#c((#c((3t99/ / / / / / b(C(((X( cd3i:#:#:::: 5# 5# 5 5 5 5rrN)rWtokenization_pythonrrutilsr get_loggerrTloggerr,r[rYrZBOSr\RESERVEDrdictr^r(__annotations__r*rr__all__r rr ris0'&&BBBBBBBB  H % %     ( l &DcN    .q-pUgUmUmUoUo-p-p-pDcNpppbbbbb)bbbJ  r