tjQ5dZddlZddlmZ ddlZn #e$rdZYnwxYwddlmZddl m Z ddl m Z m Z mZddlmZmZmZejeZd d iZd Zee Gd d e ZGddZdS)zT SentencePiece-based tokenization class for loading from sentencepiece.model files. N)copyfile)import_protobuf)PreTrainedTokenizer)INIT_TOKENIZER_DOCSTRING AddedTokengenerate_merges)add_end_docstringsloggingrequires_backends vocab_fileztokenizer.modelu▁c 2eZdZdZeZfdZedefdZ dZ dde e e e zdedefd Zdd e e d zfd Zd ZdZdZde e de fdZdde de d zdee fdZ ddee ezdeded zdede f fd ZxZS)SentencePieceBackendaJ Base class for SentencePiece-based tokenizers that load from sentencepiece.model files. Inherits from [`~tokenization_utils.PreTrainedTokenizer`]. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). c t|d|d|_|dd|_|di|_d|vrd|d<t jdi|j}||j|jsyt}|j | }|j jr3d|j _||||_|j|_|j|d<t)jdi||dS) N sentencepiecer legacyTsp_model_kwargsbackendF)r getr rpoprspmSentencePieceProcessorLoadr ModelProto FromStringserialized_model_protonormalizer_specadd_dummy_prefixLoadFromSerializedProtoSerializeToStringsp_modelget_piece_sizetotal_vocab_sizesuper__init__ _update_trie)selfkwargs tokenizer model_pb2proto __class__s k/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/tokenization_utils_sentencepiece.pyr&zSentencePieceBackend.__init__<sr$000!**\22jj400 %zz*;R@@ F " " /F9 .FF1EFF t'''{ M'))I(33I4T4T4V4VWWE$5 M9>%611%2I2I2K2KLLL! !% < < > >%)$8 ! ""6""" returnc4|jS)zReturns vocab size)r"r#)r(s r. vocab_sizezSentencePieceBackend.vocab_sizecs}++---r/c|fdtjD}|j|S)zReturns vocab as a dictc<i|]}||Sr)convert_ids_to_tokens).0ir(s r. z2SentencePieceBackend.get_vocab..js)RRRa++A..RRRr/)ranger2updateadded_tokens_encoder)r(vocabs` r. get_vocabzSentencePieceBackend.get_vocabhs@RRRR5;Q;QRRR T./// r/F new_tokensspecial_tokensc n|sdSt|}d}|D]}t|ttfs#t d|dt |dt|dkrVt|tr+||jvru||jvp|}t|dd| |}n|r|d|j d ||j vr|j s6|j r/t|d dr|j|_|j|j}||jko"|j||jk}|r|} n |} |d z }|d z }|j r0t||jvr|j|||j | <| |j|j<|jrt.d |d |||S)a Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Special tokens are sometimes already in the vocab which is why they have to be handled specifically. Args: new_tokens (`list[str]`or `list[tokenizers.AddedToken]`): Token(s) to add in vocabulary. A token is counted as added if it's not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). If a token is part of the vocabulary then we simply mark this token as an `AddedToken` which allows to control the stripping and normalization of this token. This is NOT possible in `tokenizers`. special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the tokens should be added as special tokens. Returns: `int`: The number of tokens actually added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased") model = BertModel.from_pretrained("google-bert/bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```rzToken z is not a string but a .F)rstriplstrip normalizedspecialT)rFrE do_lower_caserzAdding z to the vocabulary)len isinstancestrr TypeErrortype_added_tokens_encoderall_special_tokens __setstate__rE_added_tokens_decodervaluesrFgetattrcontentlowerr" piece_to_idr# IdToPiece_extra_special_tokensappendverboseloggerinfor'_update_total_vocab_size) r(r>r? next_index num_addedtoken is_specialtok_id in_base_vocab token_indexs r. _add_tokensz SentencePieceBackend._add_tokensns~< 1YY  ' A' AEec:%677 W U U UtE{{ U U UVVV5zzR%%% VD666"d&==O "5uU_Q_isttt V""t5CS#T#TUUU299;;;;= 6U%5 6'$Y^:_:_ 6 % 3 3 5 5 ]..u}==F55777lDM Hey", out_type = str)[4:]`.  )out_typeN)r startswithSPIECE_UNDERLINEr"encoderJ unk_tokenrH)r(textr)tokensunk_token_lengths r. _tokenizezSentencePieceBackend._tokenizes ; tjd|||d|S)z Decode token ids to string. Uses the generic decode path from PreTrainedTokenizer which works for all vocabularies, including custom vocabularies that override _convert_id_to_token. )rrrr)r%_decode)r(rrrrr)r-s r.rzSentencePieceBackend._decode s<uww  3)E      r/)Frg)FNF)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESrr&propertyintr2r=listrJrboolrdr'rurwrzrtuplerr __classcell__)r-s@r.rr,s  *%%%%%N.C...X. NNd3i$z2B&BNTXNehNNNN` , ,49t3C , , , ,XXX&000 tCyS !!c!C$J!Z_`cZd!!!!@%*48.3   c? " '+Tk  (,             r/rcveZdZdZdefdZddeeeefe eee fe effdZ dS)SentencePieceExtractorzl Extractor implementation for SentencePiece trained models. https://github.com/google/sentencepiece modelct|dddlm}||_|j|dS)Nrr)r)r rrspr)r(rrs r.r&zSentencePieceExtractor.__init__&sN$000888888((**  Ur/Nr0cJ|jfdtD}fdtD}t||}fdtD}|||fS)z By default will return vocab and merges with respect to their order, by sending `vocab_scores` we're going to order the merges with respect to the piece scores instead. c<i|]}||Sr) id_to_piece)r6ryrs r.r8z2SentencePieceExtractor.extract..3s'XXXeR^^E**EXXXr/cbi|]+}||,Srr get_scorer6r7rs r.r8z2SentencePieceExtractor.extract..5s1bbbAR^^A.. Qbbbr/cdg|],}||f-Srrrs r. z2SentencePieceExtractor.extract..9s4dddabnnQ//aAdddr/)rr9 GetPieceSizer )r( vocab_scores vocab_idsvocab_scores_dictmergesvocab_scores_listrs @r.extractzSentencePieceExtractor.extract-s WXXXXuR__EVEV?W?WXXX bbbbrO`O`IaIabbb ,=>>dddd5QSQ`Q`QbQbKcKcddd+V33r/rg) rrrrrJr&rdictrrfloatrrr/r.rr!sc44E$sCx.$uSRWZGXBY[_`e[f2f,g444444r/r)rrshutilrrr ImportErrorconvert_slow_tokenizerrtokenization_pythonrtokenization_utils_baserrr utilsr r r get_loggerrrZrrorrrr/r.rs  CCC433333444444 BAAAAAAAAA  H % %!#45,--q q q q q .q q .-q h4444444444s