~tjddlZddlZddlmZmZmZmZmZddlm Z m Z m Z m Z m Z mZmZddlmZddlmZGddeZdS) N)IteratorListOptionalUnionTuple) AddedTokenRegex Tokenizerdecoders normalizerspre_tokenizerstrainers)Unigram) BaseTokenizerceZdZdZ ddeeeeefdede ffd Z dd e eeefd e d e d eee ee fdeeedeef dZ dde eeeeefd e d e d eee ee fdeeedeedee fdZedefdZxZS)SentencePieceUnigramTokenizerzzSentencePiece Unigram Tokenizer Represents the Unigram algorithm, with the pretokenization used by SentencePiece N▁Tvocab replacementadd_prefix_spacec |tt|}ntt}tjtjtjtjtddg|_|rdnd}tj |||_ tj |||_ d||d}t||dS)N {2,} alwaysneverrprepend_schemeSentencePieceUnigram)modelrr)r rr SequenceNmtNFKCReplacer normalizerr Metaspace pre_tokenizerr decodersuper__init__)selfrrr tokenizerr parameters __class__s n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/tokenizers/implementations/sentencepiece_unigram.pyr*z&SentencePieceUnigramTokenizer.__init__s  !'%..11II!')),,I*3 _   0 2 2K4GgX[4\4\ ]  &6B7"0":{cq"r"r"r $.;Wefff ,& 0  J/////@files vocab_size show_progressspecial_tokensinitial_alphabet unk_tokenc|g}|g}tj|||||}t|tr|g}|j||dS)a Train the model using the given files Args: files (:obj:`List[str]`): A list of path to the files that we should use for training vocab_size (:obj:`int`): The size of the final vocabulary, including all tokens and alphabet. show_progress (:obj:`bool`): Whether to show progress bars while training. special_tokens (:obj:`List[Union[str, AddedToken]]`, `optional`): A list of special tokens the model should know of. initial_alphabet (:obj:`List[str]`, `optional`): A list of characters to include in the initial alphabet, even if not seen in the training dataset. If the strings contain more than one character, only the first one is kept. unk_token (:obj:`str`, `optional`): The unknown token to be used by the model. Nr3r5r4r6r7)trainer)rUnigramTrainer isinstancestr _tokenizertrain)r+r2r3r4r5r6r7r:s r/r?z#SentencePieceUnigramTokenizer.train,s}<  !N  #! )!)'-     eS ! ! GE eW55555r0iteratorlengthc|g}|g}tj|||||}|j|||dS)a Train the model using the given iterator Args: iterator (:obj:`Union[Iterator[str], Iterator[Iterator[str]]]`): Any iterator over strings or list of strings vocab_size (:obj:`int`): The size of the final vocabulary, including all tokens and alphabet. show_progress (:obj:`bool`): Whether to show progress bars while training. special_tokens (:obj:`List[Union[str, AddedToken]]`, `optional`): A list of special tokens the model should know of. initial_alphabet (:obj:`List[str]`, `optional`): A list of characters to include in the initial alphabet, even if not seen in the training dataset. If the strings contain more than one character, only the first one is kept. unk_token (:obj:`str`, `optional`): The unknown token to be used by the model. length (:obj:`int`, `optional`): The total number of sequences in the iterator. This is used to provide meaningful progress tracking Nr9)r:rA)rr;r>train_from_iterator) r+r@r3r4r5r6r7rAr:s r/rCz1SentencePieceUnigramTokenizer.train_from_iterator\svD  !N  #! )!)'-     ++  ,     r0filenamec ddl}|jdddl}n#t$rt dwxYw|}|t|d|j j }d|j D}|j j }|j j}|j j}|dkrt dd} d } t!t#|||} |rNt%jt%j|t%jt-d d g| _n:t%jt%jt-d d g| _| rd nd } t1j| | | _t7j| | | _ddi} t;jt>| | }t;j || | |S)Nr.a\You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required.rbc*g|]}|j|jfS)piecescore).0rJs r/ z:SentencePieceUnigramTokenizer.from_spm..s!BBB%+u{+BBBr0rz]You're trying to run a `Unigram` model but you're file was trained with a different algorithmrTrrrrrr r)!syspathappendsentencepiece_model_pb2 Exception ModelProtoParseFromStringopenreadnormalizer_specprecompiled_charsmappieces trainer_specunk_id model_type byte_fallbackr rr r! Precompiledr$r r%r r&r'r r(r__new__rr*)rDrNr mrXrr[r\r]rrr,rr-objs r/from_spmz&SentencePieceUnigramTokenizer.from_spms  JJJ HOOC 3 3 3 3 3   o        $x..3355666 0EBBBBB&^. 4 ??o  geV]CCDD  d#.#7+,@AA'g<<$$I $/#79LUSZ^^]`9a9a8b#c#cI %5B7"0":{cq"r"r"r $.;Wefff  + #$A9jYYsIz::: s"%?)NrT)r1TNNN)r1TNNNN)__name__ __module__ __qualname____doc__rrrr=floatboolr*rintrr?rrC staticmethodrb __classcell__)r.s@r/rr s48 !% 00U3:./000 000000<"AE04#'.6.6S$s)^$.6.6 .6 !eCO& .6 #49- .6C=.6.6.6.6f"AE04#' $4 4  x '>>?4 4  4 !eCO& 4 #49- 4 C=4  4 4 4 4 l13111\11111r0r)jsonostypingrrrrr tokenizersrr r r r r rtokenizers.modelsrbase_tokenizerrrrIr0r/rrs 99999999999999dddddddddddddddddd%%%%%%))))))yyyyyMyyyyyr0