tjdZddlmZmZmZmZmZmZddlm Z ddl m Z ddl m Z e jeZddd ZGd d e Zd gZd S) z%Tokenization class for model PEGASUS.)Regex Tokenizerdecoders normalizerspre_tokenizers processors)Unigram)TokenizersBackend)loggingz spiece.modelztokenizer.json) vocab_filetokenizer_filecteZdZdZeZddgZeZ d d e e e e e fzdzffd Z xZS)PegasusTokenizera Construct a PEGASUS tokenizer (backed by HuggingFace's *tokenizers* library). Based on [Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models). This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. mask_token (`str`, *optional*, defaults to `""`): The token used for masking single token values. This is the token used when training this model with masked language modeling (MLM). This is the token that the PEGASUS encoder will try to predict during pretraining. It corresponds to *[MASK2]* in [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://huggingface.co/papers/1912.08777). mask_token_sent (`str`, *optional*, defaults to `""`): The token used for masking whole target sentences. This is the token used when training this model with gap sentences generation (GSG). This is the sentence that the PEGASUS decoder will try to predict during pretraining. It corresponds to *[MASK1]* in [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://huggingface.co/papers/1912.08777). additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. If no additional_special_tokens are provided and are used as additional special tokens corresponding to the [original PEGASUS tokenizer](https://github.com/google-research/pegasus/blob/939830367bcf411193d2b5eca2f2f90f3f9260ca/pegasus/ops/pretrain_parsing_ops.cc#L66) that uses the tokens 2 - 104 only for pretraining offset (`int`, *optional*, defaults to 103): Offset for additional special tokens. vocab (`str` or `list[tuple[str, float]]`, *optional*): Custom vocabulary with `(token, score)` tuples. If not provided, a blank vocabulary is initialized. input_idsattention_maskNgvocabc | |_|||vr||gng}ng}|dtd|jDz }|Bt|dft|dft|dft|dfg}||_t t ||jt|dfd|_|Stj tj |tj tddg|j_ n`tj tj tddtj tddg|j_ tjd d d |j_t#jd d d |j_t'jd|||||| |d | t+jd|d|t||t|fg|j_dS)Ncg|]}d|d S)z).0is n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/pegasus/tokenization_pegasus.py z-PegasusTokenizer.__init__..cs %R%R%Rqlalll%R%R%Rg)runk_idz {2,} z\nu▁alwaysT) replacementprepend_schemesplit) pad_token eos_token unk_token mask_tokenmask_token_sentoffsetadditional_special_tokensz$A z$A $B )singlepairspecial_tokensr)r0rangestr_vocabrr index _tokenizerrSequence PrecompiledReplacer normalizerr Metaspace pre_tokenizerrdecodersuper__init__rTemplateProcessingconvert_tokens_to_idspost_processor) selfrr+r,r-r.r/_spm_precompiled_charsmapr1r0kwargs __class__s r rBzPegasusTokenizer.__init__Psa $ ,G`0`0`=L=X(9(9^` % %(* %!%R%RE!T[r]s,+ZZZZZZZZZZZZZZZZ%%%%%%>>>>>>  H % %#1EUVVh h h h h (h h h V  r"