tj d Z ddlmZmZmZmZmZmZ ddlm Z ddl
m Z ddl m
Z
e
j e ZddiZ G d d
e Zd
gZd S ) z Tokenization classes for XGLM. )Regex Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )TokenizersBackend)loggingtokenizer_fileztokenizer.jsonc e Zd ZdZeZddgZeZ dd
e e
e e e f z dz d e d e d
e de de de de dz de
f fd
Z xZS )
XGLMTokenizeraW
Construct a XGLM tokenizer (backed by HuggingFace's tokenizers library). Based on BPE.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
tokenizer_file (`str`, *optional*):
Path to a tokenizers JSON file containing the serialization of a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary. If not provided, a minimal vocabulary is created.
merges (`list[tuple[str, str]]`, *optional*):
Custom merge rules for BPE. If not provided, merges are generated from the vocabulary.
add_prefix_space (`bool`, *optional*, defaults to `True`):
Whether to add a prefix space before encoding.
input_idsattention_maskN