tjM d dl mZmZmZmZmZ d dlmZ ddlm Z ddl
m Z e j e
ZdddZ G d d
e Zd
gZd S ) ) Tokenizerdecoders normalizerspre_tokenizers
processors)BPE )TokenizersBackend)loggingz
vocab.jsonz
merges.txt)
vocab_file merges_filec e Zd ZdZeZddgZeZ dd
e e
e e f z dz d e e e z dz d e d
e de de de de dz de dz f fd
Z
xZS )HerbertTokenizera
Construct a BPE tokenizer for HerBERT (backed by HuggingFace's tokenizers library).
Peculiarities:
- uses BERT's pre-tokenizer: BertPreTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of
a punctuation character will be treated separately.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the methods. Users should refer to the
superclass for more information regarding methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The padding token.
mask_token (`str`, *optional*, defaults to `""`):
The mask token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary.
merges (`str` or `list[str]`, *optional*):
Custom merges list.
input_idsattention_maskN