tj v d dl Z d dlZd dlmZmZmZmZmZ d dlm Z ddl
m Z dddZ G d d
e Z
d
gZdS ) N) Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )TokenizersBackendz spiece.modelztokenizer.json)
vocab_filetokenizer_filec e Zd ZdZeZddgZeZ d fd Z d
Z
d Z dde e
e z dededz dedef
fd
Z xZS )
LasrTokenizera
Construct a LASR tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dict. If not provided, a minimal vocabulary is created using the special tokens.
input_idsattention_maskNd c || _ |ld |D }
t |
dk r |d t | D z
}nK|dk r)|t |
k rt d| d| d nd t | D }
|
}||| _ not | d ft | d ft | d fd
g| _ t |dz
d d D ]"} | j d | d
d f #t t | j dd | _ | t j | | j _ t j t j t j ddd g | j _ t% j ddd | j _ t) j d|||||d| t- j ddgg dd| j fg | j _ d S )Nc 4 g | ]}d t | v |S )
z*LasrTokenizer.__init__..Z s, [[[!LTWXYTZTZDZDZADZDZDZ c g | ]}d | d S r > r is r r z*LasrTokenizer.__init__..\ s$ -Z-Z-ZA.?1.?.?.?-Z-Z-Zr r zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to LasrTokenizer. In this case the additional_special_tokens must include the extra_ids tokensc g | ]}d | d S r r" r# s r r z*LasrTokenizer.__init__..d s$ HHH! - - - -HHHr g ) ▁g r r! r F)unk_id
byte_fallbackr& alwaysT) replacementprepend_schemesplit) eos_token unk_token pad_token extra_idsadditional_special_tokens$Ar )r3 r z$Br )singlepairspecial_tokensr" )
_extra_idslenrange
ValueError
_vocab_scoresr appendr r
_tokenizerr Precompiled
normalizerr SequenceWhitespaceSplit Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessing eos_token_idpost_processor)
selfr. r/ r0 _spm_precompiled_charsmapr1 r2 vocabr kwargs extra_tokensr$ __class__s
r rF zLasrTokenizer.__init__J s $ % 0[['@[[[L< 1$$)-Z-ZyIYIY-Z-Z-ZZ))Q 9L0A0A#A#A y Sl IHuY7G7GHHHL(4 % !&D Y %Y %Y % "D 9q="b11
D
D"))+<+<+<+---*+ *
*
*
&&&r c b t t t d | j S )zQGet the list of sentinel tokens (extra_id tokens) from additional_special_tokens.c J t t j d| d uS )Nz)boolresearch)r s r z3LasrTokenizer.get_sentinel_tokens.. s bi0A1&E&E!F!Fd!R r )listsetfilterr2 rJ s r get_sentinel_tokensz!LasrTokenizer.get_sentinel_tokens s1 RRTXTrss t t
r c D fd D S )z&Get the token IDs for sentinel tokens.c : g | ]} | S r" )convert_tokens_to_idsr tokenrJ s r r z8LasrTokenizer.get_sentinel_token_ids.. s' ZZZe**511ZZZr )rZ rY s `r get_sentinel_token_idsz$LasrTokenizer.get_sentinel_token_ids s) ZZZZt?W?W?Y?YZZZZr FT token_idsskip_special_tokens clean_up_tokenization_spaces group_tokensreturnc t |t r|g}|r d t j | D } fd|D } t j d|||d|S )Nc g | ]
}|d S )r r" )r token_groups r r z)LasrTokenizer._decode.. s XXXKQ XXXr c * g | ]}|j k
|S r" ) pad_token_idr^ s r r z)LasrTokenizer._decode.. s& PPPuUd>O5O5OU5O5O5Or )ra rb rc r" )
isinstanceint itertoolsgroupbyrE _decode)rJ ra rb rc rd rM rO s ` r ro zLasrTokenizer._decode s i % % $" I YXX9;LY;W;WXXXI QPPP PPP uww
3)E
r )r r r Nr NNN)FNT)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelrF rZ r` rl rV rR r ro
__classcell__)rO s @r r r ! s
" "H *$&67 E "&"&K
K
K
K
K
K
Z
[ [ [ %*48 !
c?
"
'+Tk
r r )rm rS
tokenizersr r r r r tokenizers.modelsr tokenization_utils_tokenizersr
rt r __all__r" r r r} s * S S S S S S S S S S S S S S % % % % % % > > > > > > $2EUVV U
U
U
U
U
% U
U
U
p
r