)__name__
__module__ __qualname____firstlineno__r% r5 __static_attributes__ r' r r r / s
A
r' r c
t ^ \ rS rSrSr\rSS/rSr S U 4S jjr \
S 5 r \
S 5 r \
S 5 r
\
S
\4S j5 rS rS
rU 4S jrS rS\\ S
\4S jrS rS rS S\S\\ S
\\ 4S jjr S S\\ S\\\ S
\\ 4S jjr S S\\ S\\\ S\ S
\\ 4U 4S jjjr S r U =r $ ) CpmAntTokenizerO a^
Construct a CPMAnt tokenizer. Based on byte-level Byte-Pair-Encoding.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bod_token (`str`, *optional*, defaults to `""`):
The beginning of document token.
eod_token (`str`, *optional*, defaults to `""`):
The end of document token.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
line_token (`str`, *optional*, defaults to `""`):
The line token.
space_token (`str`, *optional*, defaults to `""`):
The space token.
input_idsattention_maskFc
L > [ U S/5 X l X0l [ U5 U l U R U U R S' U R U U R S' U R U U R U [
R " [ U R R 5 S S95 U l U R R 5 V V
s0 sH u pX_M sn
n U l [ U R US9U l [ TU ]4 " SUUUUUUUU U
S. U D6 g s sn
n f ) Njieba r c U S $ Nr* r> xs r *CpmAntTokenizer.__init__.. Z[\]Z^r' key)r r" ) bod_token eod_token bos_token eos_token pad_tokenr"
line_token space_token padding_sider> )r rP rQ r encoderr r sorteditemsdecoderr wordpiece_tokenizersuperr% )r$ r rP rQ rR rS rT r" rU rV rW kwargskv __class__s r r% CpmAntTokenizer.__init__l s
$ *""!*- LL 5 S !\\* 5 T LL % LL $"..vdll6H6H6JP^/_` )- ););)=>)=)=> #5DLLT]#^
!#%
?s
D c 4 U R U R $ r! )rX rP r$ s r bod_token_id CpmAntTokenizer.bod_token_id ||DNN++r' c 4 U R U R $ r! )rX rQ rd s r eod_token_id CpmAntTokenizer.eod_token_id rg r' c U R S $ )Nr rX rd s r
newline_idCpmAntTokenizer.newline_id s ||D!!r' returnc , [ U R 5 $ r! )r, rX rd s r
vocab_sizeCpmAntTokenizer.vocab_size s 4<< r' c B [ U R 40 U R D6$ r! )dictrX added_tokens_encoderrd s r get_vocabCpmAntTokenizer.get_vocab s DLL>D$=$=>>r' c / n[ R " USS9 H- nUR U R R U5 5 M/ U$ )zTokenize a string.F)cut_all)rE cutextendr\ r5 )r$ text
output_tokensrJ s r _tokenizeCpmAntTokenizer._tokenize sA
4/A !9!9!B!B1!E F 0 r' c > U Vs/ sH o3S: d M
UPM nnU Vs/ sH8 oDU R :w d M X@R :w d M% X@R :w d M6 UPM: nn[ TU ] " U40 UD6$ s snf s snf )zDecode ids into a string.r ) pad_token_id eos_token_id bos_token_idr] _decode)r$ token_idsr^ irJ ra s r r CpmAntTokenizer._decode s} )4 1!VQ 4
!):):$:AqDUDU?UAZ[_p_pZpAy
wy3F33 5
s A/A/A4
A4
A4A4c XR ; $ r! rl r$ r s r checkCpmAntTokenizer.check s $$r' r c $ SR U5 $ )Nr) )r- )r$ r s r convert_tokens_to_string(CpmAntTokenizer.convert_tokens_to_string s wwv r' c ~ U R R XR R U R 5 5 $ )z0Converts a token (str) in an id using the vocab.)rX getr" r s r _convert_token_to_id$CpmAntTokenizer._convert_token_to_id s* || ||'7'7'GHHr' c L U R R XR 5 $ )z=Converts an index (integer) in a token (str) using the vocab.)r[ r r" )r$ r s r _convert_id_to_token$CpmAntTokenizer._convert_id_to_token s || ~~66r' save_directoryfilename_prefixc D [ R R U5 ( a6 [ R R X( a US- OS[ S - 5 nOU( a US- OSU- nSnSU R
; a) U R
S U R
S' U R
S SU R
; a) U R
S U R
S' U R
S [ R " [ U R
R 5 S S
95 U l [ US S S
9 nU R
R 5 H> u pgXG:w a [ R SU S35 UnUR US- 5 US-
nM@ S S S 5 U4$ ! , ( d f U4$ = f)N-r) r r rF r c U S $ rH r> rI s r rK 1CpmAntTokenizer.save_vocabulary.. rM r' rN wr r
zSaving vocabulary to z\: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!r* )ospathisdirr- VOCAB_FILES_NAMESrX r r rY rZ r loggerwarningwrite)r$ r r r r writerr token_indexs r save_vocabulary CpmAntTokenizer.save_vocabulary sp
77== ( ( / 3!6rUfgsUt tJ 4C/C/n\J $,, #'<<#4DLL S! 4<< #'<<#5DLL T""..vdll6H6H6JP^/_`
*cG
4&*ll&8&8&:"'NN/
| U( a [ TU ] XSS9$ Ub' S/S/[ U5 - - S/- S/[ U5 - - $ S/S/[ U5 - - $ )aT
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`list[int]`): List of IDs.
token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r r r r* r )r] get_special_tokens_maskr, )r$ r r r ra s r r 'CpmAntTokenizer.get_special_tokens_mask sw " &72 ']a 3
"31#K 001QC7A3[AQ;QR RsqcC ,,--r' )rP r[ rX rQ r\ ) zzzzzr7 r r leftr! )NF)!r9 r: r; r< __doc__r vocab_files_namesmodel_input_namesadd_prefix_spacer% propertyre ri rm intrq rv r~ r r r+ strr r r r tupler r boolr r=
__classcell__)ra s @r r@ r@ O sr 0 *$&67
(
T , , , , " " !C ! !? 4% tCy S I7 c HSM ]bcf]g 8 JNU 9U3;DI3FU
cU* sx. 9.3;DI3F.ko.
c. .r' r@ )r r r typingr transformers.utilsr r rE tokenization_utilsr utilsr
get_loggerr9 r r r r r@ __all__r> r' r r sq ' D 5
H %!;/ @~.) ~.B
r'