tj 8 d dl mZmZmZmZmZmZ d dlmZ ddl m
Z
m Z ddl m
Z
ddlmZ ej e Zddd Zg d
Z G d d e
Zd gZeZd
S ) )Regex Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )
AddedToken
BatchEncoding)TokenizersBackend)loggingzsentencepiece.bpe.modelztokenizer.json)
vocab_filetokenizer_file)4ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNaf_ZAaz_AZbn_INfa_IRhe_ILhr_HRid_IDka_GEkm_KHmk_MKml_INmn_MNmr_INpl_PLps_AFpt_XXsv_SEsw_KEta_INte_INth_THtl_XXuk_UAur_PKxh_ZAgl_ESsl_SIc e Zd ZU dZeZddgZeZg Z e
e e d< g Z
e
e e d< d"d eez e
z dz d
edz f fd
Zd Zd Zedefd Zej deddfd Z d#de
e dede
e dz dedef
fd
Zd Zd Zdeddfd Zdeddfd Zd ededz dedz fd!Z xZ S )$MBart50Tokenizeru
Construct a MBart50 tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
Path to the vocabulary file.
src_lang (`str`, *optional*):
A string representing the source language.
tgt_lang (`str`, *optional*):
A string representing the target language.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
Examples:
```python
>>> from transformers import MBart50Tokenizer
>>> tokenizer = MBart50Tokenizer.from_pretrained("facebook/mbart-large-50", src_lang="en_XX", tgt_lang="ro_RO")
>>> src_text = " UN Chief Says There Is No Military Solution in Syria"
>>> tgt_text = "Şeful ONU declară că nu există o soluţie militară în Siria"
>>> model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt")
>>> # model(**model_inputs) should work
``` input_idsattention_mask
prefix_tokens
suffix_tokensNvocab_spm_precompiled_charsmapc
t |
t rt |
dd n|
}
t |t rd |D }d |D t fdt
D } | r || _ nPt | dft | dft | dft | dfg}
|
|dd t
D ]&}|
t | df '|
t |
df |
| _ nt | dft | dft | dft | dfd g| _ t
D ] }| j |df | j t |
df t t | j dd
| _ t j
t d d g}|t j | g|z }t j | | j _ t% j d
dd | j _ t+ j d
dd | j _ | dg pg }| t
t1 j d||||||| |
|d | d| _ d t
D }n$# t6 $ r t t
}Y nw xY wt9 | dg pg }d |D t | fd|D z }|| _ ||nd| _ || _ | | j! | j | _" | # | j d S )NTF)lstriprstripc b g | ],}t |d t |d f-S )r )strfloat.0items n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mbart50/tokenization_mbart50.py
z-MBart50Tokenizer.__init__..n s1 FFFc$q'llE$q'NN3FFF c g | ]
}|d S )r rY s r\ r] z-MBart50Tokenizer.__init__..p s 666DG666r^ c 3 K | ]}|v V d S Nr` )rZ lang_code vocab_tokenss r\ z,MBart50Tokenizer.__init__..q s( $g$g9Y,%>$g$g$g$g$g$gr^ g r
) ▁g )unk_id
byte_fallbackz {2,} rf always) replacementprepend_schemesplitadditional_special_tokens) src_langtgt_lang eos_token sep_token cls_token unk_token pad_token
mask_tokenrn rV c 0 g | ]}t |d S )T)special)r )rZ codes r\ r] z-MBart50Tokenizer.__init__.. s% ]]]d:dD999]]]r^ _extra_special_tokensc , h | ]}t | S r` rW )rZ ts r\ z,MBart50Tokenizer.__init__.. s 888AQ888r^ c 6 g | ]}t | v|S r` r| )rZ r}
existing_strss r\ r] z-MBart50Tokenizer.__init__.. s* .e.e.eQQWdIdIdqIdIdIdr^ r r` )$
isinstancerW r listanyFAIRSEQ_LANGUAGE_CODES
_vocab_scoresextendappendr r
_tokenizerr Replacer PrecompiledSequence
normalizerr Metaspace
pre_tokenizerr decoderpopsuper__init__fairseq_offset Exceptiongetattrrz _src_langrp _build_language_code_mappingslang_code_to_idcur_lang_code_idset_src_lang_special_tokens)selfrP rQ ro rp rq rr rs rt ru rv kwargshas_language_codes
vocab_listrc normalizers_rn lang_tokensexisting_extra merged_extrar rd __class__s @@r\ r zMBart50Tokenizer.__init__U s< KUU_adJeJeuZ
4FFFFku
eT " " - > GFFFFE66666L!$$g$g$g$gPf$g$g$g!g!g !
0%*"" ^^S)^^S)^^S)^^S)
!!%),,, "8 = =I %%s9~~s&;<<<< !!3z??C"8999%/"" Y %Y %Y %Y % "D 4
<
< "))9c*:;;;; % %s:&< = = = $ " #
$+E(OOSAAB $ 0 ' 34M N NOR^^L%0%9,%G%G"(6(@Ucksw(x(x(x%"*"4W_gk"l"l"l $*JJ/JB$O$O$USU!!(()?@@@
!&?
7]]F\]]]KK 7 7 7 566KKK 7 '>CCIr88 888
N++.e.e.e.e+.e.e.ee %1"%-%9w
**,,, $ 4T^ D (( 88888s 8L
L+*L+c fdt D _ d j D _ ddddd _ j j t
dd } t | j d < d
j D _ d S ) zFBuild language code to ID mappings and fairseq compatibility mappings.c < i | ]}| | S r` )convert_tokens_to_ids)rZ rc r s r\
zBMBart50Tokenizer._build_language_code_mappings.. s6
AJIt11)<<
r^ c i | ]\ }}|| S r` r` rZ kvs r\ r zBMBart50Tokenizer._build_language_code_mappings.. s N N NA1 N N Nr^ r rV r
)rL rN rK rM rv rO c i | ]\ }}|| S r` r` r s r\ r zBMBart50Tokenizer._build_language_code_mappings.. s %Z%Z%Ztq!a%Z%Z%Zr^ N)
r r itemsid_to_lang_codefairseq_tokens_to_idsupdater r rW fairseq_ids_to_tokens)r rv s ` r\ r z.MBart50Tokenizer._build_language_code_mappings s
Nd
O N1E1K1K1M1M N N N &
&
"
"))$*>??? T<::
/3/I/I#j///Z/Z"8,%Z%Zt7Q7W7W7Y7Y%Z%Z%Z"""r^ c | t | d r3| j | j | _ | | j dS dS )z9Called after tokenizer.json is loaded in from_pretrained.r N)r hasattrr r r r r s r\
_post_initzMBart50Tokenizer._post_init s`
**,,, 4 % % =$($8 $HD ! , ,T^ < < < < < = =r^ returnc | j S rb )r r s r\ ro zMBart50Tokenizer.src_lang s
~ r^ new_src_langc H || _ | | j d S rb )r r )r r s r\ ro zMBart50Tokenizer.src_lang s% % (( 88888r^ r r$ src_textsro tgt_textsrp c V || _ || _ t j ||fi |S rb )ro rp r prepare_seq2seq_batch)r r ro r rp r r s r\ r z&MBart50Tokenizer.prepare_seq2seq_batch s4 !
,uww,Y LLVLLLr^ c 6 | | j S rb )r ro r s r\ _switch_to_input_modez&MBart50Tokenizer._switch_to_input_mode s //
>>>r^ c \ | j | j | _ | | j S rb )rp r set_tgt_lang_special_tokensr s r\ _switch_to_target_modez'MBart50Tokenizer._switch_to_target_mode s) = NDM//
>>>r^ c | | | _ | j g| _ | j g| _ | | j }| | j }t
j |dgz |z |ddgz |z t t ||z | j | j z | j
_ dS )z]Reset the special tokens to the source lang setting. prefix=[src_lang_code] and suffix=[eos].$A$Bsinglepairspecial_tokensN r r rI eos_token_idrJ convert_ids_to_tokensr TemplateProcessingr zipr post_processor)r ro prefix_tokens_strsuffix_tokens_strs r\ r z,MBart50Tokenizer.set_src_lang_special_tokens
$ : :8 D D " 3 4 " / 0 6 6t7I J J 6 6t7I J J)3)F$v-0AA"dD\14EE $58I$I4K]`d`rKr s stt*
*
*
&&&r^ c | | | _ | j g| _ | j g| _ | | j }| | j }t
j |dgz |z |ddgz |z t t ||z | j | j z | j
_ dS )zaReset the special tokens to the target language setting. prefix=[tgt_lang_code] and suffix=[eos].r r r Nr )r rp r r s r\ r z,MBart50Tokenizer.set_tgt_lang_special_tokens r r^ return_tensorsc ||t d || _ | |fd|d|}| | }||d< |S )zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensr forced_bos_token_id)
ValueErrorro r )r
raw_inputsr ro rp extra_kwargsinputs tgt_lang_ids r\ _build_translation_inputsz*MBart50Tokenizer._build_translation_inputs( sg x / `aa a
jiT.ii\hii00:: (3$%
r^ )
NNNNrK rK rL rM rN rO )r Nr$ ) __name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelrI r int__annotations__rJ rW dictr r r propertyro setterr r r r r r r
__classcell__)r s @r\ rF rF ! sd * * X *$&67 E !M49!!! !M49!!! +/04z9 z9TzD 4'z9 $':z9 z9 z9 z9 z9 z9x[ [ [&= = = # X _9S 9T 9 9 9 _9 &*
M
M9
M
M 9t#
M
M
M
M
M
M
M
M? ? ?? ? ?
C
D
C
D
*-
9r s [ Z Z Z Z Z Z Z Z Z Z Z Z Z Z Z % % % % % % < < < < < < < < > > > > > >
H % % $=P`aa n n n Q Q Q Q Q( Q Q Qh
( r^