` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc :>[U[5(a [USSS9OUnU c0OU Ul[R "S 0URD6UlUR R[U55 XlSSSSS.Ul SUl [UR 5Ul [[5VVs0sH#unnUURU-UR-_M% snnUlURR!5VVs0sH unnUU_M snnUl[UR 5[UR5-UR-URS 'URR%UR5 URR!5VVs0sH unnUU_M snnUl[)URR+55nU b)UR-U Vs/sH nUU;dM UPM sn5 [.TU]`"S UUUUUUUSU U UURS . UD6 U bU OS UlURUR2UlXlUR9UR25 gs snnfs snnfs snnfs snf) NTF)lstrip normalizedrr)) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr/r) isinstancestrrr/spmSentencePieceProcessorsp_modelLoadr fairseq_tokens_to_idsfairseq_offsetlen sp_model_size enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langcur_lang_code_idrCset_src_lang_special_tokens)selfr r:r;r=r>r<r?r@rArBrCr/rDkwargsicodekv_additional_special_tokenst __class__s d/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/mbart/tokenization_mbart.pyr[MBartTokenizer.__init__Asz&FPPZ\_E`E`Jz$5 Afp &5%+>> >No 261E1E1K1K1MN1MA11MN/24==/ACH\H\D]/]`d`s`s/s""8, ""))$*>*>?7;7Q7Q7W7W7Y%Z7Ytq!ad7Y%Z"%)$*>*>*C*C*E%F" $ 0 & - -5]5qB\9\5]   !&@ 00  &.%9w $ 4 4T^^ D  ((8G  O&[ ^s>)J J ,J9 JJc~URR5nSUS'URR5US'U$)NrJsp_model_proto)__dict__copyrJserialized_model_proto)r_states rh __getstate__MBartTokenizer.__getstate__s; ""$ j"&--"F"F"H cXl[US5(d0Ul[R"S0URD6UlUR R UR5 g)Nr/rE)rlhasattrr/rHrIrJLoadFromSerializedProtork)r_ds rh __setstate__MBartTokenizer.__setstate__sR t.//#%D 22JT5I5IJ  --d.A.ABrrcx[UR5[UR5-UR-S-$)Nr3)rNrJrRrMr_s rh vocab_sizeMBartTokenizer.vocab_sizes24==!C(<(<$==@S@SSVWWWrrreturncUR$N)r\rzs rhrBMBartTokenizer.src_langs ~~rr new_src_langcFXlURUR5 gr)r\r^)r_rs rhrBrs% ((8rr token_ids_0 token_ids_1already_has_special_tokensc >U(a[TU]XSS9$S/[UR5-nS/[UR5-nUcUS/[U5--U-$US/[U5--S/[U5--U-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr3r)rZget_special_tokens_maskrNr-r.)r_rrr prefix_ones suffix_onesrgs rhr&MBartTokenizer.get_special_tokens_masks& &72']a3 cC 2 233 cC 2 233  1#K(8"89KG GqcC $445!s;?O9OPS^^^rrc~UcURU-UR-$URU-U-UR-$)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r-r.)r_rrs rh build_inputs_with_special_tokens/MBartTokenizer.build_inputs_with_special_tokenssG,  %% 3d6H6HH H!!K/+=@R@RRRrrcUR/nUR/nUc[XA-U-5S/-$[XA-U-U-U-U-5S/-$)a{ Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. r) sep_token_id cls_token_idrN)r_rrsepclss rh$create_token_type_ids_from_sequences3MBartTokenizer.create_token_type_ids_from_sequencessi$  !  !  s(3./1#5 53$s*S0;>DEKKrrreturn_tensorsrBrCc vUbUc [S5eX0lU"U4SUS.UD6nURU5nXvS'U$)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrBconvert_tokens_to_ids)r_ raw_inputsrrBrC extra_kwargsinputs tgt_lang_ids rh_build_translation_inputs(MBartTokenizer._build_translation_inputssU  x/`a a jiT.i\hi00: (3$% rrc[UR5Vs0sHoRU5U_M nnURUR5 U$s snfr)ranger{convert_ids_to_tokensrUadded_tokens_encoder)r_ravocabs rh get_vocabMBartTokenizer.get_vocab sL;@;QR;Qa++A.1;QR T../ SsAtextc>URRU[S9$)N)out_type)rJencoderG)r_rs rh _tokenizeMBartTokenizer._tokenizes}}##D3#77rrcXR;aURU$URRU5nU(aX R-$UR$)z0Converts a token (str) in an id using the vocab.)rLrJ PieceToIdrM unk_token_id)r_tokenspm_ids rh_convert_token_to_id#MBartTokenizer._convert_token_to_idsQ .. .--e4 4((/06v+++L4;L;LLrrcXR;aURU$URRXR- 5$)z=Converts an index (integer) in a token (str) using the vocab.)rVrJ IdToPiecerM)r_indexs rh_convert_id_to_token#MBartTokenizer._convert_id_to_tokens= .. .--e4 4}}&&u/B/B'BCCrrclSRU5R[S5R5nU$)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r_tokens out_strings rhconvert_tokens_to_string'MBartTokenizer.convert_tokens_to_string!s,WWV_,,-=sCIIK rrsave_directoryfilename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RRUR5[RRU5:waG[RRUR5(a[URU5 U4$[RRUR5(dC[US5nURR5nURU5 SSS5 U4$U4$!,(df  U4$=f)NzVocabulary path (z) should be a directory-rr wb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathr isfileropenrJrnwrite)r_rrout_vocab_fileficontent_spiece_models rhsave_vocabularyMBartTokenizer.save_vocabulary&s,ww}}^,, LL,^,<>X lX@l[TU] "X40UD6$r)rBrCrZprepare_seq2seq_batch)r_rrBrrCr`rgs rhr$MBartTokenizer.prepare_seq2seq_batch7s$!  w,YLVLLrrc8URUR5$r)r^rBrzs rh_switch_to_input_mode$MBartTokenizer._switch_to_input_modeC// >>rrc8URUR5$r)set_tgt_lang_special_tokensrCrzs rh_switch_to_target_mode%MBartTokenizer._switch_to_target_modeFrrrctURUUl/UlURUR/Ulg)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrR cur_lang_coder- eos_token_idr.)r_rBs rhr^*MBartTokenizer.set_src_lang_special_tokensIs6!11(;"//1C1CDrrlangctURUUl/UlURUR/Ulg)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r_rs rhr*MBartTokenizer.set_tgt_lang_special_tokensOs6!11$7"//1C1CDrr)rlr\rr]rVrMrLrTrRr-rJr/rOrBr.rCr ) r5r7r7r5r8r6r9NNNNN)NFr)rNr )r}N),__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesr-rWint__annotations__r.rdictrGrr[rprwpropertyr{rBsetterboolrrrrrrrrrtuplerrrrrr^r__static_attributes__ __classcell__)rgs@rhr)r)%s (*$&67!M49!!M49! 48"&L9"$sCx.1L9L9\ CXX#__9S9T99 sx_9_3;DI3F_ko_ c__>JNS9S3;DI3FS cS8JNL9L3;DI3FL cL2 *- 9A# RZ[^R_  8c8d3i8MD  !c!HSM!]bcf]g!( )- M9 M MDI& M  M  M M??E EEEErrr))rshutilrtypingrrr&rHtokenization_utilsrrr utilsr utils.import_utilsr get_loggerrrrrrQr)__all__rErrrhrs   PP*   H %!#<={ %&mE(mE'mE`  rr