"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`list[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ``` input_idsattention_maskNsp_model_kwargsreturnc >U c0OU Ul[U5R5(d SU35eXl[ U5Ul[ U5UR ;a [S5e[ U 5UR ;deU (aK[ U5UlURR5VVs0sHupX_M snnUl /Ul OUR R5VVs0sHupX_M snnUl UR Vs/sH4oRS5(dMURS5(dM2UPM6 snUl XPlX`lX/Ul[#XR5Ul[#X R5UlUR$UlUR UlUR-5 [.TU]`"SUUUUU U URUU S. U D6 gs snnfs snnfs snf)Nzcannot find spm source z token must be in the vocab>><<) source_lang target_lang unk_token eos_token pad_tokenmodel_max_lengthrrseparate_vocabs)rrexistsr# load_jsonencoderstrKeyErrortarget_encoderitemsdecodersupported_language_codes startswithendswithrr spm_filesload_spm spm_source spm_target current_spmcurrent_encoder_setup_normalizersuper__init__)selfr r rrrrrr r!r"rr#kwargskv __class__s f/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/marian/tokenization_marian.pyr8MarianTokenizer.__init__ks &5% -=> >9~--- "+,=">D -1-@-@-F-F-HI-HTQAD-HIDL,.D )-1\\-?-?-AB-ATQAD-ABDL>Bll2vlll[_N`1efeoeopteu1l2vD )&&$1#:/C/CD":/C/CD??#||    ##- 00/+  )JC2vs? G?: HH :H H cSSKJn U"UR5RUlg![ [ 4a! [R"S5 SUlgf=f)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.cU$Nr$)xs r>3MarianTokenizer._setup_normalizer..sQ) sacremosesrAr normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)r9rAs r>r6!MarianTokenizer._setup_normalizersM / 7#78H8H#I#S#SD ./ / MM@ A#.D  /s'*.AArDc6U(aURU5$S$)zHCover moses empty string edge case. They return empty list for '' input!)rJ)r9rDs r>rIMarianTokenizer.normalizes*+t##A&33rGcfURRXRUR5$rC)r5getr)r9tokens r>_convert_token_to_id$MarianTokenizer._convert_token_to_ids(##''/C/CDNN/STTrGtextc/nURS5(a5URS5=nS:waURUSUS-5 XS-SnX!4$)z6Remove language codes like >>fr<< before sentencepiecerrN)r.findappend)r9rXcodeend_locs r>remove_language_code$MarianTokenizer.remove_language_codesW ??4 4&@gR%G KK]w{+ ,! &DzrGclURU5up!URRU[S9nX#-$)N)out_type)r`r4encoder()r9rXr^piecess r> _tokenizeMarianTokenizer._tokenizes7..t4 !!(((<}rGindexcLURRXR5$)z?Converts an index (integer) in a token (str) using the decoder.)r,rTr)r9rhs r>_convert_id_to_token$MarianTokenizer._convert_id_to_tokens||~~66rGc &>[TU]"U40UD6$)a Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[list[int], list[list[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `list[str]`: The list of decoded sentences. )r7 batch_decode)r9 sequencesr:r=s r>rmMarianTokenizer.batch_decodes*w#I888rGc &>[TU]"U40UD6$)aj Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )r7decode)r9 token_idsr:r=s r>rqMarianTokenizer.decodes0w~i2622rGtokenscZUR(a URO URn/nSnUH@nXPR;aXBR U5U-S-- n/nM/UR U5 MB XBR U5- nUR [S5nUR5$)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserQ ) _decode_use_source_tokenizerr2r3all_special_tokens decode_piecesr]replaceSPIECE_UNDERLINEstrip)r9rtsp_modelcurrent_sub_tokens out_stringrUs r>convert_tokens_to_string(MarianTokenizer.convert_tokens_to_strings&*&G&G4??T__ E///445GH5PSVVV %'""))%0  ,,-?@@ ''(8#> !!rGcJUcXR/-$X-UR/-$)z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r9 token_ids_0 token_ids_1s r> build_inputs_with_special_tokens0MarianTokenizer.build_inputs_with_special_tokens s1  "3"3!44 4(D,=,=+>>>rGcHURUlURUlgrC)r2r4r'r5r9s r>_switch_to_input_mode%MarianTokenizer._switch_to_input_modes??#||rGclURUlUR(aURUlggrC)r3r4r#r*r5rs r>_switch_to_target_mode&MarianTokenizer._switch_to_target_modes*??   #'#6#6D  rGc,[UR5$rC)lenr'rs r> vocab_sizeMarianTokenizer.vocab_sizes4<<  rGsave_directoryfilename_prefixc[RRU5(d[R SUS35 g/nUR (a[RR UU(aUS-OS[S-5n[RR UU(aUS-OS[S-5n[URU5 [URU5 URU5 URU5 O\[RR X(aUS-OS[S-5n[URU5 URU5 [[S[S/URURUR/5GH$upxn [RR X(aUS-OSU-5n [RR!U5[RR!U 5:waB[RR#U5(a[%X5 URU 5 M[RR#U5(aM['U S 5n U R)5n U R+U 5 SSS5 URU 5 GM' [-U5$!,(df  N/=f) NzVocabulary path (z) should be a directory-rQrrr r wb)ospathisdirloggererrorr#joinVOCAB_FILES_NAMES save_jsonr'r*r]zipr0r2r3abspathisfileropenserialized_model_protowritetuple) r9rr saved_filesout_src_vocab_fileout_tgt_vocab_fileout_vocab_filespm_save_filename spm_orig_path spm_model spm_save_pathficontent_spiece_models r>save_vocabularyMarianTokenizer.save_vocabularysEww}}^,, LL,^,<   1 2   1 2WW\\/3!6rUfgnUo oN dllN 3   ~ .;> | ,.? .M N NN __doo .< 7 i GGLL/3!6rUf fMww}-1OOTVT[T[TbTbcpTqTq6""=1WW^^M22-."+4+K+K+M(HH12/""=1< "[!! /.s "K K( c"UR5$rC) get_src_vocabrs r> get_vocabMarianTokenizer.get_vocabLs!!##rGcB[UR40URD6$rC)dictr'added_tokens_encoderrs r>rMarianTokenizer.get_src_vocabOsDLL>D$=$=>>rGcB[UR40URD6$rC)rr*added_tokens_decoderrs r> get_tgt_vocabMarianTokenizer.get_tgt_vocabRsD''E4+D+DEErGcURR5nUR[R /SQ55 U$)N)r2r3r4rJr)__dict__copyupdaterfromkeys)r9states r> __getstate__MarianTokenizer.__getstate__Us4 ""$ MMm n  rGdc^UTl[TS5(d0TlU4SjTR5uTlTlTRTlTR5 g)Nrc3N># UHn[UTR5v M g7frC)r1r).0fr9s r> /MarianTokenizer.__setstate__..cs$+fWeRSHQ8L8L,M,MWes"%)rhasattrrr0r2r3r4r6)r9rs` r> __setstate__MarianTokenizer.__setstate__\sT t.//#%D +fW[WeWe+f(??  rGcg)zJust EOSr$)r9argsr:s r>num_special_tokens_to_add)MarianTokenizer.num_special_tokens_to_addgsrGc[UR5nURUR5 UVs/sH o3U;aSOSPM sn$s snf)Nrr)setall_special_idsremove unk_token_id)r9seqrrDs r>_special_token_mask#MarianTokenizer._special_token_maskksHd223t001:=>#Q/)q0#>>>sA rralready_has_special_tokenscU(aURU5$UcURU5S/-$URX-5S/-$)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.r)r)r9rrrs r>get_special_tokens_mask'MarianTokenizer.get_special_tokens_maskpsQ &++K8 8  ++K8A3> >++K,EF!L LrG)rr5r4r,r'rJr#rrr0r2r3r-r*r) NNNzzziNFrC)NF)*__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesrrr(rr8r6rIrVr`listrfintrjrmrqrrrrpropertyrrrrrrrrrrboolr__static_attributes__ __classcell__)r=s@r>rr,s8 t*$&6748< "$sCx.1<  < < |/43434Ucd3i 7#7#79.34"tCy"S" ?QUVYQZ?,7 !C!!+"c+"HSM+"]bcf]g+"Z$4$?Fd !d !t !? in M M.6tn Mae M c M MrGrrrrcT[R"S0UD6nURU5 U$)Nr$)rSentencePieceProcessorLoad)rrspms r>r1r1|s%  . . A ACHHTN JrGcz[US5n[R"XSS9 SSS5 g!,(df  g=f)Nwr[)indent)rjsondump)datarrs r>rrs% dCA $!$ s, :c|[US5n[R"U5sSSS5 $!,(df  g=f)Nr)rrload)rrs r>r&r&s" dCAyy| s- ;) rrrMpathlibrshutilrtypingrrrrtokenization_utilsr utilsr utils.import_utilsr get_loggerrrrr{rr(rrr1rrr&__all__r$rGr>rs ''5*   H % ,4   %&LM)LM'LM^ 3c3hM<`<` %#%$% CE$*-  rG