"`): The token used for padding, for example when batching sequences of different lengths. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequences when encoding. Sequences longer than this will be truncated. offset (`int`, *optional*, defaults to 0): The offset of the tokenizer. input_idsattention_mask pad_token unk_token max_lengthoffsetc >[U[5(a [U5OUn[U[5(a [U5OUnSUlU[S5[S5S.UlX@l[ TU]"SUUUS.UD6 g)Nz[S1]z[S2])r)rr r) isinstancestrr_utf_vocab_size_added_tokens_decoderrsuper__init__)selfr rrrkwargs __class__s `/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/dia/tokenization_dia.pyrDiaTokenizer.__init__/s.8 3-G-GJy)Y -7 3-G-GJy)Y #)2z&7IjY_N`%a"   !   cUR$N)r)rs r vocab_sizeDiaTokenizer.vocab_sizeEs###r!c[URUR-5Vs0sHoRU5U_M nnUR UR 5 U$s snfr#)ranger$rconvert_ids_to_tokensupdateadded_tokens_encoder)rivocabs r get_vocabDiaTokenizer.get_vocabIsX;@SWS^S^A^;_`;_a++A.1;_` T../ asAtextreturncbURS5Vs/sHn[U5PM nnU$s snf)zPTake as input a string and return a list of strings (tokens) for words/sub-wordsutf-8)encodechr)rr/r+tokenss r _tokenizeDiaTokenizer._tokenizeNs/"&++g"67"6Q#a&"67 8s,c\[U5S:waSnU$[U5UR-nU$)z0Converts a token (str) in an id using the vocab.rN)lenordr)rtokentoken_ids r_convert_token_to_id!DiaTokenizer._convert_token_to_idSs4 u:?H5zDKK/Hr!c4[XR- 5nU$)z=Converts an index (integer) in a token (str) using the vocab.)r4r)rindexr;s r_convert_id_to_token!DiaTokenizer._convert_id_to_token]sEKK'( r!r5cSnUHrnX0R;a*URUn[U5RS5nO2X0R;aURS5nOURS5nX%- nMt UR SSS9nU$)z:Converts a sequence of tokens (string) in a single string.r!r2ignore)errors)added_tokens_decoderrr3r*decode)rr5bstringr;added_token_obj tok_stringstrings rconvert_tokens_to_string%DiaTokenizer.convert_tokens_to_stringbsE111"&";";E"B 188A 333"\\'2 "\\'2  !G9 r!save_directoryfilename_prefixcg)Nrr)rrNrOs rsave_vocabularyDiaTokenizer.save_vocabularyrsr!)rrr)rSirr#)__name__ __module__ __qualname____firstlineno____doc__model_input_namesrrintrpropertyr$r-listr6r=rArLtuplerQ__static_attributes__ __classcell__)rs@rr r s$%&67$+#*$(  C= C= SM     ,$$ cd3i  tCy S  cHSM]bcf]gr!r N) rXtypingrtokenization_utilsrrutilsr get_loggerrTloggerr __all__rr!rrfs?"A   H %Y&Yx  r!