ó <±h[ãóJ•SSKJr SSKJr SSKJr SSKJrJr S\S\ 4Sjr g) é)ÚPath)ÚAny)ÚTikTokenConverter)ÚTIKTOKEN_VOCAB_FILEÚTOKENIZER_FILEÚencodingÚ output_dircóì•[U5nURSS9 US- [- nU[- n[ UR 55n[ UR 55nSSKJn SSKJ n [U[5(aU"U5nU"URU5 [X@RUR S9R#5nUR%U5 g ![a [S5ef=f) a— Converts given `tiktoken` encoding to `PretrainedTokenizerFast` and saves the configuration of converted tokenizer on disk. Args: encoding (`str` or `tiktoken.Encoding`): Tokenizer from `tiktoken` library. If `encoding` is `str`, the tokenizer will be loaded with `tiktoken.get_encoding(encoding)`. output_dir (`str`): Save path for converted tokenizer configuration file. T)Úexist_okÚtiktokenr)Ú get_encoding)Údump_tiktoken_bpezY`tiktoken` is required to save a `tiktoken` file. Install it with `pip install tiktoken`.)Ú vocab_fileÚpatternÚadditional_special_tokensN)rÚmkdirrrÚstrÚabsoluter r Ú tiktoken.loadrÚ isinstanceÚ_mergeable_ranksÚ ImportErrorÚ ValueErrorrÚ_pat_strÚ_special_tokensÚ convertedÚsave) rr Ú save_fileÚtokenizer_fileÚsave_file_absoluteÚoutput_file_absoluter rÚ tokenizers ÚZ/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/integrations/tiktoken.pyÚconvert_tiktoken_to_fastr$së€ôjÓ!€JØ×јdÐÑ#à˜ZÑ'Ô*=Ñ=€Iؤ.Ñ0€Nä˜Y×/Ñ/Ó1Ó2Ðܘ~×6Ñ6Ó8Ó9Ðð vÝ)Ý3ä h¤× $Ñ $Ù# HÓ-ˆHá˜(×3Ñ3Ð5GÔHô"Ø%×/@Ñ/@Ð\d×\tÑ\tñçiƒkðð‡NNÐ'Õ(øô óvÜÐtÓuÐuðvús Á#r+s$ðÝÝåAßTð#) sð#)¸õ#)r*