j UddlmZddlZddlZddlZddlZddlmZmZm Z ddl Z ddl Z ddl m Z ejZiZded<daded<ej&d d Zdd Zdd Zdd Zy)) annotationsN)AnyCallableSequence)Encodingzdict[str, Encoding] ENCODINGSz.dict[str, Callable[[], dict[str, Any]]] | NoneENCODING_CONSTRUCTORScg}tjtjtjdz}|D]\}}}|j ||S)N.)pkgutil iter_modules tiktoken_ext__path____name__append)mods plugin_mods_mod_names [/var/www/html/Fitness-lenito-AI-main/venv/lib/python3.12/site-packages/tiktoken/registry.py_available_plugin_modulesrsN D&&|'<'S>SVY>YZK%8Q H& Kc t5t dddyia tD]^}tj|} |j}|jD]'\}}|tvrt d|d||t|<)` dddy#t $r}t d|d|d}~wwxYw#t$rdawxYw#1swYyxYw)Nztiktoken plugin z& does not define ENCODING_CONSTRUCTORSzDuplicate encoding name z in tiktoken plugin ) _lockr r importlib import_moduleAttributeError ValueErroritems Exception)rmod constructorseenc_name constructors r_find_constructorsr&!s  ,  !# 57--h7#&#<#>CC c t|tstdt||tvr t|St 5|tvrt|cdddSt tt J|t vr+td|dtdtjdt |}tdi|}|t|<|cdddS#1swYyxYw)Nz'Expected a string in get_encoding, got zUnknown encoding z. Plugins found: z tiktoken version: z (are you on latest?)) isinstancestrrtyperrr r&rtiktoken __version__r) encoding_namer%encs r get_encodingr0?s mS )B4 CVBWXYY !''  I %]+  ! (  (4 44  5 5#M?3"";"=!>?%%-%9%9$::OQ  ,M: ''#& - % sCA.CCct5tttJttcdddS#1swYyxYw)N)rr r&listr(rrlist_encoding_namesr3[s0  (  (4 44)* s '8A)returnz Sequence[str])r4None)r.r*r4r)r4z list[str]) __future__r functoolsrr threadingtypingrrrrr, tiktoken.corerRLockrr__annotations__r lru_cacherr&r0r3r(rrr>sq"**" !# #HLEL   <8+r