tjQ<ddlZddlmZGddeZdgZdS)N)TokenizersBackendc XeZdZdZ d deeezdededzded ef fd ZxZ S) ParakeetTokenizera  Inherits all methods from [`PreTrainedTokenizerFast`]. Users should refer to this superclass for more information regarding those methods, except for `_decode` which is overridden to adapt it to CTC decoding: 1. Group consecutive tokens 2. Filter out the blank token FNT token_idsskip_special_tokensclean_up_tokenization_spaces group_tokensreturnc t|tr|g}|rdtj|D}fd|D}t jd|||d|S)Ncg|] }|d S)r).0 token_groups p/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/parakeet/tokenization_parakeet.py z-ParakeetTokenizer._decode..'sXXXKQXXXc*g|]}|jk |Sr) pad_token_id)rtokenselfs rrz-ParakeetTokenizer._decode..*s&PPPuUd>O5O5OU5O5O5Or)rrr r) isinstanceint itertoolsgroupbysuper_decode)rrrr r kwargs __class__s` rrzParakeetTokenizer._decodes i % % $" I  YXX9;LY;W;WXXXIQPPP PPP uww  3)E      r)FNT) __name__ __module__ __qualname____doc__rlistboolstrr __classcell__)rs@rrrs%*48!   c? " '+Tk               rr)rtokenization_utils_tokenizersrr__all__rrrr*s_>>>>>>     )   @  r