tjHdZddlmZdddZGddeZeZddgZd S) z$Tokenization classes for DistilBERT.) BertTokenizerz vocab.txtztokenizer.json) vocab_filetokenizer_filec2eZdZddgZdddeffdZxZS)DistilBertTokenizer input_idsattention_maskT) do_lower_caser c>tj|d|i|dS)a Construct a DistilBERT tokenizer (backed by HuggingFace's tokenizers library). Based on WordPiece. This tokenizer inherits from [`BertTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. r N)super__init__)selfr argskwargs __class__s t/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/distilbert/tokenization_distilbert.pyr zDistilBertTokenizer.__init__s, $FmFvFFFFF)__name__ __module__ __qualname__model_input_namesboolr __classcell__)rs@rrrsj$&6748 G G GT G G G G G G G G G GrrDistilBertTokenizerFastN)__doc__models.bert.tokenization_bertrVOCAB_FILES_NAMESrr__all__rrr sy+*::::::$/BRSSGGGGG-GGG$. "; <r