tjC_ dZddlZddlZddlZddlZddlmZddlmZej e Z dddZ d Z Gd d eZ d Zd Zededdddddf ZejddezejejzejzZejdZejeejejzejzZejdZd%dZd&dZGdd Zd!Zd"Zd'd$Z d gZ!dS)(z!Tokenization classes for BERTweetN)PreTrainedTokenizer)logging vocab.txt bpe.codes) vocab_file merges_filect}|d}|ddD]}|||f|}t|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rN)setadd)wordpairs prev_charchars p/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairsr#s[ EEEQIQRR 9d#$$$ JJE Lc eZdZdZeZ dfd Zed Zd Z d Z d Z d Z dZ dZdZdZddededzdeedffdZdZxZS)BertweetTokenizera Constructs a BERTweet tokenizer, using Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. normalization (`bool`, *optional*, defaults to `False`): Whether or not to apply a normalization preprocess. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. Fc   ddlm} | |_n1#t$r$tdd|_YnwxYw||_||_i|_d|jt|<d|jt| <d|jt|<d|jt|<| |d|j D|_ t|d 5} | d dd }dddn #1swxYwYd |D}t!t#|t%t'||_i|_||_t/|_d dd|_t5jd||||||| | dddd | dS)Nr)demojizezsemoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0r rci|]\}}|| Sr ).0kvs r z.BertweetTokenizer.__init__..s>>>A1>>>rutf-8encoding c`g|]+}t|dd,S)Nr))tuplesplit)r!merges r z.BertweetTokenizer.__init__..s1@@@% crc*++@@@r'z...)u’u… all_zerosTcls_double_sep) normalization bos_token eos_token sep_token cls_token unk_token pad_token mask_tokentoken_type_ids_pattern%token_type_ids_include_special_tokensspecial_tokens_patternr )emojir demojizer ImportErrorloggerwarningrr encoderstr add_from_fileitemsdecoderopenreadr,dictziprangelen bpe_rankscacher2TweetTokenizertweetPreprocessorspecial_punctssuper__init__)selfrr r2r3r4r5r6r7r8r9kwargsr merges_handlemerges __class__s rrSzBertweetTokenizer.__init__hsK " & & & & & &%DNN " " " NN(   "DNNN  "%& '( S^^$'( S^^$'( S^^$'( S^^$ :&&&>>););)=)=>>> + 0 0 0 ;M"''))//55crc:F ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;@@@@@c&%F *<*<==>> *!/!1!1&)%88 '!#.26#3       s +??0D==EEc*t|jSN)rLrBrTs r vocab_sizezBertweetTokenizer.vocab_sizes4<   rc0t|jfi|jSrZ)rIrBadded_tokens_encoderr[s r get_vocabzBertweetTokenizer.get_vocabsDL>>D$=>>>rc|jvr j|St|}tt|dd|ddzgz}t|}|s|S t |fd}|jvrn8|\}}g}d}|t |kr |||} |||| | }n-#t$r |||dYnwxYw|||krC|t |dz kr-||dz|kr| ||z|dz }n | |||dz }|t |kt|}|}t |dkrnt|}Wd |}|dd }|j|<|S) Nr)zTcTj|tdS)Ninf)rMgetfloat)pairrTs rz'BertweetTokenizer.bpe..s 1C1CD%PU,,1W1Wrkeyrr r@@ ) rNr+listrminrMrLindexextend ValueErrorappendjoin) rTtokenrrbigramfirstsecondnew_wordijs ` rbpezBertweetTokenizer.bpes# DJ  :e$ $U||T$ss)__R6(9'::;;$ L ($W$W$W$WXXXFT^++"ME6HAc$ii-- 5!,,A OOD1I...AA "OODH---E7e##CIIM(9(9d1q5kV>S>SOOEFN333FAAOODG,,,FAc$ii-- XHD4yyA~~!$9 (:zz$CRCy  5 s(C'DDc|jr||}g}tjd|}|D]J}|t ||dK|S)zTokenize a string.z\S+\n? )r2normalizeTweetrefindallrnrkryr,)rTtext split_tokenswordsrrs r _tokenizezBertweetTokenizer._tokenizes   -&&t,,D  9d++ B BE   TXXe__%:%:3%?%? @ @ A A A ArcjD]#}||j|}$j|}dfd|D}|dddddddd d d }|d d dddddddddd}|dddddddd}d|S) z' Normalize a raw Tweet r{c:g|]}|Sr )normalizeToken)r!rrrTs rr.z4BertweetTokenizer.normalizeTweet..s'MMMUd11%88MMMrzcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rQreplacerPtokenizerqr,)rTtweetpuncttokens normTweets` rr|z BertweetTokenizer.normalizeTweets( E EEMM%))rTrrlowercased_tokens rrz BertweetTokenizer.normalizeTokens!;;==   C 7  ( ( 0 0 4D4O4OPU4V4V 9 ZZ1__+++*511~)~~e,,, Lrcr|j||j|jS)z0Converts a token (str) in an id using the vocab.)rBrcr7)rTrrs r_convert_token_to_idz&BertweetTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHrcB|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rFrcr7)rTrms r_convert_id_to_tokenz&BertweetTokenizer._convert_id_to_tokens|t~666rc|d|dd}|S)z:Converts a sequence of tokens (string) in a single string.r{ri)rqrstrip)rTr out_strings rconvert_tokens_to_stringz*BertweetTokenizer.convert_tokens_to_string#s5XXf%%--eR88>>@@ rNsave_directoryfilename_prefixreturn.cntj|s td|ddSt |di}|r|dnd}tj|||ddz}t|d d 5}t|j d D]&\}}|dkr| |d|d' dddn #1swxYwYtj|||ddz} t| d d 5} | dt|j d Ddddn #1swxYwY|| fS)zF Save the vocabulary and merges files to a directory. zVocabulary path (z) should be a directoryr vocab_files_names-rrrwr%r&c|dSNr r kvs rrfz3BertweetTokenizer.save_vocabulary..=s rRSurrgr{r(Nr rc3LK|]\}}d|dzV dS)r{r(N)rq)r! bpe_tokens token_indexs r z4BertweetTokenizer.save_vocabulary..EsK+J $$t+rc|dSrr rs rrfz3BertweetTokenizer.save_vocabulary..Gs ]_`a]br)ospathisdirr@errorgetattrrqrcrGsortedrBrEwrite writelinesrM) rTrrrprefixrfrrtoken_id merge_filewriters rsave_vocabularyz!BertweetTokenizer.save_vocabulary.sXw}}^,,  LLT^TTT U U U2#D*=rBB*9AO&&&&rW\\.&;L;P;PQ]_j;k;k2kll *cG 4 4 4 5#)$,*<*<*>*>DTDT#U#U#U 5 5xq==GGu33x333444 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5W\\.&;L;P;PQ^`k;l;l2lmm *cG 4 4 4    /5dn6J6J6L6LRbRb/c/c/c                   J''s&#ADDDAF((F,/F,ct|trs t|dd5}||dddn #1swxYwYn0#t$r}|d}~wt $rt d|dwxYwdS|}|D]f}|}| d}|dkrtd |d|}t|j |j |<gdS) zi Loads a pre-existing dictionary from a text file and adds its symbols to this instance. rr%r&NzIncorrect encoding detected in z, please rebuild the datasetr{r)z5Incorrect dictionary format, expected ' ') isinstancerCrGrDFileNotFoundError UnicodeError Exception readlinesrrfindrorLrB) rTrfdfnfelineslineTmplineidxrs rrDzBertweetTokenizer.add_from_fileLsq a    c!S7333+r&&r***+++++++++++++++$     c c c a! a a abbb c F  3 3G==??D**S//Cbyy !XYYY:D!$T\!2!2DL    3 3s9AA  A AAAA B!A##!B)FrrrrrrrrZ)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESrrSpropertyr\r_ryrr|rrrrrCr+rrD __classcell__)rXs@rrr3sQ00d* > > > > > > @!!X!???***X    + + +D&III777((c(C$J(Z_`ceh`hZi((((<3333333rrac (?: [<>]? [:;=8] # eyes [\-o\*\']? # optional nose [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth | [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth [\-o\*\']? # optional nose [:;=8] # eyes [<>]? | <3 # heart )u # Capture 1: entire matched URL (?: https?: # URL protocol and colon (?: /{1,3} # 1-3 slashes | # or [a-z0-9%] # Single letter or digit or '%' # (Trying not to match e.g. "URI::Escape") ) | # or # looks like domain name followed by a slash: [a-z0-9.\-]+[.] (?:[a-z]{2,13}) / ) (?: # One or more: [^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?\s]+>z [\-]+>|<[\-]+z (?:@[\w_]+)z(?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]a (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes. | (?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals. | (?:[\w_]+) # Words without apostrophes or dashes. | (?:\.(?:\s*\.){1,}) # Ellipsis dots. | (?:\S) # Everything else that isn't whitespace. z(%s)|z([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);strictcd|d}t|tr|||S|S)Nr%)rbytesdecode)rr'errorss r_str_to_unicoders8$-{{8V,,, Krr Tr%cdfd}t|t||S)u Remove entities from text by converting them to their corresponding unicode character. Args: text: A unicode string or a byte string encoded in the given *encoding* (which defaults to 'utf-8'). keep (list): List of entity names which should not be replaced. This supports both numeric entities (`&#nnnn;` and `&#hhhh;`) and named entities (such as ` ` or `>`). remove_illegal (bool): If `True`, entities that can't be converted are removed. Otherwise, entities that can't be converted are kept "as is". Returns: A unicode string with the entities removed. See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py Examples: ```python >>> from nltk.tokenize.casual import _replace_html_entities >>> _replace_html_entities(b"Price: £100") 'Price: \xa3100' >>> print(_replace_html_entities(b"Price: £100")) Price: £100 ```cP|d}|dr} |drt|d}nt|d}d|cxkrdkr&nn#t|fdSnO#t$rd}YnAwxYw|vr|d St jj|}|' t|S#ttf$rYnwxYwrd n|d S) Nrr r cp1252rr) groupintrrrohtmlentitiesname2codepointrcchr OverflowError)match entity_bodynumberkeepremove_illegals r_convert_entityz/_replace_html_entities.._convert_entity;sOkk!nn ;;q>> G ;;q>>2 b11FF b11F 6))))T))))) &++228<<<    d""{{1~~%599+FF   6{{" .    $7rrQ7s$A(B B&%B&)C88D  D )ENT_REsubr)rrrr'rs `` r_replace_html_entitiesrsB:888888: ::otX'F'F G GGrc eZdZdZddZdZdS)rOa Examples: ```python >>> # Tokenizer for tweets. >>> from nltk.tokenize import TweetTokenizer >>> tknzr = TweetTokenizer() >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--" >>> tknzr.tokenize(s0) ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--'] >>> # Examples using *strip_handles* and *reduce_len parameters*: >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True) >>> s1 = "@remy: This is waaaaayyyy too much for you!!!!!!" >>> tknzr.tokenize(s1) [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!'] ```TFc0||_||_||_dSrZ preserve_case reduce_len strip_handles)rTrrrs rrSzTweetTokenizer.__init__rs*$*rc t|}|jrt|}|jrt |}t d|}t|}|j s d|D}|S)z Args: text: str Returns: list(str) A tokenized list of strings; concatenating this list returns the original string if `preserve_case=False` \1\1\1cng|]2}t|r|n|3Sr ) EMOTICON_REsearchr)r!xs rr.z+TweetTokenizer.tokenize..s7NNN1+,,Q//>QQQWWYYNNNr) rrremove_handlesrreduce_lengtheningHANG_RErWORD_REr~r)rTr safe_textrs rrzTweetTokenizer.tokenizews&d++   (!$''D ? ,%d++DKK 400  **! ONNNNNE rNTFF)rrrrrSrr rrrOrO^sA  &++++ rrOcVtjd}|d|S)za Replace repeated character sequences of length 3 or greater with sequences of length 3. z (.)\1{2,}rregexcompilerrpatterns rrrs'mL))G ;;y$ ' ''rcVtjd}|d|S)z4 Remove Twitter username handles from text. zv(?rs(' 666666  H % %    n3n3n3n3n3+n3n3n3x L $)\  (. A+ ` %- chhw&7&779PSXS`9` a a %-/ 0 0emIu}uw'>'NOO  . / /:H:H:H:H@00000000p((("""   r