tjJ/dZddlZddlZddlmZddlmZejeZ dddZ d Z Gd d eZ d gZ dS) z Tokenization classes for BioGPT.N)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filec~t}|d}|ddD]}|||f|}|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairsrsP EEEQIQRR 9d#$$$ Lc BeZdZdZeZddgZ dfd Zed Z d Z d Z d Z d Z d dZdZdZdZ d!deedeedzdeefdZ d"deedeedzdedeeffd Zd!dededzdeefdZdZdZxZS)#BioGptTokenizera: Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc  ddl} n#t$rtdwxYwd|_| |_i|_i|_ t |d5} tj| |_ dddn #1swxYwYd|j D|_ t |d5} | ddd} dddn #1swxYwYd | D} tt| t!t#| |_i|_t)jd |||||d |dS) NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encodingci|]\}}|| Sr!).0kvs r z,BioGptTokenizer.__init__..vs>>>A1>>>r c`g|]+}t|dd,S)N)tuplesplit)r"merges r z,BioGptTokenizer.__init__..ys1???u% bqb)**???r) bos_token eos_token sep_token unk_token pad_tokenr!) sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadr+dictziprangelen bpe_rankscachesuper__init__)selfrrr1r.r/r0r2kwargsr3 vocab_handle merges_handlemerges __class__s rrGzBioGptTokenizer.__init__Zs         M    %'"')$ *w / / / 3<9\22DL 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3>>););)=)=>>> + 0 0 0 ;M"''))//55crc:F ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;?????c&%F *<*<==>>           s,"A99A=A==0C99C=C=c*t|jS)zReturns vocab size)rCr<rHs r vocab_sizezBioGptTokenizer.vocab_sizes4<   rc0t|jfi|jSN)r@r<added_tokens_encoderrOs r get_vocabzBioGptTokenizer.get_vocabsDL>>D$=>>>rc||jvr%|j|}||j|<|j||dddS)Nr5TF)aggressive_dash_splits return_strescape)r7r6MosesTokenizertokenize)rHtextr5moses_tokenizers rmoses_tokenizezBioGptTokenizer.moses_tokenizesc t1 1 1"g44$4??O/>D &t ,)$/88 %9   rc||jvr%|j|}||j|<|j||S)NrV)r8r6MosesDetokenizer detokenize)rHtokensr5moses_detokenizers rmoses_detokenizez BioGptTokenizer.moses_detokenizesR t3 3 3 $ 8 8d 8 C C 1BD ( .+D1<TcTj|tdS)Ninf)rDgetfloat)pairrHs rz%BioGptTokenizer.bpe..s 1C1CD%PU,,1W1Wrkeyrr r) z z ) r*rErminrDrCindexextend ValueErrorappendjoin) rHtokenr r bigramfirstsecondnew_wordijs ` rbpezBioGptTokenizer.bpesU3B3Z  E"I$6#88 DJ  :e$ $$ "6> ! ($W$W$W$WXXXFT^++"ME6HAc$ii-- 5!,,A OOD1I...AA "OODH---E7e##CIIM(9(9d1q5kV>S>SOOEFN333FAAOODG,,,FAc$ii-- XHD4yyA~~!$9 (:xx~~ :  D  5 sC'C/.C/Fc |r|}n|||j}g}|D]L}|rH|t ||dM|S)zReturns a tokenized string.ro)r+r^r5rrlistr})rHr\bypass_tokenizer split_tokensrvs r _tokenizezBioGptTokenizer._tokenizes  8::<)>s)C)C$D$DEEErcr|j||j|jS)z0Converts a token (str) in an id using the vocab.)r<rir1)rHrvs r_convert_token_to_idz$BioGptTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHrcB|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r>rir1)rHrqs r_convert_id_to_tokenz$BioGptTokenizer._convert_id_to_tokens|t~666rcd|D}d|}|||j}|S)z:Converts a sequence of tokens (string) in a single string.cbg|],}|dddd-S)rorf)replace)r"ts rr-z.s6JJJa!))C$$,,VS99JJJrr)rur+rdr5)rHrbr\s rconvert_tokens_to_stringz(BioGptTokenizer.convert_tokens_to_stringsOKJ6JJJ&&(($$VTY77 rN token_ids_0 token_ids_1returncB| |jg|zS|jg}||z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_id)rHrrseps r build_inputs_with_special_tokensz0BioGptTokenizer.build_inputs_with_special_tokenss;&  %&4 4 ![ 3&44ralready_has_special_tokensc|r$t||dS|/dgdgt|zzdgzdgt|zzSdgdgt|zzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNr r)rFget_special_tokens_maskrC)rHrrrrMs rrz'BioGptTokenizer.get_special_tokens_masks$ & 7722'[]a3   "31#K 0 001QC7A3[AQAQ;QR RsqcC ,,,--rsave_directoryfilename_prefixc ztj|s td|ddStj||r|dzndt dz}tj||r|dzndt dz}t|dd 5}|tj |j d d d dzdddn #1swxYwYd}t|dd 5}t|j dD][\}} || kr td|d| }|d|dz|dz }\ dddn #1swxYwY||fS)NzVocabulary path (z) should be a directory-rrrwrrr)TF)indent sort_keys ensure_asciir&rc|dS)Nr r!)kvs rrlz1BioGptTokenizer.save_vocabulary..*s Y[\]Y^rrmzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!ror )ospathisdirloggererrorruVOCAB_FILES_NAMESr9writer:dumpsr<sortedrDr=warning) rHrrr merge_filefrqwriter bpe_tokens token_indexs rsave_vocabularyzBioGptTokenizer.save_vocabularysqw}}^,,  LLT^TTT U U U FW\\ oM_s222QbcoQpp  W\\ oM_s222QbcpQqq  *cG 4 4 4 c GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c *cG 4 4 4 +1$.2F2F2H2HN^N^+_+_+_  ' KK''NNM MMM(E SXXj11D8999                 :%%s%<4C<<DDBF..F25F2cB|j}d|d<|S)Nr6)__dict__copy)rHstates r __getstate__zBioGptTokenizer.__getstate__6s# ""$$d  rch||_ ddl}n#t$rtdwxYw||_dS)NrznYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.)rr3r4r6)rHdr3s r __setstate__zBioGptTokenizer.__setstate__;s]          M   s()rrrrr)FrR)NF)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrGpropertyrPrTr^rdr}rrrrrintrboolrstrr*rrr __classcell__)rMs@rrr,s((T*$&67 * * * * * * X!!X!???   EEE ***X    III777GK5595379t3C5 c55552pu..9.379t3C.hl. c......6&&c&C$J&Z_`cZd&&&&8       rr)rr:rtokenization_pythonrutilsr get_loggerrrrrr__all__r!rrrs'& 666666  H % %   ZZZZZ)ZZZz  r