tj]<dZddlZddlZddlZddlmZddlmZmZddl m Z e j e Z ddiZGd d ZGd d eZd gZdS) z"Tokenization class for model MyT5.N) defaultdict) AddedTokenPreTrainedTokenizer)logging vocab_filezbyte_maps.jsonceZdZdZdZdeeeefzfdZdeeeeezfdedefdZ deeefd eeeeezffd Z d eed d eezfd Z ddeed eefdZ d S) ByteRewriteraZ Byte rewriter class for MyT5 tokenizer. This class is used to rewrite bytes using a hash tree. The hash tree is constructed from a set of rewriting rules. Args: rewriting_rules (`str` or `dict[str, str]`): A path to a json file containing the rewriting rules or a dictionary containing the rewriting rules. z[LEAF]rewriting_rulesct|tr=t|d5}tj|}dddn #1swxYwYn4t|t st dt||||_ d| D}|||_ dS)NrzDrewriting_rules should be either a path to json file or a dict, got ci|]\}}|| Sr).0kvs h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/myt5/tokenization_myt5.py z)ByteRewriter.__init__..6s"L"L"LDAq1a"L"L"L) isinstancestropenjsonloaddict TypeErrortypeconstruct_hash_tree hash_treeitemsreverse_hash_tree)selfr freverse_rewriting_ruless r__init__zByteRewriter.__init__,s os + + os++ /q"&)A,, / / / / / / / / / / / / / / /OT22 nW[\kWlWlnn 11/BB"L"LO4I4I4K4K"L"L"L!%!9!9:Q!R!RsAA A rbyte_in_sequencebyte_out_sequencec|d}|d}|}|D]}||vri||<||}|||j<dS)zL Add a leaf with the output byte sequence to the hash tree.  N)splitLEAF)r"rr&r' byte_in_list byte_out_list tree_pointerbs radd_leafzByteRewriter.add_leaf9so(--c22 )//44   + +A $$"$ Q'?LL"/ TYrreturnctt}dtdDD]}|g|||j<|D]\}}|||||S)zE Construct a hash tree for rewritten byte sequences. c3K|]}|dV dS)02xNr)rxs r z3ByteRewriter.construct_hash_tree..Ms&11Q**111111r)rrranger+r r0)r"r rr/ in_sequence out_sequences rrz ByteRewriter.construct_hash_treeHs %% 11eCjj111 * *A'(cIaL # #)8)>)>)@)@ @ @ %K MM)[, ? ? ? ?r byte_sequenceNcR|j}|D]}||vr ||}dS||jS)zW Search the hash tree and return the rewritten byte sequence if found. N)rr+)r"r;r.r/s rsearch_hash_treezByteRewriter.search_hash_treeUsD~   AL  +A ttDI&&rFin_bytescjg}d}d}|t|kr|s|jn|j}t|t|D]>}||}||vr ||}n||kr|g} |}nn|j|vr||j} |}?|| |dz}|t|k|S)a6 Rewrite a sequence of bytes using the hash tree. Args: in_bytes (`list[str]`): A list of bytes to be rewritten. reverse (`bool`): If True, decoding is performed with the reverse hash tree. Returns: `list[str]`: The rewritten byte sequence. r)lenrr!r8r+extend) r"r>reverse out_bytesb_startb_endr.jr/cur_leafs r rewrite_byteszByteRewriter.rewrite_bytesbs H %%18T4>>d>TL7CMM22  QK $$#/?LL'\\ !sHEEE9 ,,+DI6HE   X & & &aiG!H %%$r)F) __name__ __module__ __qualname____doc__r+rrr%listr0rr=rIrrrr r s/ D Sd38n(< S S S S 0$sD49,<'<"= 0QT 0il 0 0 0 0 4S> d3PTW[\_W`P`K`Fa     'd3i 'D49"`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 125): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary like in ByT5 preprocessing see [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)). additional_special_tokens (`list[str]`, *optional*): Additional special tokens used by the tokenizer. input_idsattention_mask}Nr1c d|dkr|dt|D}nb|dkr\|Zt|dkrGtttd|}||krt d|d|dt |t rt|ddn|}t |t rt|ddn|}t |t rt|ddn|}|||d |_t|j|_ d |_ tj t|d |_t|jd |_t|jd |_t%jd|||d|d|dS)Nrcg|]}d|d S)z rris r z*MyT5Tokenizer.__init__..s$(U(U(Uq):a):):):(U(U(Urc>tdt|vS)Nextra_id)boolr)r5s rz(MyT5Tokenizer.__init__..sDs1vv9M4N4NrzBoth extra_ids (z!) and additional_special_tokens (zm) are provided to MyT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)rr@r7r decompose_map merge_map) eos_token unk_token pad_token extra_idsadditional_special_tokensr)r8rAsetfilter ValueErrorrrr_added_tokens_decoderoffset_utf_vocab_sizerrr byte_mapsr decompose_rewritermerge_rewritersuperr%) r"rrfrgrhrirjkwargs extra_tokens __class__s rr%zMyT5Tokenizer.__init__s q==6>(U(UE)DTDT(U(U(U % % ]]8DMfIgIgjkIkIks6*N*NPi#j#jkkllLy(( (y((Sl((( HRR[]`GaGapJydCCCCgp GQR[]`GaGapJydCCCCgp GQR[]`GaGapJydCCCCgp )2yY%O%O"$455 #4 C#8#899".t~o/N"O"O*4>++FGG &?         rc|jSN)rp)r"s r vocab_sizezMyT5Tokenizer.vocab_sizes ##rcfdtjjzD}|j|S)Nc<i|]}||Sr)convert_ids_to_tokens)rr[r"s rrz+MyT5Tokenizer.get_vocab..s)```a++A..```r)r8rzroupdateadded_tokens_encoder)r"vocabs` r get_vocabzMyT5Tokenizer.get_vocabsI````5SWS^A^;_;_``` T./// rF token_ids_0 token_ids_1already_has_special_tokensc|r$t||dS|dgt|zdgzSdgt|zdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr@)rtget_special_tokens_maskrA)r"rrrrws rrz%MyT5Tokenizer.get_special_tokens_masks$ & 7722'[]a3   C#k***qc1 1c+&&&1#-!s;7G7G1GHA3NNr token_idsct|dkr0|d|jkrtjd|jd|S||jgzS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)rA eos_token_idwarningswarnrf)r"rs r_add_eos_if_not_presentz%MyT5Tokenizer._add_eos_if_not_presentsi y>>A  )B-43D"D"D M+T^+++     122 2rcz|jg}|t||zdgzSt||z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. MyT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. Nr)rrA)r"rreoss r$create_token_type_ids_from_sequencesz2MyT5Tokenizer.create_token_type_ids_from_sequencessS  !  {S())QC/ /;${2S899QC??rch||}||S||}||zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r)r"rrs r build_inputs_with_special_tokensz.MyT5Tokenizer.build_inputs_with_special_tokenssA&22;??   66{CCK, ,rtextc nd|dD}||}|S)zTake as input a string and return a list of strings (tokens) for words/sub-words. Represents tokens in two character hex formatcg|]}|dS)r4rrZs rr\z+MyT5Tokenizer._tokenize..8s;;;Q**;;;rutf-8)encodemorphological_encode)r"rrutokenss r _tokenizezMyT5Tokenizer._tokenize4s;<;dkk'&:&:;;;**622 rcbt|dkrd}nt|d|jz}|S)z0Converts a token (str) in an id using the vocab.rcN)rAintro)r"tokentoken_ids r_convert_token_to_idz"MyT5Tokenizer._convert_token_to_id<s3 u::??HH5"~~ 3Hrc||jz d}|S)z=Converts an index (integer) in a token (str) using the vocab.r4)ro)r"indexrs r_convert_id_to_tokenz"MyT5Tokenizer._convert_id_to_tokenFs4;&,, rindicescv|j|d}|j|d}|S)NFrC)rrrIrsr"rs rrz"MyT5Tokenizer.morphological_encodeKs=)777OO%33GU3KKrcv|j|d}|j|d}|S)NTr)rsrIrrrs rmorphological_decodez"MyT5Tokenizer.morphological_decodeQs=%33GT3JJ)777NNrcd}g}|D]`}||jvr!||j|,||jvr||K||a||}t |jt |jz}|D]7}||vr|t|dz }|t |z }8| dd}|S)z:Converts a sequence of tokens (string) in a single string.rrignore)errors) rnappend_added_tokens_encoderrrkadded_tokens_decodervaluesrbytesfromhexdecode)r"rbstring out_tokensr _added_tokensstrings rconvert_tokens_to_stringz&MyT5Tokenizer.convert_tokens_to_stringWs%  ) )E222!!$">??#dF_B`B``  0 0E %%50005==///99 rsave_directoryfilename_prefixc|tj|r6tj||r|dzndtdz}n |r|dznd|z}t |dd5}|tj|j dd dddn #1swxYwY|fS) N-rwr)encodingrcF)indent ensure_ascii) ospathisdirjoinVOCAB_FILES_NAMESrwriterdumpsrq)r"rrrwriters rsave_vocabularyzMyT5Tokenizer.save_vocabularyns 7== ( ( ]/!Q3!6!6rUfgsUt tJJ4CJ/C//n\J *cG 4 4 4 S LLDN15QQQ R R R S S S S S S S S S S S S S S S}s40B00B47B4)rSrTrUrVN)r1N)NFry)rJrKrLrMmodel_input_namesrvocab_files_namesr%propertyrzrrNrr_rrrrrrrrrrrtupler __classcell__)rws@rrPrPs4%&67) "&,  , , , , , , \$$X$puOO9O379t3COhlO cOOOOOO8 3c 3tCy 3 3 3 3GK@@9@379t3C@ c@@@@0GK--9-379t3C- c----4cS  DI$s) DI$s) .  c C$J Z_`cZd        rrP)rMrrr collectionsrtokenization_pythonrrutilsr get_loggerrJloggerrr rP__all__rrrrs)( ######BBBBBBBB  H % %"#34ccccccccLrrrrr'rrrj  r