tj d Z ddlZddlmZ ddlmZmZmZ ddlm Z m
Z
ddl m Z m
Z
G d d ed
Z e
j e Ze G d d
e Zd
gZdS )z
Processor class for Donut.
N )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringloggingc e Zd Zi ZdS )DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/donut/processing_donut.pyr
r
s IIIr r
F)totalc e Zd Zd
fd Ze d
dedz deee z ez e z dz de
e fd Z e
d Zd d Z xZS ) DonutProcessorNc L t || d S )N)super__init__)selfimage_processor tokenizerkwargs __class__s r r zDonutProcessor.__init__# s#
)44444r imagestextr c : ||t d | j t fd| j j i|}| | j |fi |d }|2| |d dd | j |fi |d }||S ||S |d |d< |d |d< |S ) NzBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs
images_kwargs text_kwargsadd_special_tokensF input_idslabels)
ValueError
_merge_kwargsr
r init_kwargsr
setdefault)r r r! r
output_kwargsinputs encodingss r __call__zDonutProcessor.__call__& s >dl abb b**
"&."<
)T)&SSM/4RSSF ! m,778LeTTT&tLL}]/KLLI <M
^ ( 5F8 "+K"8F; Mr c B | j j }t |ddgz S )Nr' r( )r model_input_nameslist)r image_processor_input_namess r r2 z DonutProcessor.model_input_namesF s' &*&:&L#/;2IIJJJr Fc || j }i }|rt j d|t j }|n|| d }d|vrn|d| d dz }|t d t d }t j | }t j d| d|t j } | | |d }n|
} t j | }
t j | } t j |
d| |t j t j z } | |
d } d| v r?d| v r;|
| d| }
|
r t |
dk r|
d
}
|
||< ng ||< | d D ]Y}| }||v r$|d
d k r|d
d dk r
|dd
}|| | Zt || dk r|| d
||< || | t | z d }|dd d k r$|g|
|dd d| z S ||r|r|gn|S |rg nd|iS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z
text_sequence)r get_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r tokensr9 r: outputpotential_start start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs r rK zDonutProcessor.token2jsonL s0 .88::K * p i
F FO & !6!6!8!8!:!:;K+%%%&B (9(9#(>(>(B&BCK c%jjCHH9 45C )C..K "8+"8"8"8&"-PPI R88 %OO - - &(i &<&<#$&Ii$8$8! )*DD1BDDfbm^`^gNg & %mmA . . 4 4 6 6G ((W-?-? $Ze f f 0 "5zzQ(-a*/F3K&(s $+MM)$<$< 5 5D#'::< K K XK
8G 8G 8G 8G 8G 8G 8G 8Gr r )__doc__r@ image_utilsr processing_utilsr r r tokenization_utils_baser r utilsr
r r
get_loggerr loggerr __all__r r r rf s
% % % % % % H H H H H H H H H H C C C C C C C C , , , , , , , , +5
H % % bG bG bG bG bG^ bG bG bGJ
r