tjs x d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
m Z m Z m
Z
ddlmZ ddlmZmZ dd lmZ eeeef eeeeeef z eeeeef z eeeeeef z Zeedz eedz eedz eedz z z z Z G d
d e d
Z G d de d
Z G d de
d
Z e G d de Z deeeeef dedeeef fdZ dededefdZ d Z!d Z"d Z#d d Z$dgZ%dS )!z Processor class for KOSMOS-2. N ) BatchFeature)
ImageInput) ImagesKwargsProcessingKwargsProcessorMixin
TextKwargsUnpack)
AddedToken)
BatchEncoding TextInput)auto_docstringc > e Zd ZU dZedz ed< eed< edz ed< dS )Kosmos2ImagesKwargsa
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional* defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
Nbboxesnum_image_tokensfirst_image_token_id)__name__
__module__ __qualname____doc__
NestedList__annotations__int l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/kosmos2/processing_kosmos2.pyr r ' sK
*$$$$$r r F)totalc e Zd ZU dZeed< dS )Kosmos2TextKwargsz
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
add_eos_tokenN)r r r r boolr r r r r r 8 s*
r r c
F e Zd ZU eed< eed< dddddddddd ddid Zd
S ) Kosmos2ProcessorKwargs text_kwargs
images_kwargsTFr ) add_special_tokenspaddingstridereturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mappingreturn_token_type_idsverboser! r @ )r% r& N)r r r r r r _defaultsr r r r$ r$ A sg """"&&&& #' ).*/&+%* "
IIIr r$ c
e Zd Zd fd Ze ddedz deee z dee de
fd Z d Z d
Z
dd eee z dedz d
ededz deee z f
dZddZddZed Zded
eee eee z defdZdeeef eeeeef z deeef fdZ xZS )Kosmos2Processor c d d|_ d| _ d| _ d| _ d| _ d| _ d| _ d| _ d | _ d
| _ d | _
d | _ | j | j | j | j | j | j | j | j | j | j
| j g | _ || _
d
t | j
D }g }| j |z D ](}| t! |ddd )| | t% || dS )z
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
Fzzzzzzz z z c Z g | ](}d t | d d)S )
)strzfill.0xs r
z-Kosmos2Processor.__init__.. s4 m m mQ @c!ffll1oo @ @ @ m m mr T)lstriprstrip
normalizedN)r- eod_token boi_token eoi_token eoc_token eol_token bop_token eop_token boo_token eoo_token dom_token grd_token
tag_tokensnum_patch_index_tokensrangeappendr
add_tokenssuper__init__) selfimage_processor tokenizerrO kwargspatch_index_tokens
tokens_to_addtoken __class__s r rT zKosmos2Processor.__init__X sA
+0 '!"##"#$#$9&
N N N N N N N N N N N
'=# m m%PTPkJlJl m m m
_'99 a aE E$uY^!_!_!_ ` ` ` ` ]+++
)44444r NimagestextrX returnc ||t d j t fd j j i|}|d dd }|d dd }|d dd }|d d d
}|d d } |d d }
|d d
d } t } |) j |fi |d }
| |
|lj
|||| }| rK|sIt |t r j j
| }n#t |t r fd|D }|d d o||d d < ||
nd
|d d < || nd |d d
< j d#d|i|d }| | | |d d < |
|d d < | |d d
< ||| j j dz }| }t! | dz }t t# |||z }dgdg|z z dgz }g }g }| d }t |t r|g}| d g| d< |D ]}|d | |z |||z d z }| | t' j | }|rdg|z }|dgt) | t) | z
z z
}| | t |t r-t+ d t- |j D d }|d \ }}|d \ }}|d d o||d d < d |d d
< j d#d|| gi|d }t) |j d | k r j j dk r5 fd|D } fd|D } fd | d D | d< nD j j d k r4 fd |D } fd |D } fd | d D | d< t |t r#| !|d }| d d | d< |d }| t3 || d |d!| " | S )$Nz*You have to specify either images or text.tokenizer_init_kwargsr& r r r/ r r% r! Fr' r( return_tensors)r c 0 g | ]}j j | S r )rW bos_token)r= srU s r r? z-Kosmos2Processor.__call__.. s) KKKt~ 7 < < <KKKr r^ r input_idsattention_maskc 6 g | ]\ }}|t | fS r len)r= idxr> s r r? z-Kosmos2Processor.__call__.. s& TTTvsAc3q66]TTTr c | d S Nr )r> s r z+Kosmos2Processor.__call__.. s
defhdi r )keyro rightc V g | ]%}|j j gt | z
z z &S r rW pad_token_idrk r= r> max_len_paddedrU s r r? z-Kosmos2Processor.__call__.. s; $v$v$vghQ$.*E)F.[^_`[a[aJa)b%b$v$v$vr c B g | ]}|d gt | z
z z S r rj r= r> rw s r r? z-Kosmos2Processor.__call__.. = 6 6 6DEA~A'> ? ?6 6 6r c B g | ]}|d gt | z
z z S ry rj rz s r r? z-Kosmos2Processor.__call__.. r{ r leftc V g | ]%}j j gt | z
z |z &S r rt rv s r r? z-Kosmos2Processor.__call__.. s< $v$v$vghdn&A%BnWZ[\W]W]F]%^ab%b$v$v$vr c B g | ]}d gt | z
z |z S ry rj rz s r r? z-Kosmos2Processor.__call__.. = 6 6 6DEQC>CFF#: ;a ?6 6 6r c B g | ]}d gt | z
z |z S ry rj rz s r r? z-Kosmos2Processor.__call__.. r r )rg rh image_embeds_position_mask)data tensor_typer )
ValueError
_merge_kwargsr$ rW init_kwargspop
setdefaultr rV updatepreprocess_examples
isinstancer: rd list unk_token_idr rP rQ copyrk sorted enumeraterg padding_sider ) rU r] r^ rX
output_kwargsr r r r! r' r( rb encodingimage_encoding
text_encodingwith_bos start_indeximage_token_ids base_image_embeds_position_maskrg r
all_input_idstext_idsmask
sorted_length_min_len_not_paddedrl rw s ` @r __call__zKosmos2Processor.__call__ sx >dl IJJ J** "
"&."<
/33HdCC( 9==>PRTUU ,_ = A ABXZ^ _ _ %m488 %PP
*= 9:N O
.y9&}5@@AQSWXX >> 1T 1& [ [M/>>DD d++ LKKKKdKKKD m,-ABT}
- ()= > BH ggUZM- ( 3OU~^^cgM- ()9 : *DN U U U
m8T U UM OOM * * *=O
m$%9:29
m$Y/9G
m$%56 2#+'+~'BQ'F$ *H h--!+K #5)=?SVf?f#g#g h hO/0cQC:J4J.JaS.P + I)+ & $[ 1M$$$
J!.
.67G.H-I)* )
8
8#L[L1OCh{]mOmOoOoFpp ***y!@AA & 3:Ds8}}s4yy 899*11$7777$%%
&TT=;R1S1STTTZiZi! ! !
)6a(8%%&r*Q!-01EFX= m,-AB BF
m,-=> . ` `T#YK `=Q^C_ ` `
!$]%>$v$v$v$v$vlu$v$v$v 6 6 6 6Ic6 6 626 6 6 6IQRbIc6 6 6!12
$$$
K )? %aL -56F-G-J)*-G-J*
OO %.*23C*D6P
!/
r c |dS t |t st d |D ]}|t |t s|g}|D ]~}t |t rXt | dk rt d |D s;t | dk rt d |D st d dS )a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list. c 3 @ K | ]}t |t V d S N)r r r< s r zAKosmos2Processor._check_bboxes_for_single_text../ s, .S.Saz!S/A/A.S.S.S.S.S.Sr r8 c 3 @ K | ]}t |t V d S r )r floatr< s r r zAKosmos2Processor._check_bboxes_for_single_text..0 s, 1X1X1*Q2F2F1X1X1X1X1X1Xr a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)r r r tuplerk all)rU r bboxelements r _check_bboxes_for_single_textz.Kosmos2Processor._check_bboxes_for_single_text s > FFD)) a _`` ` D| d++
v
!'511 \\Q&&3.S.S7.S.S.S+S+S&G ))c1X1XPW1X1X1X.X.X)$@
r c l | }|| d| }| || }|S )N )strip_insert_patch_index_tokens)rU r^ imager img_info_tokenss r _preprocess_single_examplez+Kosmos2Processor._preprocess_single_example9 sD zz|| %....D ..tV<< r r/ textsr r c p j g|z }d j g|z j gz d}t |t rd}|g}|dgt | z }nt |t s|g}t | t | k r0t dt | dt | d |s | |g}nT|?t |t st d |D ]} | ndgt | z }t | t | k r0t d t | dt | d fd
t ||| D }|s|d }|S ) a- Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, list[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, list[TextInput]]`: The processed texts with image and patch index tokens.
r TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got c H g | ] \ }}} ||| S r )r )r= r^ r r r rU s r r? z8Kosmos2Processor.preprocess_examples..{ sC
!eT
+ +D% O O
r r )
rD joinrE r r: rk r r r zip)
rU r r] r r
img_tokensbatchedr> resultr s
` @r r z$Kosmos2Processor.preprocess_examplesC s; , n%(88
((DN#3j#@DNCS#STT eS ! ! GGE >Vc%jj(FFFD)) XF u::V $ $ CZ]^cZdZd C Clopvlwlw C C C
) . .v 6 6 6 XFF
fd++
x !vwww
6
62215555
6 Vc%jj(F v;;#e** $ $ CZ]^cZdZd C Clopvlwlw C C C
%(%?%?
AYF
r Tc h | | j d }|rt | S |S rn )splitrE +clean_text_and_extract_entities_with_bboxes)rU r^ cleanup_and_extractcaptions r post_process_generationz(Kosmos2Processor.post_process_generation s6 **T^,,R0 H>wGG Gr c @ j |fd|i|} fd|D S )a
Post-process the output of the model to decode the text.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.
Returns:
`list[str]`: The decoded text.
skip_special_tokensc > g | ]} |d S )F)r )r )r= r^ rU s r r? zDKosmos2Processor.post_process_image_text_to_text.. s- jjjRV,,Tu,MMjjjr ) batch_decode)rU generated_outputsr rX generated_textss ` r post_process_image_text_to_textz0Kosmos2Processor.post_process_image_text_to_text sB ,$+,=qqSfqjpqqjjjjZijjjjr c D | j j }| j j }||z dgz S )Nr )rW model_input_namesrV )rU tokenizer_input_namesimage_processor_input_namess r r z"Kosmos2Processor.model_input_names s. $ @ &*&:&L#$'BBFbEcccr c |t | dk r|S t t j d| }t | t | k r0t dt | dt | d d}g }t || D ]\ }}| \ }} | ||| | }|>t |t r|g}g }
t d |D st d |D ]4} | | \ } }
|
| d |
5t |
dk rd
|
}| d | d |t | k r | ||d d
| }|S )Nr z.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c 3 K | ]}|d uV d S r r )r= boxs r r z>Kosmos2Processor._insert_patch_index_tokens.. s& 773s$777777r zTThe multiple bounding boxes for a single phrase should not contain any `None` value.r z z )
rk r refinditerr r spanrQ r r r #_convert_bbox_to_patch_index_tokensr )rU r^ r matched_phrasescurr_posbuffermatchedr r endpatch_index_stringsr
patch_index_1
patch_index_2 position_strs r r z+Kosmos2Processor._insert_patch_index_tokens sS >S[[A - -K r{+B4 P P PQQ 3v;; . . { IL M\ I] I] { { eh io ep ep { { {
& 9 9 @ @MGT \\^^FAs MM$x|, - - -H|$&&
v"$ 77$77777
j
O
O/3/W/WX[/\/\,
}#**m+M+Mm+M+MNNNN&''1,,=BBCVWWL MM>l>>> ? ? ? ? c$ii MM$xyy/ * * *wwv r r c F t | dk r|\ }}n9t t j | j }t || \ }}dt
| d d}dt
| d d}||fS )Nr r7 r8 r9 )rk r mathsqrtrO coordinate_to_patch_indexr: r; )rU r idx_1idx_2num_patches_per_sidetoken_1token_2s r r z4Kosmos2Processor._convert_bbox_to_patch_index_tokens s t99>> LE55 $'ty1L'M'M#N#N 4T;OPPLE58#e**"2"21"5"58888#e**"2"21"5"5888 r )r3 )NN)NNr/ )T)r r r rT r r r
r r
r$ r r r r BboxInputr r: r r r propertyr r r r r
__classcell__)r\ s @r r2 r2 V s /5 /5 /5 /5 /5 /5b %),0J JT!J $y/)J /0 J
J J J ^JX! ! !F %) ') @ @4 ?*@ T!@ @
* @
tCy
@ @ @ @D k k k k& d d Xd
+s +Ds x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.rf )r r floorceil) r r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxs r r r s RR GGRoppp :b// 0 0D :b// 0 0D 9R..2 3 3D 9R..2 3 3D
(
(4
/F
(
(4
/F 6>r r r c ( d|z }| |z }| |z }||z }||z }| |k r||z }||z } ||z |z }
||z |z } nS||k s||k r||z }||z } ||z |z }
||z |z } n,||z |dz z }||z |dz z } ||z |dz z }
||z |dz z } || |
| fS )a
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?r r ) r r r cell_sizer r r r r r r r s r patch_index_to_coordinater s **I ( (D ) )D ( (D ) )D
I
I
I
)
I
)
I
I
I
)
I
)
I
A
-
I
A
-
I
A
-
I
A
-
r2r>r c
d}t j || }g }|D ]}| d }| \ }}}|s8d}| d d | d d f}| d } g }
| D ]} t j d| } t j d| dd }
| r|
r|rX|
t | d t |
d f |
t | d t |
d f |r| |||
f |
D ]0}d|d d |d d
}| |||gf 1|S ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This function is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?r Nr r5 zrf r7 z> yr entitys r #extract_entities_with_patch_indicesr / s kG k'4((G #% K Kzz!}},1LLNN)
FM 8FJJqMM!$ejjmmA&67D * / /0P Q Q
% M MD 0$77A 0$qrr(;;A
MQ
M M!((#aggajj//3qwwqzz??)KLLLL!((#aggajj//3qwwqzz??)KLLL K ' . .m/L M M M M%
K
KJaJJQJJJ+22FD4&3IJJJJ
K
'&r c | \ }\ }}t t j dd|d| }t t j dd|d| }|||ff}|S )zfAdjust the positions of the entities in `text` to be relative to the text with special fields removed.<.*?>r N)rk r sub)r
r^ entity_namestartr adjusted_start adjusted_endadjusted_entitys r adjust_entity_positionsr i sk & K %T&5&\::;;Nrvgr4:6677L"^\$BCO r c | }t | t | z
}g }|D ]\ }\ }}}t | t | z
} t | t | z
}
||z
| z }||z
|
z
}| }| |||f|f ||fS )z9Remove the spaces around the text and the entities in it.)r rk r@ rA rQ ) r^ entitiesnew_textleading_spaces new_entitiesr r r r entity_name_leading_spacesentity_name_trailing_spacess r _cleanup_spacesr s s zz||HYYT[[]]!3!33NL-5 A A) \eS6%(%5%5K>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r r r r c J g | ] }t |d |d S )r rf )r )r= r r s r r? z?clean_text_and_extract_entities_with_bboxes.. s0 qqqbf 5d1gtAwH\ ] ]qqqr )r r r r rQ r )
r^ r processed_textr r itemr
r r bboxes_in_coordss
` r r r s VGR..N"Ed"K"K H+ ? ? acDG1&$??qqqqjpqqq+;*==>>>> >8 4 44r )r )&r r r r image_processing_utilsr image_utilsr processing_utilsr r r r r
tokenization_pythonr tokenization_utils_baser r
utilsr r r r r r r r r r$ r2 r r r r r r __all__r r r r- s $ # 2 2 2 2 2 2 % % % % % % b b b b b b b b b b b b b b - - - - - - ? ? ? ? ? ? ? ? # # # # # # sCx
5ue+ ,-.
4c3h !" 4eUE)* +,-
%$,edlT%$,eVZlI[:[5\&\!]]
^
% % % % %,e % % % %"
% -U * I I I I I ~ I I I X E%u*D$E ]` ejknpskset >(c (3 (c ( ( ( (Z7' 7' 7't " " "*5 5 5 5:
r