iz4>z3cX2-U-UUS3$)a Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_token image_seq_len image_token num_imagess r'build_string_from_inputr35s"&)J6 7 {6(" MMr&c ^\rSrSrSrSS/rSrSrS S\S\4U4S jjjr S!S \ S \ \ \ \\ \\ 4S \\S \4SjjrS"SjrSrSr\S5r\S \4Sj5rS"S \ S \\S \4SjjrS \ \ \\ 4S \\S \4SjrS#S\ S\S4S\ S\S4S\S\SS\ S\4S S4 SjjrSrU=r$)$ColPaliProcessorKa Constructs a ColPali processor which wraps a PaliGemmaProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score. [`ColPaliProcessor`] offers all the functionalities of [`PaliGemmaProcessor`]. See the [`~PaliGemmaProcessor.__call__`] for more information. Args: image_processor ([`SiglipImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`LlamaTokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. visual_prompt_prefix (`str`, *optional*, defaults to `"Describe the image."`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Question: "`): A prefix to be used for the query. image_processor tokenizer)SiglipImageProcessorSiglipImageProcessorFast)GemmaTokenizerGemmaTokenizerFastvisual_prompt_prefix query_prefixc>[TU]XUS9 Uc [S5eUc [S5e[US5(d [S5eURUl[US5(dK[ [ SSS 9nS U/0nURU5 UR[ 5Ul [ Ul O"URUl URUl UR[5 SUl SUlX@lXPlg) N) chat_templatez)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.image_seq_lengthz;Image processor is missing an `image_seq_length` attribute.r1FT) normalizedspecialadditional_special_tokens)super__init__ ValueErrorhasattrrAr IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idr1 add_tokens EXTRA_TOKENS add_bos_token add_eos_tokenr=r>) selfr7r8r@r=r>r1 tokens_to_add __class__s r'rFColPaliProcessor.__init__ds =Q  "HI I  AB B(:;;Z[ [ / @ @y-00$[UDQK8;-HM  ( ( 7"+"A"A+"ND *D "+":":D (44D \*"' "' $8!(r&imagestextkwargsreturnc xUR"[4SURR0UD6nUSR SS5nUSLnUcUc [ S5eUbUb [ S5eUGb[ U5(aU/nOw[U[5(a[ US5(aON[U[5(a.[US[5(a[ USS5(d [ S5eUR/[U5-n UV s/sHoRS 5PM nn [X5V V s/sHTup[U URRUR[ [U [5(a [U 5OS S 9PMV n n n [#U5nUR$"U40US D6S nUSR'SS5bUSS==UR- ss'UR"U 4SS0USD6n0UES U0EnU(a.USR)USS:HS5nUR+SU05 [-US9$Ub[U[.5(aU/nO8[U[5(a[US[.5(d [ S5eUcUR0S-n/nUH@nURRUR2-U-U-S-nUR5U5 MB USR'SS5USS'UR"U4SS0USD6nU$gs sn fs sn n f)a Main method to prepare for the model either (1) one or several texts, either (2) one or several image(s). This method is a custom wrapper around the PaliGemmaProcessor's [`~PaliGemmaProcessor.__call__`] method adapted for the ColPali model. It cannot process both text and images at the same time. When preparing the text(s), this method forwards the `text` and `kwargs` arguments to LlamaTokenizerFast's [`~LlamaTokenizerFast.__call__`]. When preparing the image(s), this method forwards the `images` and `kwargs` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNz&Either text or images must be providedz5Only one of text or images can be processed at a timerzAimages must be an image, list of images or list of list of imagesRGBr-r pixel_values max_lengthreturn_token_type_idsF input_idstoken_type_idsilabels)dataz*Text must be a string or a list of strings r,2) _merge_kwargsrr8 init_kwargspoprGr isinstancelistr=lenconvertzipr3r/rArIr r7get masked_fillupdaterstrquery_augmentation_tokenr>append)rQrUrVaudiovideosrW output_kwargsr[r` texts_docimager. image_list input_stringsr^inputs return_datarc texts_queryquery batch_querys r'__call__ColPaliProcessor.__call__s|Z** " "&.."<"<  }-11(DA &d 2 ?ummE*F?+.i*@ +A&F(!"nn66"&"7"7 +2S>SS:^^&+ .F CVB^\BK$ ,88@P9QUV9VX\]""Hf#56[1 1  $$$v t,,DGS1I1I !MNN~66;%'K0043D3DDuLvUX\\""5):G}9U9Y9YZfhj9kM- ( 6..&+ .K  -C@ s L1>AL6c 0nUb;UR/[U5-nS/[U5-nURXES.5 [S0UD6$)ax Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. r])num_image_tokensnum_image_patchesr)rArlrqr )rQ image_sizesrW vision_datarrs r'_get_num_multimodal_tokens+ColPaliProcessor._get_num_multimodal_tokenssZ  " $ 5 56[9II !"c+&6 6    4Dm n, ,,r&c:URR"U0UD6$)z This method forwards all its arguments to GemmaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r8 batch_decoderQargsrWs r'rColPaliProcessor.batch_decodes ~~**D;F;;r&c:URR"U0UD6$)z This method forwards all its arguments to GemmaTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r8decoders r'rColPaliProcessor.decodes ~~$$d5f55r&cURRnURRn[[R X-55$N)r8model_input_namesr7rkdictfromkeys)rQtokenizer_input_namesimage_processor_input_namess r'r"ColPaliProcessor.model_input_names#s< $ @ @&*&:&:&L&L#DMM"7"UVWWr&c.URR$)zr Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )r8 pad_token)rQs r'rs)ColPaliProcessor.query_augmentation_token)s~~'''r&c *UR"SSU0UD6$)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. rUrr)rQrUrWs r'process_imagesColPaliProcessor.process_images2sB}}5F5f55r&c *UR"SSU0UD6$)a? Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). rVrr)rQrVrWs r'process_queries ColPaliProcessor.process_queriesUs@}}1$1&11r&query_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec [U5S:Xa [S5e[U5S:Xa [S5eUSRUSR:wa [S5eUSRUSR:wa [S5eUcUSRn/n[ S[U5U5GHn/n[ R RRRXXs-SSS9n [ S[U5U5H}n [ R RRRX*X-SSS9n UR[ R"SX5RS S 9SRS S 95 M UR[ R"US S 9RU5RU55 GM [ R"USS 9$) a Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColPali, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeT) batch_first padding_valuez bnd,csd->bcnsr)dimr])rlrGdevicedtyperangetorchnnutilsrnn pad_sequencerteinsummaxsumcatto) rQrrrrrscoresi batch_scores batch_queriesjbatch_passagess r'score_retrieval ColPaliProcessor.score_retrievalws@  A %23 3 ! "a '34 4 A  % %);A)>)E)E ENO O A  $ $(:1(=(C(C CLM M  +A.44L%'q#./:\]"A"##LL-PTTYZT[\]^bbghbi C MM%))La8;;LILL][ \=yyQ''r&)rAr1rLr>r=)NNNzDescribe the image.z Question: )NNNNr)r*Ncpu) r r!r"r#__doc__ attributesimage_processor_classtokenizer_classrrrFrrrrrkr rrrrrrpropertyrrsrrintrrr% __classcell__)rSs@r'r5r5Ks($[1JP>O$9( ) " )  ) )H"^b {{I0$y/4HYCZZ[{ /0 { {z-$<6XX (#(("!6!6/0!6  !6F 2ItI./ 2/0 2  2L0449 >(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(>(r&r5)typingrrfeature_extraction_utilsr image_utilsrrr processing_utilsr r r r tokenization_utils_baserrrrrrrrIrrNr3r5__all__)rs0r'rs.#4OOXXOO' -U  ).t5A$qgQ5RWX[R\8]R\Q4#waR\8]] N,j(~j(Z  M 68]s B7B#