tjR2ddlmZmZddlmZmZmZddlmZddl m Z m Z ddl m Z mZddlmZmZddlmZmZerdd lZejeZGd d e d ZGddeZdgZd S))OptionalUnion) IMAGE_TOKENPaliGemmaProcessorbuild_string_from_input) BatchFeature) ImageInputmake_flat_list_of_images)ProcessingKwargsUnpack)PreTokenizedInput TextInput)is_torch_availableloggingNc(eZdZddidddddidZd S) ColPaliProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsi/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/colpali/modular_colpali.pyrr!sB y ,"  +D1  IIIr#rF)totalc eZdZ ddedeffd ZedefdZ dd edzd ee ze eze e zd e e de fd Z dd edzd e e de fd Zd ee ezd e e de fdZ ddede dfdede dfdededdedefddf dZxZS)ColPaliProcessorNDescribe the image. Question: visual_prompt_prefix query_prefixcl||_||_t|||dS)a! visual_prompt_prefix (`str`, *optional*, defaults to `"Describe the image."`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Question: "`): A prefix to be used for the query. )image_processor tokenizer chat_templateN)r*r+super__init__)selfr-r.r/r*r+ __class__s r$r1zColPaliProcessor.__init__/s;%9!( I]jkkkkkr#returnc|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )r. pad_token)r2s r$query_augmentation_tokenz)ColPaliProcessor.query_augmentation_tokenAs~''r#imagestextkwargsc jtfdjji|}|ddd}d}||t d||t d|)j|}t|}j gt|z}fd|D}fd t||D}j|fi|d d } |d d d|dd xxj z cc<j|fd |i|d} i| d | i} |r=| d| ddkd} | d| it!| S|t#|t$r|g}n?t#|t&rt#|dt$st d| jdz}g} |D]4}jjjz|z|zdz}| |5|d d d|dd <j| fd |i|d}|SdS)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNTz&Either text or images must be providedz5Only one of text or images can be processed at a timecDg|]}j|Sr")r- process_image).0imager2s r$ z-ColPaliProcessor.__call__..ls*TTTEd*88??TTTr#c g|]Q\}}t|jjjtt |t rt|ndRS))prompt bos_token image_seq_len image_token num_images)rr.rFimage_seq_lengthr isinstancelistlen)r@rE image_listr2s r$rBz-ColPaliProcessor.__call__..nso   'FJ(!"n6"&"7 +2>F-f55F23c&kkAITTTTVTTTF    +.i*@*@   M04/YY-:XYYZhiL]+// dCCOm,\:::d>SS:::#T^&; .F CVB^\BBK$ 7 ,88@P9QUV9VX\]]""Hf#5666[111 1  $$$ Ov t,, ODGS1I1I O !MNNN~6;%'K * *043DDuLvUX\\""5))))9F}9U9Y9YZfhj9k9kM- ( 6($.&; .K  - r#c |jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. r8r"rl)r2r8r:s r$process_imageszColPaliProcessor.process_imagess!>t}55F5f555r#c |jdd|i|S)ag Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). r9r"rn)r2r9r:s r$process_queriesz ColPaliProcessor.process_queriess!<t}11$1&111r#cpuquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec 8t|dkrtdt|dkrtd|dj|djkrtd|dj|djkrtd| |dj}g}t dt||D]:}g}t jjj ||||zdd} t dt||D]} t jjj || | |zdd} | t j d | |  d d d | t j|d || usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeNT) batch_first padding_valuez bnd,csd->bcnsr)dimrD)rMr\devicedtyperangetorchnnutilsrnn pad_sequencerceinsummaxsumcatto) r2rtrurvrwrxscoresi batch_scores batch_queriesjbatch_passagess r$score_retrievalz ColPaliProcessor.score_retrievals*@  A % %233 3 ! " "a ' '344 4 A  %);A)>)E E ENOO O A  $(:1(=(C C CLMM M  +A.4L%'q#.//<< ] ]A/1L!HN.;; Q^!34$VW<M1c"455zBB  !&!3!@!@&q1z>'9:\]"A""##L-PPTTYZT[[\]^bbghbii MM%)La888;;LIILL][[ \ \ \ \yQ''''r#)NNNr(r))NN)N)rrNrs)rrr rbr1propertyr7r rrrLr rr rlrorqrintrr __classcell__)r3s@r$r'r'.s$9( ll " l  llllll$(#(((X(%)Z^XXT!X++d9o=EV@WWX/0 X  XXXXx%)66T!6/06  6666B2$y/)2/02  2222H0449 >(>(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(>(>(>(>(>(>(>(r#r')typingrr2transformers.models.paligemma.processing_paligemmarrrfeature_extraction_utilsr image_utilsr r processing_utilsr r tokenization_utils_baserrrrrr get_loggerrloggerrr'__all__r"r#r$rsS #"""""""wwwwwwwwww444444????????88888888CCCCCCCC00000000LLL  H % %     -U    u(u(u(u(u()u(u(u(r r#