tjx<ddlmZddlmZddlmZddlmZmZddl m Z m Z m Z m Z ddlmZmZddlmZmZmZmZmZd d lmZmZd d lmZd d lmZerddlZeje Z!Gdde dZ"GddeZ#GddeZ$edeGddeZ%edGddeZ&gdZ'dS)) dataclass)Cache) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput) ModelOutputauto_docstringcan_return_tupleis_torch_availablelogging)ColPaliForRetrievalColPaliPreTrainedModel)ColPaliProcessor)ColQwen2ConfigNc(eZdZddidddddidZd S) ColQwen2ProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsk/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/colqwen2/modular_colqwen2.pyrr"sB y ,"  +D1  IIIr+rF)totalc eZdZ d dedzdedzfdZ d dedzdeezeezeezde e de fd Z dd Z ed ZdS)ColQwen2ProcessorNvisual_prompt_prefix query_prefixc tj||||t|dsdn|j|_t|dsdn|j|_|pd|_|pd|_dS) ar visual_prompt_prefix (`str`, *optional*, defaults to `"<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>"`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Query: "`): A prefix to be used for the query. ) chat_template image_tokenz <|image_pad|> video_tokenz <|video_pad|>zf<|im_start|>user <|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>zQuery: N)r __init__hasattrr4r5r0r1)selfimage_processor tokenizerr3r0r1kwargss r,r6zColQwen2Processor.__init__0s oyP]^^^^29)]2S2Sn??YbYn29)]2S2Sn??YbYn$8% u !)5Ir+imagestextr;returnc &|jtfd|jji|}|ddd}|du}||t d||t d|t |r|g}nt|trt |drnZt|tr6t|dtrt |ddst d|j gt|z}|j dd |i|d }|d } | |j j d z} d} tt|D]} |j|| vrW|| |jd | | | zzd|| <| dz } |j|| vW|| d |j|| <|j|fddi|d} t#i| |}|d dddf|d ddd fz}tt%j|d|}t$jjj|d|d<|r=|d|ddkd}|d|i|S|t|t6r|g}n?t|trt|dt6st d| |jdz}g}|D]$}|j|z|z}||%|j|fddi|d}|SdS)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsr#suffixNz&Either text or images must be providedz5Only one of text or images can be processed at a timerzAimages must be an image, list of images or list of list of imagesr<r$image_grid_thwrz<|placeholder|>rreturn_token_type_idsF)data pixel_valuesT) batch_first input_idstoken_type_idsilabelsz*Text must be a string or a list of strings r*) _merge_kwargsrr: init_kwargspop ValueErrorr isinstancelistr0lenr9 merge_sizeranger4replaceprodrtorchsplittolistnnutilsrnn pad_sequence masked_fillupdatestrquery_augmentation_tokenr1append)r8r<r=r; output_kwargsrArC texts_doc image_inputsrB merge_lengthindexi text_inputs return_dataoffsetsrErI texts_queryqueryaugmented_query batch_querys r,__call__zColQwen2Processor.__call__Hsg +* #  "&."<   }-11(DAA &d 2 A s9~~..]]A*il::'0|';'; ,.?>RWCXC]C]C_C_coCo.prs(( !   *il:: $-Q<#7#78I4K[#\#\IaLL($.&+ .K ',K{,Kl,KLLLK""23AAAqD9KHX[_`>`bfgg""Hf#5666   $$$ Ov t,, ODGS1I1I O !MNNN~6;%'K 4 4"&"3e";f"D""?3333($.&+ .K  + r+c @i}|tjdi|ddp jjfd|D}fd|D}|||dt di|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. Nr$rRc8g|]}jjg|RSr*)r9get_number_of_image_patches).0 image_sizer$r8s r, z@ColQwen2Processor._get_num_multimodal_tokens..sE!!!A$@\*\m\\\!!!r+c g|] }|dzz S)rr*)rs num_patchesrRs r,ruz@ColQwen2Processor._get_num_multimodal_tokens..s"ddd; A !=dddr+)num_image_tokensnum_image_patchesr*)rr)getr^r9rRr )r8 image_sizesr; vision_dataryrxr$rRs` @@r,_get_num_multimodal_tokensz,ColQwen2Processor._get_num_multimodal_tokenss  "3=AA/SUVVM   ( ( (&**<>>a$BVBaJ!!!!!"-!!!  edddRcddd    4D[lmm n n n,, ,,,r+cT|jj}|jj}d|D}||zS)Ncg|]}|dv| S))pixel_values_videosvideo_grid_thwr*)rsnames r,ruz7ColQwen2Processor.model_input_names..s*' ' ' DHqc | u| s| dddf| dddfz}tj| jd|j}|d|dk}| |} ||n |jj}| | n |jj} | | n |jj} ||j |}| |j | | dj }||jj jkd}||j|j}|||}| d||||||| | }| r|jnd}|d}|jjj}|||}||dd z }|||dz}t-||j||j S) z image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. Nrr)devicerT)grid_thwr) rGrrrrrrrr)dimkeepdim)rrrr)rVarangeshaper unsqueezerrrrvlmget_input_embeddingsvisual pooler_output vlm_configimage_token_idtodtypemasked_scatterrembedding_proj_layerweightnormrrr)r8rGrrrrIrrrrrrErBr;rjrmask image_embeds image_mask vlm_outputvlm_hidden_stateslast_hidden_states proj_dtypers r,forwardzColQwen2ForRetrieval.forwardsF.  #(B$QQQT*^AAAqD-AAG\,"4Q"7OOOF##A&&):):1)=)==D'-L1B1N--TXT_Tq%9$D $+Jj &1%&>r&B&BBJ)!&6+!,     r+) NNNNNNNNNNNN)r&r'r(rr6rrrV LongTensorrrrboolrr __classcell__)rs@r,rrsz$~$$$$$$.2.204(,*.26!%)-,0#',026I I #d*I  t+I &- I  I  4' I (4/I $;I  $;I #TkI D[I lT)I (4/I  $I I I ^I I I I I r+r)rrr/)( dataclassesr cache_utilsrfeature_extraction_utilsr image_utilsrrprocessing_utilsr r r r tokenization_utils_baser rrZrrrrrcolpali.modeling_colpalirrcolpali.processing_colpalirconfiguration_colqwen2rrV get_loggerr&loggerrr/rrr__all__r*r+r,rs"!!!!! 44444455555555XXXXXXXXXXXXCCCCCCCC______________RRRRRRRR999999222222LLL  H % %     .e    eCeCeCeCeC(eCeCeCP     4     7777777   7(    P P P P P .P P   P f   r+