tjLjdZddlmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z mZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZmZmZmZddlm Z ddl!m"Z"ej#e$Z%edeGddeZ&edeGddeZ'Gddej(Z)eGddeZ*edGdd e*Z+edGd!d"e*e Z,gd#Z-dS)$zPyTorch PaliGemmamodel.) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_causal_maskcreate_masks_for_generate!create_sliding_window_causal_mask)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introc2eZdZUdZdZejdzed<dS)PaligemmaModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr,s7 59*T188888r(rzU Base class for PaliGemma causal language model (or autoregressive) outputs. ceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZejdzed<dS) PaliGemmaCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)r r!r"r#r,r$r%r&r-r.rr/tupler0rr'r(r)r+r+<s  &*D% d ")))'+FE  $+++$(OUT\(((59M5*+d299926Je'(4/66648*T188888r(r+c*eZdZdeffd ZdZxZS)PaliGemmaMultiModalProjectorconfigcttj|jj|jjd|_dS)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr4 __class__s r)r9z%PaliGemmaMultiModalProjector.__init__[sB i 4 @&BVBelpqqq r(c0||}|SN)r>)r@image_featuresr/s r)forwardz$PaliGemmaMultiModalProjector.forward_s N33 r()r r!r"rr9rE __classcell__rAs@r)r3r3ZsZrrrrrrrr(r3cFeZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dZ dZdS) PaliGemmaPreTrainedModelr4model)imagetextTr3r.FN)r r!r"rr&base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr'r(r)rIrIes_ (&*#78#4"5"N"&r(rIz| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., ceZdZdZdeffd Zeeddej de e de e zfd Zd ejd ej d ej fd Zee dd ejdzdej dzdejdzdejdzdedzdejdzd ej dzdejdzdedzde ede ezfdZxZS)PaliGemmaModelFr4ct|tj|j|_t ||_|jj |_ tj|j}||_ |j j p|j |_|dS)N)r4)r8r9r from_configr; vision_towerr3multi_modal_projector text_config vocab_sizelanguage_modelr4get_text_configdtypetext_config_dtype post_init)r@r4r_rAs r)r9zPaliGemmaModel.__init__}s    %19MNNN%A&%I%I" ,7".f6HIII,!%!!>!D!R  r(zWObtains image last hidden states from the vision tower and apply multimodal projection.r pixel_valueskwargsreturnc h|j|fi|}|j}||}||_|SrC)r[last_hidden_stater\ pooler_output)r@rdre image_outputsselected_image_featurerDs r)get_image_featuresz!PaliGemmaModel.get_image_featuressI*),AA&AA !.!@334JKK&4 #r( input_ids inputs_embedsrDc |e||tj|jjtj|jk}|d}n||jjk}|}|j d|j dz}| d |j}t||j dz| kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)radevicerrz6Image features and image tokens do not match, tokens: z , features: )get_input_embeddingsr$tensorr4image_token_idlongrpallsumshape unsqueezetornumel)r@rmrnrDspecial_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskz#PaliGemmaModel.get_placeholder_masks  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11)/2^5I!5LL/99"==@@AUVV ]04 48L8L8N8N N s^ s saq s s   "!r(Nattention_mask position_idsr.token_type_idslabels use_cachec 2|du|duz rtd|?|jj|jkr*||jjk} |} d| | <n|} ||| }|Y||nd} tj|j d|j | z}| ddz}|h| |j }||j |j}||||} || |}|j||||d}|dup |j p|du}||rtj|dkdd|d <t+di|}t-|jjd d#|}|t3di|d }|jd||||| d | }t7|j|j|j|j||nd S) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrrrp)rnrD)r4rnrr.rrqblock_sequence_idssliding_window)full_attentionsliding_attention)rrr.rnr)rhr.r/r0rr') ValueErrorr4rtr^clonerrget_seq_lengthr$arangerxrpryrlrirzrarmasked_scatterr`is_initializedwherer getattrr]copyr r_rrhr.r/r0)r@rmrdrrr.rrnrrrer| llm_input_idspast_seen_tokensrD mask_kwargsis_first_iteration causal_masksliding_mask_kwargsoutputss r)rEzPaliGemmaModel.forwardsZ  -t"; < [YZZ Z  T[%?4?%R%R!*dk.H!H %OO--M01M, - -%M  7D5577 FFM  CRC^==???de  < (;A(>}G[\\\_ooL'11!44q8L  #!44\BBPN+..}/C]EXYYN!%!:!:~";"" *889K^\\Mk1133*,.(   -4vO2JPT     r() NNNNNNNNN)r r!r"accepts_loss_kwargsrr9rrr$r%rrr1rrl LongTensorrTensorrboolr rrErFrGs@r)rXrXts        ^n!-9?@R9S + +")":?:K"]b]n""""0.215.204(,2626*.!%l l #d*l '$.l  t+ l &- l  l (4/l (4/l  4'l $;l -.l  - -l l l ^l l l l l r(rXceZdZddiZdeffd Zedejde e fdZ e e dd ej dzdejdzd ejdzd ej dzd edzdej dzdejdzdej dzdedzdeejzde e deezfdZ dfd Ze ddedejd ejdzd edzd ejdzdejdzdedzdefdZxZS)!PaliGemmaForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightr4ct|t||_t j|jj|jjd|_ | dS)NFr6) r8r9rXrJrr:r]r<r^lm_headrcr?s r)r9z*PaliGemmaForConditionalGeneration.__init__(se    #F++ y!3!?ASA^ejkkk  r(rdrec (|jj|fi|SrC)rJrl)r@rdres r)rlz4PaliGemmaForConditionalGeneration.get_image_features.s,tz,\DDVDDDr(Nrrmrrr.rrnrrlogits_to_keeprfc n|jd|||||||| |d | } | d} t| trt| dn| }|| dd|ddf}d}| |jd|||jjjd| }t||| j | j | j | j S)r) rmrdrrrr.rnrrrN)r-rr^)r,r-r.r/r0rr')rJ isinstanceintslicer loss_functionr4r]r^r+r.r/r0r)r@rmrdrrr.rrnrrrrerr/ slice_indicesr-r,s r)rEz)PaliGemmaForConditionalGeneration.forward2sZ$*  %))%+'        8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD/#3!/) ' ;     r(TFc tj|f|||||| || d| } | d| ddz| d<| s|s|| d<| S)N)r.rnrrrrrrrrrd)r8prepare_inputs_for_generationget)r@rmr.rnrrdrrrrrrre model_inputsrAs r)rz?PaliGemmaForConditionalGeneration.prepare_inputs_for_generations =uww<   +')%))1         N + + 7+7+G!+KL (  8Y 8+7L (r(rc tjg|ddd|j}|tj|dkdd}t ||||||S)Nrqrr)r4rnrrr.r)r$fullsizerprr r`) r4rnrr.rrrre group_idss r)r z;PaliGemmaForConditionalGeneration.create_masks_for_generatesJ;!3!3!5!5crc!:;R H\]]]  % Na$7B??I())++'()+%     r() NNNNNNNNNr) NNNNNNTNNF)NF)r r!r"_tied_weights_keysrr9rr$r%rrrlrrrrrrr1r+rEr staticmethodrdictr rFrGs@r)rr s +,VW Eu/@EFSeLfEEE^E.215.204(,2626*.!%-.J J #d*J '$.J  t+ J &- J  J (4/J (4/J  4'J $;J el*J +,J  0 0J J J ^J ^ ))))))V/3*/    |  t+   lT)   t+  !4K     \     r(r)rrIrX).r# dataclassesrr$r cache_utilsrconfiguration_utilsr generationr masking_utilsr r r modeling_flash_attention_utilsr modeling_outputsr rmodeling_utilsrprocessing_utilsrutilsrrrrrrautorconfiguration_paligemmar get_loggerr loggerrr+Moduler3rIrXr__all__r'r(r)rsE!!!!!!  333333))))))mmmmmmmmmmBBBBBBSSSSSSSS------&&&&&&444444  H % %  99999#:99   9  99999k99   9029 ' ' ' ' ' ' ' ' d d d d d -d d  d N ^ ^ ^ ^ ^ (@/^ ^  ^ B ^ ] ]r(