tj:;hdZddlZddlmZddlmZddlmZddlmZm Z ddl m Z dd l m Z dd lmZdd lmZmZmZmZmZd d lmZejeZeGdde ZedGddeZedGddeeZgdZdS)zPyTorch Fuyu model.N)nn)Cache)GenerationMixin)BaseModelOutputWithPoolingCausalLMOutputWithPast)PreTrainedModel) AutoModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) FuyuConfigc@eZdZUeed<dZdZdZdZdZ dZ dZ gZ dgZ dS)FuyuPreTrainedModelconfigmodel)imagetextTpast_key_valuesN)__name__ __module__ __qualname__r__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_no_split_modules_skip_keys_device_placementd/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/fuyu/modeling_fuyu.pyrr sV (&*#"&N#4"5r(rzt The Fuyu model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introceZdZdeffd ZdejdeejdejdejfdZe e dej d e e deezfd Zd ejd ej d ej fdZe e dd ejdzdejdzdejdzdejdzdejdzdedzd ej dzdedzd e e deezfdZxZS) FuyuModelrc^t||j|_|jj|_t j|j|_tj |j |j z|j z|j |_d|_|dS)NF)super__init__ pad_token_id padding_idx text_config vocab_sizer from_configlanguage_modelrLinear patch_size num_channels hidden_sizevision_embed_tokensgradient_checkpointing post_initselfr __class__s r)r/zFuyuModel.__init__4s    !. ,7'3F4FGG#%9   1 1F4G GI[$ $  ',# r(word_embeddingscontinuous_embeddingsimage_patch_input_indicesreturnc R|jdt|ks-tdt|d|jd|}t |jdD]}t j||dkdd}|||}|jd||jdkr)td||jd|jd|d ||||j|||f<|S) aThis function places the continuous_embeddings into the word_embeddings at the locations indicated by image_patch_input_indices. Different batch elements can have different numbers of continuous embeddings. Args: word_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Tensor of word embeddings. continuous_embeddings (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Tensor of continuous embeddings. The length of the list is the batch size. Each entry is shape [num_image_embeddings, hidden], and num_image_embeddings needs to match the number of non-negative indices in image_patch_input_indices for that batch element. image_patch_input_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Tensor of indices of the image patches in the input_ids tensor. rz7Batch sizes must match! Got len(continuous_embeddings)=z and word_embeddings.shape[0]=T)as_tuplezGNumber of continuous embeddings continuous_embeddings[batch_idx].shape=zA does not match number of continuous token ids src_indices.shape=z in batch element .) shapelen ValueErrorclonerangetorchnonzerotodevice)r>r@rArBoutput_embeddings batch_idx dst_indices src_indicess r)gather_continuous_embeddingsz&FuyuModel.gather_continuous_embeddingsAs{( %a(C0E,F,FFFls3H/I/IllQ`QfghQill ,11334Q788  I -(A)(LPQ(Q\`aaabcdK4I>{KK #&;I&F&LQ&OOO i7LY7W7]ii6A6Gii\eiii9Ni8XYd8e8h8h!(99 i4 5 5! r( pixel_valueskwargsc L||}t|S)z pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. )last_hidden_state)r:r)r>rUrVpatch_embeddingss r)get_image_featureszFuyuModel.get_image_featuresms* 33LAA)r[r\r]special_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskzFuyuModel.get_placeholder_maskys  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11)/2^5I!5LL/99"==@@AUVV ]04 48L8L8N8N N s^ s saq s s   "!r(N image_patchesimage_patches_indicesattention_mask position_idsr use_cachec ^|du|duz rtd|"|j|}|jd} |b||jn|j} ||nd} t j| | | zt j| }| d}|j| |dj } | |j|j } |||| }||| }|jd |||||d | }|S) a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. Nz:You must specify exactly one of input_ids or inputs_embedsrrr_T) return_dict)r\r])r\rprqrrrr')rIr5rbrGrOget_seq_lengthrLarangererhrZrXrNr`rmmasked_scatter)r>r[rnrorprqrr\rrrVseq_lenrOpast_key_values_lengthrYrjoutputss r)forwardzFuyuModel.forwards,  -t"; < [YZZ Z  FD/DDFFyQQM%a(  )2)>Y%%MDXFIXId_%C%C%E%E%Ejk " <&2H(HPUPZciL(11!44L  $#66}RV6WWi /22=3GI\]] !%!:!:GW";"" *889KM]^^M%$% ')%+      r()NNNNNNNN)rrrrr/rLTensorlistrTrr FloatTensorr r tuplerrZ LongTensorrmrboolrr{ __classcell__r?s@r)r,r,.s z      *!*! $EL1*!$)< *!  *!*!*!*!XN!-N9?@R9SN + +NNN^N")":?:K"]b]n""""0.2-159.204(,26!%55#d*5|d* 5 %|d2 5  t+ 5&-55(4/5$;5+,5 ' '555^55555r(r,zz Fuyu Model with a language modeling head on top for causal language model conditioned on image patches and text. cFeZdZddiZdeffd Zee ddej dzdej dzd ej dzd ej dzd ej dzd e dzd ej dzde dzdej dzdedzdeedeezfdZ dfd ZxZS)FuyuForCausalLMzlm_head.weightz(model.language_model.embed_tokens.weightrct|t||_t j|jj|jjd|_ | dS)NF)bias) r.r/r,rrr6r2r9r3lm_headr<r=s r)r/zFuyuForCausalLM.__init__se    v&& y!3!?ASA^ejkkk  r(Nrr[rnrorprqrr\rrlabelslogits_to_keeprVrCc `|jd||||||||d| } | d} t| trt| dn| }|| dd|ddf}d}| |jd|| |jjjd| }t||| j | j | j S)a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Examples: ```python >>> from transformers import FuyuProcessor, FuyuForCausalLM >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> processor = FuyuProcessor.from_pretrained("adept/fuyu-8b") >>> model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b") >>> url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> prompt = "Generate a coco-style caption.\n" >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> generated_ids = model.generate(**inputs, max_new_tokens=7) >>> generation_text = processor.batch_decode(generated_ids[:, -7:], skip_special_tokens=True) >>> print(generation_text[0]) A blue bus parked on the side of a road. ```)r[rnror\rprqrrrrN)logitsrr3)lossrr hidden_states attentionsr') r isinstanceintslicer loss_functionrr2r3rrrr)r>r[rnrorprqrr\rrrrrVrzr slice_indicesrrs r)r{zFuyuForCausalLM.forwards j$*  '"7')%+        8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD&#3!/)     r(Fc tj|f||||||d|} |s |ddr d| d<d| d<| S)N)rrpr\rnrois_first_iterationrrTrorn)r.prepare_inputs_for_generationget) r>r[rrpr\rnrorrV model_inputsr?s r)rz-FuyuForCausalLM.prepare_inputs_for_generation-s=uww<   +)''"71      " 1fjjd&C&C 148L0 1,0L )r() NNNNNNNNNr)NNNNNF)rrr_tied_weights_keysrr/rr rLrr|rr~rrr r rrr{rrrs@r)rrs +,VWz .2-159.204(,26!%&*%&P P #d*P |d* P %|d2 P  t+ P &-P P (4/P $;P  t#P d P +,P  ' 'P P P ^P j" r(r)rrr,)__doc__rLr cache_utilsr generationrmodeling_outputsrrmodeling_utilsr models.auto.modeling_autor processing_utilsr utilsr r rrrconfiguration_fuyur get_loggerrloggerrr,r__all__r'r(r)rs  ))))))RRRRRRRR------222222&&&&&&jjjjjjjjjjjjjj******  H % % 6 6 6 6 6/ 6 6 6 UUUUU#UU  Up zzzzz)?zz  zz B A Ar(