..N)__name__ __module__ __qualname____firstlineno__r__annotations__base_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_no_split_modules_skip_keys_device_placementr'__static_attributes__r*r)r&rr#s< &*#"&N"3 ?r)rzt The Fuyu model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introcD^\rSrSrSS0rS\4U4SjjrSrSrSr S r S \ RS \ \ RS \ RS \ R4SjrS\ R4SjrS\ R"S\ RS\ R4Sjr\S S\ R"S\ RS\ RS\\ RS\\ R"S\\S\\ RS\\S\\S\\S\\S \\\44Sjj5rSrU=r$)! FuyuModel;zlanguage_model.modellanguage_modelrc>[TU]U5 URUlURR Ul[ R"UR5Ul[R"URUR-UR-UR5UlSUlUR!5 g)NF)super__init__ pad_token_idr# text_config vocab_sizer from_configr=rr patch_size num_channels hidden_sizevision_embed_tokensgradient_checkpointing post_initr$r __class__s r&r@FuyuModel.__init__Cs  !.. ,,77'33F4F4FG#%99    1 1 1F4G4G GI[I[$  ',# r)c6URR5$N)r=get_input_embeddingsr$s r&rPFuyuModel.get_input_embeddingsPs""7799r)c:URRU5 grO)r=set_input_embeddingsr$values r&rTFuyuModel.set_input_embeddingsSs 007r)cXlgrOr=r$decoders r& set_decoderFuyuModel.set_decoderVs%r)cUR$rOrYrQs r& get_decoderFuyuModel.get_decoderYs"""r)word_embeddingscontinuous_embeddingsimage_patch_input_indicesreturnc URS[U5:Xd)[S[U5<SURS<35eUR5n[ URS5Hn[ R "X5S:SS9SnX5UnURSX%RS:a-[SX%R<SUR<SUS 35eX%URUR5XEU4'M U$) ayThis function places the continuous_embeddings into the word_embeddings at the locations indicated by image_patch_input_indices. Different batch elements can have different numbers of continuous embeddings. Args: word_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Tensor of word embeddings. continuous_embeddings (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Tensor of continuous embeddings. The length of the list is the batch size. Each entry is shape [num_image_embeddings, hidden], and num_image_embeddings needs to match the number of non-negative indices in image_patch_input_indices for that batch element. image_patch_input_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Tensor of indices of the image patches in the input_ids tensor. rz7Batch sizes must match! Got len(continuous_embeddings)=z and word_embeddings.shape[0]=T)as_tuplezGNumber of continuous embeddings continuous_embeddings[batch_idx].shape=zA does not match number of continuous token ids src_indices.shape=z in batch element .) shapelen ValueErrorclonerangetorchnonzerotodevice)r$rarbrcoutput_embeddings batch_idx dst_indices src_indicess r&gather_continuous_embeddings&FuyuModel.gather_continuous_embeddings\sI( %%a(C0E,FFJs3H/I.KKjQ`QfQfghQiPkl ,11344Q78I --(A(LPQ(Q\`abcdK4>{KK  #&;&F&L&LQ&OO ^7L7W7]7]6_`I6A6G6G5II[\e[ffgi9N8XYd8e8h8h!((9 4 59 ! r) pixel_valuesc UVs/sHPnURURURRR55R S5PMR nnU$s snf)z Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. r)rHrordtypesqueeze)r$rwkwargspatchpatch_embeddingss r&get_image_featuresFuyuModel.get_image_featuressc& %  $ $UXXd.F.F.M.M.S.S%T U ] ]^_ `%     sAA input_ids inputs_embedsimage_featuresc JUcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnUR5nURS5RU5RUR 5nURSURS-nX$R5UR5:wa[SUSU35eU$)z Obtains multimodal placeholdr mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. ryrprrz6Image features and image tokens do not match: tokens: z , features )rPrmtensorrimage_token_idlongrpallsum unsqueeze expand_asrorhnumelrj)r$rrrspecial_image_maskn_image_tokensn_image_featuress r&get_placeholder_maskFuyuModel.get_placeholder_masks  !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H +//1/99"=GG VYYZgZnZno)//2^5I5I!5LL  , 2 2 48L8L8N NHHXXcdtcuv "!r) image_patchesimage_patches_indicesattention_mask position_idsr use_cacheoutput_attentionsoutput_hidden_states return_dictc U bU OURRn U bU OURRn UbUOURRnU bU OURRn UbUb [ S5eUbUR upOUbUR upnO [ S5eUciUb URO URnUbUR5OSn[R"UUU-[RUS9nURS5nUc URR5"U5nUblURU5n[R "USS9R#URUR$5nUR'XUS9nUR)UU5nUR"SUUUUU U UU S.U D6nU$) a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. zDYou cannot specify both input_ids and inputs_embeds at the same timez4You have to specify either input_is or inputs_embedsrr)dim)rr)rrrrrrrrr*)rrrruse_return_dictrjrhrpget_seq_lengthrmarangerrr=rPr~catroryrmasked_scatter)r$rrrrrrrrrrrr{ batch_size seq_length_rppast_key_values_lengthr}routputss r&forwardFuyuModel.forwards.2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "%.__ "J  &(5(;(; %JAST T  )2)>Y%%MDXDXFIXId_%C%C%Ejk " <<& 5K(KSXS]S]flL(11!4L   //DDFyQM  $#66}E $yy)9qADD]EYEY[h[n[no !%!:!:GW";" *889KM]^M%%  ')%+/!5#    r))rIr=r#rHrC) NNNNNNNNNNN)r+r,r-r._checkpoint_conversion_mappingrr@rPrTr\r_rmTensorlistru FloatTensorr~ LongTensorrr rrboolrtupler rr8 __classcell__rLs@r&r;r;;s '=>N%O" z :8&#*!*! $ELL1*!$)<< *!  *!X  u/@/@  "))":?:K:K"]b]n]n"0'+&*.21537+/59$(,0/3&*F##F||F %|| F !. F u//0 F"%F  1 12FD>F$D>F'tnFd^F u,, -FFr)r;zz Fuyu Model with a language modeling head on top for causal language model conditioned on image patches and text. c!^\rSrSrSSSS.rS/rS\4U4SjjrS rS r S r S r \ \ SS \RS\R S\R S\\R S\\RS\\S\\R&S\\S\\R S\\S\\S\\S\\S\\\44Sjj55rSU4SjjrSrU=r$) FuyuForCausalLMzmodel.language_modelzmodel.vision_embed_tokenslm_head)z^language_model.modelz^vision_embed_tokensz^language_model.lm_headzlm_head.weightrc>[TU]U5 [U5Ul[R "UR RUR RSS9Ul UR5 g)NF)r ) r?r@r;modelrrrBrGrCrrJrKs r&r@FuyuForCausalLM.__init__sS  v& yy!3!3!?!?ASASA^A^ejk  r)c6URR5$rO)rrPrQs r&rP$FuyuForCausalLM.get_input_embeddings szz..00r)c:URRU5 grO)rrTrUs r&rT$FuyuForCausalLM.set_input_embeddingss ''.r)c:URRU5 grO)rr\rZs r&r\FuyuForCausalLM.set_decoders w'r)c6URR5$rO)rr_rQs r&r_FuyuForCausalLM.get_decoderszz%%''r)rrrrrrrrlabelsrrrlogits_to_keeprdc ZU bU OURRn U bU OURRn UbUOURRnU bU OURRn UR UUUUUUUU U USS9 nUSn[ U [5(a [U *S5OU nURUSS2USS245nSnU b3UR"SUXRRRS.UD6n[UUURURUR S9$)ap image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Examples: ```python >>> from transformers import FuyuProcessor, FuyuForCausalLM >>> from PIL import Image >>> import requests >>> processor = FuyuProcessor.from_pretrained("adept/fuyu-8b") >>> model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b") >>> url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" >>> image = Image.open(requests.get(url, stream=True).raw) >>> prompt = "Generate a coco-style caption.\n" >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> generated_ids = model.generate(**inputs, max_new_tokens=7) >>> generation_text = processor.batch_decode(generated_ids[:, -7:], skip_special_tokens=True) >>> print(generation_text[0]) A blue bus parked on the side of a road. ```NT) rrrrrrrrrrrr)logitsrrC)lossrr hidden_states attentionsr*)rrrrrrrintslicer loss_functionrBrCr rrr)r$rrrrrrrrrrrrrr{rr slice_indicesrrs r&rFuyuForCausalLM.forwardsMj2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKSV8W8W~ot4]k mA}a,?@A  %%f9P9P9[9[_eD&#33!//))   r)c ^>[T U]"U4UUUUUUS.UD6n USS:wa SU S'SU S'U $)N)rrrrrcache_positionrrr)r?prepare_inputs_for_generation) r$rrrrrrrr{ model_inputsrLs r&r-FuyuForCausalLM.prepare_inputs_for_generationvsbw<   +)''"7)     !  !48L0 1,0L )r))rr) NNNNNNNNNNNNr)NNNNNN)r+r,r-r.r_tied_weights_keysrr@rPrTr\r_r r rmrrrrrrrrrr rrr8rrs@r&rrs"8 ;#,&" ++z 1/(('+&*.21537+/59$()-,0/3&*()[ ##[ ||[  %|| [ !. [ u//0 [ "%[   1 12[ D>[ &[ $D>[ 'tn[ d^[ ! [ u,, -![ [ @"r)r)rrr;)__doc__typingrrrmtorch.utils.checkpointr cache_utilsr generationrmodeling_outputsr modeling_utilsr models.auto.modeling_autor utilsr r rconfiguration_fuyur get_loggerr+loggerrr;r__all__r*r)r&rs"  )6-2>>*   H %?/??. u#u  up V)?V  Vr Br)