tj4dZddlmZddlmZddlZddlmZddlmZ ddl m Z dd l m Z mZdd lmZdd lmZdd lmZdd lmZddlmZmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.e$j/e0Z1e"deGddeZ2e"deGddeZ3Gdd ej4Z5 dRd"ej4d#ej6d$ej6d%ej6d&ej6dzd'e7d(e7d)ee!fd*Z8Gd+d,ej4Z9Gd-d.ej4Z:Gd/d0ej4Z;Gd1d2ej4Z<Gd3d4eZ=Gd5d6ej4Z>e"Gd7d8eZ?e"d9Gd:d;e?Z@dej6fd?ZBGd@dAej4ZCGdBdCej4ZDGdDdEej4ZEe"dFGdGdHe?ZFGdIdJej4ZGe"dKGdLdMe?ZHe"dNGdOdPe?eZIgdQZJdS)SzPyTorch Idefics2 model.)Callable) dataclassN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling ModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs) AutoModel)Idefics2ConfigIdefics2PerceiverConfigIdefics2VisionConfigz| Base class for Idefics2 model's outputs that may also contain a past key/values (to speed up sequential decoding).  custom_introceZdZUdZdZejdzed<dZe dzed<dZ e ejdzed<dZ e ejdzed<dZ e ejdzed<dS)Idefics2BaseModelOutputWithPasta last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlast_hidden_statepast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__r%torch FloatTensor__annotations__r&r r'tupler(r)l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/idefics2/modeling_idefics2.pyr$r$*s  37u(4/666$(OUT\(((59M5*+d299926Je'(4/666;?u01D8?????r3r$zT Base class for Idefics2 causal language model (or autoregressive) outputs. ceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZe ejdzed<dS) Idefics2CausalLMOutputWithPastae loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlosslogitsr&r'r(r))r*r+r,r-r7r.r/r0r8r&r r'r1r(r)r2r3r4r6r6Cs"&*D% d ")))'+FE  $+++$(OUT\(((59M5*+d299926Je'(4/666;?u01D8?????r3r6cZeZdZdZdeffd Zdejdejdej fdZ xZ S)Idefics2VisionEmbeddingsaX This is a modified version of `siglip.modelign_siglip.SiglipVisionEmbeddings` to enable images of variable resolution. The modifications are adapted from [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://huggingface.co/papers/2307.06304) which allows treating images in their native aspect ratio and without the need to resize them to the same fixed size. In particular, we start from the original pre-trained SigLIP model (which uses images of fixed-size square images) and adapt it by training on images of variable resolutions. configct|j|_|j|_|j|_t j|j|j|j|jd|_ |j|jz|_ |j dz|_ |j |_ t j |j |j|_dS)Nvalid) in_channels out_channels kernel_sizestridepaddingr)super__init__ hidden_size embed_dim image_size patch_sizerConv2d num_channelspatch_embeddingnum_patches_per_side num_patches num_positions Embeddingposition_embeddingselfr; __class__s r4rDz!Idefics2VisionEmbeddings.__init__os + + +!y+?    %)Ot$F!4a7!-"$,t/A4>"R"Rr3 pixel_valuespatch_attention_maskreturncz|j\}}}}||}|ddd}||jz||jz} } t jd|jz dd|jz |j} t j || | zfd|j} |dddddf d} |dddddf d}d| z }d|z }| d}| d}t j|| jt j }t j|| jt j }|dddf|dddfz}|dddf|dddfz}t j |d }t j |d }||j}||j}t j|| d }t j|| d }|dddddf|jz|dddddfz}||d }|||d | ||d <||| z}|S)Nrrg?)devicer)size fill_valuerXdim)rXdtypeg !?)maxT)right)shaperKflatten transposerHr.arangerLrXfullsumrYfloat32clamptor] bucketizereshapeviewrP)rRrTrU batch_size_max_im_hmax_im_w patch_embeds embeddingsmax_nb_patches_hmax_nb_patches_w boundaries position_ids nb_patches_h nb_patches_wstep_hstep_w max_patches_h max_patches_w h_indices w_indicesfractional_coords_hfractional_coords_wbucket_coords_hbucket_coords_wpos_idss r4forwardz Idefics2VisionEmbeddings.forwards,8,>) Ax++L99 !))!,,66q!<< -5-H(VZVeJe*\ ) )3D4M0MVbVi   z.1AABqYeYl   ,AAAqqq!G488Q8?? +AAAq!!!G488Q8?? |#|#,11!44 ,11!44 L|7JRWR_``` L|7JRWR_``` 'aaa06!!!T'?B'aaa06!!!T'?B#k*=JPPP#k*=JPPP144\5GHH144\5GHH/*=zQUVVV/*=zQUVVV!!!!QQQ*-0IIO\]\]\]_cefefef\fLgg//*b11BIJ^JcJcdnprJsJsBt )..z2>>?$"9"9,"G"GG r3) r*r+r,r-r rDr.r/ BoolTensorTensorr __classcell__rSs@r4r:r:dsS3SSSSSS&+E$5+UM]+bgbn++++++++r3r:modulequerykeyvalueattention_maskscalingdropoutkwargsc 4t|dr*t||j}t||j}tj||dd|z}|||z}t j|dtj  |j }t j |||j }tj||} | dd} | |fS)Nnum_key_value_groupsrrr`)r\r])ptrainingr)hasattr repeat_kvrr.matmulrcr functionalsoftmaxrgrir]rr contiguous) rrrrrrrr attn_weights attn_outputs r4eager_attention_forwardrsv-..>V899%!<==<s}}Q':':;;gEL!#n4 =((2U](SSVVW\WbccL=((6?([[L,|U33K''1--88::K  $$r3c ~eZdZdZfdZ ddejdejdzdeejejdzffdZxZ S) Idefics2VisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ |j |_ tj |j|j|_tj |j|j|_tj |j|j|_tj |j|j|_d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).F)rCrDr;rErFnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutrrLineark_projv_projq_projout_proj is_causalrQs r4rDz Idefics2VisionAttention.__init__s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( / i?? i?? i??  $.$.AA r3Nr'rrVc |jdd}g|d|jR}|||dd}|||dd}|||dd}tj|j j t} | ||||||j |j |jsdn|j\} } | jg|dR} || } | | fS)z#Input shape: Batch x Time x ChannelNr`rrrrrr)rarrrlrcrrr get_interfacer;_attn_implementationrrrrrrkrr) rRr'rr input_shape hidden_shapequerieskeysvaluesattention_interfacerrs r4rzIdefics2VisionAttention.forwardsg$)#2#. 88b8$-88 ++m,,11,??II!QOO{{=))..|<<FFq!LL]++00>>HHANN(?(M K ,.E) ) %8$7     nJ#}>CC$, % % % ! \*k);;;;;;FFHH mmK00 L((r3N) r*r+r,r-rDr.rr1rrrs@r4rrsGG2/3!)!)|!) t+!) u|U\D00 1 !)!)!)!)!)!)!)!)r3rcBeZdZfdZdejdejfdZxZS)Idefics2VisionMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) rCrDr;r hidden_act activation_fnrrrEintermediate_sizefc1fc2rQs r4rDzIdefics2VisionMLP.__init__ sf  #F$569V/1IJJ9V5v7IJJr3r'rVc||}||}||}|Sr)rrr)rRr's r4rzIdefics2VisionMLP.forwards=// **=99 // r3)r*r+r,rDr.rrrrs@r4rr scKKKKKU\elr3rc6eZdZdedededeffd ZdZxZS) Idefics2MLPrEr output_sizercttj||d|_tj||d|_tj||d|_t||_dSNFbias) rCrDrr gate_projup_proj down_projract_fn)rRrErrrrSs r4rDzIdefics2MLP.__init__sv ;0ANNNy.?eLLL #4kNNNZ( r3c||||||zSr)rrrr)rRxs r4rzIdefics2MLP.forward's;~~dkk$..*;*;<STTT*$6(*    r3c|jd}|j|dd}||||d}|}||}|||z}|dddfS)Nrr)rarrepeatrrr)rR hidden_statermrresiduals r4rz-Idefics2MultiheadAttentionPoolingHead.forward=s!'*  !!*a33~~e\<HHK ~~l33 $((<"8"88 AAAqD!!r3)r*r+r,r-r rDrrrs@r4rr,sZ&&  3        " " " " " " "r3rc veZdZdeffd Zedejdejdee dej fdZ xZ S)Idefics2EncoderLayerr;cDt|j|_t ||_t j|j|j|_ t||_ t j|j|j|_ dSNr) rCrDrErFr self_attnrrr layer_norm1rr layer_norm2rQs r4rDzIdefics2EncoderLayer.__init__Ks +088<F ')     q !=0  ((77 //  =0 r3) r*r+r,r rDrr.rrrr/rrrs@r4rrJsS3SSSSSS| +,    ^r3rcleZdZdZdeffd Ze d dejdzde e de fdZ xZ S) Idefics2Encoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Idefics2EncoderLayer`]. Args: config: Idefics2Config r;ct|_tjfdt jD|_d|_dS)Nc.g|]}tSr2)r).0rnr;s r4 z,Idefics2Encoder.__init__..zs"$k$k$ka%9&%A%A$k$k$kr3F) rCrDr;r ModuleListrangenum_hidden_layerslayersgradient_checkpointingrQs `r4rDzIdefics2Encoder.__init__wsa  m$k$k$k$k5QWQiKjKj$k$k$kll &+###r3NrrrVc N|}|jD] }|||fi|} t|S)Nr%)rr)rR inputs_embedsrrr' encoder_layers r4rzIdefics2Encoder.forward~sU& ![  M)MMM ????r3r)r*r+r,r-rrDrr.rrrrrrrs@r4rrns,~,,,,,,/3@@ t+@+, @  @@@^@@@@@r3rc~eZdZUeed<dZdZdZgdZdgZ dZ dZ dZ dZ ejfdZxZS)Idefics2PreTrainedModelr;model)imagetextT)rrIdefics2PerceiverLayerIdefics2DecoderLayerr&c t|t|trt j|jdSt|trt j|j dSdSr) rC _init_weights isinstancerinitnormal_rIdefics2PerceiverResamplerones_latents)rRrrSs r4rz%Idefics2PreTrainedModel._init_weightss{ f%%% fC D D ' L & & & & &  : ; ; ' Jv~ & & & & & ' 'r3)r*r+r,rr0base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr.no_gradrrrs@r4rrs (&*#ttt#4"5N"&U]__''''_'''''r3rzK Idefics2 vision encoder model that returnss raw image embeddings. c eZdZUeed<dZeedZdeffd Z dZ dZ e e de dd ejd zd eed eezfd ZxZS)Idefics2VisionTransformerr;r)r'r(ct||j}||_t ||_t ||_tj ||j |_ | dSr) rCrDrEr;r:rrrencoderrrrpost_layernorm post_init)rRr;rFrSs r4rDz"Idefics2VisionTransformer.__init__sx    &  26::&v..  l9&:OPPP r3c|jSrrrrRs r4get_input_embeddingsz.Idefics2VisionTransformer.get_input_embeddingss r3c||_dSrrrRrs r4set_input_embeddingsz.Idefics2VisionTransformer.set_input_embeddingss r3F)tie_last_hidden_statesNrUrrVc |d}|u|jj}tj||d|z|d|zf}|tj|j}|||}| |d}t|j||}|j d ||d |}|j }| |}t| S) z patch_attention_mask (`torch.BoolTensor` of shape `(batch_size, num_patches_height, num_patches_width)`, *optional*): The attention mask for the patches. rNrrr]rXrTrUr`r;rr)rrrr2)rYr;rHr.onesriboolrXrrrlr rr%rr) rRrTrUrrmrHr'encoder_outputsr%s r4rz!Idefics2VisionTransformer.forwards;"&&q))  '/J#(: %%a((J6 %%a((J6$$ $8#:#:T`Tg#:#h#h \Xlmm 388RHH8;'/    ,84<, '/, , , ,  ,= //0ABB1BCCCCr3r)r*r+r,r r0r rr_can_record_outputsrDrr"rrrr.rrrr1rrrrs@r4rrs !   !-- 3          _E2229=(D(D$.5(D+, (D  (D(D(D^32 (D(D(D(D(Dr3rr'n_reprVc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)raexpandrk)r'r,batchnum_key_value_headsslenrs r4rrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr3cTeZdZd deddffd ZdejdejfdZdZxZ S) Idefics2RMSNormư>rrVNcttjt j||_||_dS)z> Idefics2RMSNorm is equivalent to T5LayerNorm N)rCrDrrr.r(weightvariance_epsilon)rRrErrSs r4rDzIdefics2RMSNorm.__init__sD l5:k#:#:;; #r3r'c|j}|tj}|ddd}|tj||jzz}|j||zS)Nrr`T)keepdim) r]rir.rgpowmeanrsqrtr7r6)rRr' input_dtypevariances r4rzIdefics2RMSNorm.forward s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r3cHt|jjd|jS)Nz, eps=)r1r6rar7rs r4 extra_reprzIdefics2RMSNorm.extra_reprs& )**II$2GIIIr3)r4) r*r+r,floatrDr.rrr@rrs@r4r3r3s$$$$$$$$$$;U\;el;;;;JJJJJJJr3r3ceZdZd dedzddffd Z d dejdejdejdzdejdzd edzd e e de ejejdzffd Z xZ S)Idefics2PerceiverAttentionN layer_idxrVct||_d|_|j|_|j|_|j|_|j |_ |j|j z|_ |j |_ |jdz|_ tj|j|j|jzd|_tj|j|j |jzd|_tj|j|j |jzd|_tj|j|jz|jd|_d|_dS)ziPerceiver Cross-Attention Module --> let long-form inputs be `context`, resampled embeddings be `latents`NrFr)rCrDr;rDrEresampler_n_headsrresampler_head_dimrr0rrrrrrrro_projrrRr;rDrSs r4rDz#Idefics2PerceiverAttention.__init__s  !-11 #)#= $(Nd6N$N!!'!9}d* i 0$.4=2PW\]]] i 0$2JT]2Zafggg i 0$2JT]2Zafggg i >@PW\]]] r3r contextrrvr&rc |\}}} ||dz} tj||gd} ||} || } || }| |||j|j dd} | || |j |j dd} ||| |j |j dd}t|d|}|| | ||j \} }tj|jjt$}||| | ||f|j|j|jsdn|jd|\}}||||j|jz}||}||fS) a Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: latents (`torch.Tensor`): Tensor of shape [bsz, n_latents, embed_dim] representing fixed length latents to compress to. context (`torch.Tensor`): Tensor of shape [bsz, seq, embed_dim] representing long-form context to resample. attention_mask (`torch.Tensor`, *optional*): Tensor of shape [bsz, 1, seq, n_latents] representing attention mask. position_ids (`torch.LongTensor`, *optional*): Tensor of shape [bsz, seq] representing position indices of each input token. past_key_values (`Cache`, *optional*): Tuple of tensors containing cached key and value states. output_attentions (`bool`, *optional*, defaults to `False`): Whether to return attention weights. use_cache (`bool`, *optional*, defaults to `False`): Whether to use past_key_values for caching. rr[rr&Nrr)rYr.concatrrrrlrrrcr0getattrupdaterDrrr;rrrrrrrkrH)rRr rJrrvr&rbszq_lenrn kv_seq_lenr'rrrrrrs r4rz"Idefics2PerceiverAttention.forward*s*  UAW\\^^A..  gw%7R@@@ ++g&&{{=))]++,,sE4>4=IISSTUWXYYyyj$*BDMRR\\]^`abbS*d.F VV``abdeff!$(9?KK  &*11$OOLD&(?(M K ,.E) ) %8$7      % nL#}HCC$2H % %  % % ! \"))#udnt}6TUU kk+.. L((r3rNNN)r*r+r,rrDr.r LongTensorr rrr1rrrs@r4rCrCs#*0/304(, :):):):) t+ :) &- :)  :)+,:) u|U\D00 1:):):):):):):):)r3rCceZdZdeffd Z d dejdejdejdzdejdzdedzd e e d ej fd Z xZ S) rrDct|j|_|j|_|j|_|j|_t|j|j|_ t|j|j|_ t|||_ t|j|j|_ t|j|jdz|j|j|_dS)Nr)rDrErrr)rCrDrEresampler_n_latents n_latentsresampler_depthdepth rms_norm_epsr3input_latents_norminput_context_normrCrpost_attention_layernormrrrrIs r4rDzIdefics2PerceiverLayer.__init__hs !-3+ "/"1$2BHY"Z"Z"Z"1$2BHY"Z"Z"Z3FiPPP(78HdN_(`(`(`%*$014*(    r3Nr rJrrvr&rrVc |}||}||}|jd|||d|\}}||z}|}||}||}||z}|S)a Args: latents (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` context (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states )r rJrr2)r^r_rr`r) rRr rJrrvr&rrrns r4rzIdefics2PerceiverLayer.forwardzs.))'22))'22#T^ )      W$//88((7##W$r3rS)r*r+r,rrDr.rrTr rrr/rrrs@r4rrgs #      ,/304(, )))) t+ ) &- )  )+,)  ))))))))r3rzi Idefics2 perceiver resampler model that performs `depth` blocks of cross-attention with a fixed c eZdZUeed<dZdZdZdZd fd Z e de j de j d e ede j fd ZxZS) rr;rTrVNctj|_j|_j|_j|_j|_tj tj |j|j|_ tjfdt|jD|_t#|j|j|_|dS)Nc0g|]}t|Sr2)r)ridxr;s r4rz7Idefics2PerceiverResampler.__init__..s$$f$f$fS%;FC%H%H$f$f$fr3r)rCrDrErrYrZr[r\r]rrr.r(r rrrr3normrrQs `r4rDz#Idefics2PerceiverResampler.__init__s    !- +3+ "/|EJt~t?O$P$PQQ m$f$f$f$fTYZ^ZdTeTe$f$f$fgg #D$4$:KLLL  r3rJrrc |jd|jdg|jR}t j|d|df|j|j}t j ||gd}t|j ||}|}|j D]}|||f|dd|}| |}|S) zw context (`torch.FloatTensor` of shape `(batch, seq_len, embed_dim)`): Input to the layer. rrr%r`r[r'N)rrv)r  unsqueezer.rarYr.r(r]rXcatr r;rrf)rRrJrrr latent_attention_maskcompressed_contextperceiver_layers r4rz"Idefics2PerceiverResampler.forwards0,((++22GM!4D3[t|GXGXGZGZ3[3[\\ %   # #W\\!__ 5^=QZhZo! ! ! N4I#JPRSSS2;!)    %#{  O!0"" .! ""  ""  "YY'9::!!r3)rVN)r*r+r,rr0r rrrrDrr.rrrrrrs@r4rrs $###!N"#"#" #"+, #"  #"#"#"^#"#"#"#"#"r3rc$eZdZfdZdZxZS)Idefics2Connectorctt|jj|jj|jj|jj|_t |j |_ dS)NrX) rCrDr vision_configrE text_configrrmodality_projectionr _from_configperceiver_configperceiver_resamplerrQs r4rDzIdefics2Connector.__init__su #.,8$0B*6)4 $ $ $  $>#J#J6Kb#c#c   r3c^||}|||}|S)N)rJr)rrru)rRr)rs r4rzIdefics2Connector.forwards8"667JKK"66?Rcq6rr""r3)r*r+r,rDrrrs@r4rnrnsLddddd#######r3rnz[ Idefics2 model consisting of a SIGLIP vision encoder and Mistral language decoder ceZdZdeffd ZdZdZdejdej dzdej dzfd Z e e dd ej d ejdzd eed eezfdZe e d ddejdzdej dzdejdzdedzdej dzd ej dzd ejdzdej dzdedzd eed eezfdZxZS) Idefics2Modelr;ct||jjj|_|jjj|_t|j |_ t||_ tj|j|_|jj|_|jj|_|dSr)rCrDr;rq pad_token_id padding_idx vocab_sizerrsrp vision_modelrn connectorr from_config text_modelrtrY image_seq_lenimage_token_idrrQs r4rDzIdefics2Model.__init__s    ;2?+1<5BB6CWXX*622#/0BCC#4H"k8 r3c4|jSr)rrrs r4rz"Idefics2Model.get_input_embeddingss33555r3c:|j|dSr)rr"r!s r4r"z"Idefics2Model.set_input_embeddingss ,,U33333r3 input_idsrNr)c|e||tj|jjtj|jk}|d}n||jjk}|d |j}| |j|j }| ||}|S)ar This method aims at merging the token embeddings with the image hidden states into one single sequence of vectors that are fed to the transformer LM. The merging happens as follows: - The text token sequence is: `tok_1 tok_2 tok_3 ... tok_4`. - We get the image hidden states for the image through the vision encoder (and potentially the perceiver), and that hidden state is then projected into the text embedding space. We thus have a sequence of image hidden states of size (1, image_seq_len, hidden_dim), where 1 is for batch_size of 1 image and hidden_dim is the hidden_dim of the LM transformer. - The merging happens so that we obtain the following sequence: `vector_tok_1 vector_tok_2 vector_tok_3 vector_fake_tok_around_image {sequence of image_seq_len image hidden states} vector_fake_toke_around_image vector_tok_4`. That sequence is fed to the LM. - To fit the format of that sequence, `input_ids`, `inputs_embeds`, `attention_mask` are all 3 adapted to insert the image hidden states. Nr%r`) rr.tensorr;rlongrXallrhrir]masked_scatter)rRrrr)special_image_masks r4 inputs_mergerzIdefics2Model.inputs_mergers  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H /99"==@@AUVV144]5I=K^__%445GI\]] r3rTpixel_attention_maskrrVc T|j\}}}}}||j}|j||zg|jddR}|jdd} |dkd| k} || }|ctj| d| d| d ftj |j }n8|j||zg|jddR}|| }|j j j} |d| | } | d| | } | d | | zk } |jd|| d |}|j}||| | dd}|d|jd|_|S)4 pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. pixel_attention_mask (`torch.LongTensor`, *optional*): The attention mask indicating padded regions in the image. )r]rNrr)r`rLr[rr)rYr]rX) dimensionrYstep)r`rLr&r`)rr2)rarir]rlnumelrfrr.r(rYr)rXr;rprHunfoldr}r%r~ pooler_output)rRrTrrrm num_imagesrJheightwidthnb_values_per_imagereal_images_indsrHpatches_subgridrU image_outputsr)image_featuress r4get_image_featuresz Idefics2Model.get_image_features1sr?K>P; J fe#TZ88 (|(j)@ZRSU>V&W&W #r3a Inputs fed to the model can have an arbitrary number of images. To account for this, pixel_values fed to the model have image padding -> (batch_size, max_num_images, 3, max_heights, max_widths) where max_num_images is the maximum number of images among the batch_size samples in the batch. Padding images are not needed beyond padding the pixel_values at the entrance of the model. For efficiency, we only pass through the vision_model's forward the real images by discarding the padding images i.e. pixel_values of size (image_batch_size, 3, height, width) where image_batch_size would be 7 when num_images_per_sample=[1, 3, 1, 2] and max_num_images would be 3. r!rrvr& use_cachec | | n |jj} |jr*|jjr| rt dd} | |j\} } n| |j\} } } ntd| r|t|j}|"|j |}||td||j ||fi| j }n#|!| |j|j}|||||}d | d <|jd ||||| d | }t#|j|j|j|j| S)a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection and perceiver resampling. NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz5You have to specify either input_ids or inputs_embeds)r;zMYou cannot specify both pixel_values and image_hidden_states at the same timer%)rrr)T return_dict)rrrvr&r)r%r&r'r(r)r2)r;rrrrlogger warning_oncerarr rrrrir]rXrr$r%r&r'r()rRrrrvr&rrTrr)rrrm seq_lengthrnoutputss r4rzIdefics2Model.forwardfsB"+!6IIDK>>O  BDO@@BB9MMM  #(;(Glmm m  %"9$"9,H\"g"g`f"g"g"u   ,"5"8"8tzR[Rb"8"c"c   *!..#+$7/M !%}!$/ ')%+      /%7#3!/) 3     r3r) NNNNNNNNN)r*r+r,rrDrr"r.rTrrrrr/rrr1rrr rr)r r$rrrs@r4rxrxsG ~      666444#|d*#\D0 89=11'1$.51+, 1 + + 111^1f^     .2.204(,26158<8t|t||_|jj|_t j|jj |jj d|_ |jj |_ | dSr) rCrDrxrr;rrrrqrEr|lm_headrrQs r4rDz)Idefics2ForConditionalGeneration.__init__s    "6** "k8y!3!?ASA^ejkkk  ,7 r3c>|jjSr)rrrrs r4rz5Idefics2ForConditionalGeneration.get_input_embeddingssz$99;;;r3cD|jj|dSr)rrr"r!s r4r"z5Idefics2ForConditionalGeneration.set_input_embeddingss! 22599999r3NrTrrrVc ,|jjd||d|S)r)rTrr2)rr)rRrTrrs r4rz3Idefics2ForConditionalGeneration.get_image_featuress5-tz, %