Ph`ddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*edGdde jVZ, dDde jVdejZdejZdejZdeejZde.de.fdZ/Gdde jVZ0ee!d !Gd"d#eZ1Gd$d%e jVZ2Gd&d'e jVZ3Gd(d)e jVZ4e jje,d*Z6Gd+d,eZ7Gd-d.e jVZ8e!Gd/d0eZ9e!Gd1d2e9Z:e!Gd3d4eZ;Gd5d6e jVZee!d=!Gd>d?eZ?e!d@!GdAdBe;eZ@gdCZAy)EN) dataclass)CallableOptionalUnion)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple torch_int)check_model_inputs) AutoModel)InternVLConfigInternVLVisionConfigRMSNormc,eZdZdfd ZdZdZxZS)InternVLVisionRMSNormct|tjt j ||_||_y)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s h/var/www/html/saasai/venv/lib/python3.12/site-packages/transformers/models/internvl/modeling_internvl.pyr#zInternVLVisionRMSNorm.__init__.s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor&float32powmeanrsqrtr)r()r* hidden_states input_dtypevariances r.forwardzInternVLVisionRMSNorm.forward6sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r/c^t|jjd|jS)Nz, eps=)tupler(shaper)r*s r. extra_reprz InternVLVisionRMSNorm.extra_repr=s* ))*+6$2G2G1HIIr/)gư>)__name__ __module__ __qualname__r#r<rA __classcell__r-s@r.r r ,s$;Jr/r modulequerykeyvalueattention_maskscalingdropoutc |}|} tj||jdd|z} |#|ddddddd|jdf} | | z} tj j | d} tj j| ||j} tj| | } | jddj} | | fS)Nrrr1dim)ptrainingr) r&matmul transposer?r$ functionalsoftmaxrMrS contiguous) rGrHrIrJrKrLrMkwargs key_states value_states attn_weights causal_mask attn_outputs r.eager_attention_forwardr_AsJL<<z';';Aq'ABWLL!$Q1.D 0@0@0D.D%DE #k1 ==((2(>L==((6??([L,,|\:K''1-88:K  $$r/cteZdZdZdeffd Z ddejdeejde e fdZ xZ S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ |j}|j}d|_ tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j|_|dkDrtj*|ntj,|_ |rt/|jntj,|_|rt/|j|_ytj,|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r"r#rbr+ embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr$Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityr q_normk_norm)r*rb proj_dropoutqk_normr-s r.r#z InternVLVisionAttention.__init___s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!900 $$ii0NU[UjUjk ii0NU[UjUjk ii0NU[UjUjk " $..$.. I>JQ>N"**\":TVT_T_Ta?F+DNN;BKKM ?F+DNN; BKKM r/r9rKrYc |j\}}}|j|}|j|}|j|} |j |}|j |}|j |||j|jjdd}|j |||j|jjdd}| j|||j|jjdd} t} |jjdk7rt|jj} | |||| |f|jsdn |j |j"dd|\} } | j |||j$} |j'| } |j)| } | | fS)NrreagerF)rMrLro)sizerrrsrtrxryreshaperhrirUviewr_rb_attn_implementationrrSrlrkrfrurm)r*r9rKrY batch_sizeseq_len_ query_statesrZr[attention_interfacer^r\outputs r.r<zInternVLVisionAttention.forward{s "/!3!3!5 GQ{{=1 [[/ {{=1 {{<0 [[, #++JQUQ^Q^_iijkmno '' GT^^T]][eefgijk #((Wdnndmm\ffghjkl (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$2H2HJJ %  % ! \"))*gt~~N &&{3((0|##r/N) rBrCrD__doc__rr#r&Tensorrrrr<rErFs@r.rara\sN5Z3Z>26'$||'$!.'$+, '$r/raz7 Class for outputs of [`InternVLVisionModel`]. ) custom_introceZdZdZy)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)rBrCrDrr/r.rrsr/rcZeZdZdZfdZdej dej fdZxZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c^t||j|j}}|j|j }}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_||_tj|||||_ y)Nrr) kernel_sizestride) r"r# image_size patch_size num_channelsr+ num_patches patch_shaper$Conv2d projection) r*rbrrrr+rrr-s r.r#z&InternVLVisionPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L+:^hir/ pixel_valuesreturnc|j\}}}}||jk7r td|j|}|jd|jd}}|j dj dd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r?rrjrflattenrU) r*rrrheightwidth embeddings patch_height patch_widths r.r<z%InternVLVisionPatchEmbeddings.forwards2>2D2D/ L&% 4,, ,w __\2 $.$4$4Q$79I9I!9Lk ''*44Q: L+666r/) rBrCrDrr#r&rr<rErFs@r.rrs) j 7ELL 7U\\ 7r/rceZdZdZdeddffd Zdejdededejfd Z d d ejd e ejdejfd Z xZ S)InternVLVisionEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rbrNc2t|tjt j dd|j |_|jr:tjt j dd|j |_ nd|_ t||_ |j|_ t|jtj j"r |jn|j|jf|_|jj$}|j&r=tjt j d|dz|j |_nd|_tj*|j,|_y)Nr)r"r#r$r%r&zerosr+ cls_tokenuse_mask_token mask_tokenrpatch_embeddingsr isinstancer collectionsabcIterabler use_absolute_position_embeddingsposition_embeddingsrvhidden_dropout_probrM)r*rbrr-s r.r#z!InternVLVisionEmbeddings.__init__s$ ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO =f E ++&++[__-E-EF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r/rrrc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j dz} ||j dz} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr1r?rrbicubicF)rmode align_cornersrP)r?rr&jit is_tracingrrrpermuter$rV interpolatercat) r*rrrr num_positionsclass_pos_embedpatch_pos_embedrQ new_height new_widthsqrt_num_positionss r.interpolate_pos_encodingz1InternVLVisionEmbeddings.interpolate_pos_encodingsj!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"tq11 T__Q// &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr/rbool_masked_posc|j\}}}}|j|\}\}}|j\} } }|K|jj | | d} |j dj | } |d| z z| | zz}|jj | dd} tj| |fd}|j||j|||z}|j|}|||ffS)Nr1rrP) r?rrrexpand unsqueezetype_asrr&rrrrM)r*rrrrrrrrrr mask_tokensw cls_tokenss r.r<z InternVLVisionEmbeddings.forwards +001fe262G2G 2U/ /\;!+!2 GQ  &//00WbIK))"-55kBA#q1u- a?J^^**:r2> YY J7Q?  # # /#d&C&CJPVX]&^^J\\*- L+666r/r)rBrCrDrrr#r&rintrr BoolTensorr<rErFs@r.rrs >3>>,&D5<<&D&DUX&D]b]i]i&DV7;7ll7"%"2"237  7r/rcVeZdZfdZdej dej fdZxZS)InternVLVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r"r#rbr hidden_act activation_fnr$rpr+intermediate_sizefc1fc2r*rbr-s r.r#zInternVLVisionMLP.__init__9sd  #F$5$5699V//1I1IJ99V55v7I7IJr/r9rcl|j|}|j|}|j|}|Sr)rrr)r*r9s r.r<zInternVLVisionMLP.forward@s4/ **=9 / r/)rBrCrDr#r&rr<rErFs@r.rr8s$KU\\ellr/r) layer_normrms_normceZdZdZdeddffd Zdejdee eje ejejfffdZ xZ S)InternVLVisionLayerz?This corresponds to the Block class in the timm implementation.rbrNct||j|_d|_t ||_t ||_t|j|j|j|_ t|j|j|j|_ |j}tj |t#j$|jzd|_tj |t#j$|jzd|_tj*|j,|_y)Nrr,T) requires_grad)r"r#chunk_size_feed_forward seq_len_dimra attentionrmlpNORM2FN norm_typer+layer_norm_epslayernorm_beforelayernorm_afterlayer_scale_init_valuer$r%r&r'lambda_1lambda_2rvrrM)r*rb init_valuesr-s r.r#zInternVLVisionLayer.__init__Ms '-'E'E$08$V, '(8(8 9&:L:LRXRgRg h&v'7'789K9KQWQfQfg33  [5::f>P>P3Q%Qaef  [5::f>P>P3Q%Qaef zz&"<"<= r/r9c|j|j|\}}|j|z}||z}|j|}|j |}|j |}|j |j |z}||z}|Sr)rrrrrrMr)r*r9attention_outputr layer_outputs r.r<zInternVLVisionLayer.forward\s#nn  ! !- 0 ! ==+;;)=8 ++M: xx - ||L1 == $==<7L$m3 r/) rBrCrDrrr#r&rrr>r<rErFs@r.rrJsZI >3 > >|| uU\\"E%,, *D$EE Fr/rc`eZdZdeddffd Zedejdee e ffdZ xZ S)InternVLVisionEncoderrbrNct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r"r#rbr$ ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r*rbir-s r.r#zInternVLVisionEncoder.__init__ysU  ]]vOgOgIh#iIhA$7$?Ih#ij &+#$jsA#r9cL|jD] }||} t|S)N)last_hidden_state)rr )r*r9 layer_modules r.r<zInternVLVisionEncoder.forwards. !JJL(7M'+  r/) rBrCrDrr#rr&rrr>r r<rErFs@r.rrxsK,3,,   ||   uo% &    r/rcVeZdZUeed<dZdZdZdgZdZ dZ dZ dZ e edZfdZxZS)InternVLVisionPreTrainedModelrbinternvl_visionrTr)r9 attentionscVt||t|tr|jj j |j$|jj j |j%|jj j yyt|trs|jj j|jj|jj j|jjyy)zInitialize the weightsN)r" _init_weightsrrrdatazero_rrrrfill_rbrr)r*rGr-s r.rz+InternVLVisionPreTrainedModel._init_weightss f% f6 7    ! ! ' ' )  ,!!&&,,.))5**//5576  3 4 OO & &t{{'I'I J OO & &t{{'I'I J5r/)rBrCrDr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrra_can_record_outputsrrErFs@r.rrsV  )$O&*#./N"&-- K Kr/rc eZdZdeddffd ZdZee d dejde ejde e effdZxZS) InternVLVisionModelrbrNc2t||||_t||_t ||_|jrtjn*tj|j|j|_ |jy)Nr)r"r#rbrrrencoderuse_mean_poolingr$rw LayerNormr+r layernorm post_initrs r.r#zInternVLVisionModel.__init__so   26:,V4 $44BKKM",,vGYGY_e_t_t:u  r/c.|jjSr)rrr@s r.get_input_embeddingsz(InternVLVisionModel.get_input_embeddingss///r/rrc|j||\}}|j|}|d}|j|}t||j|j S)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr)rr9r)rrrrr9r)r*rrembedding_outputrencoder_outputssequence_outputs r.r<zInternVLVisionModel.forwardse#oolOo\!,,'78)!,..93-)77&11  r/r)rBrCrDrr#rrrr&rrrrr>rr<rErFs@r.rrss 3  07; ll "%"2"23  u:: ;   r/rc8eZdZUeed<dZdZdZdZdZ dZ dZ dZ y)InternVLPreTrainedModelrbTpast_key_valuesN) rBrCrDrrrr_skip_keys_device_placementr r _can_compile_fullgraphr r rr/r.rrs7 &*#"3N!"&r/rc*eZdZdeffd ZdZxZS)InternVLMultiModalProjectorrbc*t|tj|jj t d|jz dzz|_tj|jj t d|jz dzz|jj |_ t|j|_tj|jj |jj |_y)Nrr)r"r#r$r vision_configr+rdownsample_ratiorrp text_configlinear_1rprojector_hidden_actactlinear_2rs r.r#z$InternVLMultiModalProjector.__init__s ,,v';';'G'G#aRXRiRiNiJjnoJo'op  , ,s1v7N7N3N/OST/T TV\VhVhVtVt  &556 &"4"4"@"@&BTBTB`B`a r/c|j|}|j|}|j|}|j|}|Sr)rr)r+r,)r*image_featuresr9s r.r<z#InternVLMultiModalProjector.forwards@7  m4 /  m4 r/)rBrCrDrr#r<rErFs@r.r$r$sb~br/r$zM Base class for InternVL outputs, with hidden states and attentions. c:eZdZUdZdZeejed<y)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) rBrCrDrr1rr& FloatTensorrrr/r.r0r0s 8<%"3"34;r/r0zx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. c.eZdZddiZdeffd ZdZdZdZdZ dd e jd e e eeefd e efd Zd e j$de jde jfdZee dd e e j$d e e jde e j,de e j$de ede e jd e e eeefd e ede e j$deede eeffdZdde j,defdZxZS) InternVLModelzlanguage_model.modellanguage_modelrbct||tj|j|_t ||_tj|j|_ |jyr) r"r#r from_configr& vision_towerr$multi_modal_projectorr(r5rrs r.r#zInternVLModel.__init__sY  %11&2F2FG%@%H"'33F4F4FG r/c6|jjSr)r5rr@s r.rz"InternVLModel.get_input_embeddingss""7799r/c:|jj|yr)r5set_input_embeddingsr*rJs r.r<z"InternVLModel.set_input_embeddingss 007r/c||_yrr5r*decoders r. set_decoderzInternVLModel.set_decoder"s %r/c|jSrr?r@s r. get_decoderzInternVLModel.get_decoder%s"""r/rvision_feature_layervision_feature_select_strategyc ||n|jj}||n|jj}|j|j}|jj }|dk(r|j |j}n|j|j|}|dk(r|ddddddf}|jd}t|dz}|jd} |j| ||d}|j|| }|j| d|jd}|j|}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. N)r3r1)rdefaultrrr) scale_factor)rbrErFr4r3r'r8r vision_modelr9r?rr pixel_shuffler9) r*rrErFrYr'vision_featureschannels feature_sizers r.get_image_featuresz InternVLModel.get_image_features(sX&%9$D $++JjJj  .9 +;; ' $TZZ8 ;;77 2 %"//\/J\\O"//\/JXXYmnO )Y 6-aQh7O#((+8S=) $**1- *11*lLZ\],,_K[,\*11*b/BWBWXZB[\44_Er/ input_ids inputs_embedsr.cP|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jdj|j|j }|jd|jdz}||j|jk7rtd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )r3devicer1rrz6Image features and image tokens do not match: tokens: z , features )rr&tensorrbimage_token_idlongrSallsumr expand_asr4r?numelrj)r*rPrQr.special_image_maskn_image_tokensn_image_featuress r.get_placeholder_maskz"InternVLModel.get_placeholder_mask^s  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1/99"=GG VYYZgZnZno)//2^5I5I!5LL + , 2 2 48L8L8N NHHXXcdtcuv "!r/rK position_idsr cache_positionrYrc 8||n|jj}||n|jj}|du|duz r td||j |}|`|j |||} | j |j|j} |j||| } |j| | }|jd||||| d| } t| j| j| j| j | SdS)Nz:You must specify exactly one of input_ids or inputs_embedsrrErF)rQr.)rKr_r rQr`)rr r9rr1r)rbrErFrjrrOr4rSr3r^masked_scatterr5r0rr r9r)r*rPrrKr_r rQrErFr`rYr.r[outputss r.r<zInternVLModel.forwardvsj %9$D $++JjJj  .9 +;; '  -t"; <YZ Z  7D557 BM  #!44)%9/M5N ,..}/C/C]EXEXYN!%!:!:~";" *889K^\M%$%% )%+')    +%77#33!//))2>2J   QU   r/rLrIc |j\}}}}||zdk7s||zdk7r td|j||t||zt||z }|j ddddj }|j|t||zt||zt||dzz }|j ddddj }|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rrjrrrrX)r*rLrIrrrrMs r.rKzInternVLModel.pixel_shuffles/>.B.B.D+ E68 L A %)=)Bjk k*.. s6L#893x,?V;W *11!Q1=HHJ*.. F\12C 8L4MsS[_kmn_nSoOp  *11!Q1=HHJr/NN) NNNNNNNNN)r) rBrCrD_checkpoint_conversion_mappingrr#rr<rBrDr&r2rrrliststrrO LongTensorr^rrrr rrr>r0r<floatrKrErFs@r.r4r4 s '=>N%O"~:8&# AE8< 4''4'uS$s)^'<=4)1 4l"))":?:K:K"]b]n]n"015481537+/59@D8<597 E,,-7 u0017 !. 7 u//0 7 "% 7   1 127 'uS$s)^'<=7 )1 7 !!1!127 +,7  u11 27 7 r!U\\!!r/r4zT Base class for InternVL causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejed<dZee ejed<dZeejed<y) InternVLCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsr r9rr1)rBrCrDrrnrr&r2rror r r9r>rr1rr/r.rmrms )-D(5$$ %,*.FHU&& '.'+OXe_+8>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```N) rPrrKr_r rQrErFr`rr)rorru)rnror r9rr1r)rbrErFrtrrslicerr loss_functionr(rurmr r9rr1)r*rPrrKr_r rQrErFrr`rrrYrdr9 slice_indicesrorns r.r<z(InternVLForConditionalGeneration.forward1s7l%9$D $++JjJj  .9 +;; ' $**  %)%+'!5+I)#      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD.#33!//)) ' ; ;   r/c Nt ||f|||||d|} |ddk(r|| d<| S)N)r rQrKr`rrr)r"prepare_inputs_for_generation) r*rPr rQrrKr`rrY model_inputsr-s r.rz>InternVLForConditionalGeneration.prepare_inputs_for_generationsVw<  +')))    !  !,8L (r/rf) NNNNNNNNNNrN)NNNNNN)&rBrCrDrg_tied_weights_keysrr#rr<r$ModuleryrBrDr&r2rrrrhrirOpropertyr5r8r9rrrjrr rrr>rmr<rrErFs@r.rqrqs_"8-"?#, &" ++~ 1/ryy(( AE8<  ''  'uS$s)^'<=  )1  ))''0015481537+/59@D8<-15934.2\ E,,-\ u001\ !. \ u//0 \ "% \   1 12\ 'uS$s)^'<=\ )1 \ ))*\ !!1!12\ c5<</0\ ell+\ +,\  u44 5\ \ Br/rq)rrrr4rq)r~)Bcollections.abcr dataclassesrtypingrrrr&torch.nnr$ activationsr cache_utilsr generationr integrationsr modeling_layersr modeling_outputsr rrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrautorconfiguration_internvlrrrr rrkr_rarrrrrrrrrrrr$r0r4rmrq__all__rr/r.rs.!,, ! )79ddF&aa/HY'JBIIJ(J6% II% <<% % << % U\\* %  %%6F$biiF$R  +E  !7BII!7L[7ryy[7|  3H I+4+\ BII (KOKK<' 7' ' T 'o ' '"))$  <"9 <  < A+A  AH  <[<  <0 u'>u  up r/