tj+ddlZddlmZddlmZddlZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-edGddej.Z/ dGdej.dej0dej0dej0dej0dzde1de1e2zfd Z3Gd!d"ej.Z4e!d#$eGd%d&eZ5Gd'd(ej.Z6Gd)d*ej.Z7Gd+d,ej.Z8ej9e/d-Z:Gd.d/eZ;Gd0d1ej.Ze!Gd6d7eZ?Gd8d9ej.Z@e!d:$eGd;dd?e?ZBe!d@$eGdAdBeZCe!dC$GdDdEe?eZDgdFZEdS)HN)Callable) dataclass)initialization)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringtorch_compilable_check torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) AutoModel)InternVLConfigInternVLVisionConfigRMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) InternVLVisionRMSNormư>epsreturnNcttjt j||_||_dS)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizer# __class__s l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/internvl/modeling_internvl.pyr'zInternVLVisionRMSNorm.__init__/sD l5:k#:#:;; # hidden_statesc|j}|tj}|ddd}|tj||jzz}|j||zS)NrT)keepdim) dtypetor*float32powmeanrsqrtr-r,)r.r3 input_dtypevariances r1forwardzInternVLVisionRMSNorm.forward7s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r2cHt|jjd|jS)Nz, eps=)tupler,shaper-r.s r1 extra_reprz InternVLVisionRMSNorm.extra_repr>s& )**II$2GIIIr2)r") __name__ __module__ __qualname__floatr'r*Tensorr?rD __classcell__r0s@r1r!r!-s$$$$$$$$$$;U\;el;;;;JJJJJJJr2r!modulequerykeyvalueattention_maskscalingdropoutc |}|} tj||dd|z} || |z} tj| d} tj| ||j} tj| | } | dd} | | fS)Nrrr5dim)ptrainingr) r*matmul transposer( functionalsoftmaxrSrX contiguous) rMrNrOrPrQrRrSkwargs key_states value_states attn_weights attn_outputs r1eager_attention_forwardrcBsJL<z';';Aq'A'ABBWLL!#n4 =((2(>>L=((6?([[L,|\::K''1--88::K  $$r2cfeZdZdZdeffd Z d dejdejdzdee fdZ xZ S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigct||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ |j |_ |j }|j }d|_ tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|_|dkrtj|ntj|_ |rt/|jntj|_|rt/|jntj|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r&r'rfr/ embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr(Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityr!q_normk_norm)r.rf proj_dropoutqk_normr0s r1r'z InternVLVisionAttention.__init___s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( !'!90 $i0NU[Ujkkk i0NU[Ujkkk i0NU[Ujkkk " $.$. I I>JQ>N>N"*\":":":TVT_TaTa?FY+DN;;;BKMM ?FY+DN;;;BKMM r2Nr3rQr^c r|\}}}||}||}||} ||}||}||||j|j dd}||||j|j dd}| |||j|j dd} tj |j jt} | |||| |f|jsdn|j|jdd|\} } | |||j} || } || } | | fS)NrrrLF)rSrRrs)sizervrwrxr|r}reshaperlrmrZviewr get_interfacerf_attn_implementationrcrXrprorjryrq)r.r3rQr^ batch_sizeseq_len_ query_statesr_r`attention_interfacerbraoutputs r1r?zInternVLVisionAttention.forward{s "/!3!3!5!5 GQ{{=11 [[// {{=11 {{<00 [[,, #++JQUQ^__iijkmnoo '' GT^T][[eefgijkk #((Wdndm\\ffghjkll (?(M K ,.E) ) %8$7      % $}HCC$2HJ % %  % % ! \"))*gt~NN &&{33((00|##r2N) rErFrG__doc__rr'r*rIrrr?rJrKs@r1rere\s55Z3ZZZZZZ>/3'$'$|'$ t+'$+, '$'$'$'$'$'$'$'$r2rez7 Class for outputs of [`InternVLVisionModel`].  custom_introceZdZdZdS)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)rErFrGrr2r1rrsr2rcFeZdZdZfdZdejdejfdZxZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct|j|j}}|j|j}}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_||_tj |||||_ dS)Nrr) kernel_sizestride) r&r' image_size patch_size num_channelsr/ num_patches patch_shaper(Conv2d projection) r.rfrrrr/rrr0s r1r'z&InternVLVisionPatchEmbeddings.__init__s !'!2F4EJ $*$79Kk !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&)L+:^hiiir2 pixel_valuesr$c|j\}}}}||jkrtd|||jjj}|ddd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rr) rBrrnrr8r,r7flattenrZ)r.rrrheightwidth embeddingss r1r?z%InternVLVisionPatchEmbeddings.forwards2>2D/ L&% 4, , ,w __\__T_5K5Q%R%RSS ''**44Q:: r2) rErFrGrr'r*rIr?rJrKs@r1rrsm j j j j j EL U\        r2rceZdZdZdeddffd Zdejdededejfd Z d d ejd ej dzdejfd Z xZ S)InternVLVisionEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rfr$Ncttjt jdd|j|_|jr3tjt jdd|j|_ nd|_ t||_ |j |_ t|jtjjr|jn |j|jf|_|j j}|jr6tjt jd|dz|j|_nd|_tj|j|_dS)Nr)r&r'r(r)r*zerosr/ cls_tokenuse_mask_token mask_tokenrpatch_embeddingsr isinstancer collectionsabcIterabler use_absolute_position_embeddingsposition_embeddingsrzhidden_dropout_probrS)r.rfrr0s r1r'z!InternVLVisionEmbeddings.__init__s( ek!Q8J&K&KLL   # l5;q!V=O+P+PQQDOO"DO =f E E +&+[_-EFF 8F  #V%67  +7  2 ,')|EK;QR?TZTf4g4g'h'hD $ $'+D $z&"<== r2rrrc|jddz }|jjddz }tjs||kr ||kr|jS|jddddf}|jddddf}|jd}||jdz} ||jdz} t |dz} |d| | |}|dddd}tj || | fdd }|dddd dd|}tj ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr5r?rrbicubicF)rmode align_cornersrU)rBrr*jit is_tracingrrrpermuter(r[ interpolatercat) r.rrrr num_positionsclass_pos_embedpatch_pos_embedrV new_height new_widthsqrt_num_positionss r1interpolate_pos_encodingz1InternVLVisionEmbeddings.interpolate_pos_encodings|!&q)A- 06q9A= y##%% ,+*F*F6UZ??+ +2111bqb592111abb59r"tq11 T_Q// &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr2rbool_masked_posc|j\}}}}||}|\}}}|R|j||d} |d| } |d| z z| | zz}|j|dd} tj | |fd}|j || |||z}| |}|S)Nr5rrU) rBrrrexpand unsqueezetype_asrr*rrrrS) r.rrrrrrrr mask_tokensw cls_tokenss r1r?z InternVLVisionEmbeddings.forwards +01fe**<88 !+!2!2 GQ  &/00WbIIK))"--55kBBA#q1u- a?J^**:r2>> Y J7Q???  # /#d&C&CJPVX]&^&^^J\\*-- r2r) rErFrGrrr'r*rIintr BoolTensorr?rJrKs@r1rrs >3>>>>>>>,&D5<&D&DUX&D]b]i&D&D&D&DV48l)D0  r2rcBeZdZfdZdejdejfdZxZS)InternVLVisionMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) r&r'rfr hidden_act activation_fnr(rtr/intermediate_sizefc1fc2r.rfr0s r1r'zInternVLVisionMLP.__init__8sf  #F$569V/1IJJ9V5v7IJJr2r3r$c||}||}||}|Sr)rrr)r.r3s r1r?zInternVLVisionMLP.forward?s=// **=99 // r2)rErFrGr'r*rIr?rJrKs@r1rr7scKKKKKU\elr2r) layer_normrms_normceZdZdZdeddffd ZdejdeejeejejfzfdZ xZ S)InternVLVisionLayerz?This corresponds to the Block class in the timm implementation.rfr$Nct|j|_d|_t ||_t ||_t|j |j |j |_ t|j |j |j |_ |j}tj|t#j|j zd|_tj|t#j|j zd|_tj|j|_dS)Nrr#T) requires_grad)r&r'chunk_size_feed_forward seq_len_dimre attentionrmlpNORM2FN norm_typer/layer_norm_epslayernorm_beforelayernorm_afterlayer_scale_init_valuer(r)r*r+lambda_1lambda_2rzrrS)r.rf init_valuesr0s r1r'zInternVLVisionLayer.__init__Ls '-'E$088$V,, '(8 9&:LRXRg h h h&v'789KQWQfggg3  [5:f>P3Q3Q%Qaefff  [5:f>P3Q3Q%Qaefff z&"<== r2r3c$|||\}}|j|z}||z}||}||}||}|j |j|z}||z}|Sr)rrrrrrSr)r.r3attention_outputr layer_outputs r1r?zInternVLVisionLayer.forward[s#nn  ! !- 0 0  ! =+;;)=8 ++M:: xx -- ||L11 = $=<7L$m3 r2) rErFrGrrr'r*rIrAr?rJrKs@r1rrIsII >3 > > > > > > >| u| uU\5<%?@ @r2rcHeZdZdeddffd ZdejdeezfdZ xZ S)InternVLVisionEncoderrfr$Nct|_tjfdt jD|_d|_dS)Nc.g|]}tSr)r).0irfs r1 z2InternVLVisionEncoder.__init__..{s"#i#i#iA$7$?$?#i#i#ir2F) r&r'rfr( ModuleListrangenum_hidden_layerslayergradient_checkpointingrs `r1r'zInternVLVisionEncoder.__init__xs`  ]#i#i#i#ivOgIhIh#i#i#ijj &+###r2r3cL|jD] }||}t|S)N)last_hidden_state)rr )r.r3 layer_modules r1r?zInternVLVisionEncoder.forward~s@!J 8 8L(L77MM+    r2) rErFrGrr'r*rIrAr r?rJrKs@r1rrwsy,3,,,,,,,  |             r2rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ eedZejfdZxZS) InternVLVisionPreTrainedModelrfinternvl_visionr)imagevideoTr)r3 attentionsct|t|tr]t j|j|jt j|j|jt j|jdSdSt|trJt j |j |j j t j |j|j j dSdS)zInitialize the weightsN)r& _init_weightsrrinitzeros_rrrr constant_rrfrr)r.rMr0s r1rz+InternVLVisionPreTrainedModel._init_weightss f%%% f6 7 7 P K( ) ) ) , F-...)5 F67777765  3 4 4 P N6?DK,N O O O N6?DK,N O O O O O P Pr2)rErFrGr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrre_can_record_outputsr*no_gradrrJrKs@r1rrs    )$O)&*#./N"&-- U]__ P P P P_ P P P P Pr2rc eZdZdeddffd ZdZeede d de j d e j dzde e zfd ZxZS) InternVLVisionModelrfr$NcNt|||_t||_t ||_|jrtj ntj |j |j |_ |dS)Nr)r&r'rfrrrencoderuse_mean_poolingr(r{ LayerNormr/r layernorm post_initrs r1r'zInternVLVisionModel.__init__s     26::,V44 $4 uBKMMM",vGY_e_t:u:u:u  r2c|jjSr)rrrCs r1get_input_embeddingsz(InternVLVisionModel.get_input_embeddingss //r2F)tie_last_hidden_statesrrc |||}||}|d}||}t||j|jS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr)rr3r)rrrrr3r)r.rrr^embedding_outputencoder_outputssequence_outputs r1r?zInternVLVisionModel.forwardsm ??<?YY,,'788)!,..993-)7&1    r2r)rErFrGrr'rrrrr*rIrrArr?rJrKs@r1rrs 3       000 _E222UY  !L ;@;Kd;R 5 5   ^32      r2rc@eZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ dS)InternVLPreTrainedModelrfmodel)rtextrTpast_key_valuesN)rErFrGrr r r r_skip_keys_device_placementrr_can_compile_fullgraphrrrr2r1r&r&sV 1&*##4"5N!"&r2r&c*eZdZdeffd ZdZxZS)InternVLMultiModalProjectorrfcttj|jjt d|jz dzz|_tj |jjt d|jz dzz|j j|_ t|j |_tj |j j|j j|_dS)Nrr)r&r'r(r vision_configr/rdownsample_ratiorrt text_configlinear_1rprojector_hidden_actactlinear_2rs r1r'z$InternVLMultiModalProjector.__init__s ,v';'G#aRXRiNiJjJjnoJo'opp  ,s1v7N3N/O/OST/T TV\VhVt   &56 &"4"@&BTB`aa r2c||}||}||}||}|Sr)rr2r4r5)r.image_featuresr3s r1r?z#InternVLMultiModalProjector.forwardsL77  m44 //  m44 r2)rErFrGrr'r?rJrKs@r1r-r-sZb~bbbbbbr2r-zM Base class for InternVL outputs, with hidden states and attentions. c2eZdZUdZdZejdzed<dS)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)rErFrGrr:r* FloatTensorr rr2r1r9r9s7  59*T188888r2r9zx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. ceZdZdeffd Zeeed ddej de e e ze e zdzde dzd e ed eezf d Zd ejd ej dej fdZee dd ejdzdej dzdejdzdejdzdedzd ej dzde e e ze e zdzde dzd e ed eezfdZddejdefdZxZS) InternVLModelrfct|tj|j|_t ||_tj|j|_ | dSr) r&r'r from_configr/ vision_towerr-multi_modal_projectorr1language_modelrrs r1r'zInternVLModel.__init__sm    %1&2FGG%@%H%H"'3F4FGG r2zWObtains image last hidden states from the vision tower and apply multimodal projection.rNrvision_feature_layervision_feature_select_strategyr^r$c ||j}|jj}|dkrd|d<|jd |dd|}|dkr|j}n |j|}|dkr|ddddddf}|jd}t|d z} |jd } | | | | d}| || }| | d|jd}| |}||_ |S) a! pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. )r7r5Toutput_hidden_states)r return_dictdefaultNrrr) scale_factorr) r8r7rfr0r@rr3rBrr pixel_shufflerA pooler_output) r.rrCrDr^r0vision_outputsvision_featureschannels feature_sizers r1get_image_featuresz InternVLModel.get_image_featuressN$$TZ88 ;7 2 % %-1F) ***a RVaaZ`aa 2 % %,>OO,:;OPO )Y 6 6-aaaQQQh7O#(+8S=)) $*1- *11*lLZ\]],,_K[,\\*11*b/BWXZB[\\44_EE'6$r2 input_ids inputs_embedsr7c |e||tj|jjtj|jk}|d}n||jjk}|}|j d|j dz}| d |j}t||j dz| kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)r7devicer5rrz6Image features and image tokens do not match, tokens: z , features: )rr*tensorrfimage_token_idlongrTallsumrBrr8rnumel)r.rQrRr7special_image_maskn_image_tokensn_image_featuress r1get_placeholder_maskz"InternVLModel.get_placeholder_maskMs  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11)/2^5I!5LL/99"==@@AUVV ]04 48L8L8N8N N s^ s saq s s   "!r2rQ position_idsr)c  |du|duz rtd|||}|l||||dj} | |j|j} |||| } || | }|j d||||d| } t| j | j | j | j|| ndS)Nz:You must specify exactly one of input_ids or inputs_embedsT)rrCrDrG)rRr7)rQr_r)rR)rr)r3rr:r)rnrrPrKr8rTr7r^masked_scatterrBr9rr)r3r) r.rQrrQr_r)rRrCrDr^r7r[outputss r1r?zInternVLModel.forwardesI  -t"; < [YZZ Z  7D5577 BBM  #!44)%9/M 5   ,..}/C]EXYYN!%!:!:~";"" *889K^\\M%$% )%+'      +%7#3!/)2>2JPT     r2rrMrIc (|\}}}}||zdks ||zdkrtd|||t||zt||z }|dddd}||t||zt||zt||dzz }|dddd}|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rrnrrrr])r.rMrIrrrrNs r1rJzInternVLModel.pixel_shuffles)/>.B.B.D.D+ E68 L A % %)=)B)Bjkk k*.. s6L#8993x,?V;W;W  *11!Q1==HHJJ*.. F\122C 8L4M4MsS[_kmn_nSoOpOp   *11!Q1==HHJJr2NN)NNNNNNNN)r)rErFrGrr'rrrr*r;rliststrrrrArrP LongTensorr^rIrr9r?rHrJrJrKs@r1r=r= sV ~ ^n DH59 ,,',"DIoS 9D@,),d , +, , + + ,,,  ,\")":?:K"]b]n""""0.215.204(,26CG59- - #d*- '$.-  t+ - &- -  - (4/- "DIoS 9D@- ),d - +,-  , ,- - - ^- ^!!U\!!!!!!!!!r2r=zT Base class for InternVL causal language model (or autoregressive) outputs. ceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZejdzed<dS) InternVLCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsr)r3rr:)rErFrGrrjr*r;r rkr)rr3rArr:rr2r1riris  &*D% d ")))'+FE  $+++$(OUT\(((59M5*+d299926Je'(4/66648*T188888r2rizV The INTERNVL model which consists of a vision backbone and a language model. ceZdZddiZdeffd ZdejfdZe dde j d e e e ze e zdzd edzd eedeezf d Zee dde jdzde j dzde jdzde jdzdedzde j dzd e e e ze e zdzd edzde jdzde e jzde jdzd eedeezfdZ dfd ZxZS) InternVLForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightrfct|t||_t j|jj|jjd|_ | dS)NFrh) r&r'r=r'r(rtr1r/ vocab_sizelm_headrrs r1r'z)InternVLForConditionalGeneration.__init__se    "6** y!3!?ASA^ejkkk  r2r$c|jSr)rprCs r1get_output_embeddingsz6InternVLForConditionalGeneration.get_output_embeddingss |r2NrrCrDr^c .|jjd|||d|S)N)rrCrDr)r'rP)r.rrCrDr^s r1rPz3InternVLForConditionalGeneration.get_image_featuress:-tz, %!5+I      r2rrQrQr_r)rRlabelslogits_to_keep image_sizesc n|jd||||||||| d | } | d}t| trt| dn| }||dd|ddf}d}| |jd|| |jjjd| }t||| j | j | j | j S)ac Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```) rQrrQr_r)rRrCrDrvrN)rkrtro)rjrkr)r3rr:r)r'rrslicerp loss_functionrfr1rorir)r3rr:)r.rQrrQr_r)rRrCrDrtrurvr^rbr3 slice_indicesrkrjs r1r?z(InternVLForConditionalGeneration.forwardsh$*  %)%+'!5+I#        8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD.#3!/) ' ;     r2Fc tj|f|||||d|} |s|dds|| d<| S)N)r)rRrQruis_first_iteration use_cacheTr)r&prepare_inputs_for_generationget) r.rQr)rRrrQrur|r^ model_inputsr0s r1r~z>InternVLForConditionalGeneration.prepare_inputs_for_generationMst=uww<  +'))1       8VZZ T%B%B 8 ,8L (r2rd) NNNNNNNNNrN)NNNNNF)rErFrG_tied_weights_keysrr'r(Modulerrrr*r;rrerfrrrArrPrrgrIrrir?r~rJrKs@r1rmrmsv +,VW~ ryDH59   '  "DIoS 9D@  ),d  +,  + +    ^  .215.204(,26CG59*.-.+/Q Q #d*Q '$.Q  t+ Q &- Q  Q (4/Q "DIoS 9D@Q ),d Q  4'Q el*Q \D(Q +,Q  / /Q Q Q ^Q l r2rm)rrr&r=rm)rL)Fcollections.abcrr dataclassesrr*torch.nnr(rr activationsr cache_utilsr generationr integrationsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.output_capturingrautorconfiguration_internvlrrrr!rIrHrrcrerrrrrrrrrrr&r-r9r=rirm__all__rr2r1rs%,$$$$$$!!!!!! &&&&&&!!!!!! ))))))777777999999ddddddddddFFFFFFFF&&&&&&ggggggggggggggIIIIIIII555555HHHHHHHHY''JJJJJBIJJ('J6%% I% <% % < % L4' %  %S[%%%%4F$F$F$F$F$biF$F$F$R  +E        BI   J[[[[[ry[[[|        3H I I+++++4+++\     BI   &PPPPPOPPP@& & & & & 7& & & R ' ' ' ' 'o ' ' '")$   9 9 9 9 9"9 9 9   9 fffff+ff  fR  99999[99   90 NNNNN'>NN  Nb   r2