\)5rI/S?QrJg)AN) dataclass)CallableOptionalUnion)ACT2FN)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tuplelogging torch_int)CLIPMLP)JanusVisionAttention) LlamaRMSNorm)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)InternVLConfigInternVLVisionConfigmodulequerykeyvalueattention_maskscalingdropoutc UnUn [R"XRSS55U-n Ub"USS2SS2SS2SURS24n X-n [R R U SS9n [R RXURS9n [R"X5n U RSS5R5n X4$)Nrrdim)ptrainingr) torchmatmul transposeshapenn functionalsoftmaxr'r. contiguous) r!r"r#r$r%r&r'kwargs key_states value_states attn_weights causal_mask attn_outputs e/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/internvl/modular_internvl.pyeager_attention_forwardr>1sJL<<';';Aq'ABWLL!$Q1.D 0@0@0D.D%DE #1 ==((2(>L==((6??([L,,|:K''1-88:K  $$c\rSrSrSrg)InternVLVisionRMSNormLN__name__ __module__ __qualname____firstlineno____static_attributes__rCr?r=rArALr?rAc ^\rSrSrS\4U4SjjrS S\RS\\RS\\RS\ \ 4Sjjr S r U=r $) InternVLVisionAttentionPconfigc0>[TU]5 U?SUlURnU(a[ UR 5O[R"5Ul U(a[ UR 5Ul g[R"5Ul gNF) super__init__num_key_value_groups is_causal use_qk_normrA embed_dimr3Identityq_normk_norm)selfrNqk_norm __class__s r=rR InternVLVisionAttention.__init__Qsb   %$$?F+DNN;BKKM ?F+DNN; BKKM r? hidden_statesr%output_attentionsr7c UR5upVnURU5nURU5n URU5n UR U5nUR U 5n UR XVURUR5RSS5nU R XVURUR5RSS5n U RXVURUR5RSS5n [n URRS:wa[URRn U "UUU U U4UR(dSO UR UR"SS.UD6upU R XVUR$5n UR'U 5nUR)U5nU(aX4nU$US4nU$)NrreagerF)r'r&rT)sizeq_projk_projv_projrXrYreshape num_headshead_dimr1viewr>rN_attn_implementationrr.attention_dropoutscalerVprojection_layerprojection_dropout)rZr^r%r_r7 batch_sizeseq_len_ query_statesr8r9attention_interfacer<r:outputoutputss r=forwardInternVLVisionAttention.forward\s"/!3!3!5 Q{{=1 [[/ {{=1 {{<0 [[, #++JQUQ^Q^_iijkmno '' T^^T]][eefgijk #((dnndmm\ffghjkl (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$2H2HJJ %  % ! "))*t~~N &&{3((0,=6(EKD>r?)rTrYrXNN)rErFrGrHr rRr/Tensorrrr rwrI __classcell__r\s@r=rLrLPsf Z3 Z2648 )||)!.)$ELL1 ) -. ))r?rLcV^\rSrSr%\\S'SrSrSrS/r Sr Sr Sr Sr U4SjrSrU=r$) InternVLVisionPreTrainedModelrNinternvl_vision pixel_valuesTInternVLVisionLayercn>[TU]U5 [U[5(aURR R 5 URb$URR R 5 URb%URR R 5 gg[U[5(asURR RURR5 URR RURR5 gg)zInitialize the weightsN)rQ _init_weights isinstanceInternVLVisionEmbeddings cls_tokendatazero_ mask_tokenposition_embeddingsrlambda_1fill_rNlayer_scale_init_valuelambda_2)rZr!r\s r=r+InternVLVisionPreTrainedModel._init_weightss f% f6 7 7    ! ! ' ' )  ,!!&&,,.))5**//5576  3 4 4 OO & &t{{'I'I J OO & &t{{'I'I J5r?rC)rErFrGrHr __annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrrIr{r|s@r=r~r~sF  )$O&*#./N"& K Kr?r~z7 Class for outputs of [`InternVLVisionModel`]. ) custom_introc\rSrSrSrSrg)$InternVLVisionModelOutputWithPoolinga2 pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. rCN)rErFrGrH__doc__rIrCr?r=rrsr?rcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cH>[TU]5 URURp2URUR pTUSUS-USUS--nUSUS-USUS-4nX lX0lX@lX`lXpl[R"XEX3S9Ul g)Nrr) kernel_sizestride) rQrR image_size patch_size num_channels hidden_size num_patches patch_shaper3Conv2d projection) rZrNrrrrrrr\s r=rR&InternVLVisionPatchEmbeddings.__init__s !'!2!2F4E4EJ$*$7$79K9Kk!!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L:ir?rreturncURup#pEX0R:wa [S5eURU5nURSURSpUR S5R SS5nXgU44$)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r2r ValueErrorrflattenr1) rZrrprheightwidth embeddings patch_height patch_widths r=rw%InternVLVisionPatchEmbeddings.forwards2>2D2D/ & ,, ,w __\2 $.$4$4Q$79I9I!9Lk''*44Q: +666r?)rrrrrr) rErFrGrHrrRr/rzrwrIr{r|s@r=rrs. j 7ELL 7U\\ 7 7r?rc^\rSrSrSrS\SS4U4SjjrS\RS\ S \ S\R4S jr SS \RS \ \RS\R4S jjr SrU=r$)rz[ Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rNrNc^>[TU]5 [R"[R "SSUR 55UlUR(a<[R"[R "SSUR 55Ul OSUl [U5Ul URUl [UR[R R"5(a UROURUR4UlURR$nUR&(a?[R"[R "SUS-UR 55UlOSUl[R*"UR,5Ulg)Nr)rQrRr3 Parameterr/zerosrruse_mask_tokenrrpatch_embeddingsrrr collectionsabcIterabler use_absolute_position_embeddingsrDropouthidden_dropout_probr')rZrNrr\s r=rR!InternVLVisionEmbeddings.__init__s' ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO =f E ++&++[__-E-EFF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r?rrrcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R S-n X0R S-n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr*r?rrbicubicF)rcmode align_cornersr+)r2rr/jit is_tracingrrrgpermuter3r4 interpolaterjcat) rZrrrr num_positionsclass_pos_embedpatch_pos_embedr, new_height new_widthsqrt_num_positionss r=interpolate_pos_encoding1InternVLVisionEmbeddings.interpolate_pos_encodings]!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r"q11 __Q// &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr?rbool_masked_poscURu p4nURU5unupxUR5upnUbIURR XS5n UR S5R U 5n USU - -X--nURR U SS5n [R"X4SS9nURbX`RXdU5-nURU5nXgU44$)Nr*rr+) r2rrcrexpand unsqueezetype_asrr/rrrr')rZrrrrrrrrrrprq mask_tokensw cls_tokenss r=rw InternVLVisionEmbeddings.forwards +001e262G2G 2U/ /\!+!2 Q  &//00bIK))"-55kBA#q1u- ?J^^**:r2> YY 7Q?  # # /#&C&CJX]&^^J\\*- +666r?)rr'rrrrrN)rErFrGrHrr rRr/rzintrr BoolTensorrwrIr{r|s@r=rrs >3>>,&D5<<&D&DUX&D]b]i]i&DV7;7ll7"%"2"237  77r?rc\rSrSrSrg)InternVLVisionMLPi5rCNrDrCr?r=rr5rJr?r) layer_normrms_normc ^\rSrSrSrS\SS4U4SjjrS S\RS\ S\ \ \R\ \R\R444S jjr S r U=r$) ri<z?This corresponds to the Block class in the timm implementation.rNrNc>[TU]5 URUlSUl[ U5Ul[ U5Ul[UR"URURS9Ul [UR"URURS9Ul URn[R "U["R$"UR5-SS9Ul[R "U["R$"UR5-SS9Ul[R*"UR,5Ulg)NrepsT) requires_grad)rQrRchunk_size_feed_forward seq_len_dimrL attentionrmlpNORM2FN norm_typerlayer_norm_epslayernorm_beforelayernorm_afterrr3rr/onesrrrrr')rZrN init_valuesr\s r=rRInternVLVisionLayer.__init__?s '-'E'E$08$V, '(8(8 9&:L:LRXRgRg h&v'7'789K9KQWQfQfg33  [5::f>P>P3Q%Qaef  [5::f>P>P3Q%Qaef zz&"<"<= r?r^r_cURURU5US9up4URU-nX1-nURU5nUR U5nUR U5nUR bUR U-nXQ-nXT4$)N)r_)rrrrrr'r)rZr^r_attention_outputattention_weights layer_outputs r=rwInternVLVisionLayer.forwardNs /3nn  ! !- 0//=/ +  ==+;;)8 ++M: xx - ||L1 == $==<7L$3 ..r?) rrr'rrrrrr)F)rErFrGrHrr rRr/rzboolrtuplerwrIr{r|s@r=rr<snI >3 > >$#(/||/ / uU\\"E%,, *D$EE F //r?rc ~^\rSrSrS\SS4U4Sjjr\S S\RS\ S\ S\ \ \ 44S jj5r S rU=r$) InternVLVisionEncoderilrNrNc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfrP) rQrRrNr3 ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rZrNir\s r=rRInternVLVisionEncoder.__init__msS  ]]vOgOgIh#iIhA$7$?Ih#ij &+#$jsA%r^r_output_hidden_statescU(aSOSnU(aSOSn[UR5H/upgU(aXA4-nU"X5nUSnU(dM'XXS4-nM1 U(aXA4-n[UUUS9$)NrCrrlast_hidden_stater^ attentions) enumeraterr ) rZr^r_r all_hidden_statesall_self_attentionsr layer_module layer_outputss r=rwInternVLVisionEncoder.forwardss#7BD$5b4(4OA#$58H$H!(JM)!,M  &91=M[TU]U5 Xl[U5Ul[ U5UlUR(a[R"5O([R"URURS9Ul UR5 g)Nr)rQrRrNrrrencoderuse_mean_poolingr3rW LayerNormrr layernorm post_initrZrNr\s r=rRInternVLVisionModel.__init__sm   26:,V4 $44BKKM",,vGYGY_e_t_t:u  r?c.URR$r)rr)rZs r=get_input_embeddings(InternVLVisionModel.get_input_embeddingss///r?rrr_r cUbUOURRnUbUOURRnURXS9upVUR UUUS9nUSnUR U5n[ UURURS9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )r)r_r rr ) rNr_r rrrrr^r) rZrrr_r embedding_outputrrencoder_outputssequence_outputs r=rwInternVLVisionModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj #oolo\,, /!5'  *!,..93-)77&11  r?)rNrrr)NNN)rErFrGrHr rRr rrr/rzrrrrrrrwrIr{r|s@r=rrs 3  07;,0/3  ll "%"2"23 $D>  'tn  u:: ;   r?rc\rSrSrSrg)InternVLPreTrainedModelirCNrDrCr?r=r(r(rJr?r(c6^\rSrSrS\4U4SjjrSrSrU=r$)InternVLMultiModalProjectorirNc0>[TU]5 [R"URR [ SUR- 5S--5Ul[R"URR [ SUR- 5S--URR 5Ul [URUl[R"URR URR 5Ulg)Nrr)rQrRr3r vision_configrrdownsample_ratiorLinear text_configlinear_1rprojector_hidden_actactlinear_2rs r=rR$InternVLMultiModalProjector.__init__s ,,v';';'G'G#aRXRiRiNiJjnoJo'op  , ,s1v7N7N3N/OST/T TV\VhVhVtVt  &556 &"4"4"@"@&BTBTB`B`a r?cURU5nURU5nURU5nURU5nU$r)rr0r2r3)rZimage_featuresr^s r=rw#InternVLMultiModalProjector.forwards@7  m4 /  m4 r?)r2rr0r3) rErFrGrHrrRrwrIr{r|s@r=r*r*sb~br?r*c\rSrSrSrg)InternVLModelOutputWithPastirCNrDrCr?r=r9r9rJr?r9c#\rSrSrSS\R S\4SjjrSS\RS\ \ \ \ \ 4S\ \ 4S jjr\\SS \R"S\RS \ \R S \ \R"S \ \S\ \RS\ \ \ \ \ 4S\ \ S\ \S\ \S\ \S\ \S\ \R"S\\S\ \\44Sjj55rSrg) InternVLModelivision_features scale_factorc UR5up4pVXR-S:wdXB-S:wa [S5eURX4[XR-5[Xb- 55nUR SSSS5R 5nURU[XR-5[XB-5[XbS-- 55nUR SSSS5R 5nU$)aPerform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rcrrjrrr6)rZr<r=rprrchannelss r= pixel_shuffleInternVLModel.pixel_shuffles/>.B.B.D+ 6  A %)=)Bjk k*.. s6#893x?V;W *11!Q1=HHJ*.. F12C8L4MsS[mn_nSoOp  *11!Q1=HHJr?Nrvision_feature_layervision_feature_select_strategyc <UbUOURRnUbUOURRnURRnUS:XaUR US9R nOUR US9RUnUS:XaUSS2SS2SS24nURSn[US-5nURSn URXUS5nURXeS9nURU SURS5nURU5nU$) a Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. Nr*)rdefaultrrr)r=) rNrBrCr- vision_towerr  vision_modelr^r2rrgr@multi_modal_projector) rZrrBrCr7r-r<r? feature_sizerps r=get_image_features InternVLModel.get_image_features s?&%9$D $++JjJj  .9 +;; '  ;;77 2 %"//\/J\\O"//\/JXXYmnO )Y 6-aQh7O#((+8S=) $**1- *11*LZ\],,_,\*11*b/BWBWXZB[\44_Er? input_idsr% position_idspast_key_values inputs_embeds use_cacher_r  return_dictcache_positionr7rc U bU OURRn U bU OURRn U bU OURRn UbUOURRnUbUOURR nUSLUSL- (a [ S5eUcUR5"U5nUbYURUUUS9nURURUR5nURXUS9nURUU5nUR"SUUUUU U U SU S. UD6n[UR UR"UR$UR&UbWS9$SS9$)Nz:You must specify exactly one of input_ids or inputs_embeds)rrBrC)rOr6T) r%rMrNrOrPr_r rQrR)r rNr^rimage_hidden_statesrC)rNr_r use_return_dictrBrCrr rJtodevicedtypeget_placeholder_maskmasked_scatterlanguage_modelr9r rNr^r)rZrLrr%rMrNrOrBrCrPr_r rQrRr7r6special_image_maskrvs r=rwInternVLModel.forward>s&2C1N-TXT_T_TqTq$8$D $++JjJj &1%2J   QU   r?rC)rry) NNNNNNNNNNNNN)rErFrGrHr/rzfloatr@ FloatTensorrrrliststrrJrr LongTensorr rrr rr9rwrIrCr?r=r;r;s!U\\!!LAE8< 3''3'uS$s)^'<=3)1 3j'+*.1537+/59@D8<$(,0/3&*59D ##D ''D !. D u//0 D "% D   1 12D 'uS$s)^'<=D )1 D D>D $D>D 'tnD d^D !!1!12D -.D u11 2!D D r?r;c\rSrSrSrg)InternVLCausalLMOutputWithPastirCNrDrCr?r=rdrdrJr?rdc(^\rSrSrU4SjrSrU=r$) InternVLForConditionalGenerationic :>[5R"S0UD6 g)ay Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", torch_dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```NrC)rQrw) super_kwargsr\s r=rw(InternVLForConditionalGeneration.forwardsH ','r?rC)rErFrGrHrwrIr{r|s@r=rfrfs $($(r?rf)r~rr(r;rf)rb)Kcollections.abcr dataclassesrtypingrrrr/torch.nnr3torch.utils.checkpoint activationsr cache_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr r modeling_utilsrrprocessing_utilsrutilsrrrrclip.modeling_cliprjanus.modeling_janusrllama.modeling_llamarllava.modeling_llavarrrrrconfiguration_internvlrr get_loggerrEloggerModulerzr^r>rArLr~rrrrrrrrrr(INTERNVL_INPUTS_DOCSTRINGr*r9r;rdrf__all__rCr?r=rs "!,, ! B9KF&II(7/I   H %% II% <<% % << % U\\* %  %%6 L 525pKOKK2  +E  !7BII!7L[7ryy[7|  3H I-/4-/`# BII# L2 72 2 j 2 !"))$ ": _ J_ D %@ %('D%(P r?