[TU]5 [R"[R "SSUR 55UlU(a6[R"[R"SSUR 55OSUl [U5Ul URRn[R"[R "SUS-UR 55Ul [R"UR5UlUR"UlXlg)Nr)super__init__r Parametertorchrandn hidden_size cls_tokenzeros mask_tokenViTPatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_sizer)selfrrr, __class__s \/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/vit/modeling_vit.pyr"ViTEmbeddings.__init__0s ekk!Q8J8J&KLQ_",,u{{1a9K9K'LMei 26 :++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++ embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rN?rr bicubicF)sizemode align_cornersdim)shaper-r$jit is_tracingr1rreshapepermuter functional interpolateviewcat) r2r7r8r9r, num_positionsclass_pos_embedpatch_pos_embedrC new_height new_widthsqrt_num_positionss r4interpolate_pos_encoding&ViTEmbeddings.interpolate_pos_encoding<sS!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr6 pixel_valuesbool_masked_posrScURupEpgURXS9nUbXURSn URRXIS5n UR S5R U 5n USU - -X--nUR RUSS5n [R"X4SS9nU(aXRXU5-nOXR-nURU5nU$)N)rSrr;?rB) rDr+r)expand unsqueezetype_asr'r$rLrSr-r0) r2rUrVrS batch_size num_channelsr8r9r7 seq_length mask_tokensmask cls_tokenss r4forwardViTEmbeddings.forwardds 3?2D2D/ &**<*k  &#))!,J//00LK",,R088ED#sTz2[5GGJ^^**:r2> YY 7Q?  $#&C&CJX]&^^J#&>&>>J\\*- r6)r'rr0r)r+r1r-FNF)__name__ __module__ __qualname____firstlineno____doc__rboolr"r$TensorintrSr BoolTensorrb__static_attributes__ __classcell__r3s@r4rr+s y $ 4  &D5<<&D&DUX&D]b]i]i&DV7;). ll"%"2"23#'   r6rcn^\rSrSrSrU4SjrS S\RS\S\R4Sjjr Sr U=r $) r*z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c>[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nX lX0lX@lX`l [R"XEX3S9Ul g)Nrr) kernel_sizestride)r!r" image_sizer1r]r& isinstance collectionsabcIterabler,rConv2d projection)r2rrwr1r]r&r,r3s r4r"ViTPatchEmbeddings.__init__s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:ir6rUrSrc URup4pVX@R:wa[SURSUS35eU(dVXPRS:wdX`RS:wa2[SUSUSURSSURSS 3 5eUR U5R S 5R SS 5nU$) NzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rrzInput image size (*z) doesn't match model (z).r=)rDr] ValueErrorrwr}flatten transpose)r2rUrSr\r]r8r9r7s r4rbViTPatchEmbeddings.forwards2>2D2D/ & ,, ,!../yaI (++u8J/J (%9+,Adooa.@-AE__\2::1=GG1M r6)rwr]r,r1r}rd) rfrgrhrirjr"r$rlrkrbrorprqs@r4r*r*s8 jELLD]b]i]ir6r*modulequerykeyvalueattention_maskscalingr0c [R"XRSS55U-n[RR US[R S9RUR5n[RRXURS9nUbX-n[R"X5n U RSS5R5n X4$)Nr;)rCdtype)ptrainingrr=) r$matmulrrrIsoftmaxfloat32torr0r contiguous) rrrrrrr0kwargs attn_weights attn_outputs r4eager_attention_forwardrs<<}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#4 ,,|3K''1-88:K  $$r6c ^\rSrSrS\SS4U4SjjrS S\\RS\ S\ \ \R\R4\ \R44Sjjr S r U=r$) ViTSelfAttentionrrNc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rgF)bias)r!r"r&num_attention_headshasattrrrrmattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasrrrr2rr3s r4r"ViTSelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=2O\M]r6) rrrrrrrrrrre)rfrgrhrirr"rr$rlrkrtuplerbrorprqs@r4rrsw]y]T].-1"' 1ELL)1 1 uU\\5<</0% 2EE F 11r6rc^\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r $) ViTSelfOutputiz The residual connection is defined in ViTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rrNc>[TU]5 [R"URUR5Ul[R "UR5UlgN) r!r"rrr&denser.r/r0rs r4r"ViTSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r6r input_tensorcJURU5nURU5nU$rrr0r2rrs r4rbViTSelfOutput.forwards$ =1  ]3 r6r) rfrgrhrirjrr"r$rlrbrorprqs@r4rrsI >y>T> U\\RWR^R^r6rc ^\rSrSrS\SS4U4SjjrS\\SS4SjrS S\ RS \ \ RS \ S\ \\ R\ R4\\ R44S jjrS rU=r$) ViTAttentioni#rrNc>[TU]5 [U5Ul[ U5Ul[ 5Ulgr)r!r"r attentionroutputset pruned_headsrs r4r"ViTAttention.__init__$s0 )&1#F+ Er6headsc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)NrrrB)lenrrrrrrrrrrrrunion)r2rindexs r4 prune_headsViTAttention.prune_heads*s u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r6rrrcfURXU5nURUSU5nU4USS-nU$)Nrr)rr)r2rrr self_outputsattention_outputrs r4rbViTAttention.forward<sC ~~m@QR ;;|A F#% QR(88r6)rrrre)rfrgrhrirr"rrmrr$rlrrkrrrbrorprqs@r4rr#s"y"T" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F  r6rcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) ViTIntermediateiJrrNc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r!r"rrr&intermediate_sizerrx hidden_actstrr intermediate_act_fnrs r4r"ViTIntermediate.__init__Ks` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r6rcJURU5nURU5nU$rrr)r2rs r4rbViTIntermediate.forwardSs& =1 00? r6r rfrgrhrirr"r$rlrbrorprqs@r4rrJs69y9T9U\\ellr6rc^\rSrSrS\SS4U4SjjrS\RS\RS\R4SjrS r U=r $) ViTOutputiZrrNc>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) r!r"rrrr&rr.r/r0rs r4r"ViTOutput.__init__[sB YYv779K9KL zz&"<"<= r6rrcRURU5nURU5nX-nU$rrrs r4rbViTOutput.forward`s, =1  ]3 %4 r6rrrqs@r4rrZsD>y>T> U\\RWR^R^r6rc ^\rSrSrSrS\SS4U4SjjrS S\RS\ \RS \ S\ \ \R\R4\ \R44S jjr S rU=r$) ViTLayeriiz?This corresponds to the Block class in the timm implementation.rrNcj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nreps)r!r"chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr&layer_norm_epslayernorm_beforelayernorm_afterrs r4r"ViTLayer.__init__ls '-'E'E$%f-+F3' " V-?-?VEZEZ [!||F,>,>FDYDYZr6rrrcURURU5UUS9nUSnUSSnXQ-nURU5nURU5nUR Xq5nU4U-nU$)N)rrr)rrrrr)r2rrrself_attention_outputsrr layer_outputs r4rbViTLayer.forwardvs "&  ! !- 0 /"0"  2!4(,)8 ++M: ((6 {{<? /G+r6)rrrrrrrre)rfrgrhrirjrr"r$rlrrkrrrbrorprqs@r4rrisI[y[T[-1"' ||ELL)  uU\\5<</0% 2EE F r6rc^\rSrSrS\SS4U4SjjrS S\RS\\RS\ S \ S \ S\ \ \ 44 S jjr S rU=r$) ViTEncoderirrNc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfre) r!r"rr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r2rrr3s r4r"ViTEncoder.__init__sR  ]]eFD\D\>]#^>]HV$4>]#^_ &+#$_sA%rrroutput_hidden_states return_dictc6U(aSOSnU(aSOSn[UR5H9upU(aXa4-nUbX(OSn U "XU5n U SnU(dM1X{S4-nM; U(aXa4-nU(d[SXU455$[UUUS9$)Nrrc3,# UH ocMUv M g7frr ).0vs r4 %ViTEncoder.forward..sm$[q$[s )last_hidden_stater attentions) enumeraterrr ) r2rrrr r all_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r4rbViTEncoder.forwards#7BD$5b4(4OA#$58H$H!.7.CilO(IZ[M)!,M  &91=M[TU]U5 Xl[XS9Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rrN)r!r"rrr7rencoderrrr&r layernorm ViTPoolerpooler post_init)r2rr7rr3s r4r"ViTModel.__init__si   'N!&) f&8&8f>S>ST+:L?? Tl+ ,, /!5# ' *!,..98< 8O O4UY ?L?XO;_n^pL!""55 5)-')77&11   r6)rr7r9r:r<)TFNNNNNNN)rfrgrhrirrkr"r*r@dictrmlistrErrr$rlrnrrrrbrorprqs@r4r6r6syTZ^&0&80C4T#Y+?CDC046:,0,0/337&*; u||,; "%"2"23; ELL) ; $D> ; 'tn ; #+4.; d^;  u00 1; ; r6r6c6^\rSrSrS\4U4SjjrSrSrU=r$)r;iGrc>[TU]5 [R"URUR 5Ul[URUl gr) r!r"rrr&pooler_output_sizerr pooler_act activationrs r4r"ViTPooler.__init__Hs> YYv1163L3LM  !2!23r6c\USS2S4nURU5nURU5nU$)Nr)rrY)r2rfirst_token_tensorrOs r4rbViTPooler.forwardMs6+1a40 #56  6 r6)rYr) rfrgrhrirr"rbrorprqs@r4r;r;Gs4y4 r6r;ac ViT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introc^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \\ S \\ S\ \ \44S jj5rSrU=r$)ViTForMaskedImageModelingiVrrNcH>[TU]U5 [USSS9Ul[R "[R "URURS-UR-SS9[R"UR55Ul UR5 g)NFT)r7rr=r) in_channels out_channelsru) r!r"r6rr Sequentialr|r&encoder_strider] PixelShuffledecoderr=rs r4r""ViTForMaskedImageModeling.__init__cs  FeDQ}} II"..#22A58K8KK  OOF11 2    r6rUrVrrr rSr c UbUOURRnUbhURRURR:wa:[ SURRSURRS35eUR UUUUUUUS9nUSn U SS2SS24n U R upn [R"U S-5=pU RSS S5RXX5n URU 5nSnUGbURRURR-nURS UU5nURURRS5RURRS 5RS5R5n[ R"R%XS S 9nUU-R'5UR'5S -- URR(- nU(dU4USS-nUbU4U-$U$[+UUUR,UR.S9$)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, ViTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k") >>> model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```NzWhen `bool_masked_pos` is provided, `patch_size` must be equal to `encoder_stride` to ensure that the reconstructed image has the same dimensions as the input. Got `patch_size` = z and `encoder_stride` = r)rVrrr rSr rrr<r=r;none) reductiongh㈵>)lossreconstructionrr)rrIr1rerrrDmathfloorrHrGrgrwrepeat_interleaverZrrrIl1_losssumr]rrr)r2rUrVrrr rSr rrNr\sequence_lengthr]r8r9reconstructed_pixel_valuesmasked_im_lossr?r`reconstruction_lossrs r4rb!ViTForMaskedImageModeling.forwardtsFL&1% Y 'tn Y #+4.Y d^Y  u// 0Y Y r6r`a ViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. c^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \\ S \\ S\ \ \ 44S jj5rSrU=r$)ViTForImageClassificationirrNc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 g)NF)r7r) r!r" num_labelsr6rrrr&Identity classifierr=rs r4r""ViTForImageClassification.__init__ss   ++Fe<OUN_N_bcNc"))F$6$68I8IJikititiv r6rUrlabelsrr rSr c tUbUOURRnURUUUUUUS9nUSn URU SS2SSS245n Sn UGbUR U R 5nURR cURS:XaSURlOoURS:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5n URS:Xa&U "U R5UR55n OU "X5n OURR S:Xa=[5n U "U RSUR5URS55n O,URR S:Xa[!5n U "X5n U(dU 4USS-n U bU 4U -$U $[#U U UR$UR&S 9$) ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrr rSr rr regressionsingle_label_classificationmulti_label_classificationr;)rllogitsrr)rrIrr}rdevice problem_typer{rr$longrmr squeezerrKrrrr)r2rUrrrr rSr rrNrrlloss_fctrs r4rb!ViTForImageClassification.forwards"&1% A 'tn A #+4.A d^A  u++ ,A A r6ry)ryr`r6r)r):rjcollections.abcryrntypingrrrr$torch.utils.checkpointrtorch.nnrrr activationsr modeling_layersr modeling_outputsr rrrmodeling_utilsrr pytorch_utilsrrutilsrrrconfiguration_vitr get_loggerrfrModulerr*rlfloatrrrrrrrrrr6r;r`ry__all__r r6r4rs ,, AA!9 GQ77(   H %UBIIUp$$\% II% <<% % << % U\\* %  %%