Qr=g)AzPyTorch DINOv2 model.N)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) BackboneMixin) Dinov2Configc^\rSrSrSrS\SS4U4SjjrS\RS\ S \ S\R4S jr SS \RS \ \RS\R4S jjr Sr U=r$)Dinov2Embeddings&zE Construct the CLS token, mask token, position and patch embeddings. configreturnNc>[TU]5 [R"[R "SSUR 55UlUR(a:[R"[R"SUR 55Ul [U5Ul URRn[R"[R "SUS-UR 55Ul[R"UR 5UlUR$UlURUlXlg)Nr)super__init__r Parametertorchrandn hidden_size cls_tokenuse_mask_tokenzeros mask_tokenDinov2PatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_sizer)selfrr- __class__s b/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/dinov2/modeling_dinov2.pyr"Dinov2Embeddings.__init__+s ekk!Q8J8J&KL   ll5;;q&:L:L+MNDO 5f =++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++$33 embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5nURn [RRUR[R5X4SS S 9RU S 9nURSSSS5RSSU5n[R "Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and interpolation at torch.float32 precision. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rr bicubicF)sizemode align_cornersdtypedim)shaper.r$jit is_tracingr2rreshapepermuterCr functional interpolatetofloat32viewcat) r3r8r9r:r- num_positionsclass_pos_embedpatch_pos_embedrE new_height new_widthsqrt_num_positions target_dtypes r5interpolate_pos_encoding)Dinov2Embeddings.interpolate_pos_encoding9s~!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r". __, &}c'9:)11!5G]`a)11!Q1=&,, --33   u}} -( 4 "<" *11!Q1=BB1b#Nyy/;CCr7 pixel_valuesbool_masked_posc>URup4pVURRRRnURUR US95nUbjUR (aY[R"URS5URR UR5RS5U5nURRUSS5n [R"X4SS9nXRXU5-nURU5nU$)NrBr<rrrD)rFr, projectionweightrCrMr(r$where unsqueezer*r'expandrPrXr1) r3rZr[ batch_size_r9r:rWr8 cls_tokenss r5forwardDinov2Embeddings.forwardas'3'9'9$ v,,77>>DD **+>))"-t/A/A*BRBR/S/]/]^_/`blJ ^^**:r2> YY 7Q?  "?"? TY"ZZ \\*- r7)r'rr1r*r,r2r.r(N)__name__ __module__ __qualname____firstlineno____doc__rr"r$TensorintrXrre__static_attributes__ __classcell__r4s@r5rr&s |  &D5<<&D&DUX&D]b]i]i&DPELL8ELLCYejeqeqr7rcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)r+wz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c>[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nX lX0lX@lX`l [R"XEX3S9Ul g)Nrr) kernel_sizestride)r!r" image_sizer2 num_channelsr& isinstance collectionsabcIterabler-rConv2dr])r3rrwr2rxr&r-r4s r5r"Dinov2PatchEmbeddings.__init__~s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:ir7rZrcURSnX R:wa[SURSUS35eURU5R S5R SS5nU$)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r=)rFrx ValueErrorr]flatten transpose)r3rZrxr8s r5reDinov2PatchEmbeddings.forwardsx#))!, ,, ,!../yaI __\2::1=GG1M r7)rwrxr-r2r]) rhrirjrkrlr"r$rmrerorprqs@r5r+r+ws. jELLU\\r7r+modulequerykeyvalueattention_maskscalingr1c [R"XRSS55U-n[RR US[R S9RUR5n[RRXURS9nUbX-n[R"X5n U RSS5R5n X4$)Nr<)rErC)ptrainingrr=) r$matmulrrrKsoftmaxrNrMrCr1r contiguous) rrrrrrr1kwargs attn_weights attn_outputs r5eager_attention_forwardrs<<}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#4 ,,|3K''1-88:K  $$r7c ^\rSrSrS\SS4U4SjjrS S\\RS\ S\ \ \R\R4\ \R44Sjjr S r U=r$) Dinov2SelfAttentionrrNc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rgFbias)r!r"r&num_attention_headshasattrrrrnattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasrrrr3rr4s r5r"Dinov2SelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=2O\M]r7) rrrrrrrrrrNF)rhrirjrkrr"rr$rmboolrtuplererorprqs@r5rrsw]|]].-1"' 1ELL)1 1 uU\\5<</0% 2EE F 11r7rc^\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r $) Dinov2SelfOutputiz The residual connection is defined in Dinov2Layer instead of here (as is the case with other models), due to the layernorm applied before each block. rrNc>[TU]5 [R"URUR5Ul[R "UR5Ulgrg) r!r"rrr&denser/r0r1rs r5r"Dinov2SelfOutput.__init__sB YYv1163E3EF zz&"<"<= r7r input_tensorcJURU5nURU5nU$rgrr1)r3rrs r5reDinov2SelfOutput.forward s$ =1  ]3 r7r) rhrirjrkrlrr"r$rmrerorprqs@r5rrsI >|>> U\\RWR^R^r7rc ^\rSrSrS\SS4U4SjjrS\\SS4SjrS S\ RS \ \ RS \ S\ \\ R\ R4\\ R44S jjrS rU=r$)Dinov2AttentionirrNc>[TU]5 [U5Ul[ U5Ul[ 5Ulgrg)r!r"r attentionroutputset pruned_headsrs r5r"Dinov2Attention.__init__s0 ,V4&v. Er7headsc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)NrrrD)lenrrrrrrrrrrrrunion)r3rindexs r5 prune_headsDinov2Attention.prune_headss u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r7rrrcfURXU5nURUSU5nU4USS-nU$)Nrr)rr)r3rrr self_outputsattention_outputrs r5reDinov2Attention.forward.sC ~~m@QR ;;|A F#% QR(88r7)rrrr)rhrirjrkrr"rrnrr$rmrrrrrerorprqs@r5rrs"|"" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F  r7rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)Dinov2LayerScalei<rc>[TU]5 [R"UR[ R "UR5-5Ulgrg) r!r"rr#layerscale_valuer$onesr&lambda1rs r5r"Dinov2LayerScale.__init__=s8 ||F$;$;ejjI[I[>\$\] r7 hidden_statecXR-$rgrr3rs r5reDinov2LayerScale.forwardAsll**r7rrN rhrirjrkr"r$rmrerorprqs@r5rr<s)^+ELL+U\\++r7rinput drop_probrrcUS:XdU(dU$SU- nURS4SURS- --nU[R"X@RUR S9-nUR 5 URU5U-nU$)a* Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)rCdevice)rFndimr$randrCrfloor_div)rrr keep_probrF random_tensorrs r5 drop_pathrFsCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr7c^\rSrSrSrS S\\SS4U4SjjjrS\RS\R4Sjr S\ 4S jr S r U=r$) Dinov2DropPathi[zXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrrc.>[TU]5 Xlgrg)r!r"r)r3rr4s r5r"Dinov2DropPath.__init__^s "r7rcB[XRUR5$rg)rrr)r3rs r5reDinov2DropPath.forwardbs FFr7c SUR3$)Nzp=rr3s r5 extra_reprDinov2DropPath.extra_represDNN#$$r7rrg)rhrirjrkrlrfloatr"r$rmrestrrrorprqs@r5rr[sQb#(5/#T##GU\\GellG%C%%r7rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $) Dinov2MLPiircz>[TU]5 UR=p#[URUR-5n[ R "X$SS9Ul[UR[5(a[URUl OURUl [ R "XCSS9Ul g)NTr)r!r"r&rn mlp_ratiorrfc1ry hidden_actrr activationfc2r3r in_features out_featureshidden_featuresr4s r5r"Dinov2MLP.__init__js %+%7%77 f0063C3CCD99[E f'' - -$V%6%67DO$//DO99_Fr7rclURU5nURU5nURU5nU$rg)rrrrs r5reDinov2MLP.forwardus2xx - |4 xx - r7)rrrrrrqs@r5rris) GELLU\\r7rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)Dinov2SwiGLUFFNi|rc$>[TU]5 UR=p#[URUR-5n[US-S- 5S-S-S-n[ R "USU-SS9Ul[ R "XCSS9Ulg)Nr=r Tr) r!r"r&rnrrr weights_in weights_outr s r5r"Dinov2SwiGLUFFN.__init__}s %+%7%77 f0063C3CCD2Q67!;AAE))K_1D4P99_Nr7rcURU5nURSSS9up#[RR U5U-nUR U5$)Nr=r<rD)rchunkrrKsilur)r3rx1x2hiddens r5reDinov2SwiGLUFFN.forwardsQ|4 ##A2#.##B'",''r7)rrrrrqs@r5rr|s)O(ELL(U\\((r7rc ^\rSrSrSrS\SS4U4SjjrS S\RS\ \RS \ S\ \ \R\R4\ \R44S jjr S rU=r$) Dinov2LayerizCThis corresponds to the Block class in the original implementation.rrNc>[TU]5 [R"URUR S9Ul[U5Ul[U5Ul URS:a[UR5O[R"5Ul[R"URUR S9UlUR (a[#U5UlO['U5Ul[U5Ulg)Nepsr)r!r"r LayerNormr&layer_norm_epsnorm1rrr layer_scale1drop_path_raterIdentityrnorm2use_swiglu_ffnrmlpr layer_scale2rs r5r"Dinov2Layer.__init__s \\&"4"4&:O:OP (0,V4BHBWBWZ]B](=(=>cecncncp\\&"4"4&:O:OP  &v.DH (DH,V4r7rrrc>URURU5UUS9nUSnURU5nUSSnURU5U-nUR U5nUR U5nUR U5nURU5U-nU4U-nU$)N)rrr)rr&r'rr*r,r-)r3rrrself_attention_outputsrr layer_outputs r5reDinov2Layer.forwards "& JJ} % /"0"  2!4,,-=>(,'78=H zz-0 xx - ((6 ~~l3mC /G+r7)rrr'r-r,r&r*r)rhrirjrkrlrr"r$rmrrrrrerorprqs@r5r r sM5|55&-1"' ||ELL)  uU\\5<</0% 2EE F r7r c^\rSrSrS\SS4U4SjjrS S\RS\\RS\ S \ S \ S\ \ \ 44 S jjr S rU=r$) Dinov2EncoderirrNc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfr) r!r"rr ModuleListrangenum_hidden_layersr layergradient_checkpointingr3rrcr4s r5r"Dinov2Encoder.__init__sR  ]]vG_G_A`#aA`AK$7A`#ab &+#$bsA%rrroutput_hidden_states return_dictc6U(aSOSnU(aSOSn[UR5H9upU(aXa4-nUbX(OSn U "XU5n U SnU(dM1X{S4-nM; U(aXa4-nU(d[SXU455$[UUUS9$)Nrrc3,# UH ocMUv M g7frgr@).0vs r5 (Dinov2Encoder.forward..sm$[q$[s )last_hidden_stater attentions) enumerater9rr) r3rrrr=r>all_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r5reDinov2Encoder.forwards#7BD$5b4(4OA#$58H$H!.7.CilO(IZ[M)!,M  &91=M[TU]U5 Xl[U5Ul[ U5Ul[R"URURS9Ul UR5 g)Nr") r!r"rrr8r4encoderrr$r&r% layernorm post_initrs r5r"Dinov2Model.__init__sW   *62$V, f&8&8f>S>ST r7rc.URR$rgr8r,rs r5get_input_embeddings Dinov2Model.get_input_embeddings&///r7heads_to_pruneNcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrjr9rr)r3rsr9rs r5 _prune_headsDinov2Model._prune_heads)s< +002LE LL  u % / / ; ;E B3r7rZr[rrr=r>cUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR X0RR 5nURXS9nURUUUUUS9nUSn URU 5n U SS2SSS24n U(d X4n XSS-$[U U URURS9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Only relevant for pre-training. Nz You have to specify pixel_values)r[rrr=r>rr)rF pooler_outputrrG) rrr=use_return_dictr get_head_maskr8r8rjrkrrrG) r3rZr[rrr=r>embedding_outputencoder_outputssequence_output pooled_output head_outputss r5reDinov2Model.forward1s2C1N-TXT_T_TqTq$8$D $++JjJj &1% 4 'tn 4 d^4  u00 14 4 r7rhz Dinov2 Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. ) custom_introc^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \\ S\ \ \ 44S jj5rS rU=r$)Dinov2ForImageClassificationiirrNc6>[TU]U5 URUl[U5UlURS:a.[ R "URS-UR5O[ R"5Ul UR5 g)Nrr=) r!r" num_labelsrhrRrrr&r) classifierrlrs r5r"%Dinov2ForImageClassification.__init__psy   ++!&) EKDUDUXYDYBIIf((1,f.?.? @_a_j_j_l  r7rZrlabelsrr=r>cUbUOURRnURUUUUUS9nUSnUSS2S4n USS2SS24n [R"XR SS9/SS9n UR U 5n Sn UGbURU R5nURRcURS:XaSURl OoURS:aNUR[R:XdUR[R:XaSURl OSURl URRS:XaI[5nURS:Xa&U"U R5UR55n OU"X5n OURRS:Xa=[!5nU"U R#S UR5UR#S 55n O,URRS:Xa[%5nU"X5n U(dU 4US S-nU bU 4U-$U$['U U UR(UR*S 9$) ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NryrrrD regressionsingle_label_classificationmulti_label_classificationr<r=)losslogitsrrG)rr{rRr$rPrTrrMr problem_typerrClongrnr squeezerrOrrrrG)r3rZrrrr=r>rrr' patch_tokens linear_inputrrloss_fctrs r5re$Dinov2ForImageClassification.forward~s &1%->1->-E!FAN .  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./Y,F)-)9TGf$ Ev E$!//))   r7)rrRrr)rhrirjrkrr"rrr$rmrrrrrerorprqs@r5rris |  04,0)-,0/3&*D u||,D ELL)D & D $D> D 'tn D d^D  u++ ,D D r7rzO Dinov2 backbone, to be used with frameworks like DETR and MaskFormer. c ^\rSrSrU4SjrS\4Sjr\S S\RS\ \ S\ \ S\ \ S\ 4 S jj5r S rU=r$) Dinov2Backboneict>[TU]U5 [TU] U5 [URS-5Vs/sHo!R PM snUl[U5Ul[U5Ul [R"UR URS9UlUR5 gs snf)Nrr")r!r"_init_backboner7r8r& num_featuresrr8r4rjrr$r%rkrlr;s r5r"Dinov2Backbone.__init__s   v&9>v?W?WZ[?[9\]9\A//9\]*62$V, f&8&8f>S>ST ^sB5rc.URR$rgrors r5rp#Dinov2Backbone.get_input_embeddingsrrr7rZr=rr>cUbUOURRnUbUOURRnUbUOURRnUR U5nUR USX4S9nU(a UR OUSnSn[URU5HupXR;dMURR(aURU 5n URR(ajU SS2SS24n URuppURRnU RXU-X-S5n U R!SSSS 5R#5n X4- nM U(dU(a U4USS-nU$U4US S-nU$[%UU(a UR OSU(aUR&S 9$SS 9$) a Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base") >>> model = AutoBackbone.from_pretrained( ... "facebook/dinov2-base", out_features=["stage2", "stage5", "stage8", "stage11"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 16, 16] ```NT)r=rr>rr@r<rr r=) feature_mapsrrG)rr{r=rr8rjrzip stage_namesr apply_layernormrkreshape_hidden_statesrFr2rIrJrr rG)r3rZr=rr>r}rrrstagerrbrcr9r:r2rs r5reDinov2Backbone.forwards@&1%>,#?L;;44#/12#6L4@3E3E0J6!%!7!7J#/#7#7 jDXZ_Zmoq#rL#/#7#71a#C#N#N#PL/ $H#&712;6M'712;6M%3G'//T->w))  EI  r7)r8rjrkr)NNN)rhrirjrkr"r+rprr$rmrrr rerorprqs@r5rrs{ 0&;004,0&* G llG 'tnG $D> G d^ G  G G r7r)rrhrQr)r)rF)>rlcollections.abcrztypingrrrr$torch.utils.checkpointrtorch.nnrrr activationsr modeling_layersr modeling_outputsr rrrmodeling_utilsrr pytorch_utilsrrutilsrrrutils.backbone_utilsrconfiguration_dinov2r get_loggerrhrModulerr+rmrrrrrrrrrrrr r4rQrhrr__all__r@r7r5rs7,, AA!9rrFQ771.   H %NryyNbBIIR% II% <<% % << % U\\* %  %%>F"))FTryy&$bii$N+ryy+U\\eTV[VbVb* %RYY % &(bii("0,0h( BII( V(DO(D(DVM 'M M ` T #8T  T n Y *MY  Y x er7