tjadZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z dd lmZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZmZmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%ej&e'Z(Gddej)Z*Gddej)Z+ d>dej)dej,dej,dej,dej,dzde-dzde-deefdZ.Gd d!ej)Z/Gd"d#ej)Z0Gd$d%ej)Z1Gd&d'ej)Z2Gd(d)ej)Z3Gd*d+ej)Z4Gd,d-ej)Z5Gd.d/eZ6eGd0d1eZ7Gd2d3e7Z8eGd4d5e7Z9ed67Gd8d9e7Z:ed:7Gd;dD ** &))"--t/A/A*BR/S/S/]/]^_/`/`blJ ^**:r2>> Y J7Q???  $"?"? FTY"Z"ZZ \\*-- r5N) __name__ __module__ __qualname____doc__rr!r#TensorintrVrb __classcell__r3s@r4rr&s |       &D5<&D&DUX&D]b]i&D&D&D&DPEL5>L=((6?([[L,|U33K''1--88::K  $$r5creZdZdeffd Zdejdeede ejejffdZ xZ S)Dinov2SelfAttentionrct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |j |_ |jdz|_ d|_ tj|j|j |j|_tj|j|j |j|_tj|j|j |j|_dS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rxrFbias)r r!r%num_attention_headshasattrryrriattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasr~rrr2rr3s r4r!zDinov2SelfAttention.__init__sB    : :a ? ?PVXhHiHi ?76#5773777   #)#= #&v'9FVV 0DHH]++0)<FFq!LL 4djj//4i@JJ1aPP 4djj//4i@JJ1aPP (?(M K ,.E) ) *=)<      * nL#}CCC$2C * *  * * & #0"4"4"6"6ss";t?Q>S"S%--.EFF o--r5) rdrerfrr!r#rhrrtuplerbrjrks@r4rrs]|]]]]]](.|.+,. u|U\) * ........r5rcZeZdZdZdeffd ZdejdejdejfdZxZ S)Dinov2SelfOutputz The residual connection is defined in Dinov2Layer instead of here (as is the case with other models), due to the layernorm applied before each block. rcttj|j|j|_tj|j|_dSrc) r r!rrr%denser.r/r0rs r4r!zDinov2SelfOutput.__init__sJ Yv163EFF z&"<== r5r input_tensorrcZ||}||}|Src)rr0)r2rrs r4rbzDinov2SelfOutput.forwards* =11  ]33 r5 rdrerfrgrr!r#rhrbrjrks@r4rrs >|>>>>>> U\RWR^r5rcXeZdZdeffd ZdejdeedejfdZ xZ S)Dinov2Attentionrctt||_t ||_dSrc)r r!r attentionroutputrs r4r!zDinov2Attention.__init__s; ,V44&v.. r5rrrc T|j|fi|\}}|||}|Src)rr)r2rrself_attn_outputr`rs r4rbzDinov2Attention.forwards< -dn]EEfEE!-}== r5) rdrerfrr!r#rhrrrbrjrks@r4rrs~/|////// |+,  r5rcDeZdZdfd ZdejdejfdZxZS)Dinov2LayerScalerNcttj|jt j|jz|_dSrc) r r!rr"layerscale_valuer#onesr%lambda1rs r4r!zDinov2LayerScale.__init__sC |F$;ejI[>\>\$\]] r5 hidden_statec||jzSrc)rr2rs r4rbzDinov2LayerScale.forwardsdl**r5rNrdrerfr!r#rhrbrjrks@r4rrsi^^^^^^+EL+U\++++++++r5rcDeZdZdfd ZdejdejfdZxZS) Dinov2MLPrNc~t|jx}}t|j|jz}t j||d|_t|j trt|j |_ n |j |_ t j||d|_ dS)NTr)r r!r%ri mlp_ratiorrfc1rr hidden_actstrr activationfc2r2r in_features out_featureshidden_featuresr3s r4r!zDinov2MLP.__init__s %+%77 lf063CCDD9[/EEE f' - - 0$V%67DOO$/DO9_lFFFr5rc||}||}||}|Src)rrrrs r4rbzDinov2MLP.forward%s;xx -- |44 xx -- r5rrrks@r4rrsi G G G G G GELU\r5rcDeZdZdfd ZdejdejfdZxZS)Dinov2SwiGLUFFNrNcDt|jx}}t|j|jz}t|dzdz dzdzdz}t j|d|zd|_t j||d|_dS)Nr;rTr) r r!r%rirrr weights_in weights_outrs r4r!zDinov2SwiGLUFFN.__init__-s %+%77 lf063CCDD2Q677!;AAE)K_1D4PPP9_lNNNr5rc||}|dd\}}tj||z}||S)Nr;r:rB)rchunkrrIsilur)r2rx1x2hiddens r4rbzDinov2SwiGLUFFN.forward6s]|44 ##A2#..B##B''",'''r5rrrks@r4rr,siOOOOOO(EL(U\((((((((r5rc^eZdZdZd deddffd ZdejdejfdZde fd Z xZ S) Dinov2DropPathzStochastic depth (DropPath) per sample, for residual blocks. Identity when ``drop_prob`` is 0 or outside training. See `Deep Networks with Stochastic Depth `_. r| drop_probrNcVt||_dSrc)r r!r)r2rr3s r4r!zDinov2DropPath.__init__Es$ "r5rc|jdks|js|Sd|jz }|jdfd|jdz zz}t j||j|j}t j||z}| ||zS)Nr|rr)r)rAdevice) rrrDndimr#randrArfloordiv)r2r keep_probrD random_tensors r4rbzDinov2DropPath.forwardIs >S   & $Q')DM4F4J,KK 5 0CML`aaa  MI$=>>   ++m;;r5cd|jS)Nzp=)rr2s r4 extra_reprzDinov2DropPath.extra_reprRs$DN$$$r5)r|) rdrerfrgfloatr!r#rhrbrrrjrks@r4rr>s ##%#$######>>cecncpcp\&"4&:OPPP   )&v..DHH ((DH,V44r5rcb||}||}||}|||z}||}||}||}|||z}|Src)rrrrrrr)r2rhidden_states_normself_attention_output layer_outputs r4rbzDinov2Layer.forwardis"ZZ 66 $/A B B $ 1 12G H H'<== M zz-00 xx -- ((66 ~~l33mC r5rrks@r4rrVs{MM5|5555555 | r5rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ eedZejdejejzejzd d ffd ZxZS) Dinov2PreTrainedModelrdinov2rW)imageTr)r attentionsr}rNct|t|tjtjfrJt j|jd|j j |j t j |j dSdSt|trut j|jd|j j t j|jd|j j |j jrt j |jdSdSt|t$r&t j|j|j jdSdS)zInitialize the weightsr|)meanstdN)r _init_weightsrrrrrvinit trunc_normal_r[rinitializer_rangerzeros_rr-r&r'r)r constant_rr)r2r}r3s r4r z#Dinov2PreTrainedModel._init_weightssC f%%% fry")4 5 5 I  v}3DK4;+G H H H H H I Ir5)rdrerfr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsr#no_gradrrrvrr rjrks@r4rrs  $O!&*#&N"&$) U]__ IBI $9BL$H IT I I I I I_ I I I I Ir5rceZdZdeffd Zeeddejde e de fdZ xZ S) Dinov2Encoderrcttjfdt jD|_|dS)Nc.g|]}tS)r.0r`rs r4 z*Dinov2Encoder.__init__..s!#a#a#aAK$7$7#a#a#ar5)r r!r ModuleListrangenum_hidden_layerslayer post_initrs `r4r!zDinov2Encoder.__init__sc    ]#a#a#a#avG_A`A`#a#a#abb  r5F)tie_last_hidden_statesrrrc L|jD] }||}t|S)N)last_hidden_state)r)r )r2rr layer_modules r4rbzDinov2Encoder.forwards7!J 8 8L(L77MM????r5)rdrerfrr!rrr#rhrrr rbrjrks@r4rrs|  _E222@U\@VDV=W@\k@@@32 @@@@@r5rc eZdZdeffd ZdefdZee d de j dzde j dzde e de fd ZxZS) Dinov2Modelrct|||_t||_t ||_tj|j |j |_ | dS)Nr) r r!rrr6rencoderrrr%r layernormr*rs r4r!zDinov2Model.__init__st     *622$V,, f&8f>STTT r5rc|jjSrcr6r+rs r4get_input_embeddingsz Dinov2Model.get_input_embeddings //r5NrWrXrc |td|||}|j|fi|}|j}||}|dddddf}t |||j|jS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Only relevant for pre-training. Nz You have to specify pixel_values)rXr)r- pooler_outputrr)ryr6r2r-r3r rr)r2rWrXrembedding_outputencoder_outputssequence_output pooled_outputs r4rbzDinov2Model.forwards  ?@@ @??<?YY+74<8H+S+SF+S+S);..99'1aaa0 )-')7&1     r5NN)rdrerfrr!r*r6rrr#rhrrr rbrjrks@r4r0r0s |      0&;0000-1/3  lT) , +,  $    ^     r5r0z Dinov2 Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. ) custom_introc eZdZdeddffd Zee d dejdzdejdzde e de fdZ xZ S) Dinov2ForImageClassificationrrNc<t||j|_t||_|jdkr"t j|jdz|jnt j|_ | dS)Nrr;) r r! num_labelsr0rrrr%r classifierr*rs r4r!z%Dinov2ForImageClassification.__init__s     +!&)) EKDUXYDYDYBIf(1,f.? @ @ @_a_j_l_l  r5rWlabelsrc N|j|fi|}|j}|dddf}|ddddf}tj||dgd}||} d} ||j|| |jfi|} t| | |j |j S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrB)losslogitsrr) rr-r#rNr rD loss_functionrrrr) r2rWrEroutputsr<r& patch_tokens linear_inputrHrGs r4rbz$Dinov2ForImageClassification.forwards/:dk,.Q.Q&.Q.Q!3#AAAqD) &qqq!""u- y)\->->1->-E-E!FANNN ..  %4%ffdkLLVLLD$!/)     r5r>)rdrerfrr!rrr#rhrrrrbrjrks@r4rArAs |       -1&*  lT)  t# +,      ^     r5rAzO Dinov2 backbone, to be used with frameworks like DETR and MaskFormer. c eZdZfdZdefdZeeede j de e de fdZxZS)Dinov2Backbonec^tfdtjdzD|_t |_t|_tj j j |_ |dS)Ncg|] }j Sr")r%r#s r4r%z+Dinov2Backbone.__init__..!s]]]AV/]]]r5rr)r r!r'r( num_featuresrr6rr2rrr%rr3r*rs `r4r!zDinov2Backbone.__init__s    ]]]]v?WZ[?[9\9\]]]*622$V,, f&8f>STTT r5rc|jjSrcr5rs r4r6z#Dinov2Backbone.get_input_embeddings*r7r5rWrc Td|d<||}|j|fi|}|j}g}t|j|D]\}}||jvr|jjr||}|jj rn|ddddf}|j \} } } } |jj } | | | | z| | zd}| dddd}||t!t#|||j S) av Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base") >>> model = AutoBackbone.from_pretrained( ... "facebook/dinov2-base", out_features=["stage2", "stage5", "stage8", "stage11"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 16, 16] ```Toutput_hidden_statesNrr:rrr;) feature_mapsrr)r6r2rzip stage_namesrrapply_layernormr3reshape_hidden_statesrDr1rGrHrappendr rr)r2rWrr:rrrUstagerr_r`r7r8r1s r4rbzDinov2Backbone.forward-sdD*.%&??<88".$,/?"J"J6"J"J,  #&t'7#G#G 2 2 E<)));.@#'>>,#?#?L;4Q#/122#6L4@3E0J65!%!7J#/#7#7 FjDXZ_cmZmoq#r#rL#/#7#71a#C#C#N#N#P#PL##L111|,,'(    r5)rdrerfr!r*r6rr rr#rhrrr rbrjrks@r4rNrNs     0&;0000 8 l8 +,8   8 8 8 ^! 8 8 8 8 8 r5rN)rAr0rrN)Nr|)=rgcollections.abcrsrr#rrr  activationsrbackbone_utilsrr modeling_layersr modeling_outputsr r r rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_dinov2r get_loggerrdloggerModulerr*rhrrrrrrrrrrrrr0rArN__all__r"r5r4rls$$$$$$ &&&&&&!!!!!!HHHHHHHH999999rrrrrrrrrrrrFFFFFFFF&&&&&&KKKKKKKKKKKKIIIIIIII555555......  H % %NNNNNryNNNbBIP!%% I% <% % < % L4' % T\ %%'(%%%%:4.4.4.4.4.")4.4.4.pry$     bi    +++++ry+++ &(((((bi((($%%%%%RY%%%0&&&&&,&&&RIIIIIOIIIB @ @ @ @ @) @ @ @, , , , , ', , , ^ / / / / / #8/ /  / d K K K K K ]$9K K  K \ e d dr5