[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nX lX0lX@lX`l [R"XEX3S9Ul g)Nrr) kernel_sizestride)super__init__ image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesnnConv2d projection)selfconfigr r!r"r#r( __class__s `/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/ijepa/modeling_ijepa.pyrIJepaPatchEmbeddings.__init__!s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:i pixel_valuesinterpolate_pos_encodingreturnc URup4pVX@R:wa[SURSUS35eU(dVXPRS:wdX`RS:wa2[SUSUSURSSURSS 3 5eUR U5R S 5R SS 5nU$) NzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rrzInput image size (*z) doesn't match model (z).)shaper" ValueErrorr r+flatten transpose)r,r2r3 batch_sizer"heightwidth embeddingss r/forwardIJepaPatchEmbeddings.forward0s2>2D2D/ & ,, ,!../yaI (++u8J/J (%9+,Adooa.@-AE__\2::1=GG1M r1)r r"r(r!r+F) __name__ __module__ __qualname____firstlineno____doc__rtorchTensorboolrA__static_attributes__ __classcell__r.s@r/rrs8 jELLD]b]i]ir1rc ^\rSrSrSrSS\S\SS4U4SjjjrS\RS \ S \ S\R4S jr SS \RS \ \RS\S\R4SjjrSrU=r$)IJepaEmbeddingsAzZ Construct the CLS token, position and patch embeddings. Optionally, also the mask token. r-use_mask_tokenr4Nc>[TU]5 U(a6[R"[R "SSUR 55OSUl[U5Ul URRn[R"[R"SX1R 55Ul [R"UR5UlUR UlXlg)Nr)rrr) ParameterrIzerosr# mask_tokenrpatch_embeddingsr(randnposition_embeddingsDropouthidden_dropout_probdropoutr!r-)r,r-rRr(r.s r/rIJepaEmbeddings.__init__Fs Q_",,u{{1a9K9K'LMei 4V <++77 #%<< A{L^L^0_#` zz&"<"<=  ++ r1r@r>r?c,URSnURRSn[RR 5(dXE:XaX#:Xa UR$URnURSnX R -nX0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SSS 9nURSSSS5RSSU5nU$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rg?rr r8bicubicF)sizemode align_corners) r9rYrIjit is_tracingr!rreshapepermuter) functional interpolateview) r,r@r>r?r( num_positionspatch_pos_embeddim new_height new_widthsqrt_num_positionss r/r3(IJepaEmbeddings.interpolate_pos_encodingPs!&&q) 0066q9 yy##%%+*F6?++ +22r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nr1r2bool_masked_posr3cnURupEpgURXS9nUbXURSn URRXIS5n UR S5R U 5n USU - -X--nU(aXR XU5-nOXR-nURU5nU$)N)r3rr_?) r9rWrVexpand unsqueezetype_asr3rYr\) r,r2rrr3r=_r>r?r@ seq_length mask_tokensmasks r/rAIJepaEmbeddings.forwardws (4'9'9$ v**<*k  &#))!,J//00LK",,R088ED#sTz2[5GGJ $#&C&CJX]&^^J#&>&>>J\\*- r1)r-r\rVrWr!rYrCNF)rDrErFrGrHrrKrrIrJintr3r BoolTensorrArLrMrNs@r/rPrPAs{DT%5<<%%UX%]b]i]i%T7;). ll"%"2"23#'   r1rPc\rSrSr%\\S'SrSrSrSS/r Sr Sr Sr Sr S\\R \R"\R$4S S 4S jrS rg ) IJepaPreTrainedModelr-ijepar2TrP IJepaLayermoduler4Nc[U[R[R45(a[RR UR RR[R5SURRS9RUR R5UR lURb%URRR5 gg[U[R 5(aJURRR5 UR RR#S5 g[U[$5(a[RR UR&RR[R5SURRS9RUR&R5UR&lUR(b%UR(RR5 ggg)zInitialize the weights)meanstdNrt)r$r)Linearr*init trunc_normal_weightdatatorIfloat32r-initializer_rangedtypebiaszero_ LayerNormfill_rPrYrV)r,rs r/ _init_weights"IJepaPreTrainedModel._init_weightss fryy"))4 5 5"$!6!6 ""%%emm43DKKDaDa"7"b$$% MM {{&   &&('  - - KK   " " $ MM   $ $S )  0 0.0gg.C.C**//225==AKK11/D/b++112  & & +   ,!!&&,,.- 1r1)rDrErFrGr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr)rr*rrrLrr1r/rrsg $O&*#*L9N"&/E"))RYY *L$M/RV/r1rrquerykeyvalueattention_maskscalingr\c [R"XRSS55U-n[RR US[R S9RUR5n[RRXURS9nUbX-n[R"X5n U RSS5R5n X4$)Nr_)rmr)ptrainingrr8) rImatmulr<r)rhsoftmaxrrrr\r contiguous) rrrrrrr\kwargs attn_weights attn_outputs r/eager_attention_forwardrs<<}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#4 ,,|3K''1-88:K  $$r1c ^\rSrSrS\SS4U4SjjrS S\\RS\ S\ \ \R\R4\ \R44Sjjr S r U=r$) IJepaSelfAttentionr-r4Nc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads r6gF)r)rrr#num_attention_headshasattrr:r-r~attention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalr)rqkv_biasrrrr,r-r.s r/rIJepaSelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=2O\M]r1) rrr-rrrrrrrr})rDrErFrGrrrrIrJrKrtuplerArLrMrNs@r/rrsw]{]t].-1"' 1ELL)1 1 uU\\5<</0% 2EE F 11r1rc^\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r $) IJepaSelfOutputiz The residual connection is defined in IJepaLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r-r4Nc>[TU]5 [R"URUR5Ul[R "UR5UlgN) rrr)rr#denserZr[r\rs r/rIJepaSelfOutput.__init__"sB YYv1163E3EF zz&"<"<= r1r input_tensorcJURU5nURU5nU$rrr\r,rrs r/rAIJepaSelfOutput.forward's$ =1  ]3 r1r) rDrErFrGrHrrrIrJrArLrMrNs@r/rrsI >{>t> U\\RWR^R^r1rc ^\rSrSrS\SS4U4SjjrS\\SS4SjrS S\ RS \ \ RS \ S\ \\ R\ R4\\ R44S jjrS rU=r$)IJepaAttentioni.r-r4Nc>[TU]5 [U5Ul[ U5Ul[ 5Ulgr)rrr attentionroutputset pruned_headsrs r/rIJepaAttention.__init__/s0 +F3%f- Er1headsc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrrm)lenrrrrrrrrrrrrunion)r,rindexs r/ prune_headsIJepaAttention.prune_heads5s u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r1rrrcfURXU5nURUSU5nU4USS-nU$)Nrr)rr)r,rrr self_outputsattention_outputrs r/rAIJepaAttention.forwardGsC ~~m@QR ;;|A F#% QR(88r1)rrrr})rDrErFrGrrrr~rrIrJrrKrrrArLrMrNs@r/rr.s"{"t" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F  r1rcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) IJepaIntermediateiUr-r4Nc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rrr)rr#intermediate_sizerr$ hidden_actstrr intermediate_act_fnrs r/rIJepaIntermediate.__init__Vs` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r1rcJURU5nURU5nU$rrr)r,rs r/rAIJepaIntermediate.forward^s& =1 00? r1r rDrErFrGrrrIrJrArLrMrNs@r/rrUs69{9t9U\\ellr1rc^\rSrSrS\SS4U4SjjrS\RS\RS\R4SjrS r U=r $) IJepaOutputier-r4Nc>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) rrr)rrr#rrZr[r\rs r/rIJepaOutput.__init__fsB YYv779K9KL zz&"<"<= r1rrcRURU5nURU5nX-nU$rrrs r/rAIJepaOutput.forwardks, =1  ]3 %4 r1rrrNs@r/rresD>{>t> U\\RWR^R^r1rc ^\rSrSrSrS\SS4U4SjjrS S\RS\ \RS \ S\ \ \R\R4\ \R44S jjr S rU=r$) ritz?This corresponds to the Block class in the timm implementation.r-r4Ncj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nreps)rrchunk_size_feed_forward seq_len_dimrrr intermediaterrr)rr#layer_norm_epslayernorm_beforelayernorm_afterrs r/rIJepaLayer.__init__ws '-'E'E$'/-f5!&) " V-?-?VEZEZ [!||F,>,>FDYDYZr1rrrcURURU5UUS9nUSnUSSnXQ-nURU5nURU5nUR Xq5nU4U-nU$)N)rrr)rrrr r)r,rrrself_attention_outputsrr layer_outputs r/rAIJepaLayer.forwards "&  ! !- 0 /"0"  2!4(,)8 ++M: ((6 {{<? /G+r1)rr r rrrr r})rDrErFrGrHrrrIrJrrKrrrArLrMrNs@r/rrtsI[{[t[-1"' ||ELL)  uU\\5<</0% 2EE F r1rc^\rSrSrS\SS4U4SjjrS S\RS\\RS\ S \ S \ S\ \ \ 44 S jjr S rU=r$) IJepaEncoderir-r4Nc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfr}) rrr-r) ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r,r-rxr.s r/rIJepaEncoder.__init__sR  ]]fF^F^@_#`@_1Jv$6@_#`a &+#$asA%rrroutput_hidden_states return_dictc6U(aSOSnU(aSOSn[UR5H9upU(aXa4-nUbX(OSn U "XU5n U SnU(dM1X{S4-nM; U(aXa4-nU(d[SXU455$[UUUS9$)Nrrrc3,# UH ocMUv M g7frr).0vs r/ 'IJepaEncoder.forward..sm$[q$[s )last_hidden_stater attentions) enumeraterrr ) r,rrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r/rAIJepaEncoder.forwards#7BD$5b4(4OA#$58H$H!.7.CilO(IZ[M)!,M  &91=M[TU]5 [R"URUR 5Ul[URUl gr) rrr)rr#pooler_output_sizerr pooler_act activationrs r/rIJepaPooler.__init__s> YYv1163L3LM  !2!23r1c\USS2S4nURU5nURU5nU$)Nr)rr5)r,rfirst_token_tensor pooled_outputs r/rAIJepaPooler.forwards6+1a40 #56  6 r1)r5r) rDrErFrGrrrArLrMrNs@r/r1r1s4{4 r1r1c"^\rSrSrSS\S\S\4U4SjjjrS\4SjrS\ \ \ \ 4SS 4S jr \ SS \\R S \\R"S \\R S\\S\\S\\S\\S\\\44Sjj5rSrU=r$) IJepaModelir-add_pooling_layerrRc>[TU]U5 Xl[XS9Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rRrN)rrr-rPr@rencoderr)rr#r  layernormr1pooler post_init)r,r-r=rRr.s r/rIJepaModel.__init__si   )&P#F+ f&8&8f>S>ST->k&)D  r1r4c.URR$r)r@rW)r,s r/get_input_embeddingsIJepaModel.get_input_embeddingss///r1heads_to_pruneNcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr?rrr)r,rGrrs r/ _prune_headsIJepaModel._prune_headss< +002LE LL  u % / / ; ;E B3r1r2rrrrrr3rcUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR X0RR 5nURRRRRnURU:waURU5nURXUS9n URU UUUUS9n U Sn URU 5n URbURU 5OSn U(dU bX4OU 4n XSS-$[!U U U R"U R$S9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rrr3)rrrrrr)r& pooler_outputrr')r-rruse_return_dictr: get_head_maskrr@rWr+rrrr?r@rAr rr')r,r2rrrrrr3rexpected_dtypeembedding_outputencoder_outputssequence_outputr9 head_outputss r/rAIJepaModel.forwards|2C1N-TXT_T_TqTq$8$D $++JjJj &1%:L?? Tl+ ,, /!5# ' *!,..98< 8O O4UY ?L?XO;_n^pL!""55 5)-')77&11   r1)r-r@r?r@rA)FFNNNNNNN)rDrErFrGrrKrrrEdictr~listrJrrrIrJrrrr rArLrMrNs@r/r<r<s{t]a$0&:0C4T#Y+?CDC046:,0,0/337&*; u||,; "%"2"23; ELL) ; $D> ; 'tn ; #+4.; d^;  u00 1; ; r1r<a IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states) e.g. for ImageNet. Note that it's possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ) custom_introc^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \\ S \\ S\ \ \ 44S jj5rSrU=r$)IJepaForImageClassificationi6r-r4Nc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 g)NF)r=r) rr num_labelsr<rr)rr#Identity classifierrBrs r/r$IJepaForImageClassification.__init__Ess   ++%@ OUN_N_bcNc"))F$6$68I8IJikititiv r1r2rlabelsrrr3rc zUbUOURRnURUUUUUUS9nUSn URU R SS95n Sn UGbUR U R 5nURRcURS:XaSURlOoURS:aNUR[R:XdUR[R:XaSURlOSURlURRS:XaI[5n URS:Xa&U "U R5UR55n OU "X5n OURRS:Xa=[5n U "U R!S UR5UR!S 55n O,URRS:Xa[#5n U "X5n U(dU 4USS-n U bU 4U -$U $[%U U UR&UR(S 9$) ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrr3rrrr regressionsingle_label_classificationmulti_label_classificationr_)losslogitsrr')r-rNrr_rrdevice problem_typer]rrIlongr~rsqueezerrjrrrr')r,r2rrarrr3rrrSrgrfloss_fctrs r/rA#IJepaForImageClassification.forwardQs"&1% A 'tn A #+4.A d^A  u++ ,A A r1r[)rr<r[)r)5collections.abcr%typingrrrrItorch.nnr)rrr activationsr modeling_layersr modeling_outputsr r rmodeling_utilsrr pytorch_utilsrrutilsrrrconfiguration_ijepar get_loggerrDrModulerrPrrJfloatrrrrrrrrr1r<r[__all__rr1r/r|s,, AA!9bbFQ77,   H %$299$NNbiiNb/?//R% II% <<% % << % U\\* %  %%