S\R>S\R>S\\R>S\ S\ S\\4Sjjr!Sr"S\R>S\#S\R>4S jr$S!\R>S"\R>S#\R>S$\R>S\%\R>\R>44 S%jr&"S&S'\R65r'"S(S)\ 5r("S*S+\R65r)"S,S-\R65r*\"S.S/\55r+\"S0S19"S2S3\+55r,S/S3/r-g)5)CallableOptionalUnionN)ACT2FN)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstring torch_int)MLCDVisionConfigcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)MLCDMLP$c>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gN) super__init__configr hidden_act activation_fnnnLinear hidden_sizeintermediate_sizefc1fc2selfr __class__s ^/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/mlcd/modeling_mlcd.pyrMLCDMLP.__init__%sb  #F$5$5699V//1I1IJ99V55v7I7IJ hidden_statesreturnclURU5nURU5nURU5nU$r)r"rr#)r%r*s r'forwardMLCDMLP.forward,s4/ **=9 / r))rrr"r#) __name__ __module__ __qualname____firstlineno__rtorchTensorr-__static_attributes__ __classcell__r&s@r'rr$s)KU\\ellr)rcf^\rSrSrS S\S\SS4U4SjjjrS\S\S\R4S jr S r U=r $) MLCDRotaryEmbedding3dimthetar+Nc >[TU]5 SU[R"SUS[RS9U- -- nUR SUSS9 g)N?rdtypeinv_freqF persistent)rrr3arangefloatregister_buffer)r%r;r<rBr&s r'rMLCDRotaryEmbedding.__init__4sK %ELLC%++$NQT$TUV ZeDr)num_patches_heightnum_patches_widthc~[R"XRRS9R S5R SU5n[R"X RRS9R S5R US5n[R "UR5UR5/SS9n[X5n[R"X`RRURRS9n[R"XpR5nXRS5n U $)aE Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicerrr;)rLrA) r3rErBrL unsqueezeexpandstackflattenmaxrAouter) r%rIrJhpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embs r'r-MLCDRotaryEmbedding.forward9s LL+MM4H4H I S STU V ] ]^`bs t  LL*==3G3G H R RST U \ \]oqs t  ++x//183C3C3EFBO.B ll=1E1ET]]M`M`a#kk#}}=-5==a@r))g@) r/r0r1r2intrFrr3r4r-r5r6r7s@r'r9r93sJECEEDEE ##%,,r)r9c^\rSrSrS\4U4SjjrS\RS\S\S\R4Sjr S \RS\R4S jr S r U=r $) MLCDVisionEmbeddingsZrcv>[TU]5 XlURUlUR UlUR Ul[R"[R"UR55Ul [R"URURUR UR SS9UlUR UR -S-UlURS-UlUR#S[R$"UR 5R'S5SS9 g)NF) in_channels out_channels kernel_sizestridebiasr?r position_ids)rrMrC)rrrr embed_dim image_size patch_sizer Parameterr3randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positionsrGrErPr$s r'rMLCDVisionEmbeddings.__init__[s  ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1 ^U\\$:L:L-M-T-TU\-]jopr) embeddingsheightwidthr+cURSS- nURRRS5nURSS- n[R R 5(d%XF:Xa X#:XaURUR5$USS2SS24nUSS2SS24nURSn X R-n X0R-n [US-5n URSXU 5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU 5n[R "Xx4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrMg?rr?bicubicF)sizemode align_cornersrN)shapeposition_embeddingweightrOr3jit is_tracingrhrkrreshapepermuter functional interpolateviewcat) r%rurvrwrrr~rsclass_pos_embedpatch_pos_embedr; new_height new_widthsqrt_num_positionss r'interpolate_pos_encoding-MLCDVisionEmbeddings.interpolate_pos_encodingpsi!&&q)A- !44;;EEaH*003a7 yy##%%+*F6?**4+<+<= =,QU3,QU3r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr) pixel_valuescHURSnURRRnURUR US95nUR S5R SS5nURRUSS5n[R"XT/SS9nU$)Nrr@r?rrMrN) r}rqrrAtorR transposernrPr3r)r%r batch_size target_dtype patch_embeds class_embedsrus r'r-MLCDVisionEmbeddings.forwards!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C r))rnrrirjrrrsrqrk)r/r0r1r2rrr3r4r^r FloatTensorr-r5r6r7s@r'r`r`Zsgq/q*'D5<<'D'DUX'D]b]i]i'DR E$5$5 %,,  r)r`modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nr?rrM)r;rA)ptrainingr) repeat_kvnum_key_value_groupsr3matmulrr}rrsoftmaxfloat32rrArr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r'eager_attention_forwardrs3 ; ;