S\ RRS\ RXS\ RXS\ RXS\\ RXS\-S\-4Sjjr."SS\ RR5r/"SS\ RR5r0"S S!\ RR5r1"S"S#\ RR5r2"S$S%\ RR5r3"S&S'\5r4"S(S)\ RR5r5\!"S*S+\55r6\!"S,S-\655r7"S.S/\ RR5r8\!"S0S19"S2S3\655r9\!"S4S19"S5S6\655r:\\!"S7S19"S8S9\ 555r;\!"S:S19"S;S<\655r[TU]5 [R"[R "SSUR 55Ul[R"[R "SSUR 55UlU(a6[R"[R "SSUR 55OSUl [U5Ul URRn[R"[R "SUS-UR 55Ul [R"UR5UlUR"Ulg)Nr)super__init__r Parametertorchzeros hidden_size cls_tokendistillation_token mask_tokenDeiTPatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_size)selfrr r/ __class__s ^/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/deit/modeling_deit.pyr%DeiTEmbeddings.__init__0s ekk!Q8J8J&KL"$,,u{{1aASAS/T"UQ_",,u{{1a9K9K'LMei 3F ;++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++ embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr#N?rr bicubicF)sizemode align_cornersdim)shaper0r'jit is_tracingr4rreshapepermuter functional interpolateviewcat) r5r:r;r<r/ num_positionsclass_and_dist_pos_embedpatch_pos_embedrE new_height new_widthsqrt_num_positionss r7interpolate_pos_encoding'DeiTEmbeddings.interpolate_pos_encoding<sU!&&q)A- 0066q9A= yy##%%+*F6?++ +#'#;#;ArrE#B 221ab59r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy2D!LLr9 pixel_valuesbool_masked_posrUcURu pEnURU5nUR5upnUbIURR XS5n UR S5R U 5n USU - -X--nURR USS5n URR USS5n [R"XU4SS9nURnU(aURXuU5nX~-nURU5nU$)Nr>?rrD)rFr.rAr,expand unsqueezetype_asr*r+r'rNr0rUr3)r5rWrXrU_r;r<r: batch_size seq_length mask_tokensmask cls_tokensdistillation_tokensposition_embeddings r7forwardDeiTEmbeddings.forwardds  +001e**<8 $.OO$5!   &//00LK",,R088ED#sTz2[5GGJ^^**:r2> "55<!>zSX!Y 4 \\*- r9)r*r+r3r,r.r4r0)FNF)__name__ __module__ __qualname____firstlineno____doc__rboolr%r'TensorintrUr BoolTensorrf__static_attributes__ __classcell__r6s@r7rr+s ,z ,4 ,D , ,&M5<<&M&MUX&M]b]i]i&MV7;). ll"%"2"23#'   r9rcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)r-z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c>[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nX lX0lX@lX`l [R"XEX3S9Ul g)Nrr) kernel_sizestride)r$r% image_sizer4 num_channelsr) isinstance collectionsabcIterabler/rConv2d projection)r5rrzr4r{r)r/r6s r7r%DeiTPatchEmbeddings.__init__s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:ir9rWr!cURup#pEX0R:wa [S5eURU5R S5R SS5nU$)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r#r)rFr{ ValueErrorrflatten transpose)r5rWr_r{r;r<xs r7rfDeiTPatchEmbeddings.forwards[2>2D2D/ & ,, ,w  OOL ) 1 1! 4 > >q! Dr9)rzr{r/r4r) rirjrkrlrmr%r'rorfrrrsrts@r7r-r-s. jELLU\\r9r-modulequerykeyvalueattention_maskscalingr3c [R"XRSS55U-n[RR US[R S9RUR5n[RRXURS9nUbX-n[R"X5n U RSS5R5n X4$)Nr>)rEdtype)ptrainingrr#) r'matmulrrrKsoftmaxfloat32torr3r contiguous) rrrrrrr3kwargs attn_weights attn_outputs r7eager_attention_forwardrs<<}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#4 ,,|3K''1-88:K  $$r9c ^\rSrSrS\SS4U4SjjrS S\\RS\ S\ \ \R\R4\ \R44Sjjr S r U=r$) DeiTSelfAttentionrr!Nc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)r$r%r)num_attention_headshasattrrrrpattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasrrrr5rr6s r7r%DeiTSelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9Frr#eagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrr3r)rFrrMrrrrrrr_attn_implementationlogger warning_oncerrrrrrArrI)r5 hidden_statesrrr_r`r^ key_layer value_layer query_layerattention_interface context_layerattention_probsnew_context_layer_shapeoutputss r7rfDeiTSelfAttention.forwards %2$7$7!  HH] # T*b$":":DO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=2O\M]r9) rrrrrrrrrrrh)rirjrkrlrr%rr'rornrtuplerfrrrsrts@r7rrsw]z]d].-1"' 1ELL)1 1 uU\\5<</0% 2EE F 11r9rc^\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r $) DeiTSelfOutputiz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rr!Nc>[TU]5 [R"URUR5Ul[R "UR5UlgN) r$r%rrr)denser1r2r3rs r7r%DeiTSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r9r input_tensorcJURU5nURU5nU$rrr3r5rrs r7rfDeiTSelfOutput.forwards$ =1  ]3 r9r) rirjrkrlrmrr%r'rorfrrrsrts@r7rrsI >z>d> U\\RWR^R^r9rc ^\rSrSrS\SS4U4SjjrS\\SS4SjrS S\ RS \ \ RS \ S\ \\ R\ R4\\ R44S jjrS rU=r$) DeiTAttentioni!rr!Nc>[TU]5 [U5Ul[ U5Ul[ 5Ulgr)r$r%r attentionroutputset pruned_headsrs r7r%DeiTAttention.__init__"s0 *62$V, Er9headsc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)NrrrD)lenrrrrrrrrrrrrunion)r5rindexs r7 prune_headsDeiTAttention.prune_heads(s u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r9rrrcfURXU5nURUSU5nU4USS-nU$)Nrr)rr)r5rrr self_outputsattention_outputrs r7rfDeiTAttention.forward:sC ~~m@QR ;;|A F#% QR(88r9)rrrrh)rirjrkrlrr%rrprr'rorrnrrrfrrrsrts@r7rr!s"z"d" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F  r9rcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) DeiTIntermediateiIrr!Nc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r$r%rrr)intermediate_sizerr| hidden_actstrr intermediate_act_fnrs r7r%DeiTIntermediate.__init__Js` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r9rcJURU5nURU5nU$rrr)r5rs r7rfDeiTIntermediate.forwardRs& =1 00? r9r rirjrkrlrr%r'rorfrrrsrts@r7rrIs69z9d9U\\ellr9rc^\rSrSrS\SS4U4SjjrS\RS\RS\R4SjrS r U=r $) DeiTOutputiZrr!Nc>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) r$r%rrrr)rr1r2r3rs r7r%DeiTOutput.__init__[sB YYv779K9KL zz&"<"<= r9rrcRURU5nURU5nX-nU$rrrs r7rfDeiTOutput.forward`s, =1  ]3 %4 r9rrrts@r7rrZsD>z>d> U\\RWR^R^r9rc ^\rSrSrSrS\SS4U4SjjrS S\RS\ \RS \ S\ \ \R\R4\ \R44S jjr S rU=r$) DeiTLayerijz?This corresponds to the Block class in the timm implementation.rr!Ncj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nreps)r$r%chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr)layer_norm_epslayernorm_beforelayernorm_afterrs r7r%DeiTLayer.__init__ms '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr9rrrcURURU5UUS9nUSnUSSnXQ-nURU5nURU5nUR Xq5nU4U-nU$)N)rrr)rrrrr)r5rrrself_attention_outputsrr layer_outputs r7rfDeiTLayer.forwardws "&  ! !- 0 /"0"  2!4(,)8 ++M: ((6 {{<? /G+r9)rrrrrrrrh)rirjrkrlrmrr%r'rorrnrrrfrrrsrts@r7rrjsI[z[d[-1"' ||ELL)  uU\\5<</0% 2EE F r9rc^\rSrSrS\SS4U4SjjrS S\RS\\RS\ S \ S \ S\ \ \ 44 S jjr S rU=r$) DeiTEncoderirr!Nc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfrh) r$r%rr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r5rr^r6s r7r%DeiTEncoder.__init__sR  ]]uVE]E]?^#_?^!If$5?^#_` &+#$`sA%rrroutput_hidden_states return_dictc6U(aSOSnU(aSOSn[UR5H9upU(aXa4-nUbX(OSn U "XU5n U SnU(dM1X{S4-nM; U(aXa4-nU(d[SXU455$[UUUS9$)Nrrc3,# UH ocMUv M g7frr).0vs r7 &DeiTEncoder.forward..sm$[q$[s )last_hidden_stater attentions) enumerater rr) r5rrrr rall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r7rfDeiTEncoder.forwards#7BD$5b4(4OA#$58H$H!.7.CilO(IZ[M)!,M  &91=M[TU]U5 Xl[XS9Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )r rN)r$r%rrr:rencoderrrr)r layernorm DeiTPoolerpooler post_init)r5rr:r r6s r7r%DeiTModel.__init__si   (O"6* f&8&8f>S>ST,=j(4  r9c.URR$r)r:r.)r5s r7get_input_embeddingsDeiTModel.get_input_embeddingss///r9cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr<r rr)r5heads_to_pruner rs r7 _prune_headsDeiTModel._prune_headss< +002LE LL  u % / / ; ;E B3r9rWrXrrr rrUcUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR X0RR 5nURRRRRnURU:waURU5nURXUS9n URU UUUUS9n U Sn URU 5n URbURU 5OSn U(dU bX4OU 4n XSS-$[!U U U R"U R$S9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rXrU)rrr rrr)r pooler_outputrr)rrr use_return_dictr get_head_maskr r:r.rr(rrr<r=r?rrr)r5rWrXrrr rrUexpected_dtypeembedding_outputencoder_outputssequence_output pooled_output head_outputss r7rfDeiTModel.forwards|2C1N-TXT_T_TqTq$8$D $++JjJj &1%:L?? Tl+ ,, /!5# ' *!,..98< 8O O4UY ?L?XO;_n^pL!""55 5)-')77&11   r9)rr:r<r=r?)TFNNNNNNF)rirjrkrlrrnr%r-rCrHrrr'rorqrrrrfrrrsrts@r7r9r9szd[_lp&0&90C046:,0,0/3&*).; u||,; "%"2"23; ELL) ; $D> ; 'tn ; d^; #';  u00 1; ; r9r9c6^\rSrSrS\4U4SjjrSrSrU=r$)r>iArc>[TU]5 [R"URUR 5Ul[URUl gr) r$r%rrr)pooler_output_sizerr pooler_act activationrs r7r%DeiTPooler.__init__Bs> YYv1163L3LM  !2!23r9c\USS2S4nURU5nURU5nU$)Nr)rrZ)r5rfirst_token_tensorrRs r7rfDeiTPooler.forwardGs6+1a40 #56  6 r9)rZr) rirjrkrlrr%rfrrrsrts@r7r>r>As4z4 r9r>ad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introc^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \\ S \ S\ \ \44S jj5rSrU=r$)DeiTForMaskedImageModelingiPrr!NcH>[TU]U5 [USSS9Ul[R "[R "URURS-UR-SS9[R"UR55Ul UR5 g)NFT)r:r r#r) in_channels out_channelsrx) r$r%r9r"r Sequentialrr)encoder_strider{ PixelShuffledecoderr@rs r7r%#DeiTForMaskedImageModeling.__init__]s  fdS }} II"..#22A58K8KK  OOF11 2    r9rWrXrrr rrUc UbUOURRnURUUUUUUUS9nUSn U SS2SS24n U Rupn [ U S-5=pU R SSS5R XX5n URU 5nSnUGbURRURR-nUR SUU5nURURRS5RURRS5RS5R5n[RRXSS 9nUU-R!5UR!5S -- URR"- nU(dU4USS-nUbU4U-$U$[%UUUR&UR(S 9$) a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```N)rXrrr rrUrrr>r?r#none) reductiongh㈵>)lossreconstructionrr)rrLr"rFrprJrIrhrzr4repeat_interleaver\rrrKl1_losssumr{rrr)r5rWrXrrr rrUrrQr_sequence_lengthr{r;r<reconstructed_pixel_valuesmasked_im_lossrArbreconstruction_lossrs r7rf"DeiTForMaskedImageModeling.forwardnsL&1% R 'tn R d^R #'R  u// 0R R r9raz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. c^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \\ S \ S\ \ \ 44S jj5rSrU=r$)DeiTForImageClassificationirr!Nc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 gNF)r:r) r$r% num_labelsr9r"rrr)Identity classifierr@rs r7r%#DeiTForImageClassification.__init__ss   ++f> OUN_N_bcNc"))F$6$68I8IJikititiv r9rWrlabelsrr rrUc tUbUOURRnURUUUUUUS9nUSn URU SS2SSS245n Sn UGbUR U R 5nURR cURS:XaSURlOoURS:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5n URS:Xa&U "U R5UR55n OU "X5n OURR S:Xa=[5n U "U RSUR5URS55n O,URR S:Xa[!5n U "X5n U(dU 4USS-n U bU 4U -$U $[#U U UR$UR&S 9$) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: Polaroid camera, Polaroid Land camera ```Nrrr rrUrr regressionsingle_label_classificationmulti_label_classificationr>)rmlogitsrr)rrLr"r}rdevice problem_typer{rr'longrpr squeezer rMrrrr)r5rWrrrr rrUrrQrrmloss_fctrs r7rf"DeiTForImageClassification.forwardsT&1% Y 'tn Y d^Y #'Y  u++ ,Y Y r9rxzC Output type of [`DeiTForImageClassificationWithTeacher`]. c\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'S rg) +DeiTForImageClassificationWithTeacherOutputi4aF logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr cls_logitsdistillation_logitsrrr)rirjrkrlrmrrr' FloatTensorr/rrrrrrrrr9r7rr4s} +/FHU&& '..2J**+27;%"3"34;8