tjdnddlmZmZddlmZddlZddlmZddlmZ ddl m Z ddl m Z dd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$Gddej%Z&Gddej%Z' d:dej%dej(dej(dej(dej(dzde)dzde)deefdZ*Gd d!ej%Z+Gd"d#ej%Z,Gd$d%eZ-eGd&d'eZ.Gd(d)ej%Z/eGd*d+e.Z0ed,-Gd.d/e.Z1ed0-Gd1d2e.Z2ed3-eGd4d5eZ3ed6-Gd7d8e.Z4gd9Z5dS);)CallableIterable) dataclassN)nn)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstring torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) DeiTConfigcLeZdZdZdeffd ZdejdejfdZxZ S)DeiTPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. configct|j}|j}t |t r|n||f}t |t r|n||f}|d|dz|d|dzz|_||_||_|j|_tj |j|j |||_ dS)Nrr) kernel_sizestride) super__init__ image_size patch_size isinstancer num_patches num_channelsrConv2d hidden_size projection)selfrr#r$ __class__s d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/deit/modeling_deit.pyr"zDeiTPatchEmbeddings.__init__/s & & #-j(#C#CaZZ*V`Ia #-j(#C#CaZZ*V`Ia &qMZ]:z!}PZ[\P]?]^$$"/)F$79KYclvwww pixel_valuesreturnc|jd}||jkrtd|jd|d||dddS)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .)shaper' ValueErrorr*flatten transpose)r+r/r's r-forwardzDeiTPatchEmbeddings.forward<s#)!, 4, , ,I!.II9EIII |,,44Q77AA!QGGGr.) __name__ __module__ __qualname____doc__rr"torchTensorr8 __classcell__r,s@r-rr(s xz x x x x x xHELHU\HHHHHHHHr.rc eZdZdZddededdffd Zdejd e d e dejfd Z dd ejd ej dzdedejfdZ xZ S)DeiTEmbeddingsa Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. Differences from ViTEmbeddings: - Adds a distillation token (for distillation pre-training). - Position embeddings include +2 slots (CLS + distillation) instead of +1. - interpolate_pos_encoding handles 2 special tokens instead of 1. - forward concatenates distillation token and handles position encoding for both. Fruse_mask_tokenr0Ncttjt jdd|j|_|r-tjt jdd|jnd|_t||_ |j j }tjt jd|dz|j|_ tj |j|_|j|_|j j|_tjt jdd|j|_dS)Nrr3)r!r"r Parameterr=zerosr) cls_token mask_tokenrpatch_embeddingsr&position_embeddingsDropouthidden_dropout_probdropoutr$r#distillation_token)r+rrCr&r,s r-r"zDeiTEmbeddings.__init__Qs ek!Q8J&K&KLLQ_i",u{1a9K'L'LMMMei 3F ; ;+7 #%< A{QPVPb0c0c#d#d z&"<==  +/:"$,u{1aAS/T/T"U"Ur. embeddingsheightwidthc|jddz }|jjddz }tjs||kr ||kr|jS|jddddf}|jddddf}|jd}||jz} ||jz} t |dz} |d| | |}|dddd}tj || | fdd }|dddd dd|}tj ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr3N?rrbicubicF)sizemode align_cornersdim)r4rJr=jit is_tracingr$rreshapepermuter functional interpolateviewcat) r+rOrPrQr& num_positionsclass_and_dist_pos_embedpatch_pos_embedrZ new_height new_widthsqrt_num_positionss r-interpolate_pos_encodingz'DeiTEmbeddings.interpolate_pos_encoding^st!&q)A- 06q9A= y##%% ,+*F*F6UZ??+ +#'#;AAArrE#B 2111abb59r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy2OD!LLLLr.r/bool_masked_posric |j\}}}}||}|\}} }|R|j|| d} |d| } |d| z z| | zz}|j|dd} |j|dd} tj | | |fd}|r|| |||z}n^||j dks||j dkr2td|d|d|j dd|j dd ||jz}||}|S) NrSg?rrYrzInput image size (*z) doesn't match model (z).)r4rIrVrHexpand unsqueezetype_asrGrNr=rbrir#r5rJrM)r+r/rjri_rPrQrO batch_size seq_length mask_tokensmask cls_tokensdistillation_tokenss r-r8zDeiTEmbeddings.forwards +01fe**<88 $.OO$5$5! J  &/00ZLLK",,R0088EED#sTz2[45GGJ^**:r2>> "5<>J\\*-- r.)F)NF)r9r:r;r<rboolr"r=r>intri BoolTensorr8r?r@s@r-rBrBFs V Vz V4 VD V V V V V V&M5<&M&MUX&M]b]i&M&M&M&MV48).  l )D0 #'          r.rBmodulequerykeyvalueattention_maskscalingrMkwargsc ||ddz}tj||dd|z}|||z}tj|dtj|j }tj |||j }tj||} | dd } | |fS)NrSr3r)rZdtype)ptrainingr) rVr=matmulr7rr_softmaxfloat32torrMr contiguous) r{r|r}r~rrrMr attn_weights attn_outputs r-eager_attention_forwardrs**R..D(<s}}Q':':;;gEL!#n4 =((2U](SSVVW\WbccL=((6?([[L,|U33K''1--88::K  $$r.c eZdZdeffd Z d dejdejdzdeede ejejffdZ xZ S) DeiTAttentionrct||_|j|_t |d|j|jz|_|j|_|jdz|_ d|_ tj |j|j|jz|j |_tj |j|j|jz|j |_tj |j|j|jz|j |_tj |j|jz|jd|_dS)Nhead_dimrF)biasT)r!r"rnum_attention_headsgetattrr)rattention_probs_dropout_probattention_dropoutr is_causalrLinearqkv_biasq_projk_projv_projo_projr+rr,s r-r"zDeiTAttention.__init__s  #)#=  F4F&Jd4dee !'!D}d* i 2F4NQUQ^4^eketuuu i 2F4NQUQ^4^eketuuu i 2F4NQUQ^4^eketuuu i :T] JFL^eijjj r.N hidden_statesrrr0c |jdd}g|d|jR}|||dd}|||dd}|||dd}tj|j j t} | |||||f|j sdn|j |jd|\} } | jg|dR} || } | | fS)NrSrr3rz)rMr)r4rrrar7rrr get_interfacer_attn_implementationrrrrr]rr) r+rrr input_shape hidden_shape query_states key_states value_statesattention_interfacerrs r-r8zDeiTAttention.forwardsw $)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r.N) r9r:r;rr"r=r>rrtupler8r?r@s@r-rrs kz k k k k k k"/3))|) t+)+, ) u|U\) * ))))))))r.rcHeZdZdeffd ZdejdejfdZxZS)DeiTMLPrct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) r!r"rr hidden_act activation_fnrrr)intermediate_sizefc1fc2rs r-r"zDeiTMLP.__init__sf  #F$569V/1IJJ9V5v7IJJr.rr0c||}||}||}|Sr)rrr)r+rs r-r8zDeiTMLP.forwards=// **=99 // r. r9r:r;rr"r=r>r8r?r@s@r-rrsqKzKKKKKKU\elr.rc peZdZdeffd Z d dejdejdzdeedejfdZ xZ S) DeiTLayerrchtt||_t j|j|j|_t j|j|j|_ t||_ t j |j |_dS)Neps)r!r"r attentionr LayerNormr)layer_norm_epslayernorm_beforelayernorm_afterrmlprKrLrMrs r-r"zDeiTLayer.__init__s &v.. " V-?VEZ [ [ [!|F,>FDYZZZ6??z&"<== r.Nrrrr0c |}||}|j||fi|\}}||}||z}|}||}||}||}||z}|Sr)rrrMrr)r+rrrresidualrps r-r8zDeiTLayer.forwards!--m<< )4>-RR6RR q ]33 %0 !,,];; //  ]33 %0 r.r) r9r:r;rr"r=r>rrr8r?r@s@r-rrs>z>>>>>>/3| t++,   r.rceZdZUeed<dZdZdZdZddgZ dZ dZ dZ dZ dZeedZd Zejd ejejzejzd d ffd ZxZS)DeiTPreTrainedModelrdeitr/)imageTrBr)r attentionsrIr{r0Ncbt|t|trs|j&t j|jd|jjt j|j d|jj|j t j |j t|trmt j |j t j |jt j |j |j t j |j dSdSdS)zInitialize the weightsNrz)meanstd) r! _init_weightsr%rBrJinit trunc_normal_rinitializer_rangerGrHzeros_rN)r+r{r,s r-rz!DeiTPreTrainedModel._init_weights9s f%%% fn - - /)5"6#=CT[Mjkkkk  v/ct{?\ ] ] ] ] , F-... fn - - / K( ) ) ) K2 3 3 3 K1 2 2 2 , F-.....  / /-,r.)r9r:r;r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backend_can_compile_fullgraphrr_can_record_outputs_input_embed_layerr=no_gradrrr(rrr?r@s@r-rr&s $O!&*#);7N"&!"#,U]__/BI $9BL$H/T/////_/////r.rcHeZdZdeffd ZdejdejfdZxZS) DeiTPoolerrcttj|j|j|_t|j|_ dSr) r!r"rrr)pooler_output_sizedenser pooler_act activationrs r-r"zDeiTPooler.__init__LsE Yv163LMM  !23r.rr0cr|dddf}||}||}|S)Nr)rr)r+rfirst_token_tensor pooled_outputs r-r8zDeiTPooler.forwardQs@+111a40 #566  66 r.rr@s@r-rrKsj4z444444 U\elr.rceZdZddededeffd Zeede dd e j dzd e j dzd edzd e j dzd e e def dZxZS) DeiTModelTFradd_pooling_layerrCct|_t||_t jfdtjD|_ t j j j |_ |rtnd|_|dS)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rCc.g|]}tS)r).0rprs r- z&DeiTModel.__init__..fs!$`$`$`1Yv%6%6$`$`$`r.rN)r!r"rrBrOr ModuleListrangenum_hidden_layerslayersrr)r layernormrpooler post_init)r+rrrCr,s ` r-r"zDeiTModel.__init__\s     (OOOm$`$`$`$`fF^@_@_$`$`$`aa f&8f>STTT,=Gj(((4  r.)tie_last_hidden_statesNr/rjrirrr0c |jjjjj}|j|kr||}||||}t |j||}|}|jD] } | ||fi|} | |} |j | | nd} t| | S)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rjri)r inputs_embedsrN)last_hidden_state pooler_output) rOrIr*weightrrr rrrrr ) r+r/rjrirrexpected_dtypeembedding_outputrlayersequence_outputrs r-r8zDeiTModel.forwardls 9DKQ   / /'??>::L?? /Tl+  3;*)    ) [ K KE!E-JJ6JJMM..778< 8O O444UY )O[hiiiir.)TFNNNN)r9r:r;rrwr"rrrr=r>ryrrr r8r?r@s@r-rrZszd[_  _E222-13704.2 j jlT) j)D0 j#'+ j  t+ j +, j $ j j j^32  j j j j jr.rad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introceZdZdeffd Zee d dejdzdej dzde dejdzd e e d e f d ZxZS) DeiTForMaskedImageModelingrcVt|t|dd|_t jt j|j|jdz|j zdt j |j|_ | dS)NFT)rrCr3r) in_channels out_channelsr) r!r"rrr Sequentialr(r)encoder_strider' PixelShuffledecoderrrs r-r"z#DeiTForMaskedImageModeling.__init__s    fdSSS } I".#2A58KK    OF1 2 2     r.NFr/rjrirrr0c 6|j|f|||d|}|j}|ddddf}|j\}} } t| dzx} } |ddd|| | | }||} d}||jj|jj z}|d||}| |jj d |jj d d }tj|| d }||z|d zz |jjz }t%|| |j|j S) a; bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```)rjrirNr3rTrrrSnone) reductiongh㈵>)lossreconstructionrr)rrr4rxr^r]r rr#r$repeat_interleavernrrr_l1_losssumr'rrr)r+r/rjrirroutputsrrqsequence_lengthr'rPrQreconstructed_pixel_valuesmasked_im_lossrVrtreconstruction_losss r-r8z"DeiTForMaskedImageModeling.forwardsL/8di / +%=) / /  / / "3*!!!QRR%04C4I1 O\_c1222)11!Q::BB:|]cejkk&*\\/%B%B"  &;)T[-CCD-55b$EEO11$+2H!LL""4;#91==1  #%-"7"7 F`lr"7"s"s 1D8==??488::PTCTUX\XcXppN(5!/)     r.)NNFN)r9r:r;rr"rrr=r>ryrwrrrr8r?r@s@r-rrsz"-137)..2 J J lT)J )D0J #' J  t+ J +, J  #J J J ^J J J J J r.ra DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune DeiT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ceZdZdeffd Zee d dejdzdejdzde dzdejdzde e d e f d Z xZS) DeiTForImageClassificationrc:t||j|_t|d|_|jdkrt j|j|jnt j|_ | dSNF)rr) r!r" num_labelsrrrrr)Identity classifierrrs r-r"z#DeiTForImageClassification.__init__s     +f>>> OUN_bcNcNc")F$68IJJJikitiviv r.Nr/labelsrirrr0c |j|f||d|}|j}|dddddf}||} d} ||j|| |jfi|} t | | |j|jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rirNr)rlogitsrr)rrr loss_functionrr rr) r+r/r rirrrrrr#rs r-r8z"DeiTForImageClassification.forwards"/8di / %=)/ /  / / "3'1aaa0 //  %4%ffdkLLVLLD$!/)     r.r)r9r:r;rr"rrr=r>rwrrr r8r?r@s@r-rrs z      -1&*04.2 # # lT)#  t## #'+ #  t+ # +, #  # # # ^# # # # # r.rzC Output type of [`DeiTForImageClassificationWithTeacher`]. ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dS)+DeiTForImageClassificationWithTeacherOutputaj logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr# cls_logitsdistillation_logitsrr) r9r:r;r<r#r= FloatTensorrr'r(rrrrr.r-r&r&Bs  (,FE  $++++/J!D(///48*T188859M5*+d299926Je'(4/66666r.r&a DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c eZdZdeddffd Zee d dejdzde dejdzd e e de f d Z xZS) %DeiTForImageClassificationWithTeacherrr0Nct||j|_t|d|_|jdkrt j|j|jnt j|_ |jdkrt j|j|jnt j|_ | dSr) r!r"rrrrrr)rcls_classifierdistillation_classifierrrs r-r"z.DeiTForImageClassificationWithTeacher.__init__gs     +f>>> AG@QTU@U@UBIf(&*; < < <[][f[h[h AG@QTU@U@UBIf(&*; < < <[][f[h[h $ r.Fr/rirrc |j|f||d|}|j}||dddddf}||dddddf}||zdz } t | |||j|jS)Nr"rrr3)r#r'r(rr)rrr-r.r&rr) r+r/rirrrrr'r(r#s r-r8z-DeiTForImageClassificationWithTeacher.forwardxs/8di / %=)/ /  / / "3((Aqqq)ABB "::?111aQRQRQR7;STT22a7:! 3!/)     r.)NFN)r9r:r;rr"rrr=r>rwrrr&r8r?r@s@r-r+r+[szd"-1)..2   lT) #'  t+  +,  5    ^     r.r+)rr+rrr)Nrz)6collections.abcrr dataclassesrr=rrr activationsr masking_utilsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_deitrModulerrBr>floatrrrrrrrrrr&r+__all__rr.r-r@s*/.......!!!!!! &&&&&&!!!!!!666666999999llllllllllFFFFFFFF&&&&&&OOOOOOOOOOOOIIIIIIII555555******HHHHH")HHH<`````RY```R!%% I% <% % < % L4' % T\ %%'(%%%%8.).).).).)BI.).).)b     bi    *@!/!/!/!/!//!/!/!/H        4j4j4j4j4j#4j4j4jn    ^ ^ ^ ^ ^ !4^ ^   ^ B    2 2 2 2 2 !42 2   2 j  77777+77   7&   0 0 0 0 0 ,?0 0   0 f   r.