tjLdZddlmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z mZejeZd ejd ejd ed ededeejejff dZd ejd ejdedeejejffdZd ejdedejfdZe deGdde ZGddejZGddejZGddejZGdd ejZe Gd!d"eZe d#Gd$d%eZd%d"gZ dS)&zPyTorch SuperPoint model.) dataclassN)nn)PreTrainedModel)BaseModelOutputWithNoAttention)SuperPointConfig) ModelOutputauto_docstringlogging keypointsscoresborderheightwidthreturnc|dddf|k|dddf||z kz}|dddf|k|dddf||z kz}||z}||||fS)zPRemoves keypoints (and their associated scores) that are too close to the borderNr)r r rrrmask_hmask_wmasks p/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/superpoint/modeling_superpoint.pyremove_keypoints_from_bordersr%s1o'IaaadOv,O PF1o'IaaadOuv~,N OF F?D T?F4L ((kcx|t|kr||fStj||d\}}|||fS)z(Keeps the k keypoints with highest scorer)dim)lentorchtopk)r r rindicess rtop_k_keypointsr"/sHC NN&  j222OFG W v %%r nms_radiuscldkrtdfd}tj|}|||k}tdD]Q}||dk}tj|||}|||k}|||zz}Rtj|||S)z)Applies non-maximum suppression on scoresrz'Expected positive values for nms_radiuscVtj|dzdzdS)Nr kernel_sizestridepadding)r functional max_pool2d)xr#s rmax_poolzsimple_nms..max_pool<s-}''zA~7IRS]g'hhhrr&) ValueErrorr zeros_likerangefloatwhere) r r#r.zerosmax_mask_ supp_mask supp_scores new_max_masks ` r simple_nmsr:7sA~~BCCCiiiii  V $ $E&)))H 1XX<<HX^^--..2 k)UF;; "hh{&;&;; | z:; ;x / //ra Base class for outputs of image point description models. Due to the nature of keypoint detection, the number of keypoints is not fixed and can vary from image to image, which makes batching non-trivial. In the batch of images, the maximum number of keypoints is set as the dimension of the keypoints, scores and descriptors tensors. The mask tensor is used to indicate which values in the keypoints, scores and descriptors tensors are keypoint information and which are padding. ) custom_introceZdZUdZdZejdzed<dZej dzed<dZ ejdzed<dZ ejdzed<dZ ej dzed<dZeejdzed<dS) #SuperPointKeypointDescriptionOutputax loss (`torch.FloatTensor` of shape `(1,)`, *optional*): Loss computed during training. keypoints (`torch.FloatTensor` of shape `(batch_size, num_keypoints, 2)`): Relative (x, y) coordinates of predicted keypoints in a given image. scores (`torch.FloatTensor` of shape `(batch_size, num_keypoints)`): Scores of predicted keypoints. descriptors (`torch.FloatTensor` of shape `(batch_size, num_keypoints, descriptor_size)`): Descriptors of predicted keypoints. mask (`torch.BoolTensor` of shape `(batch_size, num_keypoints)`): Mask indicating which values in keypoints, scores and descriptors are keypoint information. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage. Nlossr r descriptorsr hidden_states)__name__ __module__ __qualname____doc__r>r FloatTensor__annotations__r IntTensorr r?r BoolTensorr@tuplerrrr=r=Is$&*D% d ")))(,Iu%,,,'+FE  $+++,0K"T)000$(D% T !(((59M5*+d299999rr=c \eZdZ d dededededdf fd Zd ejdejfd Z xZ S) SuperPointConvBlockFconfig in_channels out_channels add_poolingrNc2ttj||ddd|_tj||ddd|_tjd|_|rtjddnd|_ dS)Nrrr'Tinplacer&r(r)) super__init__rConv2dconv_aconv_bReLUrelu MaxPool2dpool)selfrLrMrNrO __class__s rrUzSuperPointConvBlock.__init__os i       i       GD))) =HRBLQq9999d rr@c|||}|||}|j||}|SN)rZrWrXr\)r]r@s rforwardzSuperPointConvBlock.forwards[ $++m"<"<==  $++m"<"<== 9  IIm44Mr)F) rArBrCrintboolrUrTensorra __classcell__r^s@rrKrKnsafSS&S58SHKSZ^S SSSSSS*U\elrrKcXeZdZdZdeddffd Z d dedzd edzdeezfd Z xZ S) SuperPointEncoderz SuperPoint encoder module. It is made of 4 convolutional layers with ReLU activation and max pooling, reducing the dimensionality of the image. rLrNc >td|_g}|t ||j|jddt dt|jdz D]A}|t ||j|dz |j|dB|t ||jd|jddtj ||_ dS)NrrT)rOF) rTrU input_dimappendrKencoder_hidden_sizesr1rr ModuleList conv_blocks)r]rLrpir^s rrUzSuperPointEncoder.__init__s7   8STU8Vdh i i i   q#f9::Q>??  A   #F7A>@[\]@^lp      3B79TUW9Xfk      =55rFToutput_hidden_states return_dictc|rdnd}|jD]}||}|r||fz}|}|std||fDSt||S)Nrc3K|]}||V dSr`r.0vs r z,SuperPointEncoder.forward..s"QQq1=====QQr)last_hidden_stater@)rprIr)r]inputrrrsall_hidden_states conv_blockoutputs rrazSuperPointEncoder.forwards #7@BBD* A AJJu%%E# A$5$@! RQQV->$?QQQQQ Q-$+    r)FT) rArBrCrDrrUrcrIrrarerfs@rrhrhs 6/6D6666662-2#'   #Tk D[  / /         rrhceZdZdZdeddffd ZdejdeejejffdZ dejdejfdZ d ejdeejejffd Z xZ S) SuperPointInterestPointDecodera The SuperPointInterestPointDecoder uses the output of the SuperPointEncoder to compute the keypoint with scores. The scores are first computed by a convolutional layer, then a softmax is applied to get a probability distribution over the 65 possible keypoint classes. The keypoints are then extracted from the scores by thresholding and non-maximum suppression. Post-processing is then applied to remove keypoints too close to the image borders as well as to keep only the k keypoints with highest score. rLrNct|j|_|j|_|j|_|j|_t jd|_t j dd|_ t j |j d|j ddd|_t j |j |jddd |_dS NTrQr&rSrkrrr'r)rTrUkeypoint_threshold max_keypointsr#border_removal_distancerrYrZr[r\rVrndecoder_hidden_size conv_score_akeypoint_decoder_dim conv_score_br]rLr^s rrUz'SuperPointInterestPointDecoder.__init__s "(";#1 +'-'E$GD))) LQq999 I  ' +  &    I  &(CQR[\fg   rencodedcd||}||\}}||fSr`)_get_pixel_scores_extract_keypoints)r]rr r s rraz&SuperPointInterestPointDecoder.forwards8''00 33F;; 6&  rc|||}||}tj|dddddf}|j\}}}}|dddd|||dd}|ddddd||dz|dz}t||j }|S) zKBased on the encoder output, compute the scores for each pixel of the imagerNrkrr&r) rZrrrr+softmaxshapepermutereshaper:r#)r]rr batch_sizer6rrs rrz0SuperPointInterestPointDecoder._get_pixel_scoress4,,W5566""6**&&vq11!!!SbS&9'-|$ Avu1a++33JqRSTT1aA..66z6A:uWXyYYFDO44 rr c|j\}}}tj|d|jk}|dt |}t |||j|dz|dz\}}|jdkrt|||j\}}tj |dg |j }||fS)z Based on their scores, extract the pixels that represent the keypoints that will be used for descriptors computation. The keypoints are in the form of relative (x, y) coordinates. rrr) rrnonzerorrItrrrr"fliptodtype)r]r r6rrr s rrz1SuperPointInterestPointDecoder._extract_keypointss "<65M&)d.E"EFF 5//0: vt;VaZQR   6   " " / 64CU V V IvJy1#..11&,?? &  r) rArBrCrDrrUrrdrIrarrrerfs@rrrs / D      (!u|!elEL6P0Q!!!!  %,    !!% el@Z:[!!!!!!!!rrceZdZdZdeddffd ZdejdejdejfdZe d d e dejfd Z xZ S) SuperPointDescriptorDecoderag The SuperPointDescriptorDecoder uses the outputs of both the SuperPointEncoder and the SuperPointInterestPointDecoder to compute the descriptors at the keypoints locations. The descriptors are first computed by a convolutional layer, then normalized to have a norm of 1. The descriptors are then interpolated at the keypoints locations. rLrNc^ttjd|_tjdd|_tj|jd|j ddd|_ tj|j |j ddd |_ dSr) rTrUrrYrZr[r\rVrnrconv_descriptor_adescriptor_decoder_dimconv_descriptor_brs rrUz$SuperPointDescriptorDecoder.__init__s GD))) LQq999 !#  ' +  & " " " "$  &  ) " " " rrr cJ||||}tj|dd}||d|dddd}tj|dd}|S)zXBased on the encoder output and the keypoints, compute the descriptors for each keypointr&rprNrr) rrZrrr+ normalize_sample_descriptorsr transpose)r]rr r?s rraz#SuperPointDescriptorDecoder.forward#s,,TYYt7M7Mg7V7V-W-WXX m--kQA-FF ..y At@TVWXXYZ[ ok1a88 rrscalec|j\}}}}||dz z dz}tj||z|dz z dz ||z|dz z dz gg}||}||z}|dzdz }ddi}||ddd}t jj||fddi|}|||d}t j |dd }|S) z-Interpolate descriptors at keypoint locationsr&g?r align_cornersTrkmodebilinearr) rrtensorrviewrr+ grid_samplerr) r r?rr num_channelsrrdivisorkwargss rrz/SuperPointDescriptorDecoder._sample_descriptors/s 3>2C/ L&% )C/ ,%%-%!)";c"AVe^V[^_V_E_beEe ghii**Y''W MA% !4(NN:q"a88 m/ YbbZb[abb !))*lBGG m--kQA-FF r)r) rArBrCrDrrUrrdra staticmethodrbrrerfs@rrrs / D      ( u|      3u|\rrcNeZdZUeed<dZdZdZdZde j de j fdZ dS) SuperPointPreTrainedModelrL superpoint pixel_values)imageFrcF|dddddddfdddddddfS)aG Assuming pixel_values has shape (batch_size, 3, height, width), and that all channels values are the same, extract the first channel value to get a tensor of shape (batch_size, 1, height, width) for SuperPoint. This is a workaround for the issue discussed in : https://github.com/huggingface/transformers/pull/25786#issuecomment-1730176446 Args: pixel_values: torch.FloatTensor of shape (batch_size, 3, height, width) Returns: pixel_values: torch.FloatTensor of shape (batch_size, 1, height, width) Nrr)r]rs r extract_one_channel_pixel_valuesz:SuperPointPreTrainedModel.extract_one_channel_pixel_valuesJs<AAAq!!!QQQJ'4AAA 66rN) rArBrCrrFbase_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointingrrErrrrrrBsb $$O!&+#7U=N7SXSd777777rrz@ SuperPoint model outputting keypoints and descriptors. c eZdZdZdeddffd Ze d dejdej dzde dzd e dzde e zf d Z xZS) SuperPointForKeypointDetectiona SuperPoint model. It consists of a SuperPointEncoder, a SuperPointInterestPointDecoder and a SuperPointDescriptorDecoder. SuperPoint was proposed in `SuperPoint: Self-Supervised Interest Point Detection and Description `__ by Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. It is a fully convolutional neural network that extracts keypoints and descriptors from an image. It is trained in a self-supervised manner, using a combination of a photometric loss and a loss based on the homographic adaptation of keypoints. It is made of a convolutional encoder and two decoders: one for keypoints and one for descriptors. rLrNct|||_t||_t ||_t||_| dSr`) rTrUrLrhencoderrkeypoint_decoderrdescriptor_decoder post_initrs rrUz'SuperPointForKeypointDetection.__init__jsg     (00 >v F F"=f"E"E rrlabelsrrrsc ld}|td||n jj}||n jj}|}|j\}}} } |||} | d} fd| D} d| D}d| D}fdt| |D}td |D}tj ||d f|j }tj ||f|j }tj ||jj f|j }tj ||f|j tj }tt|||D]Y\}\}}}|||d|jdf<|||d|jdf<|||d|jdf<d ||d|jdf<Z|tj| | g|j z }|r| d nd}|st!d||||||fDSt#||||||S)a Examples: ```python >>> from transformers import AutoImageProcessor, SuperPointForKeypointDetection >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint") >>> model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint") >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) ```Nz-SuperPoint does not support training for now.)rrrsrcFg|]}|dS)N.)r)rwrzr]s r z:SuperPointForKeypointDetection.forward..s:! ! ! DUD ! !"3I"> ? ?! ! ! rcg|] }|d S)rrrwkeypoints_scoress rrz:SuperPointForKeypointDetection.forward..s\\\2B*1-\\\rcg|] }|d S)rrrs rrz:SuperPointForKeypointDetection.forward..sYYY/?'*YYYrcZg|]'\}}|d|d(Sr)r)rwrzr r]s rrz:SuperPointForKeypointDetection.forward..sI   ,!9  # #$5i$@)IBV W W   rc30K|]}|jdVdS)rN)r)rwr s rryz9SuperPointForKeypointDetection.forward..s)#W#W9IOA$6#W#W#W#W#W#Wrr&)device)rrrc3K|]}||V dSr`rrvs rryz9SuperPointForKeypointDetection.forward..s(qqqcdcpcpcpcpcpqqr)r>r r r?rr@)r/rLrrrsrrrzipmaxrr4rrrb enumeraterrIr=)r]rrrrrsrr>rr6rrencoder_outputsrzlist_keypoints_scoreslist_keypoints list_scoreslist_descriptorsmaximum_num_keypointsr r r?rrq _keypoints_scores _descriptorsr@s` rraz&SuperPointForKeypointDetection.forwardus :  LMM M%9$D $+Jj &1%|GZbgbklll6?NT_aq@r@r6s6s , , 2A2 G\2rsD !!!!!! ((((((UTTTTT  H % %)|)%*\);>)HK)TW) 5< %&))))&u|&U\&c&eTYT`bgbnTnNo&&&&0u|000000$ :::::+:: :6")q q  q h ,-H Ir