tjddlmZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZmZdd lmZdd lmZmZmZmZd dlmZddlmZedeGddeZGddejZ dZ!dAdZ"dej#de$dej#fdZ% dBdejdej#d ej#d!ej#d"ej#dzd#e&d$e&d%eefd&Z'Gd'd(ejZ(Gd)d*ejZ)Gd+d,ejZ*d-ej#d.ej#d/ej#dej#fd0Z+Gd1d2ejZ,Gd3d4ejZ-eGd5d6eZ.d7ej#d8e&de/ej#ej#ffd9Z0d:ej#d;e$dGd?d@e.Z2d6d@gZ3dS)C)Callable) dataclassN)nn pad_sequence)ACT2FN)create_bidirectional_mask)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple)AutoModelForKeypointDetection)LightGlueConfiga Base class for outputs of LightGlue keypoint matching models. Due to the nature of keypoint detection and matching, the number of keypoints is not fixed and can vary from image to image, which makes batching non-trivial. In the batch of images, the maximum number of matches is set as the dimension of the matches and matching scores. The mask tensor is used to indicate which values in the keypoints, matches, matching_scores and prune tensors are keypoint matching information. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ ejdzed<dZ ej dzed<dZ ejdzed<dZeejdzed <dZeejdzed <dS) LightGlueKeypointMatchingOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*): Loss computed during training. matches (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Index of keypoint matched in the other image. matching_scores (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Scores of predicted matches. keypoints (`torch.FloatTensor` of shape `(batch_size, num_keypoints, 2)`): Absolute (x, y) coordinates of predicted keypoints in a given image. prune (`torch.IntTensor` of shape `(batch_size, num_keypoints)`): Pruning mask indicating which keypoints are removed and at which layer. mask (`torch.BoolTensor` of shape `(batch_size, num_keypoints)`): Mask indicating which values in matches, matching_scores, keypoints and prune are keypoint matching information. hidden_states (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(batch_size, 2, num_channels, num_keypoints)` returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True` attentions (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, 2, num_heads, num_keypoints, num_keypoints)` returned when `output_attentions=True` is passed or when `config.output_attentions=True` Nlossmatchesmatching_scores keypointsprunemask hidden_states attentions)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrr IntTensorrr tupler!n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/lightglue/modeling_lightglue.pyrr&s0&*D% d ")))(,GU  %,,,04OU&-444*.Iu 4'...$(E5?T !(((%)D% d ")))59M5*+d299926Je'(4/66666r,rc eZdZdeffd Z d dejdedzdeejeejejfzfdZ xZ S) LightGluePositionalEncoderconfigcttjd|j|jzdzd|_dS)NrFbias)super__init__rLineardescriptor_dimnum_attention_heads projectorselfr0 __class__s r-r5z#LightGluePositionalEncoder.__init__TsG 1f&;v?Y&Y]^&^ejkkkr,Froutput_hidden_statesNreturnc||}|dd}tj|}tj|}||f}|r||fn|f}|S)Nrdim)r9repeat_interleaver&cossin)r;rr=projected_keypoints embeddingscosinessinesoutputs r-forwardz"LightGluePositionalEncoder.forwardXst#nnY77(::1":EE )J'' *%%u% 6J]*122Q[P] r,F) r"r#r$rr5r&Tensorboolr*rK __classcell__r<s@r-r/r/Sslllllll LQ   =AD[  u| uU\5<%?@ @        r,r/c|ddddf}|ddddf}tj| |gdd}|S)N.rrr@rA)r&stackflatten)xx1x2rot_xs r- rotate_halfrYdsU 3!8B 319B K"b r * * * 2 22 6 6E Lr,cl|j}|}|}||}||}||zt||zz}||zt||zz}||||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. dtype)r\float unsqueezerYto)qkrDrE unsqueeze_dimr\q_embedk_embeds r-apply_rotary_pos_embrels$ GE  A  A -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G ::E: " "GJJUJ$;$; ;;r,r n_repr>c|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)shapeexpandreshape)r rfbatchnum_key_value_headsslenhead_dims r- repeat_kvros 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr,modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nrrr@)rBr\)ptrainingr)ronum_key_value_groupsr&matmul transposer functionalsoftmaxfloat32r_r\rwr{ contiguous) rqrrrsrtrurvrwrx key_states value_states attn_weights attn_outputs r-eager_attention_forwardrs3 ;<&B\$\!}d* !'!9i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    r,Nr position_embeddingsruencoder_hidden_statesencoder_attention_maskrxr>c |jdd}g|d|jR}|||dd} |du} | r|n|} | r|n|} || |dd} || |dd}||\}}t| | ||\} } tj |j j t}||| | || f|j sdn|j|jd|\}}|jg|dR}||}||fS)Nr@rrrp)rwrv)rhrnrviewr~rrrer get_interfacer0_attn_implementationrr{rrvrjrr)r;r rrurrrx input_shape hidden_shape query_statesis_cross_attentioncurrent_statescurrent_attention_maskrrrDrEattention_interfacerrs r-rKzLightGlueAttention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT 2$>2DW..-;M!a!7!7Sa[[0055lCCMMaQRSS {{>2277 EEOOPQSTUU  **HC';L*VY[^'_'_ $L*(?(M K ,.E) ) %8$7     " % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r,)NNNN)r"r#r$r%rintr5r&rMr*rr rKrOrPs@r-rrsGG  3      4IM.2596: *)*)|*)#5<#=>E*) t+ *) %|d2 *) !& t 3 *)-.*) u|U\D00 1*)*)*)*)*)*)*)*)r,rcHeZdZdeffd ZdejdejfdZxZS) LightGlueMLPr0cTt||_t|j|_t j|j|j|_ t j|j|j |_ t j |jd|_ dS)NT)elementwise_affine)r4r5r0r hidden_act activation_fnrr6intermediate_sizefc1rfc2 LayerNorm layer_normr:s r-r5zLightGlueMLP.__init__s  #F$569V5v7OPP9V5v7IJJ,v'?TXYYYr,r r>c||}||}||}||}|SN)rrrr)r;r s r-rKzLightGlueMLP.forwardsN//  66 **=99 // r, r"r#r$rr5r&rMrKrOrPs@r-rrsqZZZZZZZU\elr,rceZdZdedeffd Z d dejdejdejded zd ed zd e eje ejd ze ejd zff d Z xZ S)LightGlueTransformerLayerr0rctt|||_t ||_t|||_t ||_dSr)r4r5rself_attentionrself_mlpcross_attention cross_mlprs r-r5z"LightGlueTransformerLayer.__init__s_ 0CC$V,, 1&)DD%f--r,F descriptorsrrur=Noutput_attentionsr>c2|rdnd}|rdnd}|r||fz}|j\}} } |||||\} } tj|| gd} || }||z}|r| |f}|dd| | d|| | }|B|dddd| d|dd| nd}|||||\}}tj||gd}||}||z}|r>||f}|||| | fz|z||| | fz|z}|r || fz|fz}|||fS)Nr+)rrurr@rArr)rrr) rhrr&catrrjfliprr)r;rrrur=rall_hidden_statesall_attentions batch_size num_keypointsr7attention_outputself_attentionsintermediate_states output_statesself_attention_descriptorsself_attention_hidden_statesrrcross_attention_outputcross_attentionscross_intermediate_statescross_output_statescross_attention_hidden_statess r-rKz!LightGlueTransformerLayer.forward sY#7@BBD0:d  C 1[N B 4?4E1 M>-1,?,?  ))/ -@- - )/ $i6F(GRPPP &9:: %0=%@"  P,?+O ( ' . .r1m^ T T T!WW WZ ? ? )  " "2q!Q > > C CA F F N Nz[\^_an o o o 483G3G &"7#9/ 4H4 4 0 0 %*I/IKa.bhj$k$k$k!"nn-FGG03FF  -FH[,\ )!-55j-Q_``bc./&&z=.QQST0 0   W+.@@DTCVVN-~==r,)FF) r"r#r$rrr5r&rMrNr*rKrOrPs@r-rrs..3......-2). H>H>\H><H> H> #Tk H>  $; H> u|U5<047u|9Lt9SS TH>H>H>H>H>H>H>H>r,r similarity matchability0 matchability1c|j\}}}tj|tj|ddz}tj|d}tj|ddddd}|||dz|dzfd} ||z|z| ddd|d|f<tj|d | dddddf<tj|d | dddddf<| S)z;create the log assignment matrix from logits and similarityrrr@rRrN) rhrr logsigmoidr~ log_softmaxrnew_fullsqueeze) rrrrnum_keypoints_0num_keypoints_1 certaintiesscores0scores1scoress r-sigmoid_log_double_softmaxrXsw4>3C0J-**=99BMc.|j\}}}||}|tj|j|jdzz }||dzd||}|dddf}|dddf}||ddz} |||dzd|}|dddfd} |dddfddd} | | z}| |dktj | j j } | |} | |dzd|d} | dddf} | dddf}t| | |}|S)Ndeviceg?rrrr@rR)rhrr&tensorr7rrjr~r^ masked_fillfinfor\minrr)r;rrrrr7 m_descriptorsm_descriptors0m_descriptors1rmask0mask1rmatchability_0matchability_1rs r-rKz%LightGlueMatchAssignmentLayer.forwardos4?4E1 M>--k:: % T5HQ^Qe(f(f(fjn(nn %--jAoq-Q_`` &qqq!t,&qqq!t,#n&>&>r2&F&FF  << aMBBDAJ((,,EAJ((,,66r2>>E5=D#// 5;zGW;X;X;\]]J((55 #++J!OQ qQQ %aaad+%aaad+,JWW r,c||}tj|d}|S)z0Get matchability of descriptors as a probabilityr@)rrrsigmoidr)r;rrs r-get_matchabilityz.LightGlueMatchAssignmentLayer.get_matchabilitys>((55 },,\::BB2FF r,) r"r#r$rr5r&rMrKrrOrPs@r-rrgsIIIIIII5<u| 4ELU\r,rcHeZdZdeffd ZdejdejfdZxZS)LightGlueTokenConfidenceLayerr0cttj|jd|_dS)Nr)r4r5rr6r7tokenr:s r-r5z&LightGlueTokenConfidenceLayer.__init__s3 Yv4a88 r,rr>c||}tj|d}|S)Nr@)rdetachrrrr)r;rrs r-rKz%LightGlueTokenConfidenceLayer.forwardsG ;--//00 %%e,,44R88 r,rrPs@r-rrsj9999999 5<ELr,rc6eZdZUdZeed<dZdZdZdZ dZ dZ dS) LightGluePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r0 lightglue pixel_values)imageFTN) r"r#r$r%rr(base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpar+r,r-rrsJ #$O!&+#NNNr,rr thresholdc`|j\}}}|ddddddfd}|ddddddfd}|j}|j}tj|jd|jd}tj|jd|jd} ||d|k} | |d|k} |j}| d} tj | || } tj | | d|| }| | |kz}| |d|z}tj ||d}tj ||d}tj ||g dd |dzd}tj | |g dd |dzd}||fS)z1obtain matches from a score matrix [Bx M+1 x N+1]Nr@rrrr)rhmaxindicesr&arangergathervaluesexp new_tensorwhererSr~rj)rrr_max0max1matches0matches1indices0indices1mutual0mutual1zeromatching_scores0matching_scores1valid0valid1rrs r-get_matches_from_scoresrs|J1 !!!SbS#2#+  " "1 % %D !!!SbS#2#+  " "1 % %D|H|H|HN1-hoFFFtLH|HN1-hoFFFtLH(//!X666G(//!X666G ;??  D ??1  D{7D$77{7,<,C,CAx,P,PRVWW (94 5F v}}Q11 1F{68R00H{68R00Hk8X.//99!Q??GG UVXZ[[Gk#35E"FGGQQRSUVWW__`jmn`nprssO O ##r,rheightwidthctj||g|j|jd}|dz }|djdz }||ddddfz |dz }|S)a Normalize keypoints locations based on image image_shape Args: keypoints (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`): Keypoints locations in (x, y) format. height (`int`): Image height. width (`int`): Image width. Returns: Normalized keypoints locations of shape (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`). rr\Nrr@.).NN)r&rrr\rr )rrrsizeshiftscales r-normalize_keypointsr$sr < 0@  X X XY] ^D 1HE HHRLL ! #EU3aaa<00E/4JJI r,zV LightGlue model taking images as inputs and outputting the matching of them. ceZdZdZdeffd ZdedefdZ d$de j d e j d e d zde e j e e j e j fffd Z d e j dede j de j de j f dZd%dZde j de j dede j fdZde j d e j de j de j de j d e j defdZdZde j de j de j de j de e j e j ff dZ d&d e j de j dedede j d zde d zd e d zde e j e j e j e e ffd Zee d&d!e jd"e jd zde d zd e d zde ezf d#ZxZS)'LightGlueForKeypointMatchingar LightGlue is a model matching keypoints in images by leveraging detections from a keypoint detector such as SuperPoint. It is based on the SuperGlue architecture and is designed to be lightweight and efficient. It consists of : 1. Keypoint Encoder 2. A Graph Neural Network with self and cross attention layers 3. Matching Assignment layers The correspondence ids use -1 to indicate non-matching points. Philipp Lindenberger, Paul-Edouard Sarlin and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV 2023. https://huggingface.co/papers/2306.13643 r0c`ttjj|_jj|_j|_j |_ j |_ j |_ j |_ |j|jkr'tj|j|jd|_ntj|_t%|_tjfdt+j D|_tjfdt+j D|_tjfdt+j dz D|_|dS)NTr2c2g|]}t|S))r)r).0ir0s r- z9LightGlueForKeypointMatching.__init__.. s' e e e &v ; ; ; e e er,c.g|]}tSr+)rr)rr0s r-r+z9LightGlueForKeypointMatching.__init__.. s" \ \ \q *6 2 2 \ \ \r,c.g|]}tSr+)rr-s r-r+z9LightGlueForKeypointMatching.__init__..s" ` ` `q *6 2 2 ` ` `r,r)r4r5r from_configkeypoint_detector_configkeypoint_detectordescriptor_decoder_dim keypoint_detector_descriptor_dimr7num_hidden_layers num_layersfilter_thresholddepth_confidencewidth_confidencerr6input_projectionIdentityr/positional_encoder ModuleListrangetransformer_layersmatch_assignment_layerstoken_confidence post_initr:s `r-r5z%LightGlueForKeypointMatching.__init__s    !>!J6Kj!k!k060O0f-$3 2 & 7 & 7 & 7  $"G G G$&Id.SUYUhos$t$t$tD ! !$&KMMD !"cxddtjd|z|jz zz}tj|ddS)z-scaled confidence threshold for a given layerg?g?grr)npr r5clip)r;rBrs r-_get_confidence_thresholdz6LightGlueForKeypointMatching._get_confidence_thresholds;#tk'9DO'K L LLL wy!Q'''r,Frrr=Nc|}||}|||}||fS)Nr=)rrr9r;)r;rrr=projected_descriptorskeypoint_encoding_outputs r-_keypoint_processingz1LightGlueForKeypointMatching._keypoint_processings["((**5577 $ 5 5k B B#'#:#:9[o#:#p#p $&>>>r,keypoint_confidencesr num_pointsc|j\}}||jdz kr||dkd}||dzd}||}d||kd|z z }||jk} n tj |tj } | S)zRevaluate whether we should stop inference based on the confidence of the keypointsrrrr@g?rAr[) rhr5rrjrFr]sumr7r&onesrN) r;rLrBrrMrrrratio_confidentearly_stopped_pairss r-_get_early_stopped_image_pairsz;LightGlueForKeypointMatching._get_early_stopped_image_pairs!s A 1, , ,$8#C#CDAIq#Q#Q #7#?#? aQS#T#T 66{CCI!%9I%E$L$L$N$N$R$RWX$R$Y$Y\f$ffO"1D4I"I  #(*Zuz"J"J"J ""r,c|||}||}|j|||}t||j\}}||fSr)r?rr6)r;rrrB early_stopsrrrs r-_get_keypoint_matchingz3LightGlueForKeypointMatching._get_keypoint_matching4sW  "%k2K $D:-k:;MM#:64CX#Y#Y ''r, confidencesrc^|d|jz k}|||||kz}|S)z#mask points which should be removedr)r8rF)r;rWrrBkeeps r-_get_pruning_maskz.LightGlueForKeypointMatching._get_pruning_mask<s<T223  " K4#A#A+#N#NN ND r,r prune_outputc|j\}} } |j||} ||| ||dkt jdfd||d|d|fD\} } } }}t|D]}||||fxxdz cc<d| | | |fD\} } } }| | f}t|dd}| ||||fS) z For a given layer, prune keypoints based on the confidence of the keypoints and the matchability of the descriptors. rFc3LK|]}dt|DVdS)c$g|] \}}||Sr+r+)r)trs r-r+zULightGlueForKeypointMatching._do_layer_keypoint_pruning...Xs G G GDQtW G G Gr,N)zip)r)rpruned_keypoints_masks r- zJLightGlueForKeypointMatching._do_layer_keypoint_pruning..WsYc c  H GC0E$F$F G G Gc c c c c c r,rc38K|]}t|dVdS)T) batch_firstNr)r) pruned_tensors r-rbzJLightGlueForKeypointMatching._do_layer_keypoint_pruning.._sKS S  D 9 9 9S S S S S S r,Tr@rd padding_value) rhr?rrZrr&rr=r)r;rrrrr[rLrBrrdescriptors_matchabilitypruned_descriptorspruned_keypoints_0pruned_keypoints_1 pruned_maskpruned_indicesr*pruned_keypointsras @r-_do_layer_keypoint_pruningz7LightGlueForKeypointMatching._do_layer_keypoint_pruningCsw', Aq#'#? #L#]#]^i#j#j $ 6 67KMegr s s 5 A A$!)U\Z_M`M` a ac c c c & ! ilDY[bcc c c _.0BKQ_z"" 4 4A N1-- . . .! 3 . . . .S S "46HJ\^i!jS S S O.0BK/0BC%n$VXYYY!#3^[R^^^r,ctjtjjd}|}|d||fD\}}d||fD\}}fd||||fD\}}}}||||fS)Nrc3:K|]}t|ddVdS)Tr@rfNrr)rs r-rbzMLightGlueForKeypointMatching._concat_early_stopped_outputs..usE3 3  T D D D3 3 3 3 3 3 r,c3:K|]}t|ddVdS)TrrfNrrrs r-rbzMLightGlueForKeypointMatching._concat_early_stopped_outputs..ysE> >  T C C C> > > > > > r,c3(K|] }|V dSrr+)r)rearly_stops_indicess r-rbzMLightGlueForKeypointMatching._concat_early_stopped_outputs..}sEg g  & 'g g g g g g r,)r&rSrrh)r;rufinal_pruned_keypoints_indices!final_pruned_keypoints_iterationsrrids order_indicess ` r-_concat_early_stopped_outputsz:LightGlueForKeypointMatching._concat_early_stopped_outputshs$k*=>>l.4Q788+C0 1-@3 3 "$BC3 3 3 //> > *,MN> > > ::g g g g .1 g g g c"@Bc./PRY[jjjr,rrrc r|j\}fd|||fD\}}}|dddf}|dddf}|dddf}|dddf} |dddf} |dddf} tjdzd|fd|j|j} tjdzd|f|j|j} t dzD]}tj||dkd||d|| d| |d||f<tj| |dkd||d| | d| |d||f<| || |d||f<| || |d||f<| | fS)Nc3LK|]}|dzddVdS)rr@N)rj)r)rrs r-rbzJLightGlueForKeypointMatching._do_final_keypoint_pruning..sH- - 7=FNN:?Ar 2 2- - - - - - r,rrrr@r )r) rhr&fullrr\zerosr=r r clamp)r;rrrrrrrrrrr_matches_matching_scoresr*rs @r-_do_final_keypoint_pruningz7LightGlueForKeypointMatching._do_final_keypoint_prunings+ A- - - - BI7TcAd- - - )/111a4=111a4=111a4=111a4=*111a40*111a40:zQ=A2gndkdqrrr ; 1_a /oNc   zQ'' F FA*/+ r!2x{'9'9!Xa[=N=NST=N=U=U'V'V++HQ8A;& '+0+ r!2x{'9'9!Xa[=N=NST=N=U=U'V'V++HQ8A;& '3C12E Q8A;. /2B12E Q8A;. / /)))r,rrrc  '|rdnd}|rdnd} |jddkr\|jdd} || dtj|| || || fS|j} |j\} } }} tj|| dd}|| dz|d}||| dz|nd}|| dz||j}tj | dz| }t|||}| ||| \}}|d}|j dk}|j dk}g}g}g}g}g}tj d|| | dzd}tj|}t!|jD]}|)t%|j|ddddddf|d }n5tj| |d f|j}|j|||||| }|\}}}|r||z}|r| |z} |r||jdz kr0|j||} || |||}!n tj| tj}!tj|!rT|!d'|'}"||||'\}#}$|t=|"|t=|#|t=|$|rP|t=|'|t=|'||!}t?'fd||d|d||fD\}}%}&}}|%|&f}|r"t?'fd||| fD\}}} tj |!rn%|r!|!|||||| |\}}}}}|r<|r:|"|||||\}}}}|#||||\}}n>||||jdz \}}tj||jz}|| d|}||||| fS)Nr+rrr@r[rrArrHcBtjdtjS)NTr[)r&rrN)argss r-z@LightGlueForKeypointMatching._match_image_pair..sELUZ4X4X4Xr,)r0 inputs_embedsruand_mask_functionrR)rur=r)rM)rUc3*K|] }|VdSrr+r)rrUs r-rbzALightGlueForKeypointMatching._match_image_pair..sFVV" |,VVVVVVr,c3*K|] }|VdSrr+rs r-rbzALightGlueForKeypointMatching._match_image_pair..#sFll &#K<0llllllr,)$rhrr&r new_zerosrrOrjr3rr$rKr7r8ri ones_liker=r5r r0rPr!r>r@rSrNanyrCrVextendlistr*allrorzr)(r;rrrrrrr=rrrhrrrinitial_num_keypointsnum_points_per_pair image_indicesrJ do_early_stopdo_keypoint_pruningrurrrvrwpruned_keypoints_indicespruned_keypoints_iterationsrBextended_attention_mask layer_outputr attentionrLrRearly_stopped_image_indicesearly_stopped_matchesearly_stopped_matching_scores keypoints_0 keypoint_1rUs( @r-_match_image_pairz.LightGlueForKeypointMatching._match_image_pairs#7@BBD0:d ?1  " "OCRC(E""5"EI">>##E**##E**!  !2;// A,a#i Z(D(D!LLL%%j1n6KQOO FJFVt||JN,ABBB\`!))*q.:OQUQvww  Z!^FCCC ' 65AA 040I0I 9M1J1 1 - --Q/ -1 #3a7 )+&,.)#(<3HQW#X#X#X#_#_`jmn`npr#s#s &+o6N&O&O# 11X X K*C;"-aaa1aaai"8#'&X&X +++''+0*j+BRBRBTBTUWBX5Ybkbr*s*s*s'?42;?6%9"3 L5A 1K # F$5 $E!  #>???#**40M+N+NOOO*q6==dC[\gCh>i>ijjj9@@FabmFnAoAoppp+>?R>R*S'PUVVVV'2IaL)A,PTVc&dVVVQQMKj$ "-j 9I*fkllll!9 ; 4+lllggc02MOc9011E" 33#!03,#d Y(@$Hc  c0 c22'25#  h *,MwXg(,'F'F.% (( $G__(,'B'B;PTVZVehiVi'j'j $G_050P0PSWSb0b -,M,U,U 0- - )   -     r,rlabelsc ld}|td||n |jj}||n |jj}|jdks|ddkrtd|j\}}} } } ||dz| | | }||} | dd\} }}}| |ddd |} ||dd|j  |}||dd}| }|dddddddf| z|dddddddf<|dddddddf| z|dddddddf<| ||| | ||| \}}}}}t|||| |||| S) Nz9LightGlue is not trainable, no labels should be provided.rrzOInput must be a 5D tensor of shape (batch_size, 2, num_channels, height, width)r@r)rrr=)rrrrrrr r!) ValueErrorr0rr=ndimr!rhrjr1r_r3clonerr)r;rrrr=rxrrrchannelsrrkeypoint_detectionsrrrabsolute_keypointsrrrr r!s r-rKz$LightGlueForKeypointMatching.forward`s@  XYY Y1B1N--TXT_Tq$8$D $+Jj    ! !\%6%6q%9%9Q%>%>noo o1=1C. Ax#++JNHfeTT "44\BB*=bqb*A' 1k4%%j!R;;>>|LL !))*aT=bccffgstt ||J2..&__..);AAAqqq!!!QJ)G%)O111aaaA:&);AAAqqq!!!QJ)G&)P111aaaA:&EIE[E[    /!5F\F F B% /+'!    r,rLr)NNN)r"r#r$r%rr5rr]rFr&rMrNr*rKrSrVrZrorzrrrrr' LongTensorrrKrOrPs@r-r&r&sq  <(S(U(((( gl?? <?49L?X\_cXc? u|U5<#=>> ?????#$)L#?B#JO,#didp# ####&((((U\5<^afkfr#_\#_<#_l #_  #_ l #_$l#_#_#_#_#_Jkkk@#*#*#* #* | #* u|U\) * #*#*#*#*V%))-,0q q <q \q  q  q lT! q  $;q #Tkq  u|U\5<E Fq q q q f+/)-,0 4 4 '4  4'4  $; 4 #Tk 4  0 04 4 4 ^4 4 4 4 4 r,r&)r)rp)4collections.abcr dataclassesrnumpyrDr&rtorch.nn.utils.rnnr activationsr masking_utilsr modeling_flash_attention_utilsr modeling_utilsr r processing_utilsrutilsrrrrauto.modeling_autorconfiguration_lightgluerrModuler/rYrerMrror]rrrrrrrrr*rr$r&__all__r+r,r-rs(%$$$$$!!!!!! ++++++!!!!!!666666BBBBBBFFFFFFFF&&&&&&VVVVVVVVVVVV>>>>>>444444  7 7 7 7 7k 7 7  7F"<<<<8 UU\ U# U%, U U U U&%% I% <% % < % L4' %  %%'(%%%%2D)D)D)D)D)D)D)D)N29"P>P>P>P>P> P>P>P>f   -2\ JO,  \    &&&&&BI&&&R     BI           $EL$U$uU\[`[gMgGh$$$$@5<SU\, o o o o o #;o o  o d &'E Fr,