tjlddlmZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZddlmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+m,Z,ddl-m.Z.ddl/m0Z0m1Z1ddl2m3Z3ej4e5Z6edeGdde Z7edeGdd eZ8Gd!d"e1Z9Gd#d$e0Z:e d%&Gd'd(e.Z;Gd)d*e j<Z=eGd+d,e+Z>Gd-d.e'Z?Gd/d0e j<Z@d1ejAd2ejAd3ejAd4ejAfd5ZBGd6d7e j<ZCGd8d9e j<ZDeGd:d;eZEdZHd?ejAd@eIdAeId4ejAfdBZJedCGdDdEeEZKgdFZLdS)G)Callable) dataclassN)strict)nn pad_sequence)PreTrainedConfig)create_bidirectional_mask)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutput TensorTypeauto_docstringcan_return_tuplelogging)no_inherit_decorator)requires)CONFIG_MAPPING AutoConfig)AutoModelForKeypointDetection)CLIPMLP)apply_rotary_pos_emb)LlamaAttentioneager_attention_forward)SuperGlueImageProcessorPil)SuperGlueImageProcessorSuperGlueImageProcessorKwargs)SuperPointConfigzETH-CVG/lightglue_superpoint) checkpointceZdZUdZdZdeiZdZee zdze d<dZ e e d<dZ e e d<d Ze e d <dZe dze d <d Zee d <dZee d<dZee d<dZee d<dZee d<dZee ze d<dZee d<fdZdZxZS)LightGlueConfigaG keypoint_detector_config (`Union[AutoConfig, dict]`, *optional*, defaults to `SuperPointConfig`): The config object or dictionary of the keypoint detector. descriptor_dim (`int`, *optional*, defaults to 256): The dimension of the descriptors. depth_confidence (`float`, *optional*, defaults to 0.95): The confidence threshold used to perform early stopping width_confidence (`float`, *optional*, defaults to 0.99): The confidence threshold used to prune points filter_threshold (`float`, *optional*, defaults to 0.1): The confidence threshold used to filter matches Examples: ```python >>> from transformers import LightGlueConfig, LightGlueForKeypointMatching >>> # Initializing a LightGlue style configuration >>> configuration = LightGlueConfig() >>> # Initializing a model from the LightGlue style configuration >>> model = LightGlueForKeypointMatching(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` lightgluekeypoint_detector_configNdescriptor_dim num_hidden_layersnum_attention_headsnum_key_value_headsgffffff?depth_confidencegGz?width_confidence皙?filter_thresholdg{Gz?initializer_rangegelu hidden_actattention_dropoutTattention_biasc |j |j|_t|jtrO|jdd|jd<t |jddi|jddi|_n"|jt dd|_|jdz|_|j|_ tj di|dS)N model_type superpointattn_implementationeager)r<r) r.r- isinstancer'dictgetrr)intermediate_size hidden_sizesuper __post_init__)selfkwargs __class__s m/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/lightglue/modular_lightglue.pyrEzLightGlueConfig.__post_init__Zs  # +'+'?D $ d3T : : f:>:W:[:[\hjv:w:wD ), 7,:4;XYe;f,g--/--EL---D ) ) * 2,:<,H]d,e,e,eD )!%!4q!8.'''''''cJ|j|jzdkrtddS)zOPart of `@strict`-powered validation. Validates the architecture of the config.rz1descriptor_dim % num_heads is different from zeroN)r)r- ValueError)rFs rIvalidate_architecturez%LightGlueConfig.validate_architecturels/  !9 9Q > >PQQ Q ? >rJ)__name__ __module__ __qualname____doc__r:r sub_configsr'r@r"__annotations__r)intr+r-r.r/floatr0r2r3r5strr7r8boolrErM __classcell__rHs@rIr%r%,sG6J-z:K?Cd%55<CCCNCs    &*t***"e""""e"""!e!!!#u###J%(us{(((ND((((($RRRRRRRrJr%a Base class for outputs of LightGlue keypoint matching models. Due to the nature of keypoint detection and matching, the number of keypoints is not fixed and can vary from image to image, which makes batching non-trivial. In the batch of images, the maximum number of matches is set as the dimension of the matches and matching scores. The mask tensor is used to indicate which values in the keypoints, matches, matching_scores and prune tensors are keypoint matching information. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ ejdzed<dZ ej dzed<dZ ejdzed<dZeejdzed <dZeejdzed <dS) LightGlueKeypointMatchingOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*): Loss computed during training. matches (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Index of keypoint matched in the other image. matching_scores (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Scores of predicted matches. keypoints (`torch.FloatTensor` of shape `(batch_size, num_keypoints, 2)`): Absolute (x, y) coordinates of predicted keypoints in a given image. prune (`torch.IntTensor` of shape `(batch_size, num_keypoints)`): Pruning mask indicating which keypoints are removed and at which layer. mask (`torch.BoolTensor` of shape `(batch_size, num_keypoints)`): Mask indicating which values in matches, matching_scores, keypoints and prune are keypoint matching information. hidden_states (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(batch_size, 2, num_channels, num_keypoints)` returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True` attentions (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, 2, num_heads, num_keypoints, num_keypoints)` returned when `output_attentions=True` is passed or when `config.output_attentions=True` Nlossmatchesmatching_scores keypointsprunemask hidden_states attentions)rNrOrPrQr]torch FloatTensorrSr^r_r`ra IntTensorrbrctuplerdr>rJrIr\r\rs0&*D% d ")))(,GU  %,,,04OU&-444*.Iu 4'...$(E5?T !(((%)D% d ")))59M5*+d299926Je'(4/66666rJr\ceZdZdS)LightGlueImageProcessorKwargsN)rNrOrPr>rJrIrjrjsDrJrjc leZdZ ddddeeezdedeeee j fffd Z xZ S) LightGlueImageProcessorr6outputsr\ target_sizes thresholdreturncJt|||SNrDpost_process_keypoint_matchingrFrmrnrorHs rIrtz6LightGlueImageProcessor.post_process_keypoint_matchings! ww55g|YWWWrJr6) rNrOrPrlistrhrUr@rVreTensorrtrXrYs@rIrlrls  XX2X!4;.X X d3 $% & XXXXXXXXXXrJrlrebackendsc eZdZed d dddeeezdedeee d fffd Z xZ S) LightGlueImageProcessorPilryrzr6rmr\rnrorpz torch.TensorcJt|||Srrrsrus rIrtz9LightGlueImageProcessorPil.post_process_keypoint_matchings!ww55g|YWWWrJrv) rNrOrPrrrwrhrUr@rVrtrXrYs@rIr}r}s Xz"""  XX2X!4;.X X d3&' ( XXXXX#"XXXXXrJr}c eZdZdeffd Z d dejdedzdeejeejejfzfdZ xZ S) LightGluePositionalEncoderconfigcttjd|j|jzdzd|_dS)NrFbias)rD__init__rLinearr)r- projectorrFrrHs rIrz#LightGluePositionalEncoder.__init__sG 1f&;v?Y&Y]^&^ejkkkrJFr`output_hidden_statesNrpc||}|dd}tj|}tj|}||f}|r||fn|f}|S)Nrdim)rrepeat_interleaverecossin)rFr`rprojected_keypoints embeddingscosinessinesoutputs rIforwardz"LightGluePositionalEncoder.forwardst#nnY77(::1":EE )J'' *%%u% 6J]*122Q[P] rJF) rNrOrPr%rrerxrWrhrrXrYs@rIrrslllllll LQ   =AD[  u| uU\5<%?@ @        rJrceZdZdedeffd Z d dejdeejejfdzdejdzdejdzd ejdzd e e d eejejdzffd Z xZ S)LightGlueAttentionr layer_idxcLt|`dSrr)rDr rotary_embrFrrrHs rIrzLightGlueAttention.__init__s"  OOOrJNrcposition_embeddingsattention_maskencoder_hidden_statesencoder_attention_maskrGrpc |jdd}g|d|jR}|||dd} |du} | r|n|} | r|n|} || |dd} || |dd}||\}}t| | ||\} } tj |j j t}||| | || f|j sdn|j|jd|\}}|jg|dR}||}||fS)Nrrr6)dropoutscaling)shapehead_dimq_projview transposek_projv_projrr get_interfacer_attn_implementationrtrainingr7rreshape contiguouso_proj)rFrcrrrrrG input_shape hidden_shape query_statesis_cross_attentioncurrent_statescurrent_attention_mask key_states value_statesrrattention_interface attn_output attn_weightss rIrzLightGlueAttention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT 2$>2DW..-;M!a!7!7Sa[[0055lCCMMaQRSS {{>2277 EEOOPQSTUU  **HC';L*VY[^'_'_ $L*(?(M K ,.E) ) %8$7     " % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((rJ)NNNN) rNrOrPr%rTrrerxrhrr rrXrYs@rIrrs3IM.2596: *)*)|*)#5<#=>E*) t+ *) %|d2 *) !& t 3 *)-.*) u|U\D00 1*)*)*)*)*)*)*)*)rJrcHeZdZdeffd ZdejdejfdZxZS) LightGlueMLPrct|tj|j|j|_tj|jd|_dS)NT)elementwise_affine)rDrrrrBfc1 LayerNorm layer_normrs rIrzLightGlueMLP.__init__sR    9V5v7OPP,v'?TXYYYrJrcrpc||}||}||}||}|Srr)rr activation_fnfc2)rFrcs rIrzLightGlueMLP.forwardsN//  66 **=99 // rJ rNrOrPr%rrerxrrXrYs@rIrrsqZZZZZZZ U\elrJrceZdZdedeffd Z d dejdejdejded zd ed zd e eje ejd ze ejd zff d Z xZ S)LightGlueTransformerLayerrrctt|||_t ||_t|||_t ||_dSrr)rDrrself_attentionrself_mlpcross_attention cross_mlprs rIrz"LightGlueTransformerLayer.__init__ s_ 0CC$V,, 1&)DD%f--rJF descriptorsr`rrNoutput_attentionsrpc2|rdnd}|rdnd}|r||fz}|j\}} } |||||\} } tj|| gd} || }||z}|r| |f}|dd| | d|| | }|B|dddd| d|dd| nd}|||||\}}tj||gd}||}||z}|r>||f}|||| | fz|z||| | fz|z}|r || fz|fz}|||fS)Nr>)rrrrrrr)rrr) rrrecatrrfliprr)rFrr`rrrall_hidden_statesall_attentions batch_size num_keypointsr)attention_outputself_attentionsintermediate_states output_statesself_attention_descriptorsself_attention_hidden_statesrrcross_attention_outputcross_attentionscross_intermediate_statescross_output_statescross_attention_hidden_statess rIrz!LightGlueTransformerLayer.forwardsY#7@BBD0:d  C 1[N B 4?4E1 M>-1,?,?  ))/ -@- - )/ $i6F(GRPPP &9:: %0=%@"  P,?+O ( ' . .r1m^ T T T!WW WZ ? ? )  " "2q!Q > > C CA F F N Nz[\^_an o o o 483G3G &"7#9/ 4H4 4 0 0 %*I/IKa.bhj$k$k$k!"nn-FGG03FF  -FH[,\ )!-55j-Q_``bc./&&z=.QQST0 0   W+.@@DTCVVN-~==rJ)FF) rNrOrPr%rTrrerxrWrhrrXrYs@rIrr s..3......-2). H>H>\H><H> H> #Tk H>  $; H> u|U5<047u|9Lt9SS TH>H>H>H>H>H>H>H>rJr similarity matchability0 matchability1rpc|j\}}}tj|tj|ddz}tj|d}tj|ddddd}|||dz|dzfd} ||z|z| ddd|d|f<tj|d | dddddf<tj|d | dddddf<| S)z;create the log assignment matrix from logits and similarityrrrrN) rr functional logsigmoidr log_softmaxrnew_fullsqueeze) rrrrnum_keypoints_0num_keypoints_1 certaintiesscores0scores1scoress rIsigmoid_log_double_softmaxr^sw4>3C0J-**=99BM--k:: % T5HQ^Qe(f(f(fjn(nn %--jAoq-Q_`` &qqq!t,&qqq!t,#n&>&>r2&F&FF  << aMBBDAJ((,,EAJ((,,66r2>>E5=D#// 5;zGW;X;X;\]]J((55 #++J!OQ qQQ %aaad+%aaad+,JWW rJc||}tj|d}|S)z0Get matchability of descriptors as a probabilityr)rrrsigmoidr)rFrrs rIget_matchabilityz.LightGlueMatchAssignmentLayer.get_matchabilitys>((55 },,\::BB2FF rJ) rNrOrPr%rrerxrr rXrYs@rIrrmsIIIIIII5<u| 4ELU\rJrcHeZdZdeffd ZdejdejfdZxZS)LightGlueTokenConfidenceLayerrcttj|jd|_dS)Nr)rDrrrr)tokenrs rIrz&LightGlueTokenConfidenceLayer.__init__s3 Yv4a88 rJrrpc||}tj|d}|S)Nr)rdetachrrr r)rFrrs rIrz%LightGlueTokenConfidenceLayer.forwardsG ;--//00 %%e,,44R88 rJrrYs@rIrrsj9999999 5<ELrJrc6eZdZUdZeed<dZdZdZdZ dZ dZ dS) LightGluePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rr& pixel_values)imageFTN) rNrOrPrQr%rSbase_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpar>rJrIrrsJ #$O!&+#NNNrJrrroc`|j\}}}|ddddddfd}|ddddddfd}|j}|j}tj|jd|jd}tj|jd|jd} ||d|k} | |d|k} |j}| d} tj | || } tj | | d|| }| | |kz}| |d|z}tj ||d}tj ||d}tj ||g dd |dzd}tj | |g dd |dzd}||fS)z1obtain matches from a score matrix [Bx M+1 x N+1]Nrrrrr)rmaxindicesrearangergathervaluesexp new_tensorwherestackrr)rror_max0max1matches0matches1indices0indices1mutual0mutual1zeromatching_scores0matching_scores1valid0valid1r^r_s rIget_matches_from_scoresr5s|J1 !!!SbS#2#+  " "1 % %D !!!SbS#2#+  " "1 % %D|H|H|HN1-hoFFFtLH|HN1-hoFFFtLH(//!X666G(//!X666G ;??  D ??1  D{7D$77{7,<,C,CAx,P,PRVWW (94 5F v}}Q11 1F{68R00H{68R00Hk8X.//99!Q??GG UVXZ[[Gk#35E"FGGQQRSUVWW__`jmn`nprssO O ##rJr`heightwidthctj||g|j|jd}|dz }|djdz }||ddddfz |dz }|S)a Normalize keypoints locations based on image image_shape Args: keypoints (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`): Keypoints locations in (x, y) format. height (`int`): Image height. width (`int`): Image width. Returns: Normalized keypoints locations of shape (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`). rrNrr.).NN)rerrrrr")r`r6r7sizeshiftscales rInormalize_keypointsr=sr < 0@  X X XY] ^D 1HE HHRLL ! #EU3aaa<00E/4JJI rJzV LightGlue model taking images as inputs and outputting the matching of them. ceZdZdZdeffd ZdedefdZ d$de j d e j d e d zde e j e e j e j fffd Z d e j dede j de j de j f dZd%dZde j de j dede j fdZde j d e j de j de j de j d e j defdZdZde j de j de j de j de e j e j ff dZ d&d e j de j dedede j d zde d zd e d zde e j e j e j e e ffd Zee d&d!e jd"e jd zde d zd e d zde ezf d#ZxZS)'LightGlueForKeypointMatchingar LightGlue is a model matching keypoints in images by leveraging detections from a keypoint detector such as SuperPoint. It is based on the SuperGlue architecture and is designed to be lightweight and efficient. It consists of : 1. Keypoint Encoder 2. A Graph Neural Network with self and cross attention layers 3. Matching Assignment layers The correspondence ids use -1 to indicate non-matching points. Philipp Lindenberger, Paul-Edouard Sarlin and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV 2023. https://huggingface.co/papers/2306.13643 rc`ttjj|_jj|_j|_j |_ j |_ j |_ j |_ |j|jkr'tj|j|jd|_ntj|_t%|_tjfdt+j D|_tjfdt+j D|_tjfdt+j dz D|_|dS)NTrc2g|]}t|S))r)r).0irs rI z9LightGlueForKeypointMatching.__init__..s' e e e &v ; ; ; e e erJc.g|]}tSr>)rrBr'rs rIrDz9LightGlueForKeypointMatching.__init__..s" \ \ \q *6 2 2 \ \ \rJc.g|]}tSr>)rrFs rIrDz9LightGlueForKeypointMatching.__init__..s" ` ` `q *6 2 2 ` ` `rJr)rDrr from_configr'keypoint_detectordescriptor_decoder_dim keypoint_detector_descriptor_dimr)r+ num_layersr2r/r0rrinput_projectionIdentityrpositional_encoder ModuleListrangetransformer_layersmatch_assignment_layerstoken_confidence post_initrs `rIrz%LightGlueForKeypointMatching.__init__s    !>!J6Kj!k!k060O0f-$3 2 & 7 & 7 & 7  $"G G G$&Id.SUYUhos$t$t$tD ! !$&KMMD !">>rJkeypoint_confidencesrb num_pointsc|j\}}||jdz kr||dkd}||dzd}||}d||kd|z z }||jk} n tj |tj } | S)zRevaluate whether we should stop inference based on the confidence of the keypointsrrrrg?rr) rrLrrrZrUsumr/reonesrW) rFr`rVrbrarr'roratio_confidentearly_stopped_pairss rI_get_early_stopped_image_pairsz;LightGlueForKeypointMatching._get_early_stopped_image_pairs's A 1, , ,$8#C#CDAIq#Q#Q #7#?#? aQS#T#T 66{CCI!%9I%E$L$L$N$N$R$RWX$R$Y$Y\f$ffO"1D4I"I  #(*Zuz"J"J"J ""rJc|||}||}|j|||}t||j\}}||fSrr)rSr5r2)rFrrbrV early_stopsrr^r_s rI_get_keypoint_matchingz3LightGlueForKeypointMatching._get_keypoint_matching:sW  "%k2K $D:-k:;MM#:64CX#Y#Y ''rJ confidencesrc^|d|jz k}|||||kz}|S)z#mask points which should be removedr)r0rZ)rFrlrrVkeeps rI_get_pruning_maskz.LightGlueForKeypointMatching._get_pruning_maskBs<T223  " K4#A#A+#N#NN ND rJr prune_outputc|j\}} } |j||} ||| ||dkt jdfd||d|d|fD\} } } }}t|D]}||||fxxdz cc<d| | | |fD\} } } }| | f}t|dd}| ||||fS) z For a given layer, prune keypoints based on the confidence of the keypoints and the matchability of the descriptors. rFc3LK|]}dt|DVdS)c$g|] \}}||Sr>r>)rBtrbs rIrDzULightGlueForKeypointMatching._do_layer_keypoint_pruning...^s G G GDQtW G G GrJN)zip)rBrpruned_keypoints_masks rI zJLightGlueForKeypointMatching._do_layer_keypoint_pruning..]sYc c  H GC0E$F$F G G Gc c c c c c rJrc38K|]}t|dVdS)T) batch_firstNr)rB pruned_tensors rIrwzJLightGlueForKeypointMatching._do_layer_keypoint_pruning..esKS S  D 9 9 9S S S S S S rJTrry padding_value) rrSr rorrerrQr)rFrr`rbrrpr`rVrr'descriptors_matchabilitypruned_descriptorspruned_keypoints_0pruned_keypoints_1 pruned_maskpruned_indicesrCpruned_keypointsrvs @rI_do_layer_keypoint_pruningz7LightGlueForKeypointMatching._do_layer_keypoint_pruningIsw', Aq#'#? #L#]#]^i#j#j $ 6 67KMegr s s 5 A A$!)U\Z_M`M` a ac c c c & ! ilDY[bcc c c _.0BKQ_z"" 4 4A N1-- . . .! 3 . . . .S S "46HJ\^i!jS S S O.0BK/0BC%n$VXYYY!#3^[R^^^rJctjtjjd}|}|d||fD\}}d||fD\}}fd||||fD\}}}}||||fS)Nrc3:K|]}t|ddVdS)Trr{NrrBrs rIrwzMLightGlueForKeypointMatching._concat_early_stopped_outputs..{sE3 3  T D D D3 3 3 3 3 3 rJc3:K|]}t|ddVdS)Trr{Nrrs rIrwzMLightGlueForKeypointMatching._concat_early_stopped_outputs..sE> >  T C C C> > > > > > rJc3(K|] }|V dSrrr>)rBrearly_stops_indicess rIrwzMLightGlueForKeypointMatching._concat_early_stopped_outputs..sEg g  & 'g g g g g g rJ)rer&r r)rFrfinal_pruned_keypoints_indices!final_pruned_keypoints_iterationsr^r_ids order_indicess ` rI_concat_early_stopped_outputsz:LightGlueForKeypointMatching._concat_early_stopped_outputsns$k*=>>l.4Q788+C0 1-@3 3 "$BC3 3 3 //> > *,MN> > > ::g g g g .1 g g g c"@Bc./PRY[jjjrJr^r_rc r|j\}fd|||fD\}}}|dddf}|dddf}|dddf}|dddf} |dddf} |dddf} tjdzd|fd|j|j} tjdzd|f|j|j} t dzD]}tj||dkd||d|| d| |d||f<tj| |dkd||d| | d| |d||f<| || |d||f<| || |d||f<| | fS)Nc3LK|]}|dzddVdS)rrN)r)rBrrs rIrwzJLightGlueForKeypointMatching._do_final_keypoint_pruning..sH- - 7=FNN:?Ar 2 2- - - - - - rJrrrrr9)r) rrefullrrzerosrQr%r!clamp)rFrr^r_rr'r,r-r*r+r1r2_matches_matching_scoresrCrs @rI_do_final_keypoint_pruningz7LightGlueForKeypointMatching._do_final_keypoint_prunings+ A- - - - BI7TcAd- - - )/111a4=111a4=111a4=111a4=*111a40*111a40:zQ=A2gndkdqrrr ; 1_a /oNc   zQ'' F FA*/+ r!2x{'9'9!Xa[=N=NST=N=U=U'V'V++HQ8A;& '+0+ r!2x{'9'9!Xa[=N=NST=N=U=U'V'V++HQ8A;& '3C12E Q8A;. /2B12E Q8A;. / /)))rJr6r7rc  '|rdnd}|rdnd} |jddkr\|jdd} || dtj|| || || fS|j} |j\} } }} tj|| dd}|| dz|d}||| dz|nd}|| dz||j}tj | dz| }t|||}| ||| \}}|d}|j dk}|j dk}g}g}g}g}g}tj d|| | dzd}tj|}t!|jD]}|)t%|j|ddddddf|d }n5tj| |d f|j}|j|||||| }|\}}}|r||z}|r| |z} |r||jdz kr0|j||} || |||}!n tj| tj}!tj|!rT|!d'|'}"||||'\}#}$|t=|"|t=|#|t=|$|rP|t=|'|t=|'||!}t?'fd||d|d||fD\}}%}&}}|%|&f}|r"t?'fd||| fD\}}} tj |!rn%|r!|!|||||| |\}}}}}|r<|r:|"|||||\}}}}|#||||\}}n>||||jdz \}}tj||jz}|| d|}||||| fS)Nr>rrrrcrrrr\cBtjdtjS)NTrc)rerrW)argss rIz@LightGlueForKeypointMatching._match_image_pair..sELUZ4X4X4XrJ)r inputs_embedsrand_mask_functionr)rrr)ra)rjc3*K|] }|VdSrrr>rBrrjs rIrwzALightGlueForKeypointMatching._match_image_pair..#sFVV" |,VVVVVVrJc3*K|] }|VdSrrr>rs rIrwzALightGlueForKeypointMatching._match_image_pair..)sFll &#K<0llllllrJ)$rrrerT new_zerosrrdrrKr r=r_r/r0expand ones_likerQrLr rrer:rRrTrhrWanyrrkextendrwrhallrrr)(rFr`rr6r7rbrrrrrrrr'initial_num_keypointsnum_points_per_pair image_indicesr^ do_early_stopdo_keypoint_pruningrr^r_rrpruned_keypoints_indicespruned_keypoints_iterationsrVextended_attention_mask layer_outputrc attentionr`rgearly_stopped_image_indicesearly_stopped_matchesearly_stopped_matching_scores keypoints_0 keypoint_1rjs( @rI_match_image_pairz.LightGlueForKeypointMatching._match_image_pairs#7@BBD0:d ?1  " "OCRC(E""5"EI">>##E**##E**!  !2;// A,a#i Z(D(D!LLL%%j1n6KQOO FJFVt||JN,ABBB\`!))*q.:OQUQvww  Z!^FCCC ' 65AA 040I0I 9M1J1 1 - --Q/ -1 #3a7 )+&,.)#(<3HQW#X#X#X#_#_`jmn`npr#s#s &+o6N&O&O# 11X X K*C;"-aaa1aaai"8#'&X&X +++''+0*j+BRBRBTBTUWBX5Ybkbr*s*s*s'?42;?6%9"3 L5A 1K # F$5 $E!  #>???#**40M+N+NOOO*q6==dC[\gCh>i>ijjj9@@FabmFnAoAoppp+>?R>R*S'PUVVVV'2IaL)A,PTVc&dVVVQQMKj$ "-j 9I*fkllll!9 ; 4+lllggc02MOc9011E" 33#!03,#d Y(@$Hc  c0 c22'25#  h *,MwXg(,'F'F.% (( $G__(,'B'B;PTVZVehiVi'j'j $G_050P0PSWSb0b -,M,U,U 0- - )   -     rJrlabelsc ld}|td||n |jj}||n |jj}|jdks|ddkrtd|j\}}} } } ||dz| | | }||} | dd\} }}}| |ddd |} ||dd|j  |}||dd}| }|dddddddf| z|dddddddf<|dddddddf| z|dddddddf<| ||| | ||| \}}}}}t|||| |||| S) Nz9LightGlue is not trainable, no labels should be provided.rrzOInput must be a 5D tensor of shape (batch_size, 2, num_channels, height, width)r,rr)rbrr)r]r^r_r`rarbrcrd)rLrrrndimr:rrrItorKclonerr\)rFrrrrrGr]rr'channelsr6r7keypoint_detectionsr`rrbabsolute_keypointsr^r_rarcrds rIrz$LightGlueForKeypointMatching.forwardfs@  XYY Y1B1N--TXT_Tq$8$D $+Jj    ! !\%6%6q%9%9Q%>%>noo o1=1C. Ax#++JNHfeTT "44\BB*=bqb*A' 1k4%%j!R;;>>|LL !))*aT=bccffgstt ||J2..&__..);AAAqqq!!!QJ)G%)O111aaaA:&);AAAqqq!!!QJ)G&)P111aaaA:&EIE[E[    /!5F\F F B% /+'!    rJrrr)NNN)rNrOrPrQr%rrTrUrZrerxrWrhr_rhrkrorrrrrrrf LongTensorr\rrXrYs@rIr?r?sq  <(S(U(((( gl?? <?49L?X\_cXc? u|U5<#=>> ?????#$)L#?B#JO,#didp# ####&((((U\5<^afkfr#_\#_<#_l #_  #_ l #_$l#_#_#_#_#_Jkkk@#*#*#* #* | #* u|U\) * #*#*#*#*V%))-,0q q <q \q  q  q lT! q  $;q #Tkq  u|U\5<E Fq q q q f+/)-,0 4 4 '4  4'4  $; 4 #Tk 4  0 04 4 4 ^4 4 4 4 4 rJr?)rr?r%rlr})Mcollections.abcr dataclassesrnumpyrXrehuggingface_hub.dataclassesrrtorch.nn.utils.rnnrconfiguration_utilsr masking_utilsr modeling_flash_attention_utilsr modeling_utilsr rprocessing_utilsrutilsrrrrr utils.genericrutils.import_utilsrautorrauto.modeling_autorclip.modeling_cliprcohere.modeling_cohererllama.modeling_llamarr(superglue.image_processing_pil_supergluer$superglue.image_processing_supergluer r!r;r" get_loggerrNloggerr%r\rjrlr}ModulerrrrrxrrrrrUrhr5rTr=r?__all__r>rJrIrs%$$$$$!!!!!! ......++++++333333666666BBBBBBFFFFFFFF&&&&&&WWWWWWWWWWWWWW111111******-------->>>>>>((((((999999JJJJJJJJQQQQQQiiiiiiii))))))  H % %9:::ARARARARAR&ARAR;:ARH  7 7 7 7 7k 7 7  7F     $A   XXXXX5XXX :XXXXX!;XXX"/)/)/)/)/)/)/)/)d     7   P>P>P>P>P> P>P>P>f   -2\ JO,  \    &&&&&BI&&&R     BI           $EL$U$uU\[`[gMgGh$$$$@5<SU\, o o o o o #;o o  o d    rJ