j\ddlZddlddlZddlZdZdZedfdZ ddZ ddZ e Z ddZ d ZeZd d Zd Zd ZeZd!d ZeZd!dZGddZd"dZGddZedfdZd#dZGddZGddeZdZeZ dZe!Z"d$dZ!y)%N)*ctj|d}|j\}}tj||fd}tj||fd}t j }t ||_t ||_||_ ||_ |j|j|t ||j||fS)zPreturn k smallest values (and their indices) of the lines of a float32 arrayfloat32dtypeint64)npascontiguousarrayshapezerosfaissfloat_maxheap_array_tswig_ptridsvalnhkheapifyaddnreorderarrayrmnIDhas ^/var/www/html/Fitness-lenito-AI-main/venv/lib/python3.12/site-packages/faiss/extra_wrappers.pykminr  i 8E ;;DAq !Qw'A !Qy)A  $ $ &B a[BF a[BF BE BDJJLGGAxJJL a4Kctj|d}|j\}}tj||fd}tj||fd}t j }t ||_t ||_||_ ||_ |j|j|t ||j||fS)zOreturn k largest values (and their indices) of the lines of a float32 arrayrrr)r r r r r float_minheap_array_trrrrrrrrrs rkmaxr$+r r!c tj|d}tj|d}|j\}}|j\}}||k(sJtj||fd}|tk(r-t ||t ||t |t ||S|tk(r||jz|dd|St||t ||t |||t ||S)zJcompute the whole pairwise distance matrix between two sets of vectorsrrN) r r r empty METRIC_L2pairwise_L2sqrrMETRIC_INNER_PRODUCTTpairwise_extra_distances) xqxbmetric metric_argnqdnbd2diss rpairwise_distancesr5=s  b 2B  b 2B HHEB XXFB 7N7 ((B89 -C q"hrlB hsmL J ' 'bddA J !  RL  RL   SM  Jr!cttj|d}tt||j||SNrr)r r& float_randrsizerseedress rrandr=Xs+ ((1I &Cx}chh- Jr!ctj|d}|"tt||j||St t||j|||SNrr)r r& int64_randrr9int64_rand_max)rr;vmaxr<s rrandintrC^sO ((1G $C |8C=#((D1 J x}chhd; Jr!cttj|d}tt||j||Sr7)r r& float_randnrr9r:s rrandnrFjs+ ((1I &C sxx. Jr!c|jd}|jdk(rt|jt |S|j \}}t j|d}t||t |t ||S)z>!$>>!$  %..$ennR&8"a a4K D  NN2  NN2    NN1  NN1   a4K"!r!c2eZdZdZdZdZddZd dZdZy) KmeansuObject that performs k-means clustering and manages the centroids. The `Kmeans` class is essentially a wrapper around the C++ `Clustering` object. Parameters ---------- d : int dimension of the vectors to cluster k : int number of clusters gpu: bool or int, optional False: don't use GPU True: use all GPUs number: use this many GPUs progressive_dim_steps: use a progressive dimension clustering (with that number of steps) Subsequent parameters are fields of the Clustring object. The most important are: niter: int, optional clustering iterations nredo: int, optional redo clustering this many times and keep best verbose: bool, optional spherical: bool, optional do we want normalized centroids? int_centroids: bool, optional round centroids coordinates to integer seed: int, optional seed for the random number generator init_method: ClusteringInitMethod, optional centroid initialization method: - ClusteringInitMethod_RANDOM: uniform random sampling (default) - ClusteringInitMethod_KMEANS_PLUS_PLUS: k-means++ D²-weighted sampling, selects centroids with probability proportional to squared distance from existing centroids. Better quality but O(nkd) complexity. - ClusteringInitMethod_AFK_MC2: Assumption-Free K-MC², MCMC-based approximation using Metropolis-Hastings. Good quality with lower complexity than k-means++ for large k. afkmc2_chain_length: int, optional chain length for AFK-MC² initialization (default 50). Longer chains give better approximation to k-means++ but are slower. c t||_|j|d|_d|vrt|_nt |_|j D]S\}}|dk(r|dk(s|dk(r t}||_'t|j|t|j||U|jy)zd: input dimension, k: nb of centroids. Additional parameters are passed on the ClusteringParameters object, including niter=25, verbose=False, spherical = False Fprogressive_dim_stepsgpuTN) r1resetr"ProgressiveDimClusteringParameterscpClusteringParametersitems get_num_gpusgetattrsetattr set_index)rxr1rkwargsvs rrzzKmeans.__init__s  1  "f ,8:DG*,DGLLNDAqEz9R$A#A&# r!c|j}|jjtk(ru|jjrt ||_nt||_|jr1tj|j |j|_yy|jrt|j}n t}||_ y)N)ngpu)r1r __class__r spherical IndexFlatIPindex IndexFlatL2rr index_cpu_to_all_gpusGpuProgressiveDimIndexFactoryProgressiveDimIndexFactoryfac)rxr1rs rrzKmeans.set_index+s FF 77   4 4ww  (^ (^ xx"88JJTXX  xx3B02DHr!NcR|t||_d|_d|_d|_y)zeprepare k-means object to perform a new clustering, possibly with another number of centroidsN)rer centroidsobjiteration_stats)rxrs rrz Kmeans.reset=s* =VDF#r!c tj|d}|j\}}||jk(sJ|jj t k(rt||j|j}|D|j\}}||k(sJtj|j|j|j||j|ng|J|J|jjrJt!||j|j}|j|t#||j$tj&|j} | j)|j||_ |j*} t-| j/D cgc]} | j1| } } tj2| D cgc]} | j4c} |_dj7} | D cgc]} | Dcic]}|t9| |c}c}} |_|j4j.dkDr|j4dSdScc} wcc} wcc}wcc}} w)aPerform k-means clustering. On output of the function call: - the centroids are in the centroids field of size (`k`, `d`). - the objective value at each iteration is in the array obj (size `niter`) - detailed optimization statistics are in the array iteration_stats. Parameters ---------- x : array_like Training vectors, shape (n, d), `dtype` must be float32 and n should be larger than the number of clusters `k`. weights : array_like weight associated to each vector, shape `n` init_centroids : array_like initial set of centroids, shape (n, d) Returns ------- final_obj: float final optimization objective rr,obj time time_search imbalance_factor nsplitrrr)r r r r1rrr Clusteringrr copy_array_to_vectorravelrtrainrrProgressiveDimClusteringrrvector_float_to_arrayreshaperrVr9atrrsplitr)rxrbweightsinit_centroidsrr1clusncr3rstatsr]st stat_fieldsfields rrz Kmeans.trainFs 6  ) 4ww1DFF{{ 77   4 4a1D)'--BQww**"((*DNN JJq$**g .? "?!) ))ww(( ((+Atvvtww?D JJq(1+txx 0//? "**46615$$&+EJJL&9:&9!&9:88e4eRVVe45DJJL KP KPRK @K5UGB& &K @5  $xx}}q0txx|9c9;4 A s$(II# I-I( &I-(I-cRtj|d}|jJd|jj |jj |j|jj |d\}}|j|jfS)Nrrzshould train before assigningrI)r r rrrrsearchr)rxrbrrs rassignz Kmeans.assigns  ) 4~~)J+JJ)  t~~&zz  A&1wwy!'')##r!rNN) rrrrrzrrrrrr!rrrs",\0$$<:|$r!rc$eZdZdZdZdZddZy) SuperKmeansaDrop-in replacement for `Kmeans` that runs `SuperKMeans` (ADSampling + PDX progressive pruning) instead of `Clustering`. Same `centroids`, `obj`, `iteration_stats`, and `assign()` surface; additionally exposes `gemm_pruning_rates`. kwargs are forwarded to `SuperKMeansParameters`. Fields not present on it (e.g. `spherical`, `int_centroids`, `nredo`, `frozen_centroids`, `init_method`, `update_index`, `early_stop_threshold`, `progressive_dim_steps`, `gpu`) raise `AttributeError`. c ||_|j|d|_t|_|j D]2\}}t |j|t|j||4|jy)NF) r1rrSuperKMeansParametersrrrrr)rxr1rrkeyrs rrzzSuperKmeans.__init__sc 1 ')llnFC DGGS ! DGGS! $% r!c8t|j|_yr)rr1rrs rrzSuperKmeans.set_indexs ( r!Nc |Jd|Jdtj|d}|j\}}||jk(sJt ||j |j }|j|tj|j}|j|j ||_ |j}t|jD cgc]} |j| }} tj |D cgc]} | j"c} |_dj%} |D cgc]} | D cic]} | t'| | c} c} } |_ tj|j(|_|j"jdkDr|j"dSdScc} wcc} wcc} wcc} } w) Nz$SuperKmeans does not support weightsz+SuperKmeans does not support init_centroidsrrrrrr)r r r r1 SuperKMeansrrrr vector_to_arrayrrrrVr9rrrrrgemm_pruning_rates) rxrbrrrr1scrrr]rrrs rrzSuperKmeans.trainsF FF  " 9 8 9 "  ) 4ww1DFF{{ DFFDGG ,  ))",,7 "**46615""&+EJJL&9:&9!&9:88e4eRVVe45DJJL KP KPRK @K5UGB& &K @5 #("7"78M8M"N#xx}}q0txx|9c9;4 A s$#F=G G  G !G G r)rrrrrzrrrr!rrrs ):r!rcJt|tjjSr) isinstance collectionsabcSequenceras r is_sequencers a11 22r!c |j\}}tj|d}t|rtj|d}|j|fk(sJt |j dzdz}tj ||fd}t||t|t|t|||S||zdzdz}tj ||fd}t|||t|t|||S)a> Pack a set integers (i, j) where i=0:n and j=0:M into n bitstrings. Output is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. If nbit is an integer: all entries takes nbit bits. If nbit is an array: entry (i, j) takes nbit[j] bits. rorrH) r r r rresumr&pack_bitstrings_cr)rOnbitrM code_sizebs rpack_bitstringsr s 77DAq Qg.A4##D8zzaT!!!aA-. HHa^7 3 q(4.(1+x{I  HX\a' HHa^7 3!Qhqk8A; J Hr!c |j\}}|tj|d}t|}t |j dzdz}||k\sJtj ||fd}t||t|t||t||S|}||zdzdz}||k\sJtj ||fd}t|||t||t||S)a Unpack a set integers (i, j) where i=0:n and j=0:M from n bitstrings (encoded as uint8s). Input is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. Two forms: - when called with (array, M, nbit): there are M entries of size nbit per row - when called with (array, nbits): element (i, j) is encoded in nbits[j] bits rorrr) r r r rrerr&unpack_bitstrings_cr)r  M_or_nbitsrrr r min_code_sizerOs runpack_bitstringsrs77LAy |##Jg> ITXXZ!^12 M))) HHaV7 + q(4.(1+y(1+  H TA!+ M))) HHaV7 +Aq$ Y L Hr!)90)rN)i)Nrr)rr)#numpyr faiss.loaderr collections.abcrrr$r'r5r=rClrandrFrQrTrSr^rcrmrgrsrprurrrrrrrr rrr rr!rrs $$'0A6   , 8 !H :L<<~:  P$8v6|f$f$R/:&/:n3$ 6( r!