qtjpO dZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZm Z ddl!m"Z"m#Z#edd\Z$Z%ee$e%d\Z$Z%e&e$Z$gdZ'dhdej(DzZ)dSdZ*ej+,dedZ-dZ.ej+,dge#e"dZ/dZ0ej+,d e'ej+,d!ed"Z1d#Z2ej+,d$d%d&Z3d'Z4d(Z5d)Z6d*Z7ej+,d+d,d-gd.Z8ej+,d/e#d0Z9ej+,d1e'd2Z:d3Z;d4Zgej>d7ggfd!d6id7d8gd8d7ggfid7d8gd9d:ggfgd;Z?ej+,d/e#d<Z@ej+,d/e#d=ZAd>ZBd?ZCd@ZDej+,dAdBdCgej+,dDddEgdFZEdGZFej+,dHdIdJgdKZGej+,dLdMdNgdOZHej+,dPd,d-gdQZIdRZJdS)TzF Tests for HDBSCAN clustering algorithm Based on the DBSCAN test code N)stats)distance)HDBSCAN)CONDENSED_dtype_condense_tree _do_labelling)_OUTLIER_ENCODING) make_blobs)fowlkes_mallows_score)_VALID_METRICSeuclidean_distances)BallTreeKDTree)StandardScaler)shuffle)assert_allcloseassert_array_equal)CSC_CONTAINERSCSR_CONTAINERS ) n_samples random_state)r)kd_tree ball_treebruteautoc$h|] \}}|dS)label).0_outs `/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/sklearn/cluster/tests/test_hdbscan.py r'&s KKKvq#c'lKKKGz?ctt|tz }|dksJt|t|ksJdS)N)lenset OUTLIER_SETr y)labels threshold n_clusterss r&check_label_qualityr3)sHS[[;.//J ????  + +i 7 7 7 7 7 7r( outlier_typectjtjd|}ddd|}t|d}t|d}t}|dg|d<||g|d<t d |}|j|k \}t|ddg||j | \}t|ddgttddttd d z} t d || } t| j|j| d S)O Tests if np.inf and np.nan data are each treated as special outliers. )infinitemissingc||kSNr"xr/s r&z#test_outlier_data..9s ar(c*tj|Sr:)npisnanr;s r&r=z#test_outlier_data..:s r(r!probrFcopyrN)r?infnanr XrErfitlabels_nonzerorprobabilities_listrange) r4outlier prob_checkr!rA X_outliermodelmissing_labels_idxmissing_probs_idx clean_indices clean_models r&test_outlier_datarX/s F6G ('++J l +G 4E \ *6 2DIQ>)Aq6222&Ju';TBBKKMM(1a&111q!%%U1c]](;(;;Mu%%%)))M*BCCK{*EM-,HIIIIIr(c`tt}|}tdd|}t ||t |d}tjt|5tddtdddn #1swxYwYd}d|d <d |d <tjt|5tdd |ddddS#1swxYwYdS) zy Tests that HDBSCAN works with precomputed distance matrices, and throws the appropriate errors when needed. precomputedTmetricrEz*The precomputed distance matrix.*has shapematchNz'The precomputed distance matrix.*valuesr)rrBrB)rBrF) r rIrEr fit_predictrr3pytestraises ValueError)D D_originalr0msgs r&test_hdbscan_distance_matrixrfOs AAJ M 5 5 5 A A! D DFAz""" 7C z - - -@@}4000<)ABB*QC000Ss__s+=@S+S'T'TTUUIu%%%)))I*>??K00l0KKL|VI%677777r(ctddtjtjdidt}t |dS)z4 Tests that HDBSCAN using `BallTree` works. r}rxrBF)r\rrEN)rr?rrIrr_r3rss r&!test_hdbscan_best_balltree_metricrsXC1D1D+EEk!nn r(ctttdz dt}t |t sJdS)z Tests that HDBSCAN correctly does not generate a valid cluster when the `min_cluster_size` is too large for the data. rBFmin_cluster_sizerEN)rr,rIr_r-issubsetr.rss r&test_hdbscan_no_clustersrsT c!ffqju = = = I I! L LF v;;   , ,,, ,,,r(c:tdttdD]v}t|dt}d|D}t|dkr,t jt j||ksJwdS)zb Test that the smallest non-noise cluster has at least `min_cluster_size` many points rzrBFrcg|] }|dk| S)rr")r#r!s r& z1test_hdbscan_min_cluster_size..s@@@ERKKuKKKr(rN)rOr,rIrr_r?minbincount)rr0 true_labelss r&test_hdbscan_min_cluster_sizers "!SVVQ//HH*:GGGSSTUVV@@&@@@ {  q 6"+k22337GGGGG HHr(ctj}t|dt}t |dS)zA Tests that HDBSCAN works when passed a callable metric. Fr[N)r euclideanrr_rIr3)r\r0s r&test_hdbscan_callable_metricrs@ F F / / / ; ;A > >Fr(treerrctd|d}d}tjt|5|t ddddS#1swxYwYdS)z Tests that HDBSCAN correctly raises an error when passing precomputed data while requesting a tree-based algorithm. rZFr\rwrEz%precomputed is not a valid metric forr]N)rr`rarbrJrI)rrres r&"test_hdbscan_precomputed_non_bruters $U C C CC 1C z - - -  sAAA csr_containerctdtj}t ||t}|}td|j}t ||tjdftj dffD]\}}t}||d<td|j}t ||dt|dksJ|}||d<td|j}t ||d}tj t| 5td d d |d d d d S#1swxYwYd S)z Tests that HDBSCAN works correctly when passing sparse feature data. Evaluates correctness by comparing against the same data passed as a dense array. FrDr7r8rrrr!z4Sparse data matrices only support algorithm `brute`.r]rrrN)rrJrIrKr3rErr?rGrHr r`rarb) r dense_labels _X_sparseX_sparse sparse_labels outlier_valr4X_denseres r&test_hdbscan_sparser s&&&**1--5L %%% a  I~~H'''++H55=M|]333(*vz&:RVY>##$U+++//99A <7777 @C z - - -UU{kFFFJJ8TTTUUUUUUUUUUUUUUUUUUs&GGGrwcddg}tdd|d\}}tdd |}t||j|jD],\}}}t ||d d t ||d d -t|dtjdd t}|jjddksJ|jjddksJdS)zj Tests that HDBSCAN centers are calculated and stored properly, and are accurate to the data. )rjrj)@rirr)rrcenters cluster_stdbothF) store_centersrErBg?)rtolatol)rwrrrEN) r rrJzip centroids_medoids_rrIr)rwrHr$rcentercentroidmedoids r&test_hdbscan_centersr.s :&G 1gSV W W WDAq U 3 3 3 7 7 : :C$'$N$N;; &qt<<<<QT:::::        c!ff  >  "a ' ' ' ' < a A % % % % % %r(ctjd}|dd}t ddddd |}tj|d \}}t|dksJ||d kd ksJt dd dddd|}tj|d \}}t|dksJ||d kdksJdS)zS Tests that HDBSCAN single-cluster selection with epsilon works correctly. rrzrCrjeomTF)rcluster_selection_epsiloncluster_selection_methodallow_single_clusterrE) return_countsrg ףp= ?r)rrrrrwrEN)r?random RandomStaterandrr_uniquer,)rng no_structurer0 unique_labelscountss r&.test_hdbscan_allow_single_cluster_with_epsilonrGsA )   " "C88C##L "%!&!   k,  IfDAAAM6 }   " " " " -2% & + + + +"&!&!  k, IfDAAAM6 }   " " " " -2% &! + + + + + +r(c6ddgddgddgddgg}td|gdd\}}td |j}t t |t d |vz }|d ksJt||d kdS)z Validate that HDBSCAN can properly cluster this difficult synthetic dataset. Note that DBSCAN fails on this (see HDBSCAN plotting example) g333333g333333?r+i)皙?gffffff?皙?rr)rrrrFrDrr)N)r rrJrKr,r-intr )rrIr/r0r2s r&test_hdbscan_better_than_dbscanrjs u~t}q!fq"g>G +++    DAq % $ $Q ' ' /FS[[!!Cf $5$55J ????&!$$t++++r(z kwargs, XrZrBrzr+rcJtdddd||dS)zo Tests that HDBSCAN works correctly for array-likes and precomputed inputs with non-finite points. rBF min_samplesrENr")rrJ)rIkwargss r&test_hdbscan_usable_inputsr~s3 0000044Q77777r(c|tjd}d}tjt|5t dd|ddddS#1swxYwYdS)zd Tests that HDBSCAN raises the correct error when there are too few non-zero distances. )rrz#There exists points with fewer thanr]rZFr[N)r?zerosr`rarbrrJrrIres r&-test_hdbscan_sparse_distances_too_few_nonzerors  bhx(())A /C z - - -99}511155a888999999999999999999s%A--A14A1c8tjd}d|ddddf<d|ddddf<||jz}||}d}tjt |5t dd |ddddS#1swxYwYdS) zu Tests that HDBSCAN raises the correct error when the distance matrix has multiple connected components. )rrBNrCz3HDBSCAN cannot be performed on a disconnected graphr]rZFr[)r?rTr`rarbrrJrs r&0test_hdbscan_sparse_distances_disconnected_graphrs AAbqb"1"fIAabb"##gJ ACA aA ?C z - - -99}511155a888999999999999999999s%BBBcd}d}tjt|5td|dt dddn #1swxYwYtjt|5td|dt dddn #1swxYwYt ttj ttj z }t|d krftjt|5td|d dt ddddS#1swxYwYdSdS) zR Tests that HDBSCAN correctly raises an error for invalid metric choices. c|Sr:r")r<s r&r=z2test_hdbscan_tree_invalid_metric..sr(zV.* is not a valid metric for a .*-based algorithm\. Please select a different metric\.r]rF)rwr\rENrr) r`rarbrrJrIrNr-rrrr,)metric_callableremetrics_not_kds r& test_hdbscan_tree_invalid_metricrsN"kO  z - - -PP)O%HHHLLQOOOPPPPPPPPPPPPPPP z - - -RR+oEJJJNNqQQQRRRRRRRRRRRRRRR #h455FQ ]:S 1 1 1 V V iq0A N N N R RST U U U V V V V V V V V V V V V V V V V V Vs5+AAA>+B55B9<B9.1E,,E03E0ctttdzd}d}tjt |5|tddddS#1swxYwYdS)zx Tests that HDBSCAN correctly raises an error when setting `min_samples` larger than the number of samples. rBFrz min_samples (.*) must be at mostr]N)rr,rIr`rarbrJ)rres r&!test_hdbscan_too_many_min_samplesrs c!ffqju 5 5 5C -C z - - -  sA,,A03A0ct}tj|d<d}t dd}t jt|5||ddddS#1swxYwYdS)zu Tests that HDBSCAN correctly raises an error when providing precomputed distances with `np.nan` values. rz(np.nan values found in precomputed-denserZFr[r]N) rIrEr?rHrr`rarbrJ)X_nanrers r&"test_hdbscan_precomputed_dense_nanrs FFHHE&E$K 4C U 3 3 3C z - - - sA::A>A>rTFepsilonrc4 d}t||ddgddgddgg\} td|}t|j|j}|dz|d z|d zh}|dzd|d zd |d zdi}t |||||  fd tt D fdtt D} tj | j  } t | dS)zR Tests that the `_do_labelling` helper function correctly assigns labels. 0rr)rrFrDrrzr+rrBcondensed_treeclusterscluster_label_maprrcZi|]'}|tj|kdd(S)r)r?where)r#_yr/s r& z+test_labelling_distinct..s3KKKBHQ"W--a03KKKr(c.i|]}||Sr"r")r#rfirst_with_labelr0s r&rz+test_labelling_distinct..s&KKK2v.r23KKKr(N) r rrJr_single_linkage_tree_rrrNr-r? vectorizerr)global_random_seedrrrrIestrrr y_to_labelsaligned_targetrr0r/s @@@r&test_labelling_distinctrsc I 'F G G   DAq u    ! !! $ $C# !C4HNA y1}i!m>S2.. . . . . %$aQ:!"# Fw'*4I y>>S2.. . . . . . .r(rrrcBtjd}|d}t|}d}t jt |5td|d|ddddS#1swxYwYdS) zCheck that we raise an error if the centers are requested together with a precomputed input matrix. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27893 rdrzz>Cannot store centers when using a precomputed distance matrix.r]rZF)r\rrEN) r?rrr r`rarbrrJ)rrrIX_disterr_msgs r&0test_hdbscan_error_precomputed_and_store_centersr+s )   " "C 8A  # #FNG z 1 1 1 '    #f+++ s!&BBB valid_algorrcZtd|dtdS)zTest that HDBSCAN works with the "cosine" metric when the algorithm is set to "brute" or "auto". Non-regression test for issue #28631 cosineFrN)rr_rI)rs r&*test_hdbscan_cosine_metric_valid_algorithmr ?s- 8z>>>JJ1MMMMMr( invalid_algoctd|d}tjtd5|t ddddS#1swxYwYdS)zTest that HDBSCAN raises an informative error is raised when an unsupported algorithm is used with the "cosine" metric. rFrzcosine is not a valid metricr]N)rr`rarbr_rI)r!hdbscans r&,test_hdbscan_cosine_metric_invalid_algorithmr$Is XEJJJG z)G H H HAsAAAc tjdd}d}tjt |5t d}||ddddS#1swxYwYdS)z\ Test that HDBSCAN raises a FutureWarning when the `copy` parameter is not set. rrzCThe default value of `copy` will change from False to True in 1.10.r]rrN)r?rrr`rrrrJ)rIrers r&!test_hdbscan_default_copy_warningr&Ts a  ''11A PC m3 / / /r***  s&BB B)r))K__doc__numpyr?r`scipyr scipy.spatialrsklearn.clusterrsklearn.cluster._hdbscan._treerrr sklearn.cluster._hdbscan.hdbscanr sklearn.datasetsr sklearn.metricsr sklearn.metrics.pairwiser r sklearn.neighborsrrsklearn.preprocessingr sklearn.utilsrsklearn.utils._testingrrsklearn.utils.fixesrrrIr/ fit_transform ALGORITHMSitemsr.r3mark parametrizerXrfrqrtrrrrrrrrrrrrrrGrrrrrrrrrr r$r&r"r(r&r;s(  """"""###### ?>>>>>''''''111111HHHHHHHH........000000!!!!!!FFFFFFFF>>>>>>>>zCb1111wq!!$$$1N""1%% dKK1H1B1H1J1JKKKK 8888 ):;;JJ<;J>AAA0-/Q/Q./QRR  SR "    ,,>22%%32-,%P 0 0 077888784   --- H H H   )[!9::;:.99UU:9UDj11&&21&0 , , ,F,,,( M "HBHq"&kBFA;-G$H$HI M "aVaV$45 q!fq!f 888.99 9 9:9 9.99 9 9:9 9 VVV0   /$??QH--!/!/.-@?!/H&/&/&/R:x*@AABA&'899NN:9N)[)ABBCB     r(