h dZddlZddlZddlZddlZddlmZmZddlZ ddl m Z ddl m Z mZmZddlmZdd lmZmZmZdd lmZmZdd lmZmZdd lmZdd lmZddl m!Z!m"Z"dZ#dZ$dZ%ddddddde j&e j'j(dd dZ)GddeZ*Gdde*Z+ ddZ,Gdde*Z-dS) zUGraphicalLasso: sparse inverse covariance estimation with an l1-penalized estimator. N)IntegralReal)linalg)empirical_covarianceEmpiricalCovariancelog_likelihood)ConvergenceWarning)_is_arraylike_not_scalarcheck_random_state check_scalar)delayedParallel)Interval StrOptions)_cd_fast)lars_path_gram)check_cvcross_val_scorecZ|jd}dt||z|tjdtjzzz}||tj|tjtj|z zz }|S)zEvaluation of the graphical-lasso objective function the objective function is made of a shifted scaled version of the normalized log-likelihood (i.e. its empirical mean over the samples) and a penalisation term to promote sparsity rr )shaper nplogpiabssumdiag)mle precision_alphapcosts [/var/www/html/Sam_Eipo/venv/lib/python3.11/site-packages/sklearn/covariance/_graph_lasso.py _objectiver&$s A .j11 1Aq25y8I8I4I IDERVJ''++--rwz7J7J0K0K0O0O0Q0QQ RRD Kctj||z}||jdz}||tj|tjtj|z zz }|S)zExpression of the dual gap convergence criterion The specific definition is given in Duchi "Projected Subgradient Methods for Learning Sparse Gaussians". r)rrrrr)emp_covr!r"gaps r% _dual_gapr+1sy &:% & &C: A C5BF:&&**,,rvbgj6I6I/J/J/N/N/P/PP QQC Jr'ctj|}d|jdd|jddz<tjtj|S)aFind the maximum alpha for which there are some non-zeros off-diagonal. Parameters ---------- emp_cov : ndarray of shape (n_features, n_features) The sample covariance matrix. Notes ----- This results from the bound for the all the Lasso that are solved in GraphicalLasso: each time, the row of cov corresponds to Xy. As the bound for alpha is given by `max(abs(Xy))`, the result follows. rNr)rcopyflatrmaxr)r)As r% alpha_maxr1=sJ A !AF  agaj1n  6"&))  r'cd-C6?dF) cov_initmodetolenet_tolmax_iterverbose return_costseps return_n_iterc  |j\} } |dkr|rytj|} dt|| z}|| t jdtjzzz }t j|| z| z }| r|| ||fdfS|| ||ffS| r|tj|dfS|tj|fS||}n|}|dz}|j dd| dz}||j dd| dz<tj |} t j | }t}|dkrtdd }ntd } tj}t j|ddddfd }t|D]r}t| D]}|dkr8|dz }||||k||<|dd|f||k|dd|f<n|ddddf|dd<||||kf}t jdi|5|dkrP| ||k|f| ||fd| zzz }t#j||d|||||t'dd \}} } } n&t)|||j|| dz z d| dd\} } }dddn #1swxYwYd|||ft j|||k|f|z z | ||f<| ||f |z| ||k|f<| ||f |z| |||kf<t j||}|||||kf<||||k|f<t j| st1dt3|| |}t5|| |}|rt7d|||fz|r|||ft j||krnJt j|s|dkrt1dtt=jd||fzt@n*#t0$r}|j!ddzf|_!|d}~wwxYw|r| r || ||dzfS|| |fS| r|| |dzfS|| fS)ac L1-penalized covariance estimator. Read more in the :ref:`User Guide `. .. versionchanged:: v0.20 graph_lasso has been renamed to graphical_lasso Parameters ---------- emp_cov : ndarray of shape (n_features, n_features) Empirical covariance from which to compute the covariance estimate. alpha : float The regularization parameter: the higher alpha, the more regularization, the sparser the inverse covariance. Range is (0, inf]. cov_init : array of shape (n_features, n_features), default=None The initial guess for the covariance. If None, then the empirical covariance is used. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where p > n. Elsewhere prefer cd which is more numerically stable. tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. Range is (0, inf]. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. Range is (0, inf]. max_iter : int, default=100 The maximum number of iterations. verbose : bool, default=False If verbose is True, the objective function and dual gap are printed at each iteration. return_costs : bool, default=False If return_costs is True, the objective function and dual gap at each iteration are returned. eps : float, default=eps The machine-precision regularization in the computation of the Cholesky diagonal factors. Increase this for very ill-conditioned systems. Default is `np.finfo(np.float64).eps`. return_n_iter : bool, default=False Whether or not to return the number of iterations. Returns ------- covariance : ndarray of shape (n_features, n_features) The estimated covariance matrix. precision : ndarray of shape (n_features, n_features) The estimated (sparse) precision matrix. costs : list of (objective, dual_gap) pairs The list of values of the objective function and the dual gap at each iteration. Returned only if return_costs is True. n_iter : int Number of iterations. Returned only if `return_n_iter` is set to True. See Also -------- GraphicalLasso : Sparse inverse covariance estimation with an l1-penalized estimator. GraphicalLassoCV : Sparse inverse covariance with cross-validated choice of the l1 penalty. Notes ----- The algorithm employed to solve this problem is the GLasso algorithm, from the Friedman 2008 Biostatistics paper. It is the same algorithm as in the R `glasso` package. One possible difference with the `glasso` R package is that the diagonal coefficients are not penalized. rrr Ngffffff?rr2raiseignore)overinvalid)rBC)orderiFTlars)XyGram n_samples alpha_min copy_Gramr<method return_pathg?z1The system is too ill-conditioned for this solverz<[graphical_lasso] Iteration % 3i, cost % 3.2e, dual gap %.3ezANon SPD result: the system is too ill-conditioned for this solverzDgraphical_lasso: did not converge after %i iteration: dual gap: %.3ez3. The system is too ill-conditioned for this solver)"rrinvr rrrrr-r.pinvharangelistdictinfrangeerrstatecd_fastenet_coordinate_descent_gramr rsizedotisfiniteFloatingPointErrorr+r&printappendrwarningswarnr args)r)r"r5r6r7r8r9r:r;r<r=_ n_featuresr!r$d_gap covariance_diagonalindicescostserrorssub_covarianceiidxdirowcoefses r%graphical_lassorpQsHMMAz zz  4G,,J.*===D JBE !2!22 2DF7Z/00:=E : T5M1<< T5M99 4 7 3 3Q66 7 3 333llnn mmoo 4K|--zA~-.H*2K&& Q&'k**Ji ##G FFE t||7H555g&&&UQRRV!4C@@@xL L AZ((2 92 977qB)4RC)HN2&,72,>w#~,NN111b5))(3ABBF(;N111%c7c>12[**6**t||'w#~s':;)#s(3dSj@B!*1)M!!*$$.t44! * *q!QQ'5"!/&)h&+zA~&>&* ##)(- ' ' ' 1e)>(+S)f[C)<=uEEF( 38$4>c3h3G2G%2O 7c>3./3=c3h3G2G%2O 33./~u5538 CC/038 GsNC/00;z~~//00 (Ggz599Egz599D R$&' , dE]+++ve}}s"";t$$ Q(W MVU#$"    &)SSU +  2 E1q58 8 E1 1  + AE1 1 * *s?-CP>/A=J8, P>8J< <P>?J< E=P>> Q%Q  Q%c eZdZUiejeedddgeedddgeedddgeddhgdgd Ze e d <e d  dfd Z xZ S)BaseGraphicalLassorNrightclosedleftr2rEr:)r7r8r9r6r:_parameter_constraintsstore_precisionr3r4Fct|||_||_||_||_||_dS)Nassume_centered)super__init__r7r8r9r6r:)selfr7r8r9r6r:r{ __class__s r%r}zBaseGraphicalLasso.__init__IsG 999      r')r3r3r4r2FF)__name__ __module__ __qualname__rrwrrrrrR__annotations__popr} __classcell__rs@r%rrrr>s$  4$q$w7778XdAtG<<<=Xh4???@T6N++,; $$$D0111  r'rrceZdZUdZiejdeedddgiZee d< dd d d d d d d fd Z ddZ xZ S)GraphicalLassoa Sparse inverse covariance estimation with an l1-penalized estimator. Read more in the :ref:`User Guide `. .. versionchanged:: v0.20 GraphLasso has been renamed to GraphicalLasso Parameters ---------- alpha : float, default=0.01 The regularization parameter: the higher alpha, the more regularization, the sparser the inverse covariance. Range is (0, inf]. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where p > n. Elsewhere prefer cd which is more numerically stable. tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. Range is (0, inf]. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. Range is (0, inf]. max_iter : int, default=100 The maximum number of iterations. verbose : bool, default=False If verbose is True, the objective function and dual gap are plotted at each iteration. assume_centered : bool, default=False If True, data are not centered before computation. Useful when working with data whose mean is almost, but not exactly zero. If False, data are centered before computation. Attributes ---------- location_ : ndarray of shape (n_features,) Estimated location, i.e. the estimated mean. covariance_ : ndarray of shape (n_features, n_features) Estimated covariance matrix precision_ : ndarray of shape (n_features, n_features) Estimated pseudo inverse matrix. n_iter_ : int Number of iterations run. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- graphical_lasso : L1-penalized covariance estimator. GraphicalLassoCV : Sparse inverse covariance with cross-validated choice of the l1 penalty. Examples -------- >>> import numpy as np >>> from sklearn.covariance import GraphicalLasso >>> true_cov = np.array([[0.8, 0.0, 0.2, 0.0], ... [0.0, 0.4, 0.0, 0.0], ... [0.2, 0.0, 0.3, 0.1], ... [0.0, 0.0, 0.1, 0.7]]) >>> np.random.seed(0) >>> X = np.random.multivariate_normal(mean=[0, 0, 0, 0], ... cov=true_cov, ... size=200) >>> cov = GraphicalLasso().fit(X) >>> np.around(cov.covariance_, decimals=3) array([[0.816, 0.049, 0.218, 0.019], [0.049, 0.364, 0.017, 0.034], [0.218, 0.017, 0.322, 0.093], [0.019, 0.034, 0.093, 0.69 ]]) >>> np.around(cov.location_, decimals=3) array([0.073, 0.04 , 0.038, 0.143]) r"rNrsrtrw{Gz?r2r3r4F)r6r7r8r9r:r{cdt||||||||_dSN)r7r8r9r6r:r{)r|r}r") r~r"r6r7r8r9r:r{rs r%r}zGraphicalLasso.__init__sE +     r'c |||dd}|jr%tj|jd|_n|d|_t||j}t||j |j |j |j |j|jd\|_|_|_|S)aFit the GraphicalLasso model to X. Parameters ---------- X : array-like of shape (n_samples, n_features) Data from which to compute the covariance estimate. y : Ignored Not used, present for API consistency by convention. Returns ------- self : object Returns the instance itself. r )ensure_min_featuresensure_min_samplesrrrzTr"r6r7r8r9r:r=)_validate_params_validate_datar{rzerosr location_meanrrpr"r6r7r8r9r:rdr!n_iter_)r~Xyr)s r%fitzGraphicalLasso.fits    qQ  O O   'Xagaj11DNNVVAYYDN&q$:NOOO:I *]]L ; ; ; 7$/4< r')rN) rrr__doc__rrrwrrrRrr}rrrs@r%rrZs]]~$  3$((4D999:$$D *########r'rc td|dz } t|} || } n|} t} t} t}|t|}|D]E} t | || ||||| \} }| | | ||t ||}n[#t$rNtj }| tj | tj YnwxYw|6tj |s tj }|||dkr!tj d|dkr*|td||fz3td|zG|| | |fS| | fS)al1-penalized covariance estimator along a path of decreasing alphas Read more in the :ref:`User Guide `. Parameters ---------- X : ndarray of shape (n_samples, n_features) Data from which to compute the covariance estimate. alphas : array-like of shape (n_alphas,) The list of regularization parameters, decreasing order. cov_init : array of shape (n_features, n_features), default=None The initial guess for the covariance. X_test : array of shape (n_test_samples, n_features), default=None Optional test matrix to measure generalisation error. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where p > n. Elsewhere prefer cd which is more numerically stable. tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. The tolerance must be a positive number. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. The tolerance must be a positive number. max_iter : int, default=100 The maximum number of iterations. This parameter should be a strictly positive integer. verbose : int or bool, default=False The higher the verbosity flag, the more information is printed during the fitting. Returns ------- covariances_ : list of shape (n_alphas,) of ndarray of shape (n_features, n_features) The estimated covariance matrices. precisions_ : list of shape (n_alphas,) of ndarray of shape (n_features, n_features) The estimated (sparse) precision matrices. scores_ : list of shape (n_alphas,), dtype=float The generalisation error (log-likelihood) on the test data. Returned only if test data is passed. rrN)r"r5r6r7r8r9r:.z/[graphical_lasso_path] alpha: %.2e, score: %.2ez"[graphical_lasso_path] alpha: %.2e)r/rr-rQrpr]r r[rrSnanrZsysstderrwriter\)ralphasr5X_testr6r7r8r9r: inner_verboser)rd covariances_ precisions_scores_ test_emp_covr"r! this_scores r%graphical_lasso_pathrs F7Q;''M"1%%Gllnn  66L&&KffG +F33 "D"D '&5$!!% ' ' ' #K    , , ,   z * * *!+L*EE ! ' ' '&J    ' ' '   rv & & & & & '  ;z** % fW NN: & & & a<< J  S ! ! ! ! q[[!Ej)* :UBCCC ['11  $$s=ACAD,+D,c eZdZUdZiejeeddddgeedddgdgedgdZee d <d d dd d d d dddd fd Z ddZ xZ S)GraphicalLassoCVaGSparse inverse covariance w/ cross-validated choice of the l1 penalty. See glossary entry for :term:`cross-validation estimator`. Read more in the :ref:`User Guide `. .. versionchanged:: v0.20 GraphLassoCV has been renamed to GraphicalLassoCV Parameters ---------- alphas : int or array-like of shape (n_alphas,), dtype=float, default=4 If an integer is given, it fixes the number of points on the grids of alpha to be used. If a list is given, it gives the grid to be used. See the notes in the class docstring for more details. Range is [1, inf) for an integer. Range is (0, inf] for an array-like of floats. n_refinements : int, default=4 The number of times the grid is refined. Not used if explicit values of alphas are passed. Range is [1, inf). cv : int, cross-validation generator or iterable, default=None Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 5-fold cross-validation, - integer, to specify the number of folds. - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs :class:`KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None changed from 3-fold to 5-fold. tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. Range is (0, inf]. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. Range is (0, inf]. max_iter : int, default=100 Maximum number of iterations. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where number of features is greater than number of samples. Elsewhere prefer cd which is more numerically stable. n_jobs : int, default=None Number of jobs to run in parallel. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. .. versionchanged:: v0.20 `n_jobs` default changed from 1 to None verbose : bool, default=False If verbose is True, the objective function and duality gap are printed at each iteration. assume_centered : bool, default=False If True, data are not centered before computation. Useful when working with data whose mean is almost, but not exactly zero. If False, data are centered before computation. Attributes ---------- location_ : ndarray of shape (n_features,) Estimated location, i.e. the estimated mean. covariance_ : ndarray of shape (n_features, n_features) Estimated covariance matrix. precision_ : ndarray of shape (n_features, n_features) Estimated precision matrix (inverse covariance). alpha_ : float Penalization parameter selected. cv_results_ : dict of ndarrays A dict with keys: alphas : ndarray of shape (n_alphas,) All penalization parameters explored. split(k)_test_score : ndarray of shape (n_alphas,) Log-likelihood score on left-out data across (k)th fold. .. versionadded:: 1.0 mean_test_score : ndarray of shape (n_alphas,) Mean of scores over the folds. .. versionadded:: 1.0 std_test_score : ndarray of shape (n_alphas,) Standard deviation of scores over the folds. .. versionadded:: 1.0 n_iter_ : int Number of iterations run for the optimal alpha. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- graphical_lasso : L1-penalized covariance estimator. GraphicalLasso : Sparse inverse covariance estimation with an l1-penalized estimator. Notes ----- The search for the optimal penalization parameter (`alpha`) is done on an iteratively refined grid: first the cross-validated scores on a grid are computed, then a new refined grid is centered around the maximum, and so on. One of the challenges which is faced here is that the solvers can fail to converge to a well-conditioned estimate. The corresponding values of `alpha` then come out as missing values, but the optimum may be close to these missing values. In `fit`, once the best parameter `alpha` is found through cross-validation, the model is fit again using the entire training set. Examples -------- >>> import numpy as np >>> from sklearn.covariance import GraphicalLassoCV >>> true_cov = np.array([[0.8, 0.0, 0.2, 0.0], ... [0.0, 0.4, 0.0, 0.0], ... [0.2, 0.0, 0.3, 0.1], ... [0.0, 0.0, 0.1, 0.7]]) >>> np.random.seed(0) >>> X = np.random.multivariate_normal(mean=[0, 0, 0, 0], ... cov=true_cov, ... size=200) >>> cov = GraphicalLassoCV().fit(X) >>> np.around(cov.covariance_, decimals=3) array([[0.816, 0.051, 0.22 , 0.017], [0.051, 0.364, 0.018, 0.036], [0.22 , 0.018, 0.322, 0.094], [0.017, 0.036, 0.094, 0.69 ]]) >>> np.around(cov.location_, decimals=3) array([0.073, 0.04 , 0.038, 0.143]) rNrvrtz array-like cv_object)r n_refinementscvn_jobsrwr3r4r2F) rrrr7r8r9r6rr:r{c t||||| | ||_||_||_||_dSr)r|r}rrrr) r~rrrr7r8r9r6rr:r{rs r%r}zGraphicalLassoCV.__init__$s[ +     * r'c  djr%tjjd_nd_tj}tj |d}t}j }tdjdz t|r8j D]&}t!|dt"dtjd 'j d}n^j}t)|} d | z} tjtj| tj| |d d d t/j} t1|D]} t3j5t3jdt8t;jjfd||D} d d d n #1swxYwYtA| \}}}tA|}tA|}|!tA||tE|tGj$dd}tj }d}tK|D]s\}\}}}tj|}|dtj&tj'j(z kr tj)}tj*|r|}||kr|}|}t|dkr|dd} |dd} n||kr6|tW|dz ks ||d} ||dzd} nX|tW|dz kr ||d} d ||dz} n"||dz d} ||dzd} t|sGtjtj| tj| |dzdd jr2|dkr,tYd| dz|t/j| z fzttA|}t|d}t|d-d|-t]t_|jtj0|}dtj0i_1t1|jdD]} |d d | fj1d| d<tj|dj1d<tj2|dj1d<|}|_3ti||j5j6j7j8d\_9_:_;S)aFit the GraphicalLasso covariance model to X. Parameters ---------- X : array-like of shape (n_samples, n_features) Data from which to compute the covariance estimate. y : Ignored Not used, present for API consistency by convention. Returns ------- self : object Returns the instance itself. r )rrrrzF) classifierr"rs)min_valmax_valinclude_boundariesrNr@)rr:c 3K|]\\}}tt||jjjt djzV]dS)皙?)rrr6r7r8r9r:N)rrr6r7r8intr9).0traintestrrrr~s r% z'GraphicalLassoCV.fit..}s O O$t2G011%% w!Y H!%!$S4=%8!9!9 -    O O O O O Or'T)keyreverserz8[GraphicalLassoCV] Done refinement % 2i out of %i: % 3is)rrr:rsplit _test_score)axismean_test_scorestd_test_scorer)? CJ47mm d1g a #%%{%       h{++ $bhv&6&67{(+,, I IA7B111a47HD 3Q333 4 4.0gk.J.J.J*+-/VKa-H-H-H)*J'   ;J ]]! ; ; ; 7$/4< s-A"HH "H r) rrrrrrrwrrrRrr}rrrs@r%rrrsggR$  3$8Haf===|L"(8QVDDDEmT" $$$D   6eeeeeeeer'r)NNr2r3r3r4F).rr^rrrnumbersrrnumpyrscipyrrrr exceptionsr utils.validationr r rutils.parallelrrutils._param_validationrr linear_modelrrVrmodel_selectionrrr&r+r1rrr<rprrrrrrMr'r%rs """"""""GGGGGGGGGG++++++ /.......::::::::/.....))))))77777777      0     j+j+j+j+j+Z,8]]]]]']]]H     t%t%t%t%nrrrrr)rrrrrr'