o tBht@sddlZddlZddlZddlZddlZddlmZddlZddlm Z ddl m Z ddl m Z ddlmZmZmZmZmZmZmZddlmZddlmZmZdd lmZmZddlZd d l m!Z!d d l"m#Z#d dl$m%Z%d dl&m'Z'dgZ(dZ)dZ*dZ+dZ,dZ-dZ.eee/e/fZ0eee/e/fZ1de/de/de/fddZ2de/dee/defddZ3 dYd!e4d"e5d#e/defd$d%Z6 dZde/dee/d!e4d"e5fd&d'Z7Gd(d)d)e8Z9  dZd#e/d*ee/dee/d!e4d"e5def d+d,Z:  dZd-e/d.ee4e/fdee/d!e4d"e5f d/d0Z;  dZd1e4dee/d!e4d"e5dee/eff d2d3Z<  dZd1e4dee/d!e4d"e5de1f d4d5Z=  dZd1e4dee/d!e4d"e5de0f d6d7Z>d8e0de4fd9d:Z?  dZd#e/dee/d;e/ded?ed@edAedBe/d!e4d"e5defdCdDZ@dddEd#e/dFeAdee/dGeAdHed=ee4d>edAeee4e4fdBe/d!e4d"e5fdIdJZBdKdLZCdMdNZD d[dOdddPdQdRdSdddT d-ee/d.ee/e4fd1ee4dee/dUeee/efdVeAdWeAdGee/eAfd!e4d"e5fdXdZEdS)\N)join)warn)closingwraps)CallableOptionalDictTupleListAnyUnion)TemporaryDirectory)urlopenRequest) HTTPErrorURLError)_arff) get_data_home)_liac_arff_parser)Bunch fetch_openmlzhttps://openml.org/z*api/v1/json/data/list/data_name/{}/limit/2zapi/v1/json/data/{}zapi/v1/json/data/features/{}zapi/v1/json/data/qualities/{}zdata/v1/download/{} openml_path data_homereturncCstj|d|dS)Nz openml.orgz.gz)ospathr)rrro/var/www/html/riverr-enterprise-integrations-main/venv/lib/python3.10/site-packages/sklearn/datasets/_openml.py_get_local_path$r!csfdd}|S)zIf the first call to the decorated function fails, the local cached file is removed, and the function is called again. If ``data_home`` is ``None``, then the function is called once. cstfdd}|S)Ncsdur |i|Sz|i|WStyty?tdtt}tj|r6t||i|YSw)Nz!Invalid cache, redownloading file) r ExceptionrRuntimeWarningr!rrexistsunlink)argskw local_path)rfrrr wrapper/s      z;_retry_with_clean_cache..decorator..wrapperrr*r+rrr*r decorator.sz*_retry_with_clean_cache..decoratorr)rrr/rr-r _retry_with_clean_cache(sr0? n_retriesdelayurlcsfdd}|S)aUIf the function call results in a network error, call the function again up to ``n_retries`` times with a ``delay`` between each call. If the error has a 412 status code, don't call the function again as this is a specific OpenML error. The url parameter is used to give more information to the user about the error. cstfdd}|S)Nc s} z|i|WSttfy@}z't|tr |jdkr |dkr%tdd|d8}tWYd}~nd}~wwq)NTrz+A network error occurred while downloading z . Retrying...r)r TimeoutError isinstancercodertimesleep)r'kwargs retry_countere)r5r*r4r6rr r+Os" z;_retry_on_network_error..decorator..wrapperrr,r5r4r6r.r r/Nsz*_retry_on_network_error..decoratorr)r4r5r6r/rr@r _retry_on_network_errorCs rAc Csdd}tt|}|dd|dur,t|||jt|}||r*tj|ddS|St||}t j |\}} t j |st j |dd zet|d V} tt|||jt|2}||rbt} ntj} | t j | | d } t|| Wdn1swYWdn1swYt| j|Wdn1swYWntyt j |rt |wt|dS) a Returns a resource from OpenML.org. Caches it to data_home if required. Parameters ---------- openml_path : str OpenML URL that will be accessed. This will be prefixes with _OPENML_PREFIX. data_home : str Directory to which the files will be cached. If None, no caching will be applied. n_retries : int, default=3 Number of retries when HTTP errors are encountered. Error with status code 412 won't be retried as they represent OpenML generic errors. delay : float, default=1.0 Number of seconds between retries. Returns ------- result : stream A stream to the OpenML resource. cSs|dddkS)NzContent-Encodingr3gzip)infoget)_fsrcrrr is_gzip_encodedr"z)_open_openml_url..is_gzip_encodedzAccept-encodingrBNrb)fileobjmodeT)exist_ok)dirwb)r_OPENML_PREFIX add_headerrAfull_urlrrBGzipFiler!rrsplitr%makedirsrropenrshutil copyfileobjmovenamer#r&) rrr4r5rFreqfsrcr)dir_name file_nametmpdiropenerfdstrrr _open_openml_urlfsJ          r_c@seZdZdZdS) OpenMLErrorzDHTTP 412 is a specific OpenML error code, indicating a generic errorN)__name__ __module__ __qualname____doc__rrrr r`sr` error_messagec s^tfdd}z|WSty.}z|jdkr!|WYd}~t|d}~ww)a Loads json data from the openml api Parameters ---------- url : str The URL to load from. Should be an official OpenML endpoint. error_message : str or None The error message to raise if an acceptable OpenML error is thrown (acceptable error is, e.g., data id not found. Other errors, like 404's will throw the native error message). data_home : str or None Location to cache the response. None if no cache is required. n_retries : int, default=3 Number of retries when HTTP errors are encountered. Error with status code 412 won't be retried as they represent OpenML generic errors. delay : float, default=1.0 Number of seconds between retries. Returns ------- json_data : json the json result from the OpenML server if the call was successful. An exception otherwise. csLttd}t|dWdS1swYdS)Nr4r5utf-8)rr_jsonloadsreaddecode)responserr5r4r6rr _load_jsons $z5_get_json_content_from_openml_api.._load_jsonr7N)r0rr:r`)r6rerr4r5rnerrorrrmr !_get_json_content_from_openml_apis%  rprWversionc Cs|dkr6t|d}d|}t|||||d}|dd}t|dkr2tdj||d d d |d Std ||}z t|d |||d}Wntye|d7}d||}t|||||d}Ynw|ddd S)a? Utilizes the openml dataset listing api to find a dataset by name/version OpenML api function: https://www.openml.org/api_docs#!/data/get_data_list_data_name_data_name Parameters ---------- name : str name of the dataset version : int or str If version is an integer, the exact name/version will be obtained from OpenML. If version is a string (value: "active") it will take the first version from OpenML that is annotated as active. Any other string values except "active" are treated as integer. data_home : str or None Location to cache the response. None if no cache is required. n_retries : int, default=3 Number of retries when HTTP errors are encountered. Error with status code 412 won't be retried as they represent OpenML generic errors. delay : float, default=1.0 Number of seconds between retries. Returns ------- first_dataset : json json representation of the first dataset object that adhired to the search criteria activez/status/active/zNo active dataset {} found.rr4r5datadatasetrzMultiple active versions of the dataset matching the name {name} exist. Versions may be fundamentally different, returning version {version}.rrq)rWrqz/data_version/{}N)rerr4r5z/status/deactivatedz%Dataset {} with version {} not found.) _SEARCH_NAMEformatrplenrr`) rWrqrr4r5r6 error_msg json_dataresrrr _get_data_info_by_namesN)      r|data_idcCs.t|}d|}t|||||d}|dS)N"Dataset with data_id {} not found.rsdata_set_description) _DATA_INFOrwrpr}rr4r5r6rerzrrr _get_data_description_by_idHs  rcCs2t|}d|}t|||||d}|ddS)Nr~rs data_featuresfeature)_DATA_FEATURESrwrprrrr _get_data_features[s   rcCs:t|}d|}t|||||d}|didgS)Nr~rsdata_qualitiesquality)_DATA_QUALITIESrwrprDrrrr _get_data_qualitiesos   rrcCs&d}dd|D}tt|d|S)aJGet the number of samples from data qualities. Parameters ---------- data_qualities : list of dict Used to retrieve the number of instances (samples) in the dataset. Returns ------- n_samples : int The number of samples in the dataset or -1 if data qualities are unavailable. cSsi|] }|d|dqS)rWvaluer).0drrr sz$_get_num_samples..NumberOfInstances)intfloatrD)rdefault_n_samples qualitiesrrr _get_num_samplessroutput_arrays_type features_dict data_columnstarget_columns col_slice_x col_slice_yshape md5_checksumc  st||| | d} t| Utfdd} | | }|dk }|dkr'tjntj}tj|||d}t||||||||\}}}}|D]}qC | krTt d|d||||fWd S1sdwYd S) z@Load arff data with url and parses arff response with parse_arffrfc3s&|D] }||dVqdS)Nrg)updaterk)rllineactual_md5_checksumrr _stream_checksum_generators  z7_load_arff_response.._stream_checksum_generatorpandassparse) return_typeencode_nominalzmd5 checksum of local file for zk does not match description. Downloaded file could have been modified / corrupted, clean cache and retry...N) r_rhashlibmd5rCOO DENSE_GENloadr hexdigest ValueError)r6rrrrrrrrrr4r5rlrstreamrrarffXyframenominal_attributes_rrr _load_arff_responses@       $rrfras_frame features_listcsdd|Dt|fdd|D} fdd|D} | D]} || }t|d}|dkr:td|d |q |r@d }n|rEd }nd }t||t|||||| | ||| | d \}}}}t||||||dS)z1Download OpenML ARFF and convert to Bunch of datacSsi|]}|d|qS)rWrrrrrr rsz+_download_data_to_bunch..cg|] }t|dqSindexrrcol_namerrr z+_download_data_to_bunch..crrrrrrr rrnumber_of_missing_valuesrz]Target column {} has {} missing values. Missing values are not supported for target columns. rWrrnumpy)rr4r5)rttargetr categories feature_names target_names)_verify_target_data_typerrrwr0rr)r6rrrrrrrrr4r5rrcol_idxfeat nr_missingrrrrrrrr _download_data_to_bunchsT    rcCst|ts tdt|t}|D]<}||vrtd||ddkr+|tjn|t ||ddkr?t d |||ddkrNt d |qt |d krYtd dS) Nz%target_column should be list, got: %szCould not find target_column={} data_typenumeric is_ignoretruez$target_column={} has flag is_ignore.is_row_identifierz,target_column={} has flag is_row_identifier.rzgCan only handle homogeneous multi-target datasets, i.e., all targets are either numeric or categorical.) r9listrtypesetKeyErroraddnpfloat64objectrrwrx)rr found_types target_columnrrr r#s&   rcCsDg}|D]}|d|vr|ddkr|ddkr||dq|S)NrWrrr)append)rrvalid_data_column_namesrrrr _valid_data_column_names>s   rrrdefault-targetTFauto) rqr}rrcache return_X_yrr4r5rrrc  Csh|durd}n t|d}t|d}|dur3|}|dur%td||t||||| d} | d}n|durD|dkrCtd ||ntd t||} | d dkrbtd | d | d| dd| vrotd| dd| vr|td| dd} | ddkrd} |dkr| }|r| rtdt||} |s| D]}d|d|dfvrq|ddkrtdq|dkrd d!| D}n!t |t r|g}n|durg}nt |t r|}n t d"t |t| |}| st||}t|t| f}nd}t| d#}t|| |t|| |||| d$|| d% }|r|j|jfSd&| d'}|j|| d(|d)|S)*a)Fetch dataset from openml by name or dataset id. Datasets are uniquely identified by either an integer ID or by a combination of name and version (i.e. there might be multiple versions of the 'iris' dataset). Please give either name or data_id (not both). In case a name is given, a version can also be provided. Read more in the :ref:`User Guide `. .. versionadded:: 0.20 .. note:: EXPERIMENTAL The API is experimental (particularly the return value structure), and might have small backward-incompatible changes without notice or warning in future releases. Parameters ---------- name : str, default=None String identifier of the dataset. Note that OpenML can have multiple datasets with the same name. version : int or 'active', default='active' Version of the dataset. Can only be provided if also ``name`` is given. If 'active' the oldest version that's still active is used. Since there may be more than one active version of a dataset, and those versions may fundamentally be different from one another, setting an exact version is highly recommended. data_id : int, default=None OpenML ID of the dataset. The most specific way of retrieving a dataset. If data_id is not given, name (and potential version) are used to obtain a dataset. data_home : str, default=None Specify another download and cache folder for the data sets. By default all scikit-learn data is stored in '~/scikit_learn_data' subfolders. target_column : str, list or None, default='default-target' Specify the column name in the data to use as target. If 'default-target', the standard target column a stored on the server is used. If ``None``, all columns are returned as data and the target is ``None``. If list (of strings), all columns with these names are returned as multi-target (Note: not all scikit-learn classifiers can handle all types of multi-output combinations). cache : bool, default=True Whether to cache the downloaded datasets into `data_home`. return_X_y : bool, default=False If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` objects. as_frame : bool or 'auto', default='auto' If True, the data is a pandas DataFrame including columns with appropriate dtypes (numeric, string or categorical). The target is a pandas DataFrame or Series depending on the number of target_columns. The Bunch will contain a ``frame`` attribute with the target and the data. If ``return_X_y`` is True, then ``(data, target)`` will be pandas DataFrames or Series as describe above. If as_frame is 'auto', the data and target will be converted to DataFrame or Series as if as_frame is set to True, unless the dataset is stored in sparse format. .. versionchanged:: 0.24 The default value of `as_frame` changed from `False` to `'auto'` in 0.24. n_retries : int, default=3 Number of retries when HTTP errors or network timeouts are encountered. Error with status code 412 won't be retried as they represent OpenML generic errors. delay : float, default=1.0 Number of seconds between retries. Returns ------- data : :class:`~sklearn.utils.Bunch` Dictionary-like object, with the following attributes. data : np.array, scipy.sparse.csr_matrix of floats, or pandas DataFrame The feature matrix. Categorical features are encoded as ordinals. target : np.array, pandas Series or DataFrame The regression target or classification labels, if applicable. Dtype is float if numeric, and object if categorical. If ``as_frame`` is True, ``target`` is a pandas object. DESCR : str The full description of the dataset. feature_names : list The names of the dataset columns. target_names: list The names of the target columns. .. versionadded:: 0.22 categories : dict or None Maps each categorical feature name to a list of values, such that the value encoded as i is ith in the list. If ``as_frame`` is True, this is None. details : dict More metadata from OpenML. frame : pandas DataFrame Only present when `as_frame=True`. DataFrame with ``data`` and ``target``. (data, target) : tuple if ``return_X_y`` is True .. note:: EXPERIMENTAL This interface is **experimental** and subsequent releases may change attributes without notice (although there should only be minor changes to ``data`` and ``target``). Missing values in the 'data' are represented as NaN's. Missing values in 'target' are represented as NaN's (numerical target) or None (categorical target). FN)ropenmlzfDataset data_id={} and name={} passed, but you can only specify a numeric data_id or a name, not both.rfdidrrzlDataset data_id={} and version={} passed, but you can only specify a numeric data_id or a version, not both.zFNeither name nor data_id are provided. Please provide name or data_id.statuszVersion {} of dataset {} is inactive, meaning that issues have been found in the dataset. Try using a newer version from this URL: {}rqrWr6rozMOpenML registered a problem with the dataset. It might be unusable. Error: {}warningzIOpenML raised a warning on the dataset. It might be unusable. Warning: {}rw sparse_arffTrz(Cannot return dataframe with sparse datarrrrstringzOSTRING attributes are not supported for array representation. Try as_frame=TruercSs g|] }|ddkr|dqS) is_targetrrWrrrrr r*s  z fetch_openml..zKDid not recognize type of target_columnShould be str, list or None. Got: {}file_idr)rrrrrrr4r5z{} Downloaded from openml.org. descriptionzhttps://www.openml.org/d/{})DESCRdetailsr6)rrlowerrrwr|rrrr9strr TypeErrorrrrrrx _DATA_FILErboolrtrpopr)rWrqr}rrrrrr4r5 data_infodata_description return_sparserrrrrrr6bunchrrrr rNs                 )r1r2r3)r1r2)N)FrBrhrrTros.pathrr;warningsr contextlibr functoolsrtypingrrr r r r r tempfilerurllib.requestrr urllib.errorrrrr externalsrr3r _arff_parserrutilsr__all__rMrvrrrrrOpenmlQualitiesTypeOpenmlFeaturesTyper!r0rrrAr_rr`rpr|rrrrrrrrrrrrrr s    $      $ J  <  ]          J    J