o ڭj0@s dZddlmZmZddlmZmZmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZddlmZdd lmZmZmZdd lmZd d lmZd ej dej!fddZ"eddZ#e#ddddddZ$de e ej!de ej!fddZ%deej&deej&e'e(gdfddfddZ)Gd d!d!eZ*d"ej&defd#d$Z+d%ee'e ej!fd&e e,d'ee'efd(e ed)ee'efdef d*d+Z-deej&d,e e e'd&e e,d-e(d.ee'efd/e(d0e(de ee effd1d2Z.  3d9d4ed%ed5e ed6e(dej!f d7d8Z/dS):z*Utilities for processing spark partitions.) defaultdict namedtuple) AnyCallableDictIteratorListOptionalSequenceTupleUnionN) csr_matrix) ArrayLikeconcat)DataIterDMatrixQuantileDMatrix)XGBModel) get_loggerseriesreturncCs|jdd}t|}|S)zStack a series of arrays.F)copy)to_numpynpstack)rarrayrO/var/www/html/CropPilot/venv/lib/python3.10/site-packages/xgboost/spark/data.py stack_seriess  r!Alias)datalabelweightmarginvalidqidvaluesr$r% baseMarginvalidationIndicatorr(seqcCs|rt|SdS)z&Concatenate the data if it's not None.Nr)r,rrr concat_or_nonesr-iteratorappendcsdtjdtddffdd }d}|D]G}|durtj|jv}|dur*tj|jvs*J|rF|j|tjddf}|j|tjddf}n|d}}||d|durY||dqdS) znExtract partitions from pyspark iterator. `append` is a user defined function for accepting new partition.partis_validrNcsJ|tj||tj||tj||tj||tj|dS)N)aliasr#r$r%r&r()r0r1r/rr make_blob,s z#cache_partitions..make_blobTF)pd DataFrameboolr2r'columnsloc)r.r/r4has_validationr0trainr'rr3r cache_partitions&s     r<cs|eZdZdZdeeefdeede ddffdd Z dee e j dee j fd d Zd edefd d ZdddZZS)PartIterz7Iterator for creating Quantile DMatrix from partitions.r# device_idkwargsrNc s*d|_||_||_||_tjdddS)NrT) release_data)_iter _device_id_data_kwargssuper__init__)selfr#r>r? __class__rr rFIs zPartIter.__init__cCsL|sdS|jdur!ddl}ddl}|jj|j|||jS||jSNr)rBcudfcupycudaruntime setDevicer6rA)rGr#rKcprrr _fetchSs  zPartIter._fetch input_datac Cs|jt|jtjkr dS|d||jtj||jtjd||jtjd||jtj d||jtj dd|j |jd7_dS)NFr#r$r% base_marginr(rTr) rAlenrCr2r#rQgetr$r%r&r(rD)rGrRrrr nextbsz PartIter.nextcCs d|_dSrJ)rA)rGrrr resetps zPartIter.reset)rN)__name__ __module__ __qualname____doc__rstrrr intrrFr r5r6rQrr7rWrX __classcell__rrrHr r=Fs " r=r0cCsgdgg}}}d}t|j|j|j|jD]B\}}}}|dkr)t|} |} |} nt|} tj| tj d} |} |dkr=| }|| ksCJ| | | |dt| | | qt |} t |} t |}t || | ft||fdS)Nr)dtype)shape)zipfeatureVectorTypefeatureVectorSizefeatureVectorIndicesfeatureVectorValuesr^rUrarangeint32r/r concatenater )r0csr_indices_listcsr_indptr_listcsr_values_list n_featuresvec_type vec_size_ vec_indices vec_valuesvec_size csr_indices csr_valuescsr_indptr_arrcsr_indices_arrcsr_values_arrrrr )_read_csr_matrix_from_unwrapped_spark_vects6      ryr# dev_ordinalmetarefparamscCsD|s ttd|dSt||fi|}t|fi|d|i}|S)z+Handle empty partition for QuantileDMatrix.rr)r|r|)rremptyr=)r#rzr{r|r}itmrrr make_qdms r feature_colsuse_qdmr?enable_sparse_data_optimhas_validation_colc sttttddtjdtdtddffdd }dtjdtdtddffd d }d ttttj fd ttt fdt fd d} |rV|} dvrSddksUJn|} dt ttt fttt tttfffffdd } | \} } dur|rt|| t|| d| }n/dur|st|| | }ndur|rt|| t|| d| }n t|| | }|r|rt|| || }n |r| nd}nd}|dur||ksJ||fS)a~Create DMatrix from spark data partitions. Parameters ---------- iterator : Pyspark partition iterator. feature_cols: A sequence of feature names, used only when rapids plugin is enabled. dev_ordinal: Device ordinal, used when GPU is enabled. use_qdm : Whether QuantileDMatrix should be used instead of DMatrix. kwargs : Metainfo for DMatrix. enable_sparse_data_optim : Whether sparse data should be unwrapped has_validation: Whether there's validation data. Returns ------- Training DMatrix and an optional validation DMatrix. rr0namer1rNcs|tjks ||jvro|tjkr!dur!|jddkr!|}n||jddkr8||}|tjkr7t|}nd}|tjkrU|durUdkrL|jd|jdksUJ|dur[dS|rf||dS||dSdSNrr)r2r#r8rbr!r/r0rr1r)rrn train_data valid_datarr append_ms*    z0create_dmatrix_from_partitions..append_mcs|tjks ||jvr>|tjkr&t|}dkr|jd|jdks%Jn||}|r5||dS||dSdSr)r2r#r8ryrbr/r)rnrrrr append_m_sparses  z7create_dmatrix_from_partitions..append_m_sparser)r?cSst|dkrtddtddtdi|St|tj}t| tj d}t| tj d}t| tj d}t| tj d}td|||||d|S)NrXGBoostPySparkz_Detected an empty partition in the training data. Consider to enable repartition_random_shuffler#r~rSr)rUrwarningrrrr-r2r#rVr$r%r&r()r)r?r#r$r%r&r(rrr makes  z,create_dmatrix_from_partitions..makemissinggcs@d}i}i}D]\}}||vr|||<q |||<q ||fS)N)max_binrsilentnthreadenable_categorical)items) non_data_keysnon_data_paramsr{kv)r?rr split_paramss  z4create_dmatrix_from_partitions..split_params)rlistr5r6r]r7rrrndarrayrrr r r^floatr<rnum_col)r.rrzrr?rrrrr append_fnrr{r}dtraindvalidr)rr?rnrrr create_dmatrix_from_partitionssB"&$,4           rFmodelrT strict_shapec CsB|d}t|||j|j|j|j|jd}|j|dd||dS)z4Predict contributions with data with the full model.N)rTrr feature_typesfeature_weightsrTF) pred_contribsvalidate_featuresiteration_ranger) _get_iteration_rangerrn_jobsrrr get_boosterpredict)rr#rTrr data_dmatrixrrr rVs"  r)NF)0r\ collectionsrrtypingrrrrrr r r r numpyrpandasr5 scipy.sparser _typingrcompatrcorerrrsklearnrutilsrSeriesrr!r"r2r-r6r]r7r<r=ryr^rrrrrrr s,      "  .+      ,