o ۭjH@sjddlmZddlmZmZddlmZmZmZm Z ddl Z ddl m Z ddl Z ddlmZmZmZmZmZddlZddlZddlmZddlmZmZmZdd lmZdd lm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&dd l'm(Z(m)Z)ddl*m+Z+m,Z,erddl-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3dZ4Gddde+Z5GdddejZ6Gddde5Z7dddZ8d ddZ9dS)!) annotations)abc defaultdict)HashableIteratorMappingSequenceN)StringIO)IO TYPE_CHECKING DefaultDictLiteralcast)lib)EmptyDataError ParserError ParserWarning)cache_readonly)find_stack_level) is_bool_dtype is_integeris_numeric_dtype) is_dict_like) dedup_namesis_potential_multi_index) ParserBaseparser_defaults) ArrayLike ReadCsvBufferScalar)Index MultiIndexucsJeZdZUded<dbfdd Zedcd d Zddd dZ dedfddZdgddZ dedhddZ did d!Z edjd#d$Z dkd&d'Z ed(d)Zdld0d1Zdmd3d4Zdnd6d7Zdod9d:Zdmd;d<Zdpd@dAZdqdCdDZdrdGdHZdrdIdJZdrdKdLZdsdOdPZdrdQdRZdtdSdTZdudVdWZdvdYdZZdedwd[d\Zdxd^d_Zdyd`daZ Z!S)z PythonParserset[int]_no_thousands_columnsfReadCsvBuffer[str] | listreturnNonec st|d_g_d_d_|d_tjr!j_nfdd_t |d_ |d_ |d_ t j trEtj _ |d _|d _|d _|d _|d _|d_d_d|vro|d_|d_|d_|d_|d_t |trttt|_n t|dsJ|_d_ \}_!_"#|j$\_%_$_&}tj%_'j(sՈ)\}_'_%d_*j$dur|_$jdurtt+t,j%_-j%_./_0t,jdkrt1ddS)zN Workhorse function for processing nested list into DataFrame Nrskiprowscs |jvSN)r))xself\/var/www/html/CropPilot/venv/lib/python3.10/site-packages/pandas/io/parsers/python_parser.pyZs z'PythonParser.__init__.. skipfooter delimiter quotechar escapechar doublequoteskipinitialspacelineterminatorquotingskip_blank_linesFhas_index_namesverbose thousandsdecimalcommentreadlineTz'Only length-1 decimal markers supported)2super__init__databufposline_posr)callableskipfunc_validate_skipfooter_argr1r2r3 isinstancestrr4r5r6r7r8r9r:r;r<r=r>listrrhasattr _make_reader _col_indices_infer_columnsnum_original_columns unnamed_cols_extract_multi_indexer_columns index_namescolumns col_names orig_names_has_complex_date_col_get_index_name_name_processedrangelen_validate_parse_dates_presence_parse_date_cols_set_no_thousand_columnsr$ ValueError)r-r%kwdsrU_rT __class__r,r/rBJsv                          zPythonParser.__init__ re.PatterncCsLt|j}|jdurd|d}nt|j}d|d|d}t|S)Nz^[\-\+]?[0-9]*(z [0-9]*)?([0-9]?(E|e)\-?[0-9]+)?$z^[\-\+]?([0-9]+z |[0-9])*()reescaper=r<compile)r-r=regexr<r.r.r/nums    zPythonParser.numIO[str] | ReadCsvBuffer[str]c sJjdus tdkrjrtdGfdddtj}|}dur)|_nf}|ggd}j s>|sZj d7_ }|ggd}j s>|r>t t t |}|d}j d7_ j d7_ t|}|j|_tjt||d}jt |tj|dd} | Sfd d } | } | S) Nr@z.MyDialect N) __name__ __module__ __qualname__r2r3r4r5r6r8r7r.r,r.r/ MyDialectsrpr)dialectT)rqstrictc3sB}t}||VD] }||VqdSr*)r?rfrhsplitstrip)linepat)r%sepr.r/_reads z(PythonParser._make_reader.._read)r2r\r7r`csvDialectr?_check_commentsrHrErrLrKrFSniffersniffreaderr rDextend) r-r%rpdiarulines lines_strsniffedline_rdrr~rxr.)r%r-rwr/rNs<  zPythonParser._make_readerNrows int | NoneRtuple[Index | None, Sequence[Hashable] | MultiIndex, Mapping[Hashable, ArrayLike]]cCsz||}Wnty|jrg}n|Ynwd|_t|j}t|sIt|jt|j|j }| ||j \}}}| ||j }|||fSt|d}d} |jrd|t|krd|d} |dd}||} || \} }|| } ||| \}} || | || \}} || | fS)NFrr@) _get_lines StopIteration _first_chunkcloserLrWr\rr index_col_get_empty_metadtype_maybe_make_multi_index_columnsrVcount_empty_valsr: _rows_to_cols_exclude_implicit_index _convert_data_do_date_conversions _make_index)r-rcontentrUnamesindexcol_dict conv_columnscount_empty_content_vals indexnamerowalldatarC conv_dataresult_columnsr.r.r/readsL         zPythonParser.readrlist[np.ndarray]8tuple[Mapping[Hashable, np.ndarray], Sequence[Hashable]]cs\t|jt|j|j}d|jrt|jt||fddt|D|fS)Nrcs&i|]\}}|kr||qSr.r.).0inamer len_alldataoffsetr.r/ As z8PythonParser._exclude_implicit_index..)rrWrr_implicit_indexr\_check_data_length enumerate)r-rrr.rr/r,s"  z$PythonParser._exclude_implicit_indexsizecCs|dur|j}|j|dS)N)r) chunksizer)r-rr.r.r/ get_chunkFs zPythonParser.get_chunkrCMapping[Hashable, np.ndarray]Mapping[Hashable, ArrayLike]c Cs||j}||j}i}i}t|jtr>|jD]#}|j|}|j|}t|tr4||jvr4|j|}|||<|||<qn|j}|j}| ||||j ||Sr*) _clean_mapping convertersrrJ na_valuesdict na_fvaluesintrW_convert_to_ndarraysr;) r-rC clean_conv clean_dtypesclean_na_valuesclean_na_fvaluescolna_value na_fvaluer.r.r/rPs.         zPythonParser._convert_databoolcCs6|jdurdS|j}t|tttjfrt|dkSdS)NFr@)headerrJrLtuplenpndarrayr\)r-rr.r.r/_have_mi_columnsss  zPythonParser._have_mi_columns9tuple[list[list[Scalar | None]], int, set[Scalar | None]]c s|jd}d}t}|jdurq|j}|j}t|tttjfr-|r,t||ddg}n|g}g}t |D]\}}z| } |j |krN| } |j |ksEWnt y} zyd|j krb|krnn2|rl||dkrttt|rw|ddn|} dd| dt| d} td | d |j d | |r|dkr|r||dgt|d|||fWYd} ~ S|jstd | |jdd} WYd} ~ nd} ~ wwggt | D])\} }|d kr|rd| d|}nd| }| |q݈|q|stt}fddttD}|D]k} | }|}||}|dkr|dkrU|d||<|d|}|vrL|d7}n||}|dks5|jdurt|jr|j|dur|j|dur|j||j|i|| <|d||<qn?|r||dkrt}|j}|durt|nd}t}||kr|||ks|dkrd}dg||jdg|_||fddDt|dkrt}q6|r|durgz| }Wn t yd}Ynw|durdnt|}tt|dkr(t|kr(tdt|dkr3td|j durA|!||nt}|j"durctt|j"krcfddt#|j"Dg}ncg}n_|!||d|}nUt|j$}|}stt|g}|!||d|}n:|j dust|kr|!g|}t}n t%|j stt|j krtdg}|!||d||||fS)NrTr@[,z ], len of z, zPassed header=z but only z lines in fileNo columns to parse from filez Unnamed: _level_csg|]}|vr|qSr.r.rr)this_unnamed_colsr.r/ s z/PythonParser._infer_columns...Fcsh|]}|qSr.r.r) this_columnsr.r/ z.PythonParser._infer_columns..zHNumber of passed names did not match number of header fields in the filez*Cannot pass names with multi-index columnscsg|]}|qSr.r.r)rr.r/rr)&rsetrrrJrLrrrr_buffered_linerF _next_linermaprKjoinr\r` _clear_bufferappendrrrr[rrgetupdaterrD TypeErrorusecols_handle_usecolsrOsorted _header_linerG)r-rQ clear_bufferrRrhave_mi_columnsrUlevelhrruerrjoimsgrccol_namecountscol_loop_orderrold_col cur_countlcsicic unnamed_count first_linelen_first_data_rowncolsr.)rrrr/rP~s                   "    $      zPythonParser._infer_columnsc Csb|jdurdSz|}W|Sty0}z|jstd||jdd}WYd}~|Sd}~ww)Nr)rrrrr)r-rurr.r.r/r7s   zPythonParser._header_linerUlist[list[Scalar | None]] usecols_keylist[Scalar | None]rQrc s|jduryt|jr||j|nYtdd|jDrTt|dkr&tdg|jD]'}t|trMz | |Wq+tyL| |j|Yq+w|q+nfdd|jD}|rht d|d |jfd d|D}t |_ |S) zb Sets self._col_indices usecols_key is used if there are string usecols. Ncss|]}t|tVqdSr*)rJrK)rur.r.r/ Usz/PythonParser._handle_usecols..r@z4If using multiple headers, usecols must be integers.csg|]}|kr|qSr.r.)rr)rQr.r/resz0PythonParser._handle_usecols..z..)r)rcolumnrr.r/ros)rrG_evaluate_usecolsanyr\r`rJrKrr_validate_usecols_namesrrrO)r-rUrrQrmissing_usecolsr.)rrQr/rFsB         zPythonParser._handle_usecols list[Scalar]cCs t|jdkr |jdS|S)zH Return a line from buffer, filling buffer if required. r)r\rDrr,r.r.r/rvs zPythonParser._buffered_line first_rowc Cs|s|St|dts |S|ds|S|dd}|tkr|S|d}t|dkrZ|d|jkrZd}|d}|dd|d}|||}t||dkrY|||dd7}n|dd}|g}||ddS)a- Checks whether the file begins with the BOM character. If it does, remove it. In addition, if there is quoting in the field subsequent to the BOM, remove it as well because it technically takes place at the beginning of the name, not the middle of it. rr@N)rJrK_BOMr\r3r) r-r first_elt first_row_bomstartquoteendnew_row new_row_listr.r.r/_check_for_boms*    zPythonParser._check_for_bomrucCs| p tdd|DS)z Check if a line is empty or not. Parameters ---------- line : str, array-like The line of data to check. Returns ------- boolean : Whether or not the line is empty. css|]}| VqdSr*r.rr+r.r.r/rsz.PythonParser._is_line_empty..)all)r-rur.r.r/_is_line_emptys zPythonParser._is_line_emptycCst|jtrg||jr"|jt|jkrn |jd7_||js z8||j|jgd}|jd7_|jsI||j|jdsG|rIWn|jrZ| |g}|rZ|d}Wn Wn t ydt wq#nV||jr|jd7_|jdus{Jt |j||jsm |j |jdd}|jd7_|dur||gd}|jr| |g}|r|d}n n||s|rnq|jdkr||}|jd7_|j||S)Nr@Trrow_num)rJrCrLrHrEr\r{r9r_remove_empty_lines IndexErrorrnext_next_iter_linerrFrDr)r-ruret orig_liner.r.r/rsf            zPythonParser._next_linerrKrcCsL|j|jjkr t||j|jjkr$tjd|d|dttddSdS)a  Alert a user about a malformed row, depending on value of `self.on_bad_lines` enum. If `self.on_bad_lines` is ERROR, the alert will be `ParserError`. If `self.on_bad_lines` is WARN, the alert will be printed out. Parameters ---------- msg: str The error message to display. row_num: int The row number where the parsing error occurred. Because this row number is displayed, we 1-index, even though we 0-index internally. zSkipping line z: rl) stacklevelN) on_bad_linesBadLineHandleMethodERRORrWARNwarningswarnrr)r-rrr.r.r/_alert_malformeds zPythonParser._alert_malformedlist[Scalar] | Nonec Csz|jdusJt|j}t|tsJ|WStjyV}z3|j|jj|jj fvrKt |}d|vs6d|vr8d}|j dkrEd}|d|7}| ||WYd}~dSd}~ww)a[ Wrapper around iterating through `self.data` (CSV source). When a CSV error is raised, we check for specific error messages that allow us to customize the error message displayed to the user. Parameters ---------- row_num: int The row number of the line being parsed. Nz NULL bytezline contains NULzNULL byte detected. This byte cannot be processed in Python's native csv library at the moment, so please pass in engine='c' insteadrzError could possibly be due to parsing errors in the skipped footer rows (the skipfooter keyword is only applied after Python's csv library has parsed all rows).. ) rCrrJrLryErrorrrrr rKr1r#)r-rruerreasonr.r.r/rs,     zPythonParser._next_iter_linerlist[list[Scalar]]cCs|jdur|Sg}|D]8}g}|D],}t|tr"|j|vs"||jvr(||q|d||j}t|dkr=||||q |S)Nr)r>rJrKrrfindr\)r-rrrurlr+r.r.r/r{Es"       zPythonParser._check_commentscCsdd|D}|S)a Iterate through the lines and remove any that are either empty or contain only one whitespace value Parameters ---------- lines : list of list of Scalars The array of lines that we are to filter. Returns ------- filtered_lines : list of list of Scalars The same array of lines with the "empty" ones removed. cSsBg|]}t|dkst|dkrt|dtr|dr|qS)r@r)r\rJrKrtrrur.r.r/rjs    z4PythonParser._remove_empty_lines..r.)r-rrr.r.r/rZs z PythonParser._remove_empty_linescCs |jdur|S|j||jddS)Nrrsearchreplace)r<_search_replace_num_columnsr-rr.r.r/_check_thousandsus zPythonParser._check_thousandsr.r/c Cs|g}|D]7}g}t|D])\}}t|tr&||vs&||jvs&|j|s,||q ||||q ||q|Sr*) rrJrKr$rjr.rtrr/) r-rr.r/rrur+rr+r.r.r/r0}s   z(PythonParser._search_replace_num_columnscCs$|jtdkr |S|j||jddS)Nr=rr-)r=rr0r1r.r.r/_check_decimals zPythonParser._check_decimalcCs g|_dSr*)rDr,r.r.r/rs zPythonParser._clear_buffer@tuple[Sequence[Hashable] | None, list[Hashable], list[Hashable]]c Csl|j}t|}t|}|jdur|j}nz|}Wn ty%d}Ynwz|}Wn ty7d}Ynwd}|dur|j}|durLt||j}|dur|jdur|durt|t||jkrtt t||_|j dd|_ t |D]}| d|qyt|}t||_|||fS|dkrd|_ |jdurtt ||_d}n |||j\}} |_|||fS)a Try several cases to get lines: 0) There are headers on row 0 and row 1 and their total summed lengths equals the length of the next line. Treat row 0 as columns and row 1 as indices 1) Look for implicit index: there are more columns on row 1 than row 0. If this is true, assume that row 1 lists index columns and row 0 lists normal columns. 2) Get index from the columns if it was listed. NrFr@T)rWrLrrrrr\rQrr[rDreversedinsertr_clean_index_names) r-rUrWru next_lineimplicit_first_colsrr index_namerbr.r.r/rYsP           zPythonParser._get_index_namercsj}jr |tj7}tdd|D}||krjdurjdurjr*jnd}g}t|}t|}g}|D]J\}} t| } | |krtj rZ | } | durY| | q:j j j j j fvr~j|||} | | | fj j j kr~nq:| | q:|D]1\} } d|d| dd| } jrtjdkrjtjkrd }| d |7} | | dqttj||d j}jrjdusJjjrfd d t|D}|Sfdd t|D}|S)Ncss|]}t|VqdSr*)r\rrowr.r.r/rsz-PythonParser._rows_to_cols..Frz Expected z fields in line r@z, saw zXError could possibly be due to quotes being ignored when a multi-char delimiter is used.r%) min_widthcs4g|]\}}|tjks|tjvr|qSr.)r\rrrarr-r.r/r1s z.PythonParser._rows_to_cols..crr.r.r>rr.r/r:s)rQrr\rmaxrr1rrGrrrrr rEr2r8ry QUOTE_NONEr#rLrto_object_arrayTrO)r-rcol_lenmax_lenfooters bad_lines iter_content content_lenr_content actual_lennew_lrrr(zipped_contentr.r@r/rsp            zPythonParser._rows_to_colsc CsJ|j}d}|dur(t|j|kr!|jd||j|d}|_n|t|j8}|durt|jtrl|jt|jkr.)r)r)r-rPr.r,r/rOs  z!PythonParser._remove_skipped_rowscCst}|jr|jr|jdusJ||j|j}|jr_|jr_|jdus%Jt|j|jD]2\}}t|jts@t |js@| |t|jtr^||jvr^t |j|rYt |j|r^| |q,|Sr*) rrU parse_datesrO_set_noconvert_dtype_columnsrziprJrraddr)r-no_thousands_columnsrrr.r.r/r_s.        z%PythonParser._set_no_thousand_columns)r%r&r'r()r're)r%rkr*)rrr'r)rrr'r)rrr'r)rCrr'r)r'r)r'r)rUrrrrQrr'r)r'r)rrr'r)rurr'r)rrKrrr'r()rrr'r$rr)r'r))rr)r.rKr/rKr'r))r'r()r'r4)rr)r'r)rrr'r))rPr)r'r))r'r#)"rmrnro__annotations__rBrrjrNrrrrrrPrrrrrrr#rr{rr2r0r3rrYrrrOr_ __classcell__r.r.rcr/r"GsH _  A 6 #  :   0 5  8  0       N V Qr"c@sDeZdZdZ  ddddZddddZ dd ddZd!ddZdS)"FixedWidthReaderz( A reader of fixed-width lines. Ndr%rkcolspecs(list[tuple[int, int]] | Literal['infer']r2 str | Noner>r)set[int] | None infer_nrowsrr'r(cCs||_d|_|r d|nd|_||_|dkr|j||d|_n||_t|jttfs4t dt |j |jD]-}t|ttfr`t |dkr`t|dt tjt dfr`t|dt tjt dfsdt d q7dS) Nz z infer)rdr)z;column specifications must be a list or tuple, input was a rrr@zEEach column specification must be 2 element tuple or list of integers)r%bufferr2r>detect_colspecsr`rJrrLrtypermr\rrinteger)r-r%r`r2r>r)rdcolspecr.r.r/rBs8     zFixedWidthReader.__init__ list[str]cCsd|durt}g}g}t|jD]\}}||vr||||t||kr*nqt||_|S)a Read rows from self.f, skipping as specified. We distinguish buffer_rows (the first <= infer_nrows lines) from the rows returned to detect_colspecs because it's simpler to leave the other locations with skiprows logic alone than to modify them to deal with the fact we skipped some rows here as well. Parameters ---------- infer_nrows : int Number of rows to read from self.f, not counting rows that are skipped. skiprows: set, optional Indices of rows to skip. Returns ------- detect_rows : list of str A list containing the rows to read. N)rrr%rr\iterrf)r-rdr) buffer_rows detect_rowsrr<r.r.r/get_rowss    zFixedWidthReader.get_rowslist[tuple[int, int]]c sdddjD}td|d}||}|s tdttt|}t j |dt d}j dur>fd d|D}|D]}| |D] } d|| | <qGq@t |d} d | d <t || Adkd } tt| ddd | ddd } | S) NrcSsg|]}d|qS)\r.rr.r.r/rsz4FixedWidthReader.detect_colspecs..z([^z]+)z(No rows from which to infer column widthr@)rcsg|] }|jdqS)r) partitionr>r;r,r.r/rrrr)rr2rfrhrorrArr\rzerosrr>finditerr r rollwhererLrX) r-rdr) delimiterspatternrrFmaskr<mshiftededges edge_pairsr.r,r/rgs$   "z FixedWidthReader.detect_colspecscs\jdurztjWntyd_tjYnwtjfddjDS)Ncs$g|]\}}||jqSr.)rtr2)rfrom_torur-r.r/r$s$z-FixedWidthReader.__next__..)rfrrr%r`r,r.rr/__next__s   zFixedWidthReader.__next__)Nr_)r%rkr`rar2rbr>rbr)rcrdrr'r(r*)rdrr)rcr'rk)r_N)rdrr)rcr'rp)r'rk)rmrnro__doc__rBrorgrr.r.r.r/r^s  &' r^c@s.eZdZdZdddZdd d ZdddZdS)FixedWidthFieldParserzl Specialization that Converts fixed-width fields into DataFrames. See PythonParser for details. r%ReadCsvBuffer[str]r'r(cKs0|d|_|d|_tj||fi|dS)Nr`rd)popr`rdr"rB)r-r%rar.r.r/rB-s  zFixedWidthFieldParser.__init__rkr^cCst||j|j|j|j|jSr*)r^r`r2r>r)rd)r-r%r.r.r/rN3sz"FixedWidthFieldParser._make_readerrr)cCsdd|DS)z Returns the list of lines without the empty ones. With fixed-width fields, empty lines become arrays of empty strings. See PythonParser._remove_empty_lines. cSs"g|] }tdd|Dr|qS)css$|] }t|t p |VqdSr*)rJrKrt)rr'r.r.r/rG"zGFixedWidthFieldParser._remove_empty_lines...)rr,r.r.r/rDs z=FixedWidthFieldParser._remove_empty_lines..r.r1r.r.r/r=sz)FixedWidthFieldParser._remove_empty_linesN)r%rr'r()r%rkr'r^r[)rmrnrorrBrNrr.r.r.r/r's    rr'rcCstdd|DS)Ncss$|] }|dks |durdVqdS)rNr@r.)rvr.r.r/rLrz#count_empty_vals..)sum)valsr.r.r/rKsrr1cCs$t|std|dkrtd|S)a Validate the 'skipfooter' parameter. Checks whether 'skipfooter' is a non-negative integer. Raises a ValueError if that is not the case. Parameters ---------- skipfooter : non-negative integer The number of rows to skip at the end of the file. Returns ------- validated_skipfooter : non-negative integer The original input if the validation succeeds. Raises ------ ValueError : 'skipfooter' was not a non-negative integer. zskipfooter must be an integerrzskipfooter cannot be negative)rr`)r1r.r.r/rIOs rI)r'r)r1rr'r): __future__r collectionsrrcollections.abcrrrrryior rftypingr r r r rr!numpyr pandas._libsr pandas.errorsrrrpandas.util._decoratorsrpandas.util._exceptionsrpandas.core.dtypes.commonrrrpandas.core.dtypes.inferencerpandas.io.commonrrpandas.io.parsers.base_parserrrpandas._typingrrrpandasr r!rr"r^rrrIr.r.r.r/sD       rv $