o tBhE=@sddlmZmZddlmZmZmZmZmZzddl m Z Wne y-ee dfZ Ynwddl mZmZmZddlmZddlmZmZddlmZdd lZdd lmZmZmZmZmZmZdd l m!Z!m"Z"m#Z#m$Z$e%d Z&e&'ej(e)Z*e*+e,d e&-e*     d&de.de/de/de0dee dee de1de1defddZ2     d&dede/de/de0dee dee de1de1defdd Z3     d&d!e de/de/de0dee dee de1de1defd"d#Z4d'd!e de/de/de0dee dee de1defd$d%Z5d S)()splitextbasename)ListBinaryIOOptionalSetUnion)PathLikezos.PathLike[str])TOO_SMALL_SEQUENCETOO_BIG_SEQUENCEIANA_SUPPORTED) mess_ratio)CharsetMatches CharsetMatch)warnN)any_specified_encodingis_multi_byte_encodingidentify_sig_or_bomshould_strip_sig_or_bom is_cp_similar iana_name)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratioscharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s皙?TF sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc/ Cs6|s ttjnttjt|}|dkr(tdtt|dddgdgS|dur=tdd |d d |D}ng}|durTtd d |d d |D}ng}|||krhtd|||d}|}|dkrx|||krxt ||}t|t k} t|t k} | rt d|g} |durt|nd} | dur| | td| t} g}g}d}d}d}d}d}t}t|\}}|dur| |tdt||| dd| vr| d| tD]`}|r||vrq|r||vrq|| vrq| |d}||k}|o t|}|dvr|durtd|qzt|}Wnttfy7td|Yqwz9| r]|dur]t|durO|dt dn |t|t d|dnt|dure|n|t|d|d}Wn=ty}ztd|t||||s|d7}WYd}~qd}~wty|||s|d7}Yqwd}|D] }t||rd}nq|rtd||qt|durdnt||t ||} |o|duot||k}!|!rtd|t t| d}"|"dkrd}"d}#g}$g}%| D]E}&||&|&|}'|r#|dur#||'}'|'j |dd }(|$|(|%t!|(||%d!|krB|#d7}#|#|"ksO|rQ|durQnq |%r_t"|%t|%})nd})|)|ksk|#|"kr|||sw|d7}td"||#t#|)d#d$d%|dd| fvrt|||dg|}*|| kr|*}n |dkr|*}n|*}qtd&|t#|)d#d$d%|st$|}+nt%|}+|+rtd'|t|+g},|$D]}(t&|(d(|+rd) |+nd}-|,|-qt'|,}.|.rtd*|.||t|||)||.||| ddfvr#|)d(kr#td+|t||gS||kr7td,|t||gS|d!j(rGtd-|||j)qt|dkr|sX|sX|r]td.|rntd/|j*|||S|rv|dus|r|j+|j+krtd0|||S|rtd1|||S)2aD Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. rzXGiven content is empty, stopping the process very early, returning empty utf_8 str matchutf_8gFNz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, cSg|]}t|dqSFr.0cpr0m/var/www/html/riverr-enterprise-integrations-main/venv/lib/python3.10/site-packages/charset_normalizer/api.py Nzfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSr*r+r,r-r0r0r1r2Wr3z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.z>Trying to detect encoding from a tiny portion of ({}) byte(s).Tz@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %szW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.ignore)errorszc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z0%s is most likely the one. Stopping the process.z[%s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.z:Using %s code page we detected the following languages: %szONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z#%s will be used as a fallback matchz&utf_8 will be used as a fallback matchz&ascii will be used as a fallback match),loggersetLevelloggingCRITICALINFOlenwarningrrjoinintr r rformatrappendinfosetrr addrrModuleNotFoundError ImportErrordebugstrUnicodeDecodeError LookupErrorrrangedecoder sumroundrrrr languages _languagesr8 fingerprint)/rr r!r"r#r$r%r&lengthis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedsingle_byte_hard_failure_countsingle_byte_soft_failure_countresults sig_encoding sig_payload encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_count md_chunks md_ratiosi cut_sequencechunkmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_mergedr0r0r1 from_bytess                   ,                 $    rfpc Cst||||||||S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rread)rr r!r"r#r$r%r&r0r0r1from_fpbsrpathc CsDt|d}t||||||||WdS1swYdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rr r!r"r#r$r%r&rr0r0r1 from_path|s $rc Cst|||||||}t|}tt|} t|dkr!td||} | dd| j7<t d| |d | d} | | Wd| S1sRwY| S)zi Take a (text-based) file path and try to create another file next to it, this time using UTF-8. rz;Unable to normalize "{}", no encoding charset seems to fit.-z{}r)wbN)rrlistrrGIOErrorrKbestr8rreplacerIwriteoutput) rr r!r"r#r$r%rjfilenametarget_extensionsresultrr0r0r1 normalizes.    r)rrrNNTF)rrrNNT)6os.pathrrtypingrrrrrosr rQrScharset_normalizer.constantr r r charset_normalizer.mdr charset_normalizer.modelsrrwarningsrrDcharset_normalizer.utilsrrrrrrcharset_normalizer.cdrrrr getLoggerrBrCDEBUG StreamHandlerhandler setFormatter Formatter addHandlerbytesrJfloatboolrrrrr0r0r0r1s        K  8