ƇtjUUddlmZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z ddlmZmZmZmZmZdd lmZdd lmZmZdd lmZmZmZmZmZmZej d Z!ej"Z#e#$ej%d e&edZ'de(d< d3d4d)Z) d3d5d,Z* d3d6d/Z+ d7d8d2Z,dS)9) annotationsN) lru_cache)PathLike)BinaryIO)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDIANA_SUPPORTED_SIMILARTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bomis_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)sc"t| SN)rencodings T/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/charset_normalizer/api.pyr .s-CH-M-M)M)keyz list[str]IANA_SUPPORTED_MB_FIRST皙?TF皙? sequencesbytes | bytearraystepsint chunk_size thresholdfloat cp_isolationlist[str] | None cp_exclusionpreemptive_behaviourboolexplainlanguage_thresholdenable_fallbackreturnrc 4't|ttfs/tdt ||rJt j} t tt tt|} | dkrwt d|r9t tt | tt!|dddgdgS|At td d |d |D}ng}|At td d |d |D}ng}| ||zkr't td||| d}| }|dkr| |z |krt'| |z }t|t(k} t|t*k} | r4t td| n5| r3t td| g}|rt-|nd}|6||t td|t1}g}g}t1}t1}i}t3dt4}t3dt6}d}t1}d}d}d}d}d}d}t} t}!t9|\}"}#|"D||"t tdt|#|"|dd|vr|d|t:zD ]}$|r|$|vr |r|$|vr|$|vr||$d}%|"|$k}&|&ot?|$}'|$dvr$|&s"t td|$l|$dvr$|&s"t td|$|$|vr"t td|$|$|vr"t td|$ tA|$}(n9#tBtDf$r%t td|$Y&wxYw|ru|(st1tG|$})nt1tI|$})|)%|s%t td|$|)||r-|(s+||kr%t td|$|||r%|(s#t td |$|( o| }* | rO|(sMtM|'s|dt'd!n#|t|#t'd!|$"n^|*s\|$d#kr,|&r*tM||$"}%|%r|%dd$kr |%dd}%n*tM|'s|n|t|#d|$"}%nx#tNtPf$rd}+t|+tPs/t td%|$tM|+||$Yd}+~+d}+~+wwxYwtS|&sdnt|#| t'| |z },|(o|%duot|%| k}-|-r!t td&|$t't|,d'z }.tU|.d(}.d}/d}0g}1g}2 tW||$|,||&|'|#|(|%|* D]q}3|1|3|2||3||odt|cxkod(knc|2d)|kr|/dz }/|/|.ks|&r|'snrn#tNtPf$r}+|*rJt td%|$tM|+||$Yd}+~+t td*|$tM|+|.}/d+}0Yd}+~+nd}+~+wwxYw|0s| r|(s |t'd,d,|$d-.n\#tN$rO}+t td/|$tM|+||$Yd}+~+|d}+~+wwxYw|2rt[|2t|2z nd}4|4|ks|/|.krA||$|$t\vr |/t\|$|%(|(s&|0tc|%|4gdft td0|$|/te|4d1zd23| r|$dd|d4d5fvr|0s|%i tM|'s|n|t|#d|$"}%n9#tNtPf$r%t td6|$YwxYw| rd}%t!||$||&g|%|7}5|$|kr|5}n |$dkr|5}n|5}|*r tM|'s|n|t|#d|$"}%nc#tNtPf$rO}+t td%|$tM|+||$Yd}+~+~d}+~+wwxYw|%T|(sQtc|%}6|3|6}7|7*|7\}8}9}:|:rt!||$|8|&|9| r|$|ddfvr|%nd|7};| |;||$t td8|$te|8d1zd23|$|ddfvr|8d9kr|8dkrmt d:|;j4|r9t tt | t|;gcS|!|;t|!r|||vrd|vrd|vr|!5}|$dkr@|=rd>|=nd}?|1D]$}3||3||?}@|>|@%tm|>}A|Ar4t td?|A|$t!||$|4|&|A| r|$|ddfvr|%nd|7}B| |B|%(|(s&|0tc|%|4|Ad+f|r |(s |4d@kr|dz }|$|ddfvr|4d9kr|4dkrmt d:|Bj4|r9t tt | t|BgcS|!|Bt|!r|||vrd|vrd|vr|!5}d|vr:d+}|/|=t tdD|$|4|C|so|(rm|-rk|%it|%| dEzkrS|$dFvrOd|vrKd|vrGd+}t tdG|$|4t|%| t|%| z d1z|$|"krnt dH|$|r9t tt | t| |$gcS t| dkr|s|s|r t tdI|r6t dJ|j4| |n{|r||r|r|j7|j7ks|0t dK| |n1|r/t dL| || rDt dM| 5j4t| dz nt dN|r9t tt | | S)Oaf Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z3Expected object of type bytes or bytearray, got: {}rzzfrom_bytes..l"DDD "e,,DDDr!zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c.g|]}t|dSr<r=r>s rrAzfrom_bytes..wrBr!z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.)maxsizezIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.>utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.zY%s is deemed too similar to a code page that was already considered unsuited. Continuing!zESkipping %s: already fast-tracked from a similar successful encoding.z2Encoding %s does not provide an IncrementalDecoderzbSkipping %s: definitive match already found, this encoding targets different languages (%s vs %s).zXSkipping %s: already accumulated %d same-family results after definitive match (cap=%d).zCSkipping single-byte %s: multi-byte definitive match already found.gArrIuz9Code page %s does not fit given bytes sequence at ALL. %szpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sTgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsrGrHz>%s does not decode the whole payload: fallback entry withheld.)preemptive_declarationzM%s fast-tracked (identical decoded payload to a prior encoding, chaos=%f %%).r'z.Encoding detection: %s is most likely the one.zZ%s fast-skipped (identical decoded payload to a prior encoding that failed chaos probing).z=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {},z We detected language {} using {}g{Gz?c3 K|] \}}|V dSr)r?_vs r zfrom_bytes..Cs&4441aQ444444r!)defaultg?zyDefinitive match found: %s (chaos=%.3f, coherence=%.2f). Encodings targeting different language families will be skipped.g\(\?> rIr9rGrH utf_16_be utf_16_le utf_32_be utf_32_le utf_8_sigzjMulti-byte definitive match: %s (chaos=%.3f, decoded=%d/%d=%.1f%%). Single-byte encodings will be skipped.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)8 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelrlendebug removeHandlerrrlogjoinr+rrrappendsetrrrrr#addrrModuleNotFoundError ImportErrorr r intersectionstrUnicodeDecodeError LookupErrorrangemaxrdecodesumr update setdefaulthashroundgetrbestr fingerprint)Dr(r*r,r-r/r1r2r4r5r6previous_logger_levellengthis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failuresoft_failure_skipsuccess_fast_trackedpayload_result_cachecached_mess_ratiocached_coherence_ratiodefinitive_match_founddefinitive_target_languages post_definitive_sb_success_countPOST_DEFINITIVE_SB_CAPmb_definitive_match_foundfallback_ascii fallback_u8fallback_specifiedresultsearly_stop_results sig_encoding sig_payload encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoder enc_languagesdeferred_decodinger_multi_byte_bonusmax_chunk_gave_upearly_stop_countlazy_str_hard_failure md_chunks md_ratioschunkmean_mess_ratiofallback_entry payload_hashcached cached_mess cached_cd cached_passed fast_matchprobable_resulttarget_languages cd_ratios lg_inclusionchunk_languagescd_ratios_merged current_matchbest_coherencesD r from_bytesr2s< i)U!3 4 4  A H HY     %+\/***i..F {{ STTT  3   1 1 1 OO1 2 2 2|IwUBPRSSTUUU  5 IIl # #    ED|DDD    6 IIl # #    ED|DDD   *u$%%  l         qyyVe^j00%(( "%i..3E"E"%i..4D"D    L S S         W ^ ^     (*.BKy)))t%$$%7888  N    uuF)+)+"%%%%(UU TV 0 $/// ;;4Yt444_EE$),/EE -.$"#',*.N'+K.2,..G)7)9)9 3I > >L+$$\222  W          )))+++$$W---.1HHQ <Q <  M==   M\99  F " "  =!!!&*%1]%B!5" :Q ; ;  0 0 09M 0 JJn     I % %.B % JJd     - - - JJk     0 0 0 JJW     *@*O*O ! !#[1    JJD    H   " ( J #$6}$E$E F F #$9-$H$H I I  --.IJJ  x!!/  # ) 14JJJ JJj0&      % -B  JJU    & % C.C*C * $ -B  0E +CII+..&s;'7'7#d))'CD* ' !G++0D+&)!!.'''O'>?1+=+I+I*9!""*=&)$4?II!*3{+;+;+=+=!>!. '''O#K0   a--  O!FF  $ * *= 9 9 9 HHHH ) ?AAs;/?/?       " .t+ .O$$v-    JJ-    "%SWWq[!1!1 1155 ! %!  5 ),$ %!      '''  %%!?A\):):$?$?$?$?a$?$?$?$?R=I--$)$$(999(:1A:E    ) ) )!  O!FF  (..}=== JJsA     1 $( ! ! ! ! ! !/ )8& % *  #d))++&--mH-MMMM%    t!FF  (..}=== EN!VY#i..!@!@SV i ' '+;?P+P+P # * *= 9 9 9 666!(()? )NOOO*3H*$//))OR+G JJ0 o+Q777     - 1!W&8(HMNN-N#*!*-(8!C %.s;/?/?/A/A%B%2 +++/ <!!! !\) ! !-/*.!-!(#+="""!$666)7&&"g--%3NN"0K    "% 0; &s;'7'7'9'9:* ###' 4    O!FF  (..}===   &/D & $_ 5 5L)--l;;F!8>5 Y  ^!-!%#,!%:&$1$6#I$J$J,O "&/A"""J"NN:...(,,];;;JJg%kC/;;; &*>>1?..*g55-;NN*8K  K  /C' 3 3 3    % D*<]*K*K  4]CC    JJ8??!3'7#8#8     G # #/?H)***D # 2 2"8"8& ##   11111)<<   JJ299$m   %    .%);Wg(NNN  #5   " }%%%  &/D & + +_%% "2D9    # 2) 2 $&& , 1 , 0'7C C C#%%#%% D!*;((999OO$9:::%}o66666  % %m 4 4 4 " # # 5#+/AV/K/K6!!6!!05577O LL@(    7$$_555 5666!?"344 4 4 4& .C $44#3444cBBBB  $$F):):w&?P?P)-&+223CDDD P!#" * % !  +O$$v}44   6!!6!!(, % JJ|O$$O$$v-3    L ( ( LL1     7$$_555 5666!7=#9":;; ; ; ; ) 7||q  . ,>  JJa     + LLI"+    NN- . . . .  ++34#4 +~/III' LLU V V V NN; ' ' ' '  + LLU V V V NN> * * *V k LLNN # LL1      TUUU/_----... Ns<T  2UUB/[  \>A\99\>'B a22d Ad3dd ,e f Aff<*j''2kk*l>>nAnnfprc Vt|||||||||| S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rread) rr*r,r-r/r1r2r4r5r6s rfrom_fprs<       r!pathstr | bytes | PathLikec t|d5} t| ||||||||| cdddS#1swxYwYdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rr*r,r-r/r1r2r4r5r6rs r from_pathrs dD    R                               s 6::fp_or_path_or_payload!PathLike | str | BinaryIO | bytesc t|ttfrt||||||||||  } nOt|tt frt ||||||||||  } nt||||||||||  } | S)a) Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string. Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match are disabled to be stricter around ASCII-compatible but unlikely to be a string. ) r*r,r-r/r1r2r4r5r6)r_rurrrar`rr) rr*r,r-r/r1r2r4r5r6guessess r is_binaryrs"'#x99,  !!%%!51+            !!%%!51+     !!%%!51+    ;r!) r$r%r&NNTFr'T)r(r)r*r+r,r+r-r.r/r0r1r0r2r3r4r3r5r.r6r3r7r)rrr*r+r,r+r-r.r/r0r1r0r2r3r4r3r5r.r6r3r7r)rrr*r+r,r+r-r.r/r0r1r0r2r3r4r3r5r.r6r3r7r) r$r%r&NNTFr'F)rrr*r+r,r+r-r.r/r0r1r0r2r3r4r3r5r.r6r3r7r3)- __future__rlogging functoolsrosrtypingrcdrr r r constantr r rrrmdrmodelsrrutilsrrrrrr getLoggerre StreamHandlerrh setFormatter Formattersortedr#__annotations__rrrrrUr!rrs""""""" 00000000  / 0 0'''))GABB&,VMM&&&%)%)!% # x x x x x z%)%)!% # @%)%)!% #      B%)%)!% #!???????r!