a Šxdº(ã@sdZddlZddlZddlZddlmZmZmZmZm Z m Z ddl m Z m Z ddlZe dej¡Zeeeedœdd„Zd Zd Zed Zed Zed ZedZdeeeeefZe eejejB¡Ze e d¡ejejB¡Ze eedœdd„Z ddddddddddddddddddddœZ!eedœdd „Z"eeed!œd"d#„Z#ej$d$fej%d%fej&d&fej'd'fej(d(fgZ)d)d*„e)DƒZ*e+e ed+eee+ffd,œd-d.„Z,e -d/d0d1„¡e+eed2œd3d4„Z.d9eee+eeee+geefeeefd6œd7d8„Z/dS):z. Functions for handling encoding of web pages éN)ÚCallableÚMatchÚOptionalÚTupleÚUnionÚcast)ÚAnyUnicodeErrorÚ StrOrByteszcharset=([\w-]+))Ú content_typeÚreturncCs$|r t |¡}|r t| d¡ƒSdS)zÅExtract the encoding in the content-type header >>> import w3lib.encoding >>> w3lib.encoding.http_content_type_encoding("Content-Type: text/html; charset=ISO-8859-4") 'iso8859-4' éN)Ú_HEADER_ENCODING_REÚsearchÚresolve_encodingÚgroup)r Úmatch©rúF/var/www/html/Ranjet/env/lib/python3.9/site-packages/w3lib/encoding.pyÚhttp_content_type_encodings  rz%s\s*=\s*["']?\s*%s\s*["']?a(?:\s+ [^=<>/\s"'-]+ # Attribute name (?:\s*=\s* (?: # ' and " are entity encoded (', "), so no need for ', " '[^']*' # attr in ' | "[^"]*" # attr in " | [^'"\s]+ # attr having no ' nor " ))? )*?)z http-equivz Content-Type)Úcontentz.(?P[^;]+);\s*charset=(?P[\w-]+))Úcharsetz(?P[\w-]+))Úencodingz(?P[\w-]+)z><\s*(?:meta%s(?:(?:\s+%s|\s+%s){2}|\s+%s)|\?xml\s[^>]+%s|body)Úascii)Ú html_body_strr cCsf|dd…}t|tƒr"t |¡}n t |¡}|rb| d¡pL| d¡pL| d¡}|rbttj  |¡ƒSdS)a‚Return the encoding specified in meta tags in the html body, or ``None`` if no suitable encoding was found >>> import w3lib.encoding >>> w3lib.encoding.html_body_declared_encoding( ... """ ... ... ... Some title ... ... ... ... ... ... ... """) 'utf-8' >>> NirZcharset2Z xmlcharset) Ú isinstanceÚbytesÚ_BODY_ENCODING_BYTES_RErÚ_BODY_ENCODING_STR_RErrÚw3libÚutilÚ to_unicode)rÚchunkrrrrrÚhtml_body_declared_encoding>s     ÿýr"Úcp1252Ú big5hkscsÚcp949Úgb18030Úcp874Úcp1254Ú mac_romanÚcp932Úcp1251)rÚbig5Úeuc_krÚgb2312Z gb_2312_80ÚgbkÚ iso8859_11Ú iso8859_9Úlatin_1Ú macintoshÚ shift_jisÚtis_620Zwin_1251Z windows_31jZwin_31jZ windows_874Zwin_874Zx_sjisÚzh_cn)rr cCs$t |¡ ¡}tttjj ||¡ƒS)z~Canonicalize an encoding name This performs normalization and translates aliases using python's encoding aliases )Ú encodingsÚnormalize_encodingÚlowerrÚstrÚaliasesÚget)rÚnormedrrrÚ_c18n_encoding„sr>)Úencoding_aliasr cCs<t|ƒ}t ||¡}zt |¡jWSty6YdS0dS)aReturn the encoding that `encoding_alias` maps to, or ``None`` if the encoding cannot be interpreted >>> import w3lib.encoding >>> w3lib.encoding.resolve_encoding('latin1') 'cp1252' >>> w3lib.encoding.resolve_encoding('gb_2312-80') 'gb18030' >>> N)r>ÚDEFAULT_ENCODING_TRANSLATIONr<ÚcodecsÚlookupÚnameÚ LookupError)r?Z c18n_encodingZ translatedrrrrŽs   rz utf-32-bez utf-32-lez utf-16-bez utf-16-lezutf-8cCsh|]\}}|d’qS)rr)Ú.0ÚcÚ_rrrÚ ©órH©NN)Údatar cCs8|r4|dtvr4tD]\}}| |¡r||fSqdS)ažRead the byte order mark in the text, if present, and return the encoding represented by the BOM and the BOM. If no BOM can be detected, ``(None, None)`` is returned. >>> import w3lib.encoding >>> w3lib.encoding.read_bom(b'\xfe\xff\x6c\x34') ('utf-16-be', '\xfe\xff') >>> w3lib.encoding.read_bom(b'\xff\xfe\x34\x6c') ('utf-16-le', '\xff\xfe') >>> w3lib.encoding.read_bom(b'\x00\x00\xfe\xff\x00\x00\x6c\x34') ('utf-32-be', '\x00\x00\xfe\xff') >>> w3lib.encoding.read_bom(b'\xff\xfe\x00\x00\x34\x6c\x00\x00') ('utf-32-le', '\xff\xfe\x00\x00') >>> w3lib.encoding.read_bom(b'\x01\x02\x03\x04') (None, None) >>> rrJ)Ú _FIRST_CHARSÚ _BOM_TABLEÚ startswith)rKÚbomrrrrÚread_bom¬s   rPZ w3lib_replacecCsdtt|ƒjfS)Nu�)rrÚend)ÚexcrrrÚÌrIrS)Údata_strrr cCs | |d¡S)z¯Convert a str object to unicode using the encoding given Characters that cannot be converted will be converted to ``\ufffd`` (the unicode replacement character). Úreplace)Údecode)rTrrrrr Ðsr Úutf8)Úcontent_type_headerrÚdefault_encodingÚauto_detect_funr cCs¨t|ƒ\}}|dur8tt|ƒ}|t|t|ƒd…|ƒfSt|ƒ}|durn|dksX|dkr`|d7}|t||ƒfSt|ƒ}|durŽ|durŽ||ƒ}|durš|}|t||ƒfS)a Convert raw html bytes to unicode This attempts to make a reasonable guess at the content encoding of the html body, following a similar process to a web browser. It will try in order: * BOM (byte-order mark) * http content type header * meta or xml tag declarations * auto-detection, if the `auto_detect_fun` keyword argument is not ``None`` * default encoding in keyword arg (which defaults to utf8) If an encoding other than the auto-detected or default encoding is used, overrides will be applied, converting some character encodings to more suitable alternatives. If a BOM is found matching the encoding, it will be stripped. The `auto_detect_fun` argument can be used to pass a function that will sniff the encoding of the text. This function must take the raw text as an argument and return the name of an encoding that python can process, or None. To use chardet, for example, you can define the function as:: auto_detect_fun=lambda x: chardet.detect(x).get('encoding') or to use UnicodeDammit (shipped with the BeautifulSoup library):: auto_detect_fun=lambda x: UnicodeDammit(x).originalEncoding If the locale of the website or user language preference is known, then a better default encoding can be supplied. If `content_type_header` is not present, ``None`` can be passed signifying that the header was not present. This method will not fail, if characters cannot be converted to unicode, ``\\ufffd`` (the unicode replacement character) will be inserted instead. Returns a tuple of ``(, )`` Examples: >>> import w3lib.encoding >>> w3lib.encoding.html_to_unicode(None, ... b""" ... ... ... ... Creative Commons France ... ... ...

Creative Commons est une organisation \xc3\xa0 but non lucratif ... qui a pour dessein de faciliter la diffusion et le partage des oeuvres ... tout en accompagnant les nouvelles pratiques de cr\xc3\xa9ation \xc3\xa0 l\xe2\x80\x99\xc3\xa8re numerique.

... ... """) ('utf-8', '\n\n\n\nCreative Commons France\n\n\n

Creative Commons est une organisation \xe0 but non lucratif\nqui a pour dessein de faciliter la diffusion et le partage des oeuvres\ntout en accompagnant les nouvelles pratiques de cr\xe9ation \xe0 l\u2019\xe8re numerique.

\n\n') >>> Nzutf-16zutf-32z-be)rPrrr Úlenrr")rXrrYrZZbom_encrOÚencrrrÚhtml_to_unicodeÙsC  r])rWN)0Ú__doc__ÚrerAr7ÚtypingrrrrrrZ w3lib._typesrr Z w3lib.utilrÚcompileÚIr r:rZ _TEMPLATEZ _SKIP_ATTRSZ _HTTPEQUIV_REZ _CONTENT_REZ _CONTENT2_REZ_XML_ENCODING_REZ_BODY_ENCODING_PATTERNÚVERBOSErÚencoderr"r@r>rÚ BOM_UTF32_BEÚ BOM_UTF32_LEÚ BOM_UTF16_BEÚ BOM_UTF16_LEÚBOM_UTF8rMrLrrPÚregister_errorr r]rrrrÚs„   ÿÿÿ0í û$ÿ ü û