a xd. @sdZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZed ejZed ejZed ejZed ejejBZed ejejBZedejZdZd6eeee eedddZ!d7eeee dddZ"d8eeeeedddZ#edejZ$d9eeeedddZ%d:eeeeeeeed d!d"Z&d;eeeeeed#d$d%Z'deeeed,d-d.Z*d?eeeeee ee+eefd0d1d2Z,eed3d4d5Z-dS)@z( Functions for dealing with markup text N)name2codepoint)IterableMatchAnyStrOptionalPatternTupleUnion)urljoin) to_unicode)safe_url_string) StrOrByteszI&((?P[a-z\d]+)|#(?P\d+)|#x(?P[a-f\d]+))(?P;?)z<[a-zA-Z\/!].*?>z5]*href\s*=\s*[\"\']\s*([^\"\'\s]+)\s*[\"\']z}]*http-equiv[^>]*refresh[^>]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)z]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)[^>]*?\shttp-equiv\s*=[^>]*refreshz<((?P.*?)(?P\]\]>))z Tutf-8)textkeepremove_illegalencodingreturncs,tttdfdd }t|t||S)uRemove entities from the given `text` by converting them to their corresponding unicode character. `text` can be a unicode string or a byte string encoded in the given `encoding` (which defaults to 'utf-8'). If `keep` is passed (with a list of entity names) those entities will be kept (they won't be removed). It supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``) and named entities (such as `` `` or ``>``). If `remove_illegal` is ``True``, entities that can't be converted are removed. If `remove_illegal` is ``False``, entities that can't be converted are kept "as is". For more information see the tests. Always returns a unicode string (with the entities removed). >>> import w3lib.html >>> w3lib.html.replace_entities(b'Price: £100') 'Price: \xa3100' >>> print(w3lib.html.replace_entities(b'Price: £100')) Price: £100 >>> mrc s|}d}|dr&t|dd}nZ|dr@t|dd}n@|dr|d}|vrh|dSt|p~t|}|durz8d|krdkrnnt|fd WSt|WSWnt t fyYn0r|d rd S|dS) Ndec hexnamedrcp1252Z semicolon) groupdictgetintlowergrouprbytesdecodechr ValueError OverflowError)rgroupsnumber entity_namerrrB/var/www/html/Ranjet/env/lib/python3.9/site-packages/w3lib/html.pyconvert_entityEs*     z(replace_entities..convert_entity)rstr_ent_resubr )rrrrr/rr-r.replace_entities$s!r3)rrrcCsttt||SN)boolr1searchr )rrrrr. has_entitiesfsr7r)rtokenrrcCst|t||S)acReplace all markup tags found in the given `text` by the given token. By default `token` is an empty string so it just removes all tags. `text` can be a unicode string or a regular string encoded as `encoding` (or ``'utf-8'`` if `encoding` is not given.) Always returns a unicode string. Examples: >>> import w3lib.html >>> w3lib.html.replace_tags('This text contains some tag') 'This text contains some tag' >>> w3lib.html.replace_tags('

Je ne parle pas fran\xe7ais

', ' -- ', 'latin-1') ' -- Je ne parle pas -- fran\xe7ais -- -- ' >>> )_tag_rer2r )rr8rrrr. replace_tagsjsr:z|$)cCst||}td|S)zRemove HTML Comments. >>> import w3lib.html >>> w3lib.html.remove_comments(b"test whatever") 'test whatever' >>> r)r _REMOVECOMMENTS_REr2)rrutextrrr.remove_commentss r=)r which_onesrrrcsrrtdddDddDttdfdd tttdfd d }d }t|tjtjB}||t ||S) a;Remove HTML Tags only. `which_ones` and `keep` are both tuples, there are four cases: ============== ============= ========================================== ``which_ones`` ``keep`` what it does ============== ============= ========================================== **not empty** empty remove all tags in ``which_ones`` empty **not empty** remove all tags except the ones in ``keep`` empty empty remove all tags **not empty** **not empty** not allowed ============== ============= ========================================== Remove all tags: >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags(doc) 'This is a link: example' >>> Keep only some tags: >>> w3lib.html.remove_tags(doc, keep=('div',)) '
This is a link: example
' >>> Remove only specific tags: >>> w3lib.html.remove_tags(doc, which_ones=('a','b')) '

This is a link: example

' >>> You can't remove some and keep some: >>> w3lib.html.remove_tags(doc, which_ones=('a',), keep=('p',)) Traceback (most recent call last): ... ValueError: Cannot use both which_ones and keep >>> z#Cannot use both which_ones and keepcSsh|] }|qSrr#.0tagrrr. zremove_tags..cSsh|] }|qSrr?r@rrr.rCrD)rBrcs |}r|vS|vSdSr4r?)rB)rr>rr. will_removesz remove_tags..will_removercs |d}|rdS|dS)Nrr)r$)rrB)rErr. remove_tags zremove_tags..remove_tagz/]+).*?>) r(r0r5rrecompileDOTALL IGNORECASEr2r )rr>rrrGregexretagsr)rr>rEr. remove_tagss1rN)rr>rrcCsFt||}|rBddd|D}t|tjtjB}|d|}|S)aRemove tags and their content. `which_ones` is a tuple of which tags to remove including their content. If is empty, returns the string unmodified. >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags_with_content(doc, which_ones=('b',)) '' >>> |c Ss$g|]}d|d|d|dqS)|rr@rrr. rDz,remove_tags_with_content..r)r joinrHrIrJrKr2)rr>rr<tagsrMrrr.remove_tags_with_contents   rT   )rr> replace_byrrcCs*t||}|D]}||t||}q|S)a$Remove escape characters. `which_ones` is a tuple of which escape characters we want to remove. By default removes ``\n``, ``\t``, ``\r``. `replace_by` is the string to replace the escape characters by. It defaults to ``''``, meaning the escape characters are removed. )r replace)rr>rYrr<Zecrrr.replace_escape_charss r[cCstttttttttfddd}t||}d}||tD]0}t|tr`|t|||d7}q>|| d7}q>|S)a` This function receives markup as a text (always a unicode string or a UTF-8 encoded string) and does the following: 1. removes entities (except the ones in `keep`) from any part of it that is not inside a CDATA 2. searches for CDATAs and extracts their text (if any) without modifying it. 3. removes the found CDATAs )txtpatternrcssLd}||D]*}|d\}}|||V|V|}q||dVdS)NrrF)finditerspan)r\r]offsetmatchZmatch_sZmatch_errr._get_fragmentssz&unquote_markup.._get_fragmentsrr-Zcdata_d) r0rrr rr _cdata_re isinstancer3r$)rrrrrbr<Zret_textfragmentrrr.unquote_markups   rf)rbaseurlrrcCsBt||d}t|}|r6tt|t|d|dSt|SdS)zReturn the base url if declared in the given HTML `text`, relative to the given base url. If no base url is found, the given `baseurl` is returned. )rrFN)r= _baseurl_rer6r r r$)rrgrr<rrrr. get_base_url2s  riscriptnoscript)rrgr ignore_tagsrcCszt||}Wnty*t|Yn0t||}tt|}t|pTt|}|rt | d}t | d d|}t ||}||fSdSdS)aYReturn the http-equiv parameter of the HTML meta element from the given HTML text and return a tuple ``(interval, url)`` where interval is an integer containing the delay in seconds (or zero if not present) and url is a string with the absolute url to redirect. If no meta redirect is found, ``(None, None)`` is returned. r"urlz "')NNN)r UnicodeDecodeErrorprintrTr=r3_meta_refresh_rer6_meta_refresh_re2floatr$r stripr )rrgrrmr<rintervalrnrrr.get_meta_refreshFs    rv)rrcCs |tS)a Strip all leading and trailing space characters (as defined in https://www.w3.org/TR/html5/infrastructure.html#space-character). Such stripping is useful e.g. for processing HTML element attributes which contain URLs, like ``href``, ``src`` or form ``action`` - HTML5 standard defines them as "valid URL potentially surrounded by spaces" or "valid non-empty URL potentially surrounded by spaces". >>> strip_html5_whitespace(' hello\n') 'hello' )rtHTML5_WHITESPACE)rrrr.strip_html5_whitespacefs rx)rTr)N)rN)N)rrN)rN)rUrN)rTN)rr)rrrj).__doc__rH html.entitiesrtypingrrrrrrr urllib.parser Z w3lib.utilr Z w3lib.urlr Z w3lib._typesr rIrKr1rJr9Irhrqrrrcrwr0r5r3r7r:r;r=rNrTr[rfrirsrvrxrrrr.s $       B I   +