a xd3@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddlmZmZddlmZmZmZddlmZed Zejd Zd Zed ZGd dde ZGdddZ e Z!ee!j"de#e e$edddZ%ee!j&ddZ&GdddZ'e#e#dddZ(dS)a `tldextract` accurately separates a URL's subdomain, domain, and public suffix, using the Public Suffix List (PSL). >>> import tldextract >>> tldextract.extract('http://forums.news.cnn.com/') ExtractResult(subdomain='forums.news', domain='cnn', suffix='com') >>> tldextract.extract('http://forums.bbc.co.uk/') # United Kingdom ExtractResult(subdomain='forums', domain='bbc', suffix='co.uk') >>> tldextract.extract('http://www.worldbank.org.kg/') # Kyrgyzstan ExtractResult(subdomain='www', domain='worldbank', suffix='org.kg') `ExtractResult` is a namedtuple, so it's simple to access the parts you want. >>> ext = tldextract.extract('http://forums.bbc.co.uk') >>> (ext.subdomain, ext.domain, ext.suffix) ('forums', 'bbc', 'co.uk') >>> # rejoin subdomain and domain >>> '.'.join(ext[:2]) 'forums.bbc' >>> # a common alias >>> ext.registered_domain 'bbc.co.uk' Note subdomain and suffix are _optional_. Not all URL-like inputs have a subdomain or a valid suffix. >>> tldextract.extract('google.com') ExtractResult(subdomain='', domain='google', suffix='com') >>> tldextract.extract('google.notavalidsuffix') ExtractResult(subdomain='google', domain='notavalidsuffix', suffix='') >>> tldextract.extract('http://127.0.0.1:8080/deployed/') ExtractResult(subdomain='', domain='127.0.0.1', suffix='') If you want to rejoin the whole namedtuple, regardless of whether a subdomain or suffix were found: >>> ext = tldextract.extract('http://127.0.0.1:8080/deployed/') >>> # this has unwanted dots >>> '.'.join(ext) '.127.0.0.1.' >>> # join part only if truthy >>> '.'.join(part for part in ext if part) '127.0.0.1' N)wraps) FrozenSetList NamedTupleOptionalSequenceUnion) DiskCache get_cache_dir)IP_RE SCHEME_RE looks_like_ip)get_suffix_listsZ tldextractZTLDEXTRACT_CACHE_TIMEOUT)z4https://publicsuffix.org/list/public_suffix_list.datzQhttps://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.datu [.。．｡]c@s`eZdZUdZeed<eed<eed<eedddZeeddd Zeedd d Z d S) ExtractResultz4namedtuple of a URL's subdomain, domain, and suffix. subdomaindomainsuffixreturncCs |jr|jr|jd|jSdS)z Joins the domain and suffix fields with a dot, if they're both set. >>> extract('http://forums.bbc.co.uk').registered_domain 'bbc.co.uk' >>> extract('http://localhost:8080').registered_domain '' .)rrselfrM/var/www/html/Ranjet/env/lib/python3.9/site-packages/tldextract/tldextract.pyregistered_domainSs zExtractResult.registered_domaincCs$|jr |jr ddd|DSdS)z Returns a Fully Qualified Domain Name, if there is a proper domain/suffix. >>> extract('http://forums.bbc.co.uk/path/to/file').fqdn 'forums.bbc.co.uk' >>> extract('http://localhost:8080').fqdn '' rcss|]}|r|VqdSNr).0irrr nz%ExtractResult.fqdn..r)rrjoinrrrrfqdnas zExtractResult.fqdncCs"|js|jst|jr|jSdS)a Returns the ipv4 if that is what the presented domain/url is >>> extract('http://127.0.0.1/path/to/file').ipv4 '127.0.0.1' >>> extract('http://127.0.0.1.1/path/to/file').ipv4 '' >>> extract('http://256.1.1.1').ipv4 '' r)rrr matchrrrrripv4qs zExtractResult.ipv4N) __name__ __module__ __qualname____doc__str__annotations__propertyrr#r%rrrrrLs  rc @seZdZdZeedddefeee ee e e ee ee dfddddZ deee ed d d Zde dd d dZeeedddZddddZdS) TLDExtractzSA callable for extracting, subdomain, domain, and suffix components from a URL.TFrN) cache_dirsuffix_list_urlsfallback_to_snapshotinclude_psl_private_domainsextra_suffixescache_fetch_timeoutrcCsr|pd}tdd|D|_||_|js:|s:|js:td||_||_d|_t|tr^t |n||_ t ||_ dS)a Constructs a callable for extracting subdomain, domain, and suffix components from a URL. Upon calling it, it first checks for a JSON in `cache_dir`. By default, the `cache_dir` will live in the tldextract directory. You can disable the caching functionality of this module by setting `cache_dir` to `None`. If the cached version does not exist (such as on the first run), HTTP request the URLs in `suffix_list_urls` in order, until one returns public suffix list data. To disable HTTP requests, set this to an empty sequence. The default list of URLs point to the latest version of the Mozilla Public Suffix List and its mirror, but any similar document could be specified. Local files can be specified by using the `file://` protocol. (See `urllib2` documentation.) If there is no cached version loaded and no data is found from the `suffix_list_urls`, the module will fall back to the included TLD set snapshot. If you do not want this behavior, you may set `fallback_to_snapshot` to False, and an exception will be raised instead. The Public Suffix List includes a list of "private domains" as TLDs, such as blogspot.com. These do not fit `tldextract`'s definition of a suffix, so these domains are excluded by default. If you'd like them included instead, set `include_psl_private_domains` to True. You can pass additional suffixes in `extra_suffixes` argument without changing list URL cache_fetch_timeout is passed unmodified to the underlying request object per the requests documentation here: http://docs.python-requests.org/en/master/user/advanced/#timeouts cache_fetch_timeout can also be set to a single value with the environment variable TLDEXTRACT_CACHE_TIMEOUT, like so: TLDEXTRACT_CACHE_TIMEOUT="1.2" When set this way, the same timeout value will be used for both connect and read timeouts rcss|]}|r|VqdSr)strip)rurlrrrr sz&TLDExtract.__init__..zThe arguments you have provided disable all ways for tldextract to obtain data. Please provide a suffix list data, a cache_dir, or set `fallback_to_snapshot` to `True`.N) tupler/r0 ValueErrorr1r2 _extractor isinstancer*floatr3r _cache)rr.r/r0r1r2r3rrr__init__s"1  zTLDExtract.__init__r5r1rc Cstd|ddddddddddd }t|}d d |D}|j||d }d ||d }|s|rt |rt d|dS|rd |d |dnd}|r||dnd} t || |S)a Takes a string URL and splits it into its subdomain, domain, and suffix (effective TLD, gTLD, ccTLD, etc.) component. >>> extract = TLDExtract() >>> extract('http://forums.news.cnn.com/') ExtractResult(subdomain='forums.news', domain='cnn', suffix='com') >>> extract('http://forums.bbc.co.uk/') ExtractResult(subdomain='forums', domain='bbc', suffix='co.uk') r/r?#@:rcSsg|] }t|qSr)_decode_punycode)rlabelrrr r!z'TLDExtract.__call__..r1Nr ) r sub partitionsplitr4rstrip_UNICODE_DOTS_RE_get_tld_extractor suffix_indexr"rr) rr5r1netloclabelsZ translationsrNrrrrrr__call__sF   zTLDExtract.__call__) fetch_nowrcCs d|_|j|r|dS)z/Force fetch the latest suffix list definitions.N)r8r;clearrM)rrRrrrupdates zTLDExtract.updatercCst|S)z Returns the list of tld's used by default This will vary based on `include_psl_private_domains` and `extra_suffixes` )listrMtldsrrrrrVszTLDExtract.tlds_PublicSuffixListTLDExtractorcCs`|jr |jSt|j|j|j|jd\}}t|||jgs@tdt ||t |j|j d|_|jS)a'Get or compute this object's TLDExtractor. Looks up the TLDExtractor in roughly the following order, based on the settings passed to __init__: 1. Memoized on `self` 2. Local system _cache file 3. Remote PSL, over HTTP 4. Bundled PSL snapshot file)cacheurlsr3r0z)No tlds set. Cannot proceed without tlds. public_tlds private_tlds extra_tldsr1) r8rr;r/r3r0anyr2r7rWrUr1)rr[r\rrrrMs"  zTLDExtract._get_tld_extractor)N)F)r&r'r(r)r PUBLIC_SUFFIX_LIST_URLS CACHE_TIMEOUTrr*rboolrr:r<rrQrTr,rrVrMrrrrr-s2  J (r-Fr=cCs t||dS)NrG) TLD_EXTRACTOR)r5r1rrrextract+srccOstj|i|Sr)rbrT)argskwargsrrrrT2srTc@sheZdZdZd eeeeeeedddZdeee eddd Z deeeee d d d Z dS)rWzAWrapper around this project's main algo for PSL lookups. FrZcCs6||_||_||_t||||_t|||_dSr)r1r[r\ frozensettlds_incl_privatetlds_excl_private)rr[r\r]r1rrrr<=s z&_PublicSuffixListTLDExtractor.__init__N)r1rcCs|dur|j}|r|jS|jS)z,Get the currently filtered list of suffixes.N)r1rgrh)rr1rrrrVKs z"_PublicSuffixListTLDExtractor.tlds) lower_splr1rc Cs||}t|}t|D]l}d||d}d|}||vrL|dS||vr\|Sdd||dd}||vr|Sq|S)zdReturns the index of the first suffix label. Returns len(spl) if no suffix is found rN!r z*.)rVlenranger") rrir1rVlengthrZ maybe_tldZ exception_tldZ wildcard_tldrrrrNXs    z*_PublicSuffixListTLDExtractor.suffix_index)F)N)N) r&r'r(r)rr*rar<rrrVintrNrrrrrW8s$    rW)rErc Cs>|}|d}|r:z t|WSttfy8Yn0|S)Nzxn--)lower startswithidnadecode UnicodeError IndexError)rEZloweredZlooks_like_punyrrrrDps  rD)F))r)loggingosre functoolsrtypingrrrrrrrqrXr r remoter r rZ suffix_listr getLoggerZLOGenvirongetr`r_compilerLrr-rbrQr*rarcrTrWrDrrrrs62      6' 8