tj=ddlZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZGdd eZdS) N)Any AsyncIteratorIteratorListOptionalSetUnion)urlparse) BeautifulSoup)Document) BaseLoader) WebBaseLoaderceZdZdZ d%ddddeded eed ed ed ed eedeeefdZdedefdZ d&de edededefdZ de ee efde fdZdedefdZdede efdZ d'dedeedee de efdZ d'ded edeedee de ef dZdeefd Zdeefd!Z d'ded"eedeefd#Zdede efd$ZdS)( GitbookLoadera Load `GitBook` data. 1. load from either a single page, or 2. load all (relative) paths in the sitemap, handling nested sitemap indexes. When `load_all_paths=True`, the loader parses XML sitemaps and requires the `lxml` package to be installed (`pip install lxml`). FNmainT) sitemap_urlallowed_domainsweb_pageload_all_pathsbase_urlcontent_selectorcontinue_on_failure show_progressrrc|p||_|jdr|jdd|_||_||_||_||_||_||_|jt|j } | r| h|_|r|p |jd|_ n||_ | |j std|j d|jdS)aTInitialize with web page and whether to load all paths. Args: web_page: The web page to load or the starting point from where relative paths are discovered. load_all_paths: If set to True, all relative paths in the navbar are loaded instead of only `web_page`. Requires `lxml` package. base_url: If `load_all_paths` is True, the relative paths are appended to this base url. Defaults to `web_page`. content_selector: The CSS selector for the content to load. Defaults to "main". continue_on_failure: whether to continue loading the sitemap if an error occurs loading a url, emitting a warning instead of raising an exception. Setting this to True makes the loader more robust, but also may result in missing data. Default: False show_progress: whether to show a progress bar while loading. Default: True sitemap_url: Custom sitemap URL to use when load_all_paths is True. Defaults to "{base_url}/sitemap.xml". allowed_domains: Optional set of allowed domains to fetch from. If None (default), the loader will restrict crawling to the domain of the `web_page` URL to prevent potential SSRF vulnerabilities. Provide an explicit set (e.g., {"example.com", "docs.example.com"}) to allow crawling across multiple domains. Use with caution in server environments where users might control the input URLs. /Nz /sitemap.xmlz Domain in z% is not in the allowed domains list: ) rendswithrrrrrrr netloc start_url_is_url_allowed ValueError) selfrrrrrrrrinitial_domains j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_community/document_loaders/gitbook.py__init__zGitbookLoader.__init__sJ!,H = ! !# & & / M#2#.DM  , 0#6 *.   '%h//6N 8(6'7$  &(Jt},J,J,JDNN%DN##DN33 *T^**'**   urlreturnc|jdS t|}|jdvrdS|jsdS|j|jvS#t$rYdSwxYw)z0Check if a URL has an allowed scheme and domain.NF)httphttps)rr schemer Exception)r"r'parseds r$r zGitbookLoader._is_url_allowedYsz   '5 c]]F}$555u= u=D$88 8   55 s<< < A  A URLurl_listurl_typec||r||dStjd|d|dS)aBSafely add a URL to a list if it's from an allowed domain. Args: url_list: The list to add the URL to url: The URL to add url_type: Type of URL for warning message (e.g., "sitemap", "content") Returns: bool: True if URL was added, False if skipped TzSkipping disallowed z URL: F)r appendwarningswarn)r"r0r'r1s r$ _safe_add_urlzGitbookLoader._safe_add_urlpsW    $ $  OOC 4 MFFFFF G G G5r& url_or_urlsc:t||j|jS)zCreate a new WebBaseLoader instance for the given URL(s). This ensures each operation gets its own isolated WebBaseLoader. )web_pathrr)rrr)r"r7s r$_create_web_loaderz GitbookLoader._create_web_loaders*  $ 8,    r&soupc0|dduS)z+Check if the soup contains a sitemap index. sitemapindexN)find)r"r;s r$_is_sitemap_indexzGitbookLoader._is_sitemap_indexsyy((44r&c|d}g}|D]<}|d}|r#|jr|||jd=|S)z*Extract sitemap URLs from a sitemap index.sitemaploc)find_allr>textr6)r"r; sitemap_tagsurlsrArBs r$_extract_sitemap_urlsz#GitbookLoader._extract_sitemap_urlssi}}Y// # > >G,,u%%C >sx >""49=== r&processed_urls web_loadercR|||j}||r||}g}|D]}||vrt jd||| |j}|g|_|d}||_| |||} | | #t$r-} |j rt jd|d| nYd} ~ d} ~ wwxYw|S| |S)aOProcess a sitemap, handling both direct content URLs and sitemap indexes. Args: soup: The BeautifulSoup object of the sitemap processed_urls: Set of already processed URLs to avoid cycles web_loader: WebBaseLoader instance to reuse for all requests, created if None Nz(Skipping already processed sitemap URL: lxml-xmlparserError processing sitemap : )r:rr?rGr4r5add web_pathsscrape_process_sitemapextendr-r _get_paths) r"r;rHrI sitemap_urlsall_content_urlsroriginal_web_paths sitemap_soup content_urlses r$rSzGitbookLoader._process_sitemaps  00@@J  ! !$ ' '& )55d;;L! +   .00MP;PP"";///)3)=&,7=J($.#4#4J#4#G#GL,>J($(#8#8$nj$$L%++L9999 / &T+&T&TQR&T&TUUUUVUUUU $ #??4(( (s?AC D"#D  DcK|||j}||r||}g}fd|D}|sgS|j}||_||dd{V} ||_t || D]\} } |  || ||d{V} | | O#t$r-} |j rtj d| d| nYd} ~ d} ~ wwxYw|S||S)a^Async version of _process_sitemap. Args: soup: The BeautifulSoup object of the sitemap base_url: The base URL for relative paths processed_urls: Set of already processed URLs to avoid cycles web_loader: WebBaseLoader instance to reuse for all requests, created if None Ncg|]}|v| Sr^).0r'rHs r$ z3GitbookLoader._aprocess_sitemap..s#QQQs.7P7P7P7P7Pr&rKrLrNrO)r:rr?rGrQ ascrape_allziprP_aprocess_sitemaprTr-rr4r5rU)r"r;rrHrIrVrWnew_urlsrXsoupsrrYrZr[s ` r$rczGitbookLoader._aprocess_sitemaps"  00@@J  ! !$ ' '& )55d;;L! RQQQ|QQQH  ",!5 #+J %00*0MMMMMMMME$6J -05-A-A  ) \"";/// )-)?)?$h **$$$$$$L%++L9999 / &T+&T&TQR&T&TUUUUVUUUU $ #??4(( (s;3C// D&9#D!!D&c#K|jsS||j}|}|||j}|r|VdSdS||j}|d}t }|||}|s#|jrtj d|jg}|D]}| ||d|sdS||} | |} t| |D]!\}}|||}|r|V"dS)zDFetch text from one single GitBook page or recursively from sitemap.rKrL$No content URLs found in sitemap at contentN)rr:rrR _get_documentrsetrSrr4r5r6 scrape_allrb) r" temp_loaderr;doc soup_inforHrelative_pathsrFr'content_loader soup_infoss r$ lazy_loadzGitbookLoader.lazy_loads"& 11$-@@K%%''D$$T4=99C    11$.AAK#****==I(+uuN!229nMMN! Wd&8 W UT^UUVVV!D% 9 9""4i8888 "44T::N(22488J"%j$"7"7   3((C88III  r&c(K|jsi||j}||jgd{V}|d}|||j}|r|WVdSdS||j}||jgdd{V}|d}t }|||j|d{V}|s#|j rtj d|jg}|D]}| ||d|sdS||} | |d{V} t| |D]"\}}|||} | | WV#dS)z/Asynchronously fetch text from GitBook page(s).NrrKrLrgrh)rr:rrarirrjrcrrr4r5r6rb) r"rlrernrmrHrorFr'rprq maybe_docs r$ alazy_loadzGitbookLoader.alazy_load?s"* $11$-@@K%114=/BBBBBBBBEaI$$Y >>C    11$.AAK%114>2B:1VVVVVVVVEaI(+uuN#'#9#94=.$$N" Wd&8 W UT^UUVVV!D% 9 9""4i8888 "44T::N .99$????????J"%j$"7"7 $ $ 3 ..y#>> (#OOOO $ $r& custom_urlc||j}|sdS|d}|d}|r|jnd}|p|j|d}t ||S)z,Fetch content from page and return Document.N ) separatorh1)sourcetitle) page_contentmetadata)r>rget_textstriprDrr )r"r;rvpage_content_rawrhtitle_if_existsr}rs r$rizGitbookLoader._get_documentms 99T%:;; 4"++d+;;AACC*//55(7?$$R(9DMEJJWx@@@@r&c|g}|dD]#}|jr||j$|S)zFetch all URLs in the sitemap.rB)rCrDr3)r"r;rFrBs r$rUzGitbookLoader._get_pathszsH=='' & &Cx & CH%%% r&)FNrFT)r/)N)__name__ __module__ __qualname____doc__strboolrrr%r rr6r rr:r r?rGrSrcrr rrrrurrirUr^r&r$rr s %"& &$)"A&*.2AAAAA3- A  A " AAc]A"#c(+AAAAF340>CS (+7: (  eCcN.C      5m55555-DI/3 9)9)9)C9)]+ 9) c 9)9)9)9)@/3 ;);););)C ;) ]+ ;) c ;);););)z(8H-((((T,$-"9,$,$,$,$^6: A A A%-c] A (  A A A AstCyr&r)r4typingrrrrrrr urllib.parser bs4r langchain_core.documentsr )langchain_community.document_loaders.baser -langchain_community.document_loaders.web_baserrr^r&r$rsKKKKKKKKKKKKKKKKKK!!!!!!------@@@@@@GGGGGGvvvvvJvvvvvr&