dh=SSKrSSKJrJrJrJrJrJrJr SSK J r SSK J r SSK Jr SSKJr SSKJr "SS \5rg) N)Any AsyncIteratorIteratorListOptionalSetUnion)urlparse) BeautifulSoup)Document) BaseLoader) WebBaseLoaderc\rSrSrSrS#SSS.S\S\S\\S\S \S \S \\S \\\4S jjjr S\S\4Sjr S$S\ \S\S\S\4Sjjr S\ \\ \4S\4SjrS\S\4SjrS\S\ \4SjrS%S\S\\S\\S\ \4SjjrS%S\S\S\\S\\S\ \4 SjjrS\\4SjrS\\4SjrS%S\S\\S\\4S jjrS\S\ \4S!jrS"rg)& GitbookLoader a Load `GitBook` data. 1. load from either a single page, or 2. load all (relative) paths in the sitemap, handling nested sitemap indexes. When `load_all_paths=True`, the loader parses XML sitemaps and requires the `lxml` package to be installed (`pip install lxml`). N) sitemap_urlallowed_domainsweb_pageload_all_pathsbase_urlcontent_selectorcontinue_on_failure show_progressrrcU=(d UUlURRS5(aURSSUlXlX lX@lXPlX`lXlURc$[U5Rn U (aU 1UlU(aU=(d URS3Ul OXl URUR5(d%[SURSUR35eg)aInitialize with web page and whether to load all paths. Args: web_page: The web page to load or the starting point from where relative paths are discovered. load_all_paths: If set to True, all relative paths in the navbar are loaded instead of only `web_page`. Requires `lxml` package. base_url: If `load_all_paths` is True, the relative paths are appended to this base url. Defaults to `web_page`. content_selector: The CSS selector for the content to load. Defaults to "main". continue_on_failure: whether to continue loading the sitemap if an error occurs loading a url, emitting a warning instead of raising an exception. Setting this to True makes the loader more robust, but also may result in missing data. Default: False show_progress: whether to show a progress bar while loading. Default: True sitemap_url: Custom sitemap URL to use when load_all_paths is True. Defaults to "{base_url}/sitemap.xml". allowed_domains: Optional set of allowed domains to fetch from. If None (default), the loader will restrict crawling to the domain of the `web_page` URL to prevent potential SSRF vulnerabilities. Provide an explicit set (e.g., {"example.com", "docs.example.com"}) to allow crawling across multiple domains. Use with caution in server environments where users might control the input URLs. /Nz /sitemap.xmlz Domain in z% is not in the allowed domains list: ) rendswithrrrrrrr netloc start_url_is_url_allowed ValueError) selfrrrrrrrrinitial_domains d/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_community/document_loaders/gitbook.py__init__GitbookLoader.__init__sJ!,H == ! !# & & MM#2.DM , 0#6 *.    '%h/66N(6'7$ (Jt}}o\,JDN%N##DNN33T^^,,Q''(* 4urlreturncURcg[U5nURS;agUR(dgURUR;$![a gf=f)z0Check if a URL has an allowed scheme and domain.F)httphttps)rr schemer Exception)r"r(parseds r$r GitbookLoader._is_url_allowedYsf    ' c]F}}$55====D$8$88 8  sAAA A$#A$url_listurl_typecURU5(aURU5 g[R"SUSU35 g)a Safely add a URL to a list if it's from an allowed domain. Args: url_list: The list to add the URL to url: The URL to add url_type: Type of URL for warning message (e.g., "sitemap", "content") Returns: bool: True if URL was added, False if skipped TzSkipping disallowed z URL: F)r appendwarningswarn)r"r1r(r2s r$ _safe_add_urlGitbookLoader._safe_add_urlpsA    $ $ OOC  MM0 &F Gr' url_or_urlsc@[UURURS9$)z|Create a new WebBaseLoader instance for the given URL(s). This ensures each operation gets its own isolated WebBaseLoader. )web_pathrr)rrr)r"r9s r$_create_web_loader GitbookLoader._create_web_loaders&  $ 8 8,,  r'soupc(URS5SL$)z+Check if the soup contains a sitemap index. sitemapindexN)find)r"r>s r$_is_sitemap_indexGitbookLoader._is_sitemap_indexsyy(44r'cURS5n/nUHLnURS5nU(dMUR(dM0URX5RS5 MN U$)z*Extract sitemap URLs from a sitemap index.sitemaploc)find_allrAtextr7)r"r> sitemap_tagsurlsrErFs r$_extract_sitemap_urls#GitbookLoader._extract_sitemap_urlssU}}Y/ #G,,u%Cssxxx""49=$ r'processed_urls web_loadercPUcURUR5nURU5(aURU5n/nUHnXb;a[R "SU35 M#UR U5 URnU/UlURSS9nXslURXU5n URU 5 M U$URU5$![a9n UR(a"[R "SUSU 35 Sn A MeSn A ff=f)aProcess a sitemap, handling both direct content URLs and sitemap indexes. Args: soup: The BeautifulSoup object of the sitemap processed_urls: Set of already processed URLs to avoid cycles web_loader: WebBaseLoader instance to reuse for all requests, created if None Nz(Skipping already processed sitemap URL: lxml-xmlparserError processing sitemap : )r<rrBrKr5r6add web_pathsscrape_process_sitemapextendr.r _get_paths) r"r>rMrN sitemap_urlsall_content_urlsroriginal_web_paths sitemap_soup content_urlses r$rXGitbookLoader._process_sitemaps5  00@J  ! !$ ' '55d;L! + 0MMB;-P"";/)3)=)=&,7=J($.#4#4J#4#GL,>($(#8#8$j$L%++L91 ,>$ #??4( (!// (A+bQRPS&TUU s?A C"" D%,-D D  D%c# UcURUR5nURU5(aURU5n/nUVs/sH owU;dM UPM nnU(d/$URn XlUR USS9IShvN n Xl[ X5HBupURU 5 URXX45IShvN n URU 5 MD U$URU5$s snfNsN3![a9nUR(a"[R"SU SU35 SnAMeSnAff=f7f)a&Async version of _process_sitemap. Args: soup: The BeautifulSoup object of the sitemap base_url: The base URL for relative paths processed_urls: Set of already processed URLs to avoid cycles web_loader: WebBaseLoader instance to reuse for all requests, created if None NrPrQrSrT)r<rrBrKrV ascrape_allziprU_aprocess_sitemaprYr.rr5r6rZ)r"r>rrMrNr[r\r(new_urlsr]soupsrr^r_r`s r$reGitbookLoader._aprocess_sitemapsZ"  00@J  ! !$ ' '55d;L! (4Q|.7P|HQ ",!5!5 #+ %00*0MME$6 -0-A) "";/ )-)?)?$*$L%++L9.B$ #??4( (CRN$!// (A+bQRPS&TUU sfA EC?C? 0ED+E=DDD(ED E -E?EEE  Ec## UR(dSURUR5nUR5nUR X R5nU(aUv ggURUR 5nURSS9n[ 5nURXE5nU(d4UR(a#[R"SUR 35 /nUHnURXxS5 M U(dgURU5n U RU5n [X5H#upHUR XH5nU(dMUv M% g7f)zDFetch text from one single GitBook page or recursively from sitemap.rPrQ$No content URLs found in sitemap at contentN)rr<rrW _get_documentrsetrXrr5r6r7 scrape_allrd) r" temp_loaderr>doc soup_inforMrelative_pathsrJr(content_loader soup_infoss r$ lazy_loadGitbookLoader.lazy_loads/""11$--@K%%'D$$T==9C 11$..AK#****=I(+uN!229MN!d&8&8 DT^^DTUV!D%""4i8&"44T:N(2248J"%j"7 ((83I#8s EE! E!cn# UR(dmURUR5nURUR/5IShvN nUSnUR X0R5nU(aU7v ggURUR 5nURUR /SS9IShvN nUSn[ 5nURX0RU5IShvN nU(d4UR(a#[R"SUR 35 /nUHnURXxS5 M U(dgURU5n U RU5IShvN n [X5H up8UR X85n U cMU 7v M" gGN^NNN;7f)z/Asynchronously fetch text from GitBook page(s).NrrPrQrjrk)rr<rrcrlrrmrerrr5r6r7rd) r"rorgrqrprMrrrJr(rsrt maybe_docs r$ alazy_loadGitbookLoader.alazy_load?s""11$--@K%114==/BBEaI$$Y >C 11$..AK%114>>2B:1VVEaI(+uN#'#9#9==.$N"d&8&8 DT^^DTUV!D%""4i8&"44T:N .99$??J"%j"7  ..y> (#O#8ICW (@sPA F5F,A+F5:F/;2F5-F1.B F57F38&F5" F5/F51F53F5 custom_urlcURUR5nU(dgURSS9R5nURS5nU(a UROSnU=(d UR US.n[ XGS9$)z,Fetch content from page and return Document.N ) separatorh1)sourcetitle) page_contentmetadata)rArget_textstriprHrr )r"r>r{page_content_rawrktitle_if_existsrrs r$rlGitbookLoader._get_documentmsx 99T%:%:;"++d+;AAC*//5(7$$R(9DMMEJW@@r'c/nURS5H1nUR(dMURUR5 M3 U$)zFetch all URLs in the sitemap.rF)rGrHr4)r"r>rJrFs r$rZGitbookLoader._get_pathszs<=='Cxxx CHH% (  r')rrrrrrrr)FNmainFT)URL)N)__name__ __module__ __qualname____firstlineno____doc__strboolrrr%r rr7r rr<r rBrKrXrerr rurryrrlrZ__static_attributes__r'r$rr s %"& &$)"A&*.2AAA3- A  A " AAc]A"#c(+AF340>CS (+7: (  eCcN.C   5m55-DI/3 9)9)C9)]+ 9) c 9)@/3 ;););)C ;) ]+ ;) c ;)z(8H-(T,$-"9,$^6: A A%-c] A (  AstCyr'r)r5typingrrrrrrr urllib.parser bs4r langchain_core.documentsr )langchain_community.document_loaders.baser -langchain_community.document_loaders.web_baserrrr'r$rs.KKK!-@GvJvr'