tj*dZddlZddlZddlmZmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZerddlmZddlmZdd lmZdd lmZmZmZejeZGd d eZGd deZGddeZdS)zQLoader that uses Playwright to load a page, then uses unstructured to parse html.N)ABCabstractmethod) TYPE_CHECKING AsyncIteratorDictIteratorListOptionalUnion)Document) BaseLoader)Browser)Page)Response)rrrc beZdZdZedddddddefd Zedd dd dd defd ZdS)PlaywrightEvaluatorzAbstract base class for all evaluators. Each evaluator should take a page, a browser instance, and a response object, process the page as necessary, and return the resulting text. pagerbrowserrresponserreturncdS)aSynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nselfrrrs q/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_community/document_loaders/url_playwright.pyevaluatezPlaywrightEvaluator.evaluates   AsyncPage AsyncBrowser AsyncResponsec KdS)aAsynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nrrs revaluate_asyncz"PlaywrightEvaluator.evaluate_async+s  rN)__name__ __module__ __qualname____doc__rstrrr"rrrrrs   V  i  :  RU    ^      *8  DS      ^    rrcheZdZdZddeeefdZddddd d d efd Zdd ddd dd efdZ dS)UnstructuredHtmlEvaluatorz@Evaluate the page HTML content using the `unstructured` library.Nremove_selectorscZ ddl}n#t$rtdwxYw||_dS)z%Initialize UnstructuredHtmlEvaluator.rNzQunstructured package not found, please install it with `pip install unstructured`) unstructured ImportErrorr*)rr*r,s r__init__z"UnstructuredHtmlEvaluator.__init__?sX         -   !1!rrrrrrrcNddlm}|jpgD]W}||}|D]+}|r|d,X|}||}dd|DS)z3Synchronously process the HTML content of the page.rpartition_htmlelement => element.remove()text c,g|]}t|Srr'.0els r z6UnstructuredHtmlEvaluator.evaluate..W777CGG777r unstructured.partition.htmlr2r*locatorall is_visiblercontentjoin rrrrr2selectorelementselement page_sources rrz"UnstructuredHtmlEvaluator.evaluateKs>>>>>>-3 D DH||H--1133H# D D%%''D$$%BCCC Dllnn !>{333{{77h777888rrrr cKddlm}|jpgD]i}||d{V}|D]7}|d{Vr|dd{V8j|d{V}||}dd|DS)z4Asynchronously process the HTML content of the page.rr1Nr3r4r6c,g|]}t|Srr8r9s rr<z.gr=rr>rEs rr"z(UnstructuredHtmlEvaluator.evaluate_asyncYs ?>>>>>-3 J JH!\\(337799999999H# J J ++--------J!**+HIIIIIIIII J!LLNN****** !>{333{{77h777888rN) r#r$r%r&r r r'r.rr"rrrr)r)<sJJ 1 1$s))< 1 1 1 1 9V 9i 9: 9RU 9 9 9 999*89DS9 999999rr)c.eZdZdZ ddeededed eeed eed ee eefd ee ee j efd ee deefdZdeefdZdeefdZdeefdZdS)PlaywrightURLLoadera Load `HTML` pages with `Playwright` and parse with `Unstructured`. This is useful for loading pages that require javascript to render. Attributes: urls (List[str]): List of URLs to load. continue_on_failure (bool): If True, continue loading other URLs on failure. headless (bool): If True, the browser will run in headless mode. proxy (Optional[Dict[str, str]]): If set, the browser will access URLs through the specified proxy. browser_session (Optional[Union[str, os.PathLike[str]]]): Path to a file with browser session data that can be used to restore the browser session. timeout (Optional[int]): Timeout in milliseconds for page navigation. wait_until (Optional[str]): When to consider navigation succeeded. Can be one of: - "load": wait for the "load" event to fire - "domcontentloaded": wait for the "DOMContentLoaded" event to fire - "networkidle": wait until there are no network connections for at least 500ms - "commit": wait for the first network request to be sent Example: .. code-block:: python from langchain_community.document_loaders import PlaywrightURLLoader urls = ["https://api.ipify.org/?format=json",] proxy={ "server": "https://xx.xx.xx:15818", # https://: "username": "username", "password": "password" } loader = PlaywrightURLLoader( urls=urls, proxy=proxy, timeout=60000, # 60 second timeout wait_until="domcontentloaded" # Wait for DOM content to load ) data = loader.load() TN0uloadurlscontinue_on_failureheadlessr* evaluatorproxybrowser_sessiontimeout wait_untilc  ddl} n#t$rtdwxYw||_||_||_||_||_||_| |_|r|rtd|pt||_ dS)z%Load a list of URLs using Playwright.rNzMplaywright package not found, please install it with `pip install playwright`z:`remove_selectors` and `evaluator` cannot be both not None) playwrightr-rQrRrSrUrVrWrX ValueErrorr)rT) rrQrRrSr*rTrUrVrWrXrZs rr.zPlaywrightURLLoader.__init__s         +    #6    . $    L  #Q&?@P&Q&Qr/rc #Kddlm}|5}|j|j|j}d}|jrbtj |jr| |j}n"t d|j|| }|j D]} |}|||j|j}|t%d|||j|j |j|||}|d |i} t/|| V#t0$r4} |jr!td |d | n| Yd} ~ d} ~ wwxYw|ddddS#1swxYwYdS)zLoad the specified URLs using Playwright and create Document instances. Returns: A list of Document instances with loaded content. r)sync_playwrightrSrUN storage_stateSession file not found: rWrX"page.goto() returned None for url rWsource page_contentmetadataError fetching or processing , exception: )playwright.sync_apir]chromiumlaunchrSrUrVospathexists new_contextloggerwarningrQnew_pagegotorWrXr[wait_for_load_staterTrcloser ExceptionrRerror) rr]prcontexturlrrr5rhes r lazy_loadzPlaywrightURLLoader.lazy_loadsz 877777 _  # !j''dj'QQGG# V7>>$"677V%11@T1UUGGNN#Td>R#T#TUUU!--//y   "++--D#yyT\do )  H '()Sc)S)STTT,,T_dl,SSS>224(KKDJJLLL (#H"xHHHHHHH    /  QCQQaQQ    MMOOOG# # # # # # # # # # # # # # # # # # s=B1GB2E87G8 F6*F1,G1F66GG!GcNKd|2d{VS)Load the specified URLs with Playwright and create Documents asynchronously. Use this function when in a jupyter notebook environment. Returns: A list of Document instances with loaded content. c"Kg|3d{V}| 6SrLr)r:docs rr<z-PlaywrightURLLoader.aload..s.77777777c7777sN) alazy_load)rs raloadzPlaywrightURLLoader.aloads887T__%6%67777777777rc dKddlm}|4d{V}|j|j|jd{V}d}|jrhtj |jr"| |jd{V}n"t d|j|| d{V}|j D]} |d{V}|||j|jd{V}|t%d|||j|j d{V|j|||d{V}|d{Vd |i} t/|| WV#t0$r5} |jr!td |d | n| Yd} ~ d} ~ wwxYw|d{Vdddd{VdS#1d{VswxYwYdS)rr)async_playwrightNr^r_rarbrcrdrerfrirj)playwright.async_apirrlrmrSrUrVrnrorprqrrrsrQrtrurWrXr[rvrTr"rwr rxrRry) rrrzrr{r|rrr5rhr}s rrzPlaywrightURLLoader.alazy_loads :99999##%%' "' "' "' "' "' "' "J--t}DJ-WWWWWWWWGG# V7>>$"677V$+$7$7&*&:%8%%GGNN#Td>R#T#TUUU ' 3 3 5 5555555y   !(!1!1!3!3333333D%)YYT\do&/&&      H '()Sc)S)STTT223"&!>!>tWh!W!WWWWWWWD**,,&&&&&&& (#H"xHHHHHHHH    /  QCQQaQQ   --// ! ! ! ! ! ! !O' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "' "s=CHCF0/H0 G/:*G*$H*G//H H),H))TTNNNNrOrP)r#r$r%r&r r'boolr rrr rnPathLikeintr.rr r~rrrrrrrNrNjsW))\%)0437*.BF!&$*#R#R3i#R"#R #R #49- #R /0 #RS#X'#R"%R[-=(=">?#R##RSM#R#R#R#RJ+8H-++++Z8T(^88880"-"90"0"0"0"0"0"rrN) r&loggingrnabcrrtypingrrrrr r r langchain_core.documentsr )langchain_community.document_loaders.baser rrrrrrr rk getLoggerr#rrrr)rNrrrrsWW ########VVVVVVVVVVVVVVVVVV------@@@@@@<<<<<<<666666>>>>>>;;;;;;;;;;  8 $ $# # # # # ## # # L+9+9+9+9+9 3+9+9+9\w"w"w"w"w"*w"w"w"w"w"r