o |gr+@s"ddlZddlZddlZddlmZddlZejdddlZddlZddl m Z ddl Z ddl m Z mZddlmZddlmZddlmZdd lmZdd lmZmZmZmZdd lmZdd lmZdd l m!Z!e"Z#dZ$GdddZ%e&dkrdZ'e%e'dddddZ(e)e(j*dSdS)N)urlparse.)RobotFileParser)Node PreOrderIter)ChatPromptTemplate) ChatOpenAI)CharacterTextSplitter)Selector)OPENAI_API_KEYOPENAI_MODEL_35OPENAI_MODEL_MINIOPENAI_TOKEN_LIMIT)SeleniumDriverChrome) parsed_url) crawl_pagesdc @s.eZdZdgZdZiZ    d2dedededed ef d d Zd d Z ddZ e fddZ e fddZ dede efddZdedejfddZde edeefddZddZdefddZd edefd!d"Zd#e fd$d%Zd&e ede efd'd(Zded)e efd*d+Zded,efd-d.Zd/d0Zd1S)3 SitemapScrapez /robots.txtTFurlscrape_sitemap url_scrape summarize exclude_blogscCst||_||_||_||_||_|t|j|_||j|_ |jr8| |_ t t |j |_ ||j |_ dSdSN)sitemap_scraperrrrsitemaps_from_robotsrdomain_sitemapremove_duplicate_sitemaps_urlsconsolidate_sitemapsr site_urlslistsetrank_urls_by_path_depth)selfrrrrrr%:/var/www/html/XCapMarket/utils/webscrape/sitemap_scrape.py__init__0s  zSitemapScrape.__init__cs$||jIdH|_||_dSr) scrape_urlsr urls_scrapedconsolidate_images image_urlsr$r%r%r&a_process_urlsJszSitemapScrape.a_process_urlscCst|dSr)asynciorunr-r,r%r%r& process_urlsNszSitemapScrape.process_urlscC dd}t||d}|d|S)NcS"t|j}tdd|dDS)NcSg|]}|r|qSr%r%.0pr%r%r& UMSitemapScrape.rank_urls_by_path_depth..path_depth../rpathlensplitrr<r%r%r& path_depthR 9SitemapScrape.rank_urls_by_path_depth..path_depthkeysortedr$urlstop_nr@ ranked_urlsr%r%r&r#Q  z%SitemapScrape.rank_urls_by_path_depthcCr1)NcSr2)NcSr3r%r%r4r%r%r&r7^r8r9r:r;r?r%r%r&r@[rArBrCrErGr%r%r&r#ZrKreturnc Cs|dg}||d|jD]+}||}t|}||}|jdvr$q|jd}|||}|r:| |qt t |S)zGet the sitemap for a urlz /sitemap.xmlz/sitemap_index.xml)ii ) appendrobots_locationsrget_url status_codetextr>parse site_mapsextendr!r") r$r sitemap_urlsrobots_location robot_urlrprlinessitemapsr%r%r&rcs        z"SitemapScrape.sitemaps_from_robotsc Csztj|ddiddd}W|Stjjy@}z$tjd||ddtdtj|d d ddiddd}WYd }~|Sd }~wt jj y]}ztjd ||ddtd |d }~ww)z Get the urlz User-Agentz Mozilla/5.0 T)headerstimeoutallow_redirectszConnectionError with %s: %sexc_infoz//www.z//NzMaxRetryError with %s: %s?) requestsget exceptionsConnectionError XCM_loggererrortimesleepreplaceurllib3 MaxRetryError)r$rrZer%r%r&rPvs2    zSitemapScrape.get_urlrcCs*t}|D] }||}||jq|S)zRemove duplicate sitemap urls)r"rPaddr)r$rrsitemaprZr%r%r&rs  z,SitemapScrape.remove_duplicate_sitemaps_urlscCstjd|jddg}|jrOtd|j}||}||j }t |}| d D] }|j ||q.| d D] }|||qA|jstdt||j|S)zScrape the sitemapzScrape sitemap for %sTrardz//sitemap/loc/text()z//url/loc/text()z#Scraped %s urls from sitemap for %s)riinforrrkrlpoprP _clean_cdatarRr xpathgetallrqclean_loc_textrNr=)r$rH sitemap_urlresponse response_textselectorlocr%r%r&rs     zSitemapScrape.scrape_sitemaprRcCsddl}|dd|S)Nrzz\1)resub)r$rRr~r%r%r&ruszSitemapScrape._clean_cdataloc_textcCs |ddd}|S)zClean the loc url textr: r)striprstripr>)r$rr%r%r&rxszSitemapScrape.clean_loc_textrHc si}fdd|D}t|ddIdH}g}|D]\}}|dd}|r3jr3|||qtj|IdH}t|D]\} }||d||djrU|| ndd ||<q@|S) zScrape the URLs asynchronouslycs<g|]}tt|jdjkrd|vrjs|qS)r:blog)r=rr<r>max_depth_to_scrapelowerr)r5rr,r%r&r7s z-SitemapScrape.scrape_urls..T) with_imagesNmarkdownimages)rRrsummary) ritemsrfrrNsummarize_websiter.gather enumerate) r$rHr)urls_to_scrapetasksr page_data page_text summariesir%r,r&r(s&      zSitemapScrape.scrape_urlstextscCs0g}|D]}||jvrd|j|<||q|S)zRemove duplicate textT)duplicate_page_text_dictrN)r$url_noderunique_page_textrRr%r%r&remove_duplicate_texts  z#SitemapScrape.remove_duplicate_textpage_text_listcs^tgd}tttddd}||B}|fdd|DIdH}dd d|D}|S) z Summarize the website text chunk))systemaYou are a research analyst with 20 years of experience and you have been asked to summarize website pages. You will first understand the website text in detail and then summarize the relevant information. Only return the summary text of the website text. Make your summary verbose but factual. Extra information is better than truncating information that could be useful. You will be provided prior summary of the website if the page had too much text. You want to only append to the summary.z2systemBring the relevant information from the text)humanz%The URL you are summarizing is: {url})rz The website text is: {page_text}g?i) model_nameapi_key temperature max_tokenscsg|]}|dqS))rrr%)r5rRrr%r&r7sz>SitemapScrape.summarize_website_text_chunk..NrMcSsg|]}|jqSr%)content)r5resultr%r%r&r7s)r from_messagesrr r abatchjoin)r$rrpromptllmchainoutput_resultsoutputr%rr&summarize_website_text_chunks  z*SitemapScrape.summarize_website_text_chunkrcsBtjdttdtdtd}||}|||IdH}|S)zSummarize the website cl100k_baseg?) encoding_name chunk_size chunk_overlapN)r from_tiktoken_encoderintr split_textr)r$rr text_splitterrrr%r%r&rs   zSitemapScrape.summarize_websitecCstd|ji}|jD]&}|j|dD]}|d}||vr'|dd||<q||dd7<qq ddt|d d d d D}|S) zConsolidate the imageszConsolidating images for %srsrcrc)rcountrcSsi|] \}}|dur||qSrr%)r5kvr%r%r& 3s z4SitemapScrape.consolidate_images..cSs |ddS)Nrcrr%)itemr%r%r&6s z2SitemapScrape.consolidate_images..T)rDreverse)rirsrr)rFr)r$r+siteimage image_urlr%r%r&r*%s z SitemapScrape.consolidate_imagesN)TTTF)__name__ __module__ __qualname__rOrrstrboolr'r-r0MAX_SITES_TO_SCRAPEr#r!rreResponserPr"rrrurxr(rrrr*r%r%r%r&r(sD     #! r__main__zhttps://munichmotorsport.com/TF)rrrr)+r.loggingsys urllib.parserrnr<rNrkurllib.robotparserrreanytreerrlangchain.promptsrlangchain_openairlangchain_text_splittersr parselr configs.configr r r rutils.selenium_driver_chromerutils.url_parserrutils.webscrape.crawl_pagesr getLoggerrirrrURLrprintr r%r%r%r&s>