j2ZddlZddlmZmZmZddlmZddlmZddl m Z GddeZ y)N)IteratorLiteralOptional) BaseLoader)Document) get_from_envc eZdZdZdedefdZdedefdZdddddd ed eed eed e d deef dZ de e fdZ y)FireCrawlLoadera! FireCrawlLoader document loader integration Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``. .. code-block:: bash pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key" Instantiate: .. code-block:: python from langchain_community.document_loaders import FireCrawlLoader loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} paramsreturncd}gd}|D]}|j|sd}n|rtjdtd|vr|ddur|d|d<|d=d|vr|ddur|d|d<|d=d |vr|d dur|d |d <|d =d |vr|d dur|d |d <|d =d |vr-t |d t r|j |d |d<|d =|S)NF)includesexcludesallowBackwardCrawlingallowExternalContentLinks pageOptionsTBDeprecated parameters detected. See Firecrawl v1 docs for updates.r includePathsr excludePathsrallowBackwardLinksrallowExternalLinksr scrapeOptions)getwarningswarnDeprecationWarning isinstancedictlegacy_scrape_options_adapter)selfr use_legacy_options legacy_keyskeys x/var/www/html/Fitness-lenito-AI-main/venv/lib/python3.12/site-packages/langchain_community/document_loaders/firecrawl.pylegacy_crawler_options_adapterz.FireCrawlLoader.legacy_crawler_options_adapterCsC" Czz#%)"  MMT" V#*%--3J-?F>*:&V#*%--3J-?F>*:&&&012d:39:Q3RF/023*f456$>39:U3VF/067&f]3T:.2.P.P}-/F?+=) cd}dg}d|vrd|dvr|dddk(s|dddk(s |dddk(rld}d |dvr,|dd r |dd |d <n|djd d |d <d |dvr|dd r |dd |d <d|dvr|ddr |dd|d <|d=gd}|D]}|j|sd}n|rtjdtd|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|d|d<|d=d|vr|ddur|d|d <|d=d!|vr||d!<|S)"NFmarkdownextractorOptionsmodezllm-extractionzllm-extraction-from-raw-htmlzllm-extraction-from-markdownTextractionPromptpromptz-Extract page information based on the schema.extractionSchemaschema userPrompt) includeMarkdown includeHtmlincludeRawHtmlincludeExtract includeLinks screenshotfullPageScreenshotonlyIncludeTags removeTagsrr0r1htmlr2rawHtmlr3extractr4linksr5r6zscreenshot@fullPager7 includeTagsr8 excludeTagsformats)rrrrremoveappend)r r r!r? scrape_keysr#s r$rz-FireCrawlLoader.legacy_scrape_options_adapterss",  ' 233-.v6:JJ01&95601&956*.&)V4F-GG!"456HI/56H/I 20F8,066H/I/M/M 2 O0F8, *V4F-GG!"456HI/56H/I 20F8,$v.@'AA!"45lC/56H/I,/WF8,12  Czz#%)"  MMT" !F*+,5NN:.,-&-(D0NN6*=)6)*+t3NN9-+,6)*+t3NN9-+,'.)T1NN7+>*v%,'4/NN<0<(#v-./47NN#89/0 F*+,4,23D,EF=),-v%,'4/,2<,@F=)<( F " 'F9  r&Ncrawl)api_keyapi_urlr*r urlrDrEr*)rCscrapemapr;searchc ddlm}|dvrtd|d|s td|xs t dd }||| |_||_||_|xsi|_y #t$r tdwxYw) aInitialize with API key and url. Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). "extract" (extracts structured data from a page). "search" (search for data across the web). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py r) FirecrawlAppzD`firecrawl` package not found, please run `pip install firecrawl-py`)rCrGrIrHr;rIInvalid mode 'zT'. Allowed: 'crawl', 'scrape', 'search', 'map', 'extract', 'search'.zUrl must be providedrDFIRECRAWL_API_KEY)rDrEN) firecrawlrK ImportError ValueErrorrrFr*r )r rFrDrEr*r rKs r$__init__zFireCrawlLoader.__init__s8  . P P"4&)RU  34 4I\)5HI%gwG l # V  s A!!A6c#Kd|jd<|jdk(rC|jj|j|j |jg}n|jdk(rk|js t d|jj|j|j|j}|jdg}n$|jdk(rI|js t d |jj|j|j}n|jd k(rT|js t d t|jj|jg|jg}ni|jd k(rA|jj|jjd |j}nt d|jd|D]}|jdk(s|jd k(r|}i}nJ|jdxs%|jdxs|jdd}|jdi}|sst||yw)N langchain integrationrG)r rCzURL is required for crawl modedatarHzURL is required for map moder;z URL is required for extract moderIquery)rVr rLzJ'. Allowed: 'crawl', 'scrape', 'map', 'extract', 'search'.r(r9r:metadata) page_contentrX)r r*rN scrape_urlrFrrP crawl_urlr%rmap_urlstrr;rIr)r firecrawl_docscrawl_responsedocrYrXs r$ lazy_loadzFireCrawlLoader.lazy_loads5%0 M" 99 ))HHT%G%G %T*N YY' !88 !ABB!^^55!D!DT[[!Q6N,//;N YY% 88 !?@@!^^33DHHT[[3QN YY) #88 !CDDDNN**DHH:dkk*JKNYY( "!^^22kkoog.{{3N "499+.HK "CyyE!TYY)%;" GGJ'T3776?TcggiQS>T77:r2)! "sJJ )__name__ __module__ __qualname____doc__rr%rr]rrrQrrrar&r$r r s7r.T.d.`jDjTj`"&!%GN!%/# /## /# # /# CD /#/#b18H-1r&r ) rtypingrrrlangchain_core.document_loadersrlangchain_core.documentsrlangchain_core.utilsrr rfr&r$rks%..6--xjxr&