dh}2ZSSKrSSKJrJrJr SSKJr SSKJr SSK J r "SS\5r g)N)IteratorLiteralOptional) BaseLoader)Document) get_from_envc \rSrSrSrS\S\4SjrS\S\4SjrSSSSS .S \S \ \S \ \S \ SS\ \4 Sjjr S\ \ 4SjrSrg)FireCrawlLoader a FireCrawlLoader document loader integration Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``. .. code-block:: bash pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key" Instantiate: .. code-block:: python from langchain_community.document_loaders import FireCrawlLoader loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} paramsreturncSn/SQnUHnURU5(dMSn O U(a[R"S[5 SU;aUSSLaUSUS'US SU;aUSSLaUSUS'US S U;aUS SLaUS US 'US S U;aUS SLaUS US 'US S U;a2[ US [ 5(aUR US 5US'US U$)NF)includesexcludesallowBackwardCrawlingallowExternalContentLinks pageOptionsTBDeprecated parameters detected. See Firecrawl v1 docs for updates.r includePathsr excludePathsrallowBackwardLinksrallowExternalLinksr scrapeOptions)getwarningswarnDeprecationWarning isinstancedictlegacy_scrape_options_adapter)selfr use_legacy_options legacy_keyskeys f/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_community/document_loaders/firecrawl.pylegacy_crawler_options_adapter.FireCrawlLoader.legacy_crawler_options_adapterCsH" Czz#%)"  MMT" V#*%--3J-?F>*:&V#*%--3J-?F>*:&&&012d:39:Q3RF/023*f456$>39:U3VF/067&f]3T::.2.P.P}-/F?+=) c`SnS/nSU;aSUS;aUSSS:XdUSSS:Xd USSS:XaSnS US;a1USS (a USS US 'OUSRS S 5US 'S US;aUSS (a USS US 'SUS;aUSS(a USSUS 'US /SQnUHnURU5(dMSn O U(Ga;[R"S[5 SU;aUSSLaUR S5 US SU;aUSSLaUR S5 US SU;aUSSLaUR S5 US SU;aUSSLaUR S5 US SU;aUSSLaUR S5 US SU;aUSSLaUR S5 US SU;aUSSLaUR S5 US SU;aUSSLaUSUS'US SU;aUSSLaUSUS 'US S!U;aX1S!'U$)"NFmarkdownextractorOptionsmodezllm-extractionzllm-extraction-from-raw-htmlzllm-extraction-from-markdownTextractionPromptpromptz-Extract page information based on the schema.extractionSchemaschema userPrompt) includeMarkdown includeHtmlincludeRawHtmlincludeExtract includeLinks screenshotfullPageScreenshotonlyIncludeTags removeTagsrr2r3htmlr4rawHtmlr5extractr6linksr7r8zscreenshot@fullPager9 includeTagsr: excludeTagsformats)rrrrremoveappend)r!r r"rA scrape_keysr$s r%r -FireCrawlLoader.legacy_scrape_options_adapterss",  ' 233-.v6:JJ01&95601&956*.&)V4F-GG!"456HI/56H/I 20F8,066H/I/M/M 2 O0F8, *V4F-GG!"456HI/56H/I 20F8,$v.@'AA!"45lC/56H/I,/WF8,12  Czz#%)"  MMT" !F*+,5NN:.,-&-(D0NN6*=)6)*+t3NN9-+,6)*+t3NN9-+,'.)T1NN7+>*v%,'4/NN<0<(#v-./47NN#89/0 F*+,4,23D,EF=),-v%,'4/,2<,@F=)<( F " '9  r(Ncrawl)api_keyapi_urlr,r urlrGrHr,)rFscrapemapr=searchc SSKJn US;a[SUS35eU(d [S5eU=(d [ SS 5nU"X#S 9UlXlX@lU=(d 0Ulg ![a [S5ef=f) aQInitialize with API key and url. Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). "extract" (extracts structured data from a page). "search" (search for data across the web). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py r) FirecrawlAppzD`firecrawl` package not found, please run `pip install firecrawl-py`)rFrJrLrKr=rLInvalid mode 'zT'. Allowed: 'crawl', 'scrape', 'search', 'map', 'extract', 'search'.zUrl must be providedrGFIRECRAWL_API_KEY)rGrHN) firecrawlrN ImportError ValueErrorrrIr,r )r!rIrGrHr,r rNs r%__init__FireCrawlLoader.__init__s8  . P P"4&)RU  34 4I\)5HI%gG l # V  s A,,Bc#$# URS:Xa@URRURUR UR 5S9/nGOURS:XamUR(d [ S5eURRURURUR 5S9nURS/5nGO(URS:XaKUR(d [ S5eURRURUR S9nOURS:XaVUR(d [ S 5e[URRUR/UR S95/nOgURS :Xa>URRUR RS 5UR S 9nO[ S URS35eUHnURS:XdURS:XaUn0nOTURS5=(d* URS5=(d URSS5nURS05nU(dM[UUS9v M g7f)NrJ)r rFzURL is required for crawl modedatarKzURL is required for map moder=z URL is required for extract moderLquery)rXr rOzJ'. Allowed: 'crawl', 'scrape', 'map', 'extract', 'search'.r*r;r<metadata) page_contentrZ)r,rQ scrape_urlrIr r rS crawl_urlr&rmap_urlstrr=rLr)r!firecrawl_docscrawl_responsedocr[rZs r% lazy_loadFireCrawlLoader.lazy_loads& 99 ))HHT%G%G %T*N YY' !88 !ABB!^^55!D!DT[[!Q6N,//;N YY% 88 !?@@!^^33DHHT[[3QN YY) #88 !CDDDNN**DHH:dkk*JKNYY( "!^^22kkoog.{{3N "499+.HK "CyyE!TYY)%;" GGJ'T3776?TcggiQS>T77:r2)! "sJJ)rQr,r rI)__name__ __module__ __qualname____firstlineno____doc__rr&r r_rrrTrrrc__static_attributes__r(r%r r s7r.T.d.`jDjTj`"&!%GN!%/# /## /# # /# CD /#/#b08H-0r(r ) rtypingrrrlangchain_core.document_loadersrlangchain_core.documentsrlangchain_core.utilsrr rkr(r%rps%..6--wjwr(