dhSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r J r SSK JrJrJrJrJrJrJrJrJrJr SSKJr SSKrSSKJr SSKJr SS KJ r SS K!J"r" SS K#J$r$ SS K%J&r& SS K'J(r(J)r)J*r*J+r+J,r,J-r-J.r.J/r/ SSK0J1r1 \(aSSK2J3r3 \Rh"\55r6"SS\15r7"SS\ \5r8"SS\85r9"SS\85r:"SS\85r;"SS\ 5r<"SS\85r="SS\85r>"S S!\85r?"S"S#\85r@"S$S%\85rA"S&S'\85rB"S(S)\$5rC"S*S+\85rD"S,S-\85rE\:rFg).N)ABC)StringIO)PathPurePath) TYPE_CHECKINGAnyBinaryIOIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)get_from_dict_or_env) BaseLoader)Blob)DedocBaseLoader)BaseImageBlobParser)_DEFAULT_PAGES_DELIMITERAmazonTextractPDFParserDocumentIntelligenceParserPDFMinerParserPDFPlumberParser PyMuPDFParserPyPDFium2Parser PyPDFParser)UnstructuredFileLoader)TextLinearizationConfigcZ^\rSrSrSrS S\\\4S\S\4U4Sjjjr S\ 4Sjr S r U=r $) UnstructuredPDFLoader2aLoad `PDF` files using `Unstructured`. You can run the loader in one of two modes: "single" and "elements". If you use "single" mode, the document will be returned as a single langchain Document object. If you use "elements" mode, the unstructured library will split the document into elements such as Title and NarrativeText. You can pass in additional unstructured kwargs after mode to apply different unstructured settings. Examples -------- from langchain_community.document_loaders import UnstructuredPDFLoader loader = UnstructuredPDFLoader( "example.pdf", mode="elements", strategy="fast", ) docs = loader.load() References ---------- https://unstructured-io.github.io/unstructured/bricks.html#partition-pdf file_pathmodeunstructured_kwargsc @>[U5n[TU]"SXS.UD6 g)z Args: file_path: The path to the PDF file to load. mode: The mode to use when loading the file. Can be one of "single", "multi", or "all". Default is "single". **unstructured_kwargs: Any kwargs to pass to the unstructured. )r%r&N)strsuper__init__)selfr%r&r' __class__s `/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_community/document_loaders/pdf.pyr,UnstructuredPDFLoader.__init__Js$ N  O9O;NOreturncJSSKJn U"SSUR0URD6$)Nr) partition_pdffilenamer))unstructured.partition.pdfr4r%r')r-r4s r/ _get_elements#UnstructuredPDFLoader._get_elements[s"<QdnnQ8P8PQQr1r))single)__name__ __module__ __qualname____firstlineno____doc__rr*rrr,listr7__static_attributes__ __classcell__r.s@r/r#r#2sU4Pd#PP # PP"RtRRr1r#c\rSrSrSrSS.S\\\4S\\ 4Sjjr SS jr \ S \S\ 4S j5r\ S \S\ 4S j5r\ S \S\ 4S j5r\S\4Sj5rSrg) BasePDFLoaderazBase Loader class for `PDF` files. If the file is a web path, it will download it to a temporary file, use it, then clean up the temporary file after completion. Nheadersr%rGc&[U5UlSUlX lSUR;a.[R R UR5Ul[R RUR5(GdURUR5(Ga[R"5Ul [R RUR5up4URUR5(a1[UR5R RS5Sn[R R!URR"SU35nURUlUR%UR5(d[&R("URURS9nUR*S:wa[-SUR*-5e[/US S 9nUR1UR25 SSS5 [U5Ulgg[R RUR5(d[-S UR-5eg!,(df  Ng=f) zInitialize with a file path. Args: file_path: Either a local, S3 or web path to a PDF file. headers: Headers to use for GET request to download a file from a web path. N~/tmprFz3Check the url of your file; returned status code %swb)r&z'File path %s is not a valid file or url)r*r%web_pathrGospath expanduserisfile _is_valid_urltempfileTemporaryDirectorytemp_dirsplitext_is_s3_presigned_urlrsplitjoinname _is_s3_urlrequestsget status_code ValueErroropenwritecontent)r-r%rG_suffixtemp_pdfrfs r/r,BasePDFLoader.__init__hsY  $.. WW//?DNww~~dnn--$2D2DT^^2T2T$779DM((8IA((88!$..166<>22LLF==C'$M--( (.!GGAII&/!$X3//FWX X0/.s J Jr2c\[US5(aURR5 gg)NrW)hasattrrWcleanupr-s r/__del__BasePDFLoader.__del__s$ 4 $ $ MM ! ! # %r1urlcz[U5n[UR5=(a [UR5$)zCheck if the url is valid.)rboolnetlocscheme)rqparseds r/rTBasePDFLoader._is_valid_urls*#FMM":tFMM'::r1c[U5nURS:XaUR(agg![a gf=f)zcheck if the url is S3s3TF)rrurtrarqresults r/r]BasePDFLoader._is_s3_urls: c]F}}$  s ,0 ==c[U5n[[R"SUR55$![ a gf=f)z'Check if the url is a presigned S3 url.z\.s3\.amazonaws\.com$F)rrsresearchrtrarzs r/rY"BasePDFLoader._is_s3_presigned_urls= c]F ":FMMJK K  s47 AAcLURb UR$UR$N)rOr%rns r/sourceBasePDFLoader.sources $ 9t}}Mt~~Mr1)r%rGrWrO)r2N)r:r;r<r=r>rr*rr dictr,ro staticmethodrsrTr]rYpropertyrr@r)r1r/rDrDasMQ$YsH}-$Y;CD>$YL$;3;4;; #$NNNr1rDc,\rSrSrSrS\\4SjrSrg)OnlinePDFLoaderzLoad online `PDF`.r2c^[[UR55nUR5$)zLoad documents.)r#r*r%load)r-loaders r/rOnlinePDFLoader.loads!&s4>>':;{{}r1r)N) r:r;r<r=r>r?rrr@r)r1r/rrsd8nr1rc^\rSrSrSrSSSS\SSS.S\\\4S \ \\\ 4S \ \ S \ S \ S S\ \S\ SS\S\ SS\ \ SS4U4SjjjjrS\\4SjrSrU=r$) PyPDFLoaderaLoad and parse a PDF file using 'pypdf' library. This class provides methods to load and parse PDF documents, supporting various configurations such as handling password-protected files, extracting images, and defining extraction mode. It integrates the `pypdf` library for PDF processing and offers both synchronous and asynchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdf Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PyPDFLoader loader = PyPDFLoader( file_path = "./example_data/layout-parser-paper.pdf", # headers = None # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_parser = RapidOCRBlobParser(), ) Lazy load documents: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) Npagetextplain)r& images_parserimages_inner_formatpages_delimiterextraction_modeextraction_kwargsr%passwordrGextract_imagesr&r9rrrrz markdown-imgzhtml-imgrrrlayoutrr2c J>[T U]XS9 [UUUUUUU U S9Ulg)u<Initialize with a file path. Args: file_path: The path to the PDF file to be loaded. headers: Optional headers to use for GET request to download a file from a web path. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extraction_mode: “plain” for legacy functionality, “layout” extract text in a fixed width format that closely adheres to the rendered layout in the source pdf extraction_kwargs: Optional additional parameters for the extraction process. Returns: This method does not directly return data. Use the `load`, `lazy_load` or `aload` methods to retrieve parsed documents with content and metadata. rF)rr&rrrrrrN)r+r,rparser) r-r%rrGrr&rrrrrr.s r/r,PyPDFLoader.__init__s:V 4!)' 3++/   r1c#<# UR(aB[R"[URS5R 5URS9nO [R "UR5nURRU5ShvN gN7fz Lazy load given path as pages. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. rbrQN rOr from_datarbr%read from_pathr lazy_parser-blobs r/ lazy_loadPyPDFLoader.lazy_load%` ==>>$t~~t"<"A"A"C$--XD>>$..1D;;))$///BBBBr)NNF)r:r;r<r=r>rrr*rr bytesrrsr rr,r rrr@rArBs@r/rrs1l15"&$ 5 +17;KQ76=,05 h'5 5e,-5 $ 5  5 &'5  345 %%GH5 5 !!235 $D>5  5 5 n 0 (  0 0r1rc^\rSrSrSrS\SSSSSS.S\\\4S \ S S \S \ \S \ S\ \ S\ SS\ \ 4U4SjjjrS\\4SjrSrU=r$)PyPDFium2Loaderi4aLoad and parse a PDF file using the `pypdfium2` library. This class provides methods to load and parse PDF documents, supporting various configurations such as handling password-protected files, extracting images, and defining extraction mode. It integrates the `pypdfium2` library for PDF processing and offers both synchronous and asynchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdfium2 Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PyPDFium2Loader loader = PyPDFium2Loader( file_path = "./example_data/layout-parser-paper.pdf", # headers = None # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazy load documents: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) rNFr)r&rrrrrrGr%r&rrrrrrrrGc F>[T U]XS9 [UUUUUUS9Ulg)a&Initialize with a file path. Args: file_path: The path to the PDF file to be loaded. headers: Optional headers to use for GET request to download a file from a web path. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) Returns: This class does not directly return data. Use the `load`, `lazy_load` or `aload` methods to retrieve parsed documents with content and metadata. rF)r&rrrrrN)r+r,rr) r-r%r&rrrrrrGr.s r/r,PyPDFium2Loader.__init__is4H 4%)' 3+   r1r2c#<# UR(aB[R"[URS5R 5URS9nO [R "UR5nURRU5ShvN gN7fr) rOrrrbr%rrrparsers r/rPyPDFium2Loader.lazy_loads` ==>>$t~~t"<"A"A"C$--XD>>$..1D;;$$T***rr)r:r;r<r=r>rrr*rr r rsrrr,r rrr@rArBs@r/rr4s2p+17"&$7;KQ"&, h', &' ,  , 3- , ,  34, %%GH, $, , \ + (  + +r1rc\rSrSrSrSSSSSSSS.S\\\4S\S \S \S \S \S \ \S\ SS\ \ S\ \ S\ SS\ \ 4Sjjjr \S\S\4Sj5rS\\4SjrSrg)PyPDFDirectoryLoaderiakLoad and parse a directory of PDF files using 'pypdf' library. This class provides methods to load and parse multiple PDF documents in a directory, supporting options for recursive search, handling password-protected files, extracting images, and defining extraction modes. It integrates the `pypdf` library for PDF processing and offers synchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdf Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PyPDFDirectoryLoader loader = PyPDFDirectoryLoader( path = "./example_data/", glob = "**/[!.]*.pdf", silent_errors = False, load_hidden = False, recursive = False, extract_images = False, password = None, mode = "page", images_to_text = None, headers = None, extraction_mode = "plain", # extraction_kwargs = None, ) Load documents: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) Nrr)rr&rrGrrrQglob silent_errors load_hidden recursiverrr&rrrGrrrcXplXlXlX lX@lXPlX0lX`lXlXl Xl Xl g)uIInitialize with a directory path. Args: path: The path to the directory containing PDF files to be loaded. glob: The glob pattern to match files in the directory. silent_errors: Whether to log errors instead of raising them. load_hidden: Whether to include hidden files in the search. recursive: Whether to search subdirectories recursively. extract_images: Whether to extract images from PDFs. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for extracting the entire document or "page" for page-wise extraction. images_parser: Optional image blob parser.. headers: Optional headers to use for GET request to download a file from a web path. extraction_mode: “plain” for legacy functionality, “layout” for experimental layout mode functionality extraction_kwargs: Optional additional parameters for the extraction process. Returns: This method does not directly return data. Use the `load` method to retrieve parsed documents with content and metadata. N) rr&rQrrrrrrrGrr) r-rQrrrrrrr&rrGrrs r/r,PyPDFDirectoryLoader.__init__sGP!    &"*,* .!2r1r2cD[SUR55(+$)Nc3@# UHoRS5v M g7f).N) startswith).0parts r/ 3PyPDFDirectoryLoader._is_visible..sC s++ s)anypartsrs r/ _is_visible PyPDFDirectoryLoader._is_visiblesC CCCCr1c 8[UR5n/nUR(aURUR5OUR UR5nUHnUR 5(dMUR URU55(dUR(dMR[[U5URURURURURUR UR"S9nUR%5nUHn[U5UR&S'M UR)U5 M U$![*a4nUR,(a[.R1U5 SnAGM1UeSnAff=f)N)rr&rrrGrrr)rrQrrglobris_filer relative_torrr*rr&rrrGrrrmetadataextend Exceptionrloggerwarning) r-pdocsitemsirsub_docsdoces r/rPyPDFDirectoryLoader.loads/ O&*nn "!&&:KAyy{{##AMM!$4559I9I9I$!,F%)]]!%+/+>+>*.*<*<$(LL,0,@,@.2.D.D "$*;;=#+C58VCLL2$, H-#. %$--"NN1--"#G $s4B!E F%&FFF) rrrrrGrrr&rrQrr)z **/[!.]*.pdfFFFF)r:r;r<r=r>rr*rrsr r rrr,rrr?rrr@r)r1r/rrs 2n##!$33#'*07;"&6=,033CM"3333 33  33  33333-33&'33 3433$33!!2333$D>33jD(DtDDd8nr1rc^\rSrSrSrSS\SSSSSS.S\\\4S \ \S \ S S \S \ S\ \ S\ SS\ \ S\ \ SS4U4SjjjrS\\4SjrSrU=r$)PDFMinerLoaderi2aLoad and parse a PDF file using 'pdfminer.six' library. This class provides methods to load and parse PDF documents, supporting various configurations such as handling password-protected files, extracting images, and defining extraction mode. It integrates the `pdfminer.six` library for PDF processing and offers both synchronous and asynchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pdfminer.six Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PDFMinerLoader loader = PDFMinerLoader( file_path = "./example_data/layout-parser-paper.pdf", # headers = None # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazy load documents: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) Nr9Fr)rr&rrrrrGconcatenate_pagesr%rr&rrrrrrrGrr2c H>[T U]XS9 [UUUU UUUS9Ulg)aInitialize with a file path. Args: file_path: The path to the PDF file to be loaded. headers: Optional headers to use for GET request to download a file from a web path. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) concatenate_pages: Deprecated. If True, concatenate all PDF pages into one a single document. Otherwise, return one document per page. Returns: This method does not directly return data. Use the `load`, `lazy_load` or `aload` methods to retrieve parsed documents with content and metadata. rF)rrrrr&rrN)r+r,rr) r-r%rr&rrrrrGrr.s r/r,PDFMinerLoader.__init__fs7N 4$)'/+ 3  r1c#<# UR(aB[R"[URS5R 5URS9nO [R "UR5nURRU5ShvN gN7frrrs r/rPDFMinerLoader.lazy_loadrrr)r:r;r<r=r>rrr*rr r rsrrr,r rrr@rArBs@r/rr2s1n#'*27$7;KQ"&,00 h'0 3- 0 &' 0  0 0  340 %%GH0 $0 $D>0  0 0 d 0 (  0 0r1rcd^\rSrSrSrSS.S\\\4S\\ 4U4Sjjjr S\ \ 4S jr S rU=r$) PDFMinerPDFasHTMLLoaderiz2Load `PDF` files as HTML content using `PDFMiner`.NrFr%rGc`>SSKJn [TU]XS9 g![a [S5ef=f)Initialize with a file path.rextract_text_to_fpzO`pdfminer` package not found, please install it with `pip install pdfminer.six`rFN)pdfminer.high_levelr ImportErrorr+r,)r-r%rGrr.s r/r, PDFMinerPDFasHTMLLoader.__init__s@  > 4  -  s-r2c #l# SSKJn SSKJn SSKJn [ 5nU"URS5nU"[[U5USU"5SS9 S S S 5 S URc[UR5O UR0n[UR5US 9v g !,(df  NY=f7f) Load file.rr)LAParams) open_filenamerhtml)codeclaparams output_typeNr page_contentr)rrpdfminer.layoutrpdfminer.utilsrrr%rr rOr*rgetvalue)r-rrr output_stringfprs r/r!PDFMinerPDFasHTMLLoader.lazy_loads:,0 4>>4 0B Xr"!"  1 T]]-Bc$..)  M$:$:$rr*rr rr,r rrr@rArBs@r/rrsM<MQ 5sH}- 5;CD> 5 5Q8H-QQr1rc^\rSrSrSrSS\SSSSSSS. S\\\4S \ \S \ S S \S \ S\ \ S\ SS\\ SS4S\ \ S\ \ \\4S\SS4U4SjjjrS\S\\4SjrS\S\\4SjrS\\4SjrSrU=r$) PyMuPDFLoaderia3Load and parse a PDF file using 'PyMuPDF' library. This class provides methods to load and parse PDF documents, supporting various configurations such as handling password-protected files, extracting tables, extracting images, and defining extraction mode. It integrates the `PyMuPDF` library for PDF processing and offers both synchronous and asynchronous document loading. Examples: Setup: .. code-block:: bash pip install -U langchain-community pymupdf Instantiate the loader: .. code-block:: python from langchain_community.document_loaders import PyMuPDFLoader loader = PyMuPDFLoader( file_path = "./example_data/layout-parser-paper.pdf", # headers = None # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_parser = TesseractBlobParser(), # extract_tables = "markdown", # extract_tables_settings = None, ) Lazy load documents: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) Load documents asynchronously: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) NrFr) rr&rrrrextract_tablesrGextract_tables_settingsr%rr&rrrrrrr)csvmarkdownrrGrkwargsr2c n>US;a [S5e[T U] XS9 [UUUU UUUUU S9 Ulg)aInitialize with a file path. Args: file_path: The path to the PDF file to be loaded. headers: Optional headers to use for GET request to download a file from a web path. password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extract_tables: Whether to extract tables in a specific format, such as "csv", "markdown", or "html". extract_tables_settings: Optional dictionary of settings for customizing table extraction. **kwargs: Additional keyword arguments for customizing text extraction behavior. Returns: This method does not directly return data. Use the `load`, `lazy_load`, or `aload` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the `mode` argument is not one of "single" or "page". rzmode must be single or pagerF) rr&r text_kwargsrrrrrN)rar+r,rr) r-r%rr&rrrrrrGrrr.s r/r,PyMuPDFLoader.__init__sR` ) ):; ; 4#+)' 3)$;   r1c+|# U(a[RSUS35 URnUR(aB[R "[ URS5R5URS9nO [R"UR5nURX1S9ShvN gN7f)zLazy load given path as pages or single document (see `mode`). Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. zReceived runtime arguments zd. Passing runtime args to `load` is deprecated. Please pass arguments during initialization instead.rr)r N) rrrrOrrrbr%rr _lazy_parse)r-rrrs r/ _lazy_loadPyMuPDFLoader._lazy_loadCs  NN-fX6WX  ==>>$t~~t"<"A"A"C$--XD>>$..1D%%d%???sB2B<4B:5B<c 8[UR"S0UD65$)Nr))r?r )r-rs r/rPyMuPDFLoader.loadTsDOO-f-..r1c#@# UR5ShvN gN7fr)r rns r/rPyMuPDFLoader.lazy_loadWs??$$$s r)r:r;r<r=r>rrr*rr r rsrrrr,r rr r?rrr@rArBs@r/rrs/4t#'*07$7;KQJN"&<@= h'= 3- = &' =  = =  34= %%GH= g&?@$FG= $= "*$sCx.!9= =  = = ~@3@8H+=@"/S/T(^/%8H-%%r1rc^\rSrSrSrSS\\\4S\S\S\ S\ \ \\ 4S \ S S4U4S jjjr \S \ \\44S j5r\S \4S j5r\S \ 4Sj5rS \4SjrS\S S4SjrS\S \4SjrS\S \4SjrS \\4SjrSrU=r$)MathpixPDFLoaderi]z)Load `PDF` files using `Mathpix` service.Nr%processed_file_formatmax_wait_time_secondsshould_clean_pdfextra_request_datarr2c >[USS5Ul[USS5UlURSS5 URSS5 [TU]"U40UD6 X lUbUO0UlX0lX@l g)aInitialize with a file path. Args: file_path: a file for loading. processed_file_format: a format of the processed file. Default is "md". max_wait_time_seconds: a maximum time to wait for the response from the server. Default is 500. should_clean_pdf: a flag to clean the PDF file. Default is False. extra_request_data: Additional request data. **kwargs: additional keyword arguments. mathpix_api_keyMATHPIX_API_KEYmathpix_api_idMATHPIX_API_IDN) rrrpopr+r,rrrr)r-r%rrrrrr.s r/r,MathpixPDFLoader.__init__`s( 4 %'8 3 $&6   $d+ #T* -f-%:""4"@ b &;" 0r1c4URURS.$)N)app_idapp_key)rrrns r/_mathpix_headers!MathpixPDFLoader._mathpix_headerss--$:N:NOOr1cg)Nzhttps://api.mathpix.com/v3/pdfr)rns r/rqMathpixPDFLoader.urls/r1cjSURS00UREnS[R"U50$)Nconversion_formatsT options_json)rrjsondumps)r-optionss r/dataMathpixPDFLoader.datas? !4#=#=t"D %%  7 344r1cb[[UR5S5nSU0n[R"UR UR X RS9nSSS5 WR5nSU;a[SUS35eSU;aUSnU$[S5e!,(df  NM=f)Nrfile)rGfilesr-errorzMathpix request failed: pdf_idzUnable to send PDF to Mathpix.) rbr*r%r^postrqr#r-r*ra)r-rir1response response_datar3s r/send_pdfMathpixPDFLoader.send_pdfs #dnn%t ,QKE}}$"7"7u99H- !  m #7 g8N7OPQ Q } $"8,FM=> >- ,s 9B  B.r3cURS-U-n[SURS5Hn[R"X R S9nUR 5nUR SS5nUR SS5nUbSU3nUb US US S 3- n[U5eUR S S5n U S :Xa gU S:Xa [S5e[RSU 5 [R"S5 M [e)zMWait for processing to complete. Args: pdf_id: a PDF id. Returns: None rJrrFr2N error_infoz%Unable to retrieve PDF from Mathpix: z (id)status completedz#Unable to retrieve PDF from Mathpixz.Status: %s, waiting for processing to complete) rqrangerr^r_r#r*rarinfotimesleep TimeoutError) r-r3rqrer5r6r2r; error_msgr>s r/wait_for_processing$MathpixPDFLoader.wait_for_processingshhnv%q$44a8A||C1F1FGH$MMOM"%%gt4E&**<>J CE7K )2j&6%7q!99I ++"&&x6F$7" !FGG LfU 1 394r1cURU5 URSUSUR3n[R"X R S9nUR RS5$)NrJrrFzutf-8)rFrqrr^r_r#rddecode)r-r3rqr5s r/get_processed_pdf"MathpixPDFLoader.get_processed_pdfs\   ( !F81T%?%?$@A<<-B-BC&&w//r1contentscfSRURS5Vs/sHo"RS5(aMUPM sn5nURSS5RSS5nURSS5RS S 5RS S 5RS S5nU$s snf)zIClean the PDF file. Args: contents: a PDF file contents. Returns:  z![]z \section{z# }rz\$$z\%%z\((z\)r=)r[rZrreplace)r-rLlines r/ clean_pdfMathpixPDFLoader.clean_pdfs99&nnT2 Q2d//%:PT2 Q ##L$7??RH   UC ( WUC WUC WUC   Rs B.B.cUR5nURU5nUR(aURU5nURURUS.n[ X#S9/$)N)rr%r3r)r7rJrrUrr)r-r3rLrs r/rMathpixPDFLoader.loadsV))&1  ~~h/H"kk vVhBCCr1)rrrrrr)mdiFN)r:r;r<r=r>rr*rintrsr rrr,rr#rqr-r7rFrJrUr?rrr@rArBs@r/rr]s33 &*%(!&7; %1h'%1 #%1 # %1  %1 %T#s(^4 %1%1 %1%1NP$sCx.PP0S005d55 ?# ?###$#J000 ##.Dd8nDDr1rc ^\rSrSrSrS S\\\4S\\ \\ 4S\ S\\ S\ S S4 U4S jjjr S \\4S jrS rU=r$)PDFPlumberLoaderiz$Load `PDF` files using `pdfplumber`.Nr%r deduperGrr2c>SSKn[TU] XS9 U=(d 0UlX0lXPlg![a [S5ef=f)rrNzMpdfplumber package not found, please install it with `pip install pdfplumber`rF) pdfplumberrr+r,r r]r)r-r%r r]rGrr_r.s r/r,PDFPlumberLoader.__init__sX   4&," , +  s 1Ac\[URURURS9nUR(aB[ R "[URS5R5URS9nO [ R"UR5nURU5$)r)r r]rrr) rr r]rrOrrrbr%rrr)r-rrs r/rPDFPlumberLoader.load sy"((;;..  ==>>$t~~t"<"A"A"C$--XD>>$..1D||D!!r1)r]rr )NFNF)r:r;r<r=r>rr*rr r rrsrr,r?rrr@rArBs@r/r\r\s. 48"&$ -h'-gc3h/0- - $ -  - --, "d8n " "r1r\c^\rSrSrSrSSS.S\\\4S\\ \S\\ S\\S \\S \\S \\ S \S SS4U4Sjjjjr S\ \4SjrS\\4Sjr\S\S\4Sj5rSrU=r$)AmazonTextractPDFLoaderiaLoad `PDF` files from a local file system, HTTP or S3. To authenticate, the AWS client uses the following methods to automatically load credentials: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html If a specific credential profile should be used, you must pass the name of the profile from the ~/.aws/credentials file that is to be used. Make sure the credentials / roles used have the required policies to access the Amazon Textract service. Example: .. code-block:: python from langchain_community.document_loaders import AmazonTextractPDFLoader loader = AmazonTextractPDFLoader( file_path="s3://pdfs/myfile.pdf" ) document = loader.load() N)linearization_configr%textract_featuresclientcredentials_profile_name region_name endpoint_urlrGrer!r2c >[TU]XS9 SSKn U(aUV s/sHoRU PM n n O/n U(dU(dU(aRSSKn UbU R US9n OU R 5n 0nU(aX^S'U(aXnS'U R"S 0UD6n[U UUS 9Ul g![a [S5ef=fs sn f![a [S5e[an[S U35UeSnAff=f) aInitialize the loader. Args: file_path: A file, url or s3 path for input file textract_features: Features to be used for extraction, each feature should be passed as a str that conforms to the enum `Textract_Features`, see `amazon-textract-caller` pkg client: boto3 textract client (Optional) credentials_profile_name: AWS profile name, if not default (Optional) region_name: AWS region, eg us-east-1 (Optional) endpoint_url: endpoint url for the textract service (Optional) linearization_config: Config to be used for linearization of the output should be an instance of TextLinearizationConfig from the `textractor` pkg rFrNztCould not import amazon-textract-caller python package. Please install it with `pip install amazon-textract-caller`.) profile_namerirjzRCould not import boto3 python package. Please install it with `pip install boto3`.zCould not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid. )rfrgre)textract) r+r,textractcallerrTextract_Featuresboto3Sessionrgrrarr)r-r%rfrgrhrirjrGretcxfeaturesrpsession client_paramsrr.s r/r, AmazonTextractPDFLoader.__init__/s86 4  ' 9JK9JA,,Q/9JHKHH #{l +7#mm9QmRG$mmoG " 3>-04@.1 DmD.&!5  O O   L, !B  //0c3  s)B3C AC3C D/C>>Dc4[UR55$zLoad given path as pages.r?rrns r/rAmazonTextractPDFLoader.load{DNN$%%r1c## UR(a4URUR5(a[URS9nOR[R"UR5n[ R U5S:a[SURS35eURRU5ShvN gN7f)zLazy load documentsrz the file z is a multi-page document, but not stored on S3. Textract requires multi-page documents to be on S3.N) rOr]rrr%rd_get_number_of_pagesrarQrrrs r/r!AmazonTextractPDFLoader.lazy_loads ==T__T]];;T]]+D>>$..1D&;;DAAE  {+HI ;;$$T***sB6C8B>9CrcSSKnSSKJnJn UR S:Xa@UR 5nURU5n[UR5sSSS5 $UR S:XaLSnURUR55n[URU55H u nUS- nM U$UR S;ag[SUR 35e![a [ S5ef=f!,(df  g=f) Nr)Image ImageSequencezcCould not import pypdf or Pilloe python package. Please install it with `pip install pypdf Pillow`.zapplication/pdfz image/tiffr~)z image/pngz image/jpegzunsupported mime type: )pypdfPILrrrmimetype as_bytes_io PdfReaderlenpagesrbas_bytes enumerater ra) rrrrinput_pdf_file pdf_reader num_pagesimgres r/r,AmazonTextractPDFLoader._get_number_of_pagess   0 ==- -!!#~"__^< :++,$#]]l *I**T]]_-C!-"8"8"=>1Q ?  ]]9 96t}}oFG G% E   $#s C#&C<#C9< D r)NNNNNN)r:r;r<r=r>rr*rr rrrr,r?rrr rrrrZrr@rArBs@r/rdrds 06: $26%)&*"&J EIJ h'J $HSM2J  J #+3- J c] J smJ $J ''@AJ  J J X&d8n&+ ( +*H4HCHHr1rdc&\rSrSrSrS\4SjrSrg)DedocPDFLoaderia DedocPDFLoader document loader integration to load PDF files using `dedoc`. The file loader can automatically detect the correctness of a textual layer in the PDF document. Note that `__init__` method supports parameters that differ from ones of DedocBaseLoader. Setup: Install ``dedoc`` package. .. code-block:: bash pip install -U dedoc Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocPDFLoader loader = DedocPDFLoader( file_path="example.pdf", # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/pdf_handling.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing is_one_column_document: detect number of columns for PDF without a textual layer, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result need_binarization: clean pages background (binarize) for PDF without a textual layer need_pdf_table_analysis: parse tables for PDF without a textual layer r2clSSKJn U"[UR5URUR S9$)Nr)make_manager_pdf_config)r%parsing_paramsrZ)dedoc.utils.langchainrr*r%parsing_parametersrZ)r-rs r/ _make_configDedocPDFLoader._make_configs.A&$..)22**  r1r)N)r:r;r<r=r>rrr@r)r1r/rrsTl d r1rc ^\rSrSrSrS S\\\4S\S\S\ \ SS4 U4S jjjr S\ \ 4S jrS\\ 4S jrS rU=r$)DocumentIntelligenceLoaderiz+Load a PDF with Azure Document IntelligenceNr%rgmodelrGr2c<>[TU]XS9 [X#S9Ulg)aInitialize the object for file processing with Azure Document Intelligence (formerly Form Recognizer). This constructor initializes a DocumentIntelligenceParser object to be used for parsing files using the Azure Document Intelligence API. The load method generates a Document node including metadata (source blob and page number) for each page. Parameters: ----------- file_path : str The path to the file that needs to be parsed. client: Any A DocumentAnalysisClient to perform the analysis of the blob model : str The model name or ID to be used for form recognition in Azure. Examples: --------- >>> obj = DocumentIntelligenceLoader( ... file_path="path/to/file", ... client=client, ... model="prebuilt-document" ... ) rF)rgrN)r+r,rr)r-r%rgrrGr.s r/r,#DocumentIntelligenceLoader.__init__s"B 40L r1c4[UR55$ryrzrns r/rDocumentIntelligenceLoader.load7r|r1c## [R"UR5nURR U5ShvN gN7f)zLazy load given path as pages.N)rrr%rrrs r/r$DocumentIntelligenceLoader.lazy_load;s1~~dnn-;;$$T***s?A AA r)zprebuilt-documentN)r:r;r<r=r>rr*rrr rr,r?rrr rr@rArBs@r/rrs5 )"& "Mh'"M"M "M $ "M  "M"MH&d8n&+ ( ++r1rc d^\rSrSrSrS S\\\4S\S\SS4U4Sjjjr S\ \ 4S jr S r U=r$) ZeroxPDFLoaderiCaDocument loader utilizing Zerox library: https://github.com/getomni-ai/zerox Zerox converts PDF document to series of images (page-wise) and uses vision-capable LLM model to generate Markdown representation. Zerox utilizes anyc operations. Therefore when using this loader inside Jupyter Notebook (or any environment running async) you will need to: ```python import nest_asyncio nest_asyncio.apply() ``` r%r zerox_kwargsr2Nc :>[TU]US9 X0lX lg)N)r%)r+r,rr)r-r%rrr.s r/r,ZeroxPDFLoader.__init__Ss' 9- $) r1c## SSKnSSKJn URU"S[ UR 5UR S.URD65n[UR5S:a[URSRnURH1n[URURURUS.S9v M3 gg7f) zLazily load pages.rN)zerox)r%rrK)rrrrr))asynciopyzeroxrrunr*r%rrrrrrrdr)r-rr zerox_outputrrs r/rZeroxPDFLoader.lazy_loados!{{  WC/tzz WTEVEV W |!! "Q &$**2.33I$**!%"&++ $ %.+ 'sCC)rr)z gpt-4o-mini)r:r;r<r=r>rr*rrr,r rrr@rArBs@r/rrCs\ $#h'   88H-r1r)Gr*loggingrPr~rUrBabcriorpathlibrrtypingrrr r r r r rrr urllib.parserr^langchain_core.documentsrlangchain_core.utilsr)langchain_community.document_loaders.baser1langchain_community.document_loaders.blob_loadersr*langchain_community.document_loaders.dedocr3langchain_community.document_loaders.parsers.imagesr0langchain_community.document_loaders.parsers.pdfrrrrrrrr1langchain_community.document_loaders.unstructuredr )textractor.data.text_linearization_configr! getLogger__file__rr#rDrrrrrrrrr\rdrrrPagedPDFSplitterr)r1r/rs_   "   "-5@BFS   UQ   8 $,R2,R^MNJMN`mw0-w0to+mo+dI:IXr0]r0j#Qm#QLK%MK%`QD}QDh%"}%"PSHmSHl^ _^ B0+0+fB]BLr1