dhQSSKrSSKrSSKrSSKJrJr SSKJrJrJ r J r J r SSK J r SSKJr "SS\\5r"SS \5r"S S \5rg) N)ABCabstractmethod)DictIteratorOptionalTupleUnion)Document) BaseLoaderc!Z\rSrSrSrSSSSSSS S S SSSS S S .S \S\S\S\\\4S\S\S\S\S\S\S\\\4S\\\4S\\\4S\ \S\ \SS 4 Sjjr S\ \ 4Sjr \S\4Sj5rS \S\4S!jrS"\S#\S\ \ 4S$jrS*S"\S\S%\ \S\ \ 4S&jjrS'\S\\\44S(jrS)rg )+DedocBaseLoadera Base Loader that uses `dedoc` (https://dedoc.readthedocs.io). Loader enables extracting text, tables and attached files from the given file: * `Text` can be split by pages, `dedoc` tree nodes, textual lines (according to the `split` parameter). * `Attached files` (when with_attachments=True) are split according to the `split` parameter. For attachments, langchain Document object has an additional metadata field `type`="attachment". * `Tables` (when with_tables=True) are not split - each table corresponds to one langchain Document object. For tables, Document object has additional metadata fields `type`="table" and `text_as_html` with table HTML representation. documentTF auto_tabbyrus+eng:autoN)split with_tableswith_attachmentsrecursion_deep_attachmentspdf_with_text_layerlanguagepagesis_one_column_documentdocument_orientationneed_header_footer_analysisneed_binarizationneed_pdf_table_analysis delimiterencoding file_pathrrrrrrrrrrrr r!r"returnct[5R5VVs0sHunnUS;dMUU_M snnUl1SkUlX R;a[ SUSURS35eX lX0lXlUR S:XaSOSnUURS 'X@RS 'g s snnf) a[ Initialize with file path and parsing parameters. Args: file_path: path to the file for processing split: type of document splitting into parts (each part is returned separately), default value "document" "document": document text is returned as a single langchain Document object (don't split) "page": split document text into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document text into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document text into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV >selfrr#r>linenodepagerGot $ for `split`, but should be one of ``r(treelinearstructure_typeneed_content_analysisN)localsitemsparsing_parametersvalid_split_values ValueErrorrrr#)r&r#rrrrrrrrrrrr r!r"keyvaluer/s b/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_community/document_loaders/dedoc.py__init__DedocBaseLoader.__init__#sF%hnn.# . UGG CJ.#  #G // /ug++,A/  &"#'::#7X4B 01;K 78## s B4B4c## SSKnSSKJn U"UR 5S9nSUR SlUR5nURUR0URESU0ES 9nSSS5 URWR5R5URS 9ShvN g![a [S5ef=f!,(df  Nf=fN/7f) Lazily load documents.rN) DedocManagerzE`dedoc` package not found, please install it with `pip install dedoc`)manager_configTloggerattachments_dir)r# parameters document_treer)tempfilededocr= ImportError _make_configconfigdisabledTemporaryDirectoryparser#r3_split_document to_api_schemadictr)r&rDr= dedoc_managertmpdirrCs r8 lazy_loadDedocBaseLoader.lazy_loadws  * %D4E4E4GH 26 X&/  ( ( *f)//..Qd55Q7H&Q0M+ '''557<<>djj(    W  + *  s@C$B88C$+C1AC$2C"3C$8CC$ CC$cg)z] Make configuration for DedocManager according to the file extension and parsing parameters. Nr&s r8rGDedocBaseLoader._make_configs paragraphcSRUSVs/sHnURU5PM sn5nU(a USSU3nU$USnU$s snf)z1Get text (recursively) of the document tree node. subparagraphstext)join _json2txt)r&rX subparagraphsubparagraphs_textr\s r8r^DedocBaseLoader._json2txts!YY%.o$> $>L|,$>  " !$6#7 8  6"   sArCdocument_metadatac## [US5S:a$USHnURX2S9ShvN M g[US0UEUSES9v gN 7f)z4Parse recursively document tree obtained by `dedoc`.r[rrCrbNr\metadata page_contentre)len_parse_subparagraphsr )r&rCrbr_s r8ri$DedocBaseLoader._parse_subparagraphssw }_- . 2 -o > 44".5!? *62K-Kz1JK  s-AA!Aadditional_metadatac## USnU(a0UEUEnUS:Xa"URUSSS9n[XTS9v GOUS:XaUSSSnUS SS nS nUHOn U SS U:XaXRU 5- nM$[U0UES U0ES9v U SS nURU 5nMQ [U0UES U0ES9v O~US :Xa7USSSH'n U Sn [URU 50UEU ES9v M) OAUS :XaURUSSUS9ShvN O[SUSURS35eUR (a8USSH,n UR U 5up[U 0U SESU S.ES9v M. USH(nURUURSS0S9ShvN M* gNN 7f)z=Split document into parts according to the `split` parameter.rercontent structure)rXrfr)r[rpage_idr'r(rdNr*r+r,tablestable)type text_as_html attachmentsrs attachment)rCrrk) r^r rir5r4r _get_tablerLr)r&rCrrkrbr\nodesro page_textr( line_metadatarr table_text table_htmlrvs r8rLDedocBaseLoader._split_documentst**5  L#4 L8K L  J >>M),D[,Q>RDI I f_!),[9/JEAhz*95GI #I.'9!55I"%.!J$5!Jy'!J#:.y9G $t 4I&B-By'B  f_%i0=oN $Z 0 !%!5C 1C]COf_00+I6{C"31   ug++,A/    &y1(;)-)?& !+ + '(2<( 6J++(jj%+\$:,  7/ 0 s%DGF< BG2F>3 G>Grrc SnUSH4nUH&nUSRSUS55- nUS- nM( US- nM6 SnUSHpnUS - nUH]nSRS US55n[R"U5nUS - nUS (aUS - nUSUSSUSSUS3- nM_ US- nMr US- nX%4$)z.Get text and HTML representation of the table.rpcells c3(# UH oSv M g7fr\NrT.0r's r8 -DedocBaseLoader._get_table..s&N F| lines rZzK z c3(# UH oSv M g7frrTrs r8rrs%M}t6l}rz z z
z
)r]htmlescape)r&rrr{rowcellr| cell_texts r8rwDedocBaseLoader._get_tables >Cchh&NW &NNN d"  $ J "  >C ( "J II%MtG}%MM  KK 2 e#  $";;J i 12Y(9+W>  ) #J" ** %%rW)r#r3rr4rN)__name__ __module__ __qualname____firstlineno____doc__strboolr intrr9rr rQrrNrGr^rirLrrw__static_attributes__rTrWr8r r s(  -2*,#/!&,$*8=.348#'"&#RLRL RL  RL  T * RL%(RL!RLRLRL!$RL"RL&+39%5RL!d+RL"'sDy!1RL C=!RL"3-#RL$ %RLh 8H- , d   4 C  ! 6:  (  &/3 III&d^ I (  IV&&sCx&rWr c&\rSrSrSrS\4SjrSrg)DedocFileLoaderia DedocFileLoader document loader integration to load files using `dedoc`. The file loader automatically detects the file type (with the correct extension). The list of supported file types is gives at https://dedoc.readthedocs.io/en/latest/index.html#id1. Please see the documentation of DedocBaseLoader to get more details. Setup: Install ``dedoc`` package. .. code-block:: bash pip install -U dedoc Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocFileLoader loader = DedocFileLoader( file_path="example.pdf", # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } r$cZSSKJn U"URURURS9$)Nr)make_manager_config)r#parsing_paramsr)dedoc.utils.langchainrr#r3r)r&rs r8rGDedocFileLoader._make_config`s)="nn22**  rWrTN)rrrrrrNrGrrTrWr8rrs@D d rWrc#^\rSrSrSrSSSSSSS S S S SSSS S S .S\S\S\S\S\\\4S\S\S\S\S\S\S\\\4S\\\4S\\\4S\ \S\ \SS 4"U4Sjjjr S\ \ 4S jr S\4S!jrS\S\S"\S\\\\\\444S#jrS$rU=r$)%DedocAPIFileLoaderijay Load files using `dedoc` API. The file loader automatically detects the file type (even with the wrong extension). By default, the loader makes a call to the locally hosted `dedoc` API. More information about `dedoc` API can be found in `dedoc` documentation: https://dedoc.readthedocs.io/en/latest/dedoc_api_usage/api.html Please see the documentation of DedocBaseLoader to get more details. Setup: You don't need to install `dedoc` library for using this loader. Instead, the `dedoc` API needs to be run. You may use Docker container for this purpose. Please see `dedoc` documentation for more details: https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker .. code-block:: bash docker pull dedocproject/dedoc docker run -p 1231:1231 Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocAPIFileLoader loader = DedocAPIFileLoader( file_path="example.pdf", # url=..., # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } zhttp://0.0.0.0:1231rTFrrrrrN)urlrrrrrrrrrrrr r!r"r#rrrrrrrrrrrrr r!r"r$cf>[TU]UUUUUUUU U U U U UUUS9 X lSURS'g)ad Initialize with file path, API url and parsing parameters. Args: file_path: path to the file for processing url: URL to call `dedoc` API split: type of document splitting into parts (each part is returned separately), default value "document" "document": document is returned as a single langchain Document object (don't split) "page": split document into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV )r#rrrrrrrrrrrr r!r"json return_formatN)superr9rr3)r&r#rrrrrrrrrrrrr r!r" __class__s r8r9DedocAPIFileLoader.__init__s\B #-'A 3#9!5(C/$;  "390rWc## URURURURS9nUR XR S9ShvN gN7f)r<)rr#rArBN) _send_filerr#r3rLr)r&doc_trees r8rQDedocAPIFileLoader.lazy_load sJ??DNNt?V?V# ''hjj'QQQsA AAAc0$rrTrUs r8rGDedocAPIFileLoader._make_configs rWrAcSSKn[RRU5n[ US5nSXV40nUR US3XsS9nSSS5 WR S:wa&[SURR535e[R"URR55n U $!,(df  Nt=f) z7Send POST-request to `dedoc` API and return the resultsrNrbfilez/upload)filesdatazError during file handling: ) requestsospathbasenameopenpost status_coder5rmdecoderloads) r&rr#rAr file_namerrrresults r8rDedocAPIFileLoader._send_files GG$$Y/ )T "di./E WoU LA# ==C ;AIIrsQ #.@H&j#H&VJ oJ ZxxrW