tj R d Z ddlmZ ddlZddlZddlZddlZddlZddlmZ ddl m
Z
ddl m Z ddl
mZmZmZmZmZmZmZmZmZmZmZ ddlmZ ddlZddlZ dd l m Z dd
l m Z dd l!m"Z" dd l#m$Z$m%Z% erddl&Z&ddl'Z'ddl(Z(ddl)Z)dd
l*m+Z+ g dZ,g dZ-d6dZ. ej/ e0 Z1dZ2dZ3dZ4dZ5h dZ6d7d Z7d8d Z8d8d!Z9d"d#gZ:d9d'Z; G d( d)e Z< G d* d+e Z= G d, d-e Z> G d. d/e Z? G d0 d1e Z@ G d2 d3e ZA G d4 d5e ZBdS ):z(Module contains common parsers for PDFs. ) annotationsN)datetime)Path)TemporaryDirectory)
TYPE_CHECKINGAnyBinaryIOIterableIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)BaseBlobParser)Blob)BaseImageBlobParserRapidOCRBlobParser)TextLinearizationConfig) DCTDecodeDCT JPXDecode)
LZWDecodeLZW FlateDecodeFl
ASCII85DecodeA85ASCIIHexDecodeAHxRunLengthDecodeRLCCITTFaxDecodeCCF JBIG2Decodeimages,Sequence[Union[Iterable[np.ndarray], bytes]]returnstrc ddl m} n # t $ r t d w xY w | }d}| D ]3} || }|r$|j r |d |j z
}4|S )zExtract text from images with RapidOCR.
Args:
images: Images to extract text from.
Returns:
Text extracted from images.
Raises:
ImportError: If `rapidocr` package is not installed.
r )RapidOCRzK`rapidocr` package not found, please install it with `pip install rapidocr`
)rapidocrr. ImportErrortxtsjoin)r) r. ocrtextimgresults n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/pdf.py!extract_from_images_with_rapidocrr: @ s
%%%%%%%
%
(**C
D + +S +fk + DIIfk** *D Ks #z
{image_text}
r0 z
> sourcecreatorproducer total_pages creationdateblobr contentformatc |rR| j pd}|dk r | dd }d| d| d}n#|dk r d t j |d
d | d
}|S )a Format the content of the image with the source of the blob.
blob: The blob containing the image.
format::
The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(`
`)
#z markdown-img]z\\]z![z]()zhtml-imgz
![T)quotez src=]()
)r; replacehtmlescape)r@ rA rB r; s r9 _format_inner_imagerK h s X# ^ # # ooc622G/7//f///GG
z
!
!W4;wd#C#C#CWW6WWWG N metadatadict[str, Any]c t | st d t | dd t st d | S )zValidate that the metadata has all the standard keys and the page is an integer.
The standard keys are:
- source
- total_page
- creationdate
- creator
- producer
Validate that page is an integer if it is present.
z3The PDF parser must valorize the standard metadata.pager z(The PDF metadata page must be a integer.)_STD_METADATA_KEYSissubsetkeys
ValueError
isinstancegetint)rM s r9 _validate_metadatarX ~ sg & &x}} 7 7 PNOOO hll61--s 3 3 ECDDD OrL c x i }ddd}| D ] \ }}t | t t fvrt | }| d r
|dd }| }|dv rV t
j | dd d
d ||< # t $ r |||< Y w xY w||v r|||| < |||< t |t r| ||< t |t r|||< |S ) zPurge metadata from unwanted keys and normalize key names.
Args:
metadata: The original metadata dictionary.
Returns:
The cleaned and normalized the key format of metadata dictionary.
r> r; )
page_count file_path/ N)r? moddate'r/ zD:%Y%m%d%H%M%S%zT)
itemstyper, rW
startswithlowerr strptimerH isoformatrT rU strip)rM new_metadatamap_keykvs r9 _purge_metadatarl sf $&L# G 1 773* $ $AA << !""A
GGII + + +
$"*"3IIc2&&(:# #)C.. Q
$
$
$"# Q
$
'\\'(L $ LOO
3
ggiiLOO
3
LO s ?CCCz
extras list[str]text_from_pagec dfd | |d
}|s=d }d t d
| }|rt d |z }||z }|S )a5 Insert extras such as image/table in a text between two paragraphs if possible,
else at the end of the text.
Args:
extras: List of extra content (images/tables) to insert.
text_from_page: The text content from the page.
Returns:
The merged text with extras inserted.
rn ro rp r, recursboolr+
Optional[str]c @ | rt D ]}| | }|dk rpd }|r | |d | d }|r|||d z }nEd}d t d | }|r||z }|d | |z ||d z } nd }n|}|S )NFr/ rm c | S N xs r9 zO_merge_text_and_extras.._recurs_merge_text_and_extras.. s ! rL )_PARAGRAPH_DELIMITERrfindr4 filter)
rn rp rr delimpos
previous_textall_text
all_extras
str_extras _recurs_merge_text_and_extrass
r9 r z=_merge_text_and_extras.._recurs_merge_text_and_extras s &-
$**511"99$(M (E(E "N4C4$8%) )
% #0>#$$3G#G%'
%+[[ V1L1L%M%M
% <).);J *4C4 0: =stt@T T ! E# & %HrL Tr/ rm c | S rx ry rz s r9 r| z(_merge_text_and_extras.. s ! rL rv )rn ro rp r, rr rs r+ rt )r4 r r} )rn rp r r r r s @r9 _merge_text_and_extrasr s < -,V^TJJH /
[[ V!J!J. OrL c J e Zd ZdZ d dedddddd! fdZd"d Zd#d Z xZS )$ PyPDFParsera Parse a blob from a PDF using `pypdf` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images.
It integrates the 'pypdf' library for PDF processing and offers synchronous blob
parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pypdf
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyPDFParser
parser = PyPDFParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# images_parser = TesseractBlobParser(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NFrP r6 plain)modepages_delimiter
images_parserimages_inner_formatextraction_modeextraction_kwargspasswordOptional[Union[str, bytes]]extract_imagesrs r Literal['single', 'page']r r, r Optional[BaseImageBlobParser]r +Literal['text', 'markdown-img', 'html-img']r Literal['plain', 'layout']r Optional[dict[str, Any]]c t |dvrt d || _ |r|st }|| _ || _ || _ || _ || _ || _
|pi | _ dS )u Initialize a parser based on PyPDF.
Args:
password: Optional password for opening encrypted PDFs.
extract_images: Whether to extract images from the PDF.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extraction_mode: “plain” for legacy functionality, “layout” extract text
in a fixed width format that closely adheres to the rendered layout in
the source pdf.
extraction_kwargs: Optional additional parameters for the extraction
process.
Raises:
ValueError: If the `mode` is not "single" or "page".
singlerP mode must be single or pageN) super__init__rT r r r r r r r r r )
selfr r r r r r r r __class__s
r9 r zPyPDFParser.__init__# s J ) ) ) :;; ; , 1- 1 . 0 0M *#6
. . !2!8b rL r@ r r+ Iterator[Document]c # n K ddl n # t $ r t d w xY wd fd }| 5 } j | j }t d
d
d d t
t |j pi z |j t |j d
z }g }t |j D ]\ }} || }
| }
t |
g| } j dk r2t# | t% |||j | dz V | | j dk r8t# j | t% | V ddd dS # 1 swxY w Y dS )m
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
Raises:
ImportError: If the `pypdf` package is not found.
Yield:
An iterator over the parsed documents.
r NzE`pypdf` package not found, please install it with `pip install pypdf`rP pypdf.PageObjectr+ r, c j d r| S | j ddj ij S )z
Extract text from image given the version of pypdf.
Args:
page: The page object to extract text from.
Returns:
str: The extracted text.
3r Nry ) __version__rc extract_textr r )rP pypdfr s r9 _extract_text_from_pagez7PyPDFParser.lazy_parse.._extract_text_from_pagel sb ++C00
((***(t( $($8, rL r PyPDFr/ r= r< r? )r; r> )rP )rP
page_label page_contentrM r )rP r r+ r, )r r2 as_bytes_io PdfReaderr rl r dictrM r; lenpages enumerateextract_images_from_pager rg r r rX page_labelsappendr r4 )
r r@ r pdf_file_obj
pdf_reader doc_metadata single_texts page_numberrP rp images_from_pager r s
` @r9
lazy_parsezPyPDFParser.lazy_parseV s LLLL W
$
# <(
NNNJ* $"MMtZ 0 6B778 #k#&z'7#8#8 L L%.z/?%@%@
2
2! T!8!8d!C!C!C#'#@#@#F#F 1%& %'' 9&&"%-!3 ((3.8.D[.Q " " !''1111yH$$ !%!5!:!:S>S>U>U3V3V(W(W X XHH NN#8999'"$*,,KOOH--22;u2MMM",,..5:: >+*>*>*@*@KXXXD!%d&8&C&CD&I&I!J!J!WJMM+D*d>VWW !''#((f)=)=>> (
rL NF)r r r rs r r r r, r r r r r r r r r@ r r+ r )rP r r+ r, ) __name__
__module__ __qualname____doc___DEFAULT_PAGES_DELIMITERr r r
__classcell__r s @r9 r r s . .d 15 $19
+1 77;KQ6=6:19 19 19 19 19 19 19 19fK K K KZ4
4
4
4
4
4
4
4
rL r c e Zd ZdZdZ d(ddeddddd) fdZed*d Zed+d Z d,d-d#Z
d.d'Z xZ S )/PDFMinerParsera Parse a blob from a PDF using `pdfminer.six` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'pdfminer.six' library for PDF processing and offers synchronous
blob parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pdfminer.six pillow
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PDFMinerParser
parser = PDFMinerParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# extract_images = True,
# images_to_text = convert_images_to_text_with_tesseract(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
FNr r6 )r r r r r concatenate_pagesr rs r rt r r r r, r r r r r Optional[bool]c h t |dvrt d |r|st }|| _ || _ || _ || _ || _ || _ |?t j s&dt _ t
d |rdnd| _ dS dS )aH Initialize a parser based on PDFMiner.
Args:
password: Optional password for opening encrypted PDFs.
mode: Extraction mode to use. Either "single" or "page" for page-wise
extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from PDF.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
concatenate_pages: Deprecated. If True, concatenate all PDF pages
into one a single document. Otherwise, return one document per page.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the `mode` is not "single" or "page".
Warnings:
`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'
instead.
r r NTzS`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'` instead.r rP )r r rT r r r r r r r r _warn_concatenate_pagesr r ) r r r r r r r r r s r9 r zPDFMinerParser.__init__ s P ) ) ) :;; ; 1- 1 . 0 0M , *#6
. (!9
9=6= %6A6DIII ) (rL sUnion[bytes, str]r+ c 0 ddl m t | t r.| d rt | dd dd S d | D }d fd
|D S # t $ r t | cY S w xY w) z
Decodes a PDFDocEncoding string to Unicode.
Adds py3 compatibility to pdfminer's version.
Args:
s: The string to decode.
Returns:
str: The decoded Unicode string.
r )PDFDocEncodings Nzutf-16beignorec 3 b K | ]*}t |t rt | n|V +d S rx )rU r, ord).0cs r9 z-PDFMinerParser.decode_text..\ s; CCAjC 0 07CFFFaCCCCCCrL r/ c 3 ( K | ] }| V
d S rx ry )r or s r9 r
z-PDFMinerParser.decode_text..] s( ;;>!,;;;;;;rL )pdfminer.utilsr rU bytesrc r, r4
IndexError)r ordsr s @r9 decode_textzPDFMinerParser.decode_textK s 211111 a 4ALL$=$= 4quj(33 3 CCCCCD77;;;;d;;;;; ; q66MMM s ,A9 9BBr r c 8 ddl m} t | d r| } t | t
r't t
t j | S t | | r t | j
S t | t t f rt | S t | t r9| D ]"\ }}t | | |< #| S | S )z
Recursively resolve the metadata values.
Args:
obj: The object to resolve and decode. It can be of any type.
Returns:
The resolved and decoded object.
r ) PSLiteralresolve)pdfminer.psparserr hasattrr rU listmapr resolve_and_decoder namer, r r ra )r r rj rk s r9 r z!PDFMinerParser.resolve_and_decodea s 0///// 3 " " ++--C c4 N =sCCDD D
Y
'
' !--ch77 7
c5\
*
* !--c22 2
T
"
"
>
>1'::1==AJ
rL r/ Tfpr cachingrN c ddl m}m}m} || } |||| }i } |j D ]}
| |
| D ]c\ } } t | | | < $# t $ r3}
t d| t |
Y d}
~
\d}
~
ww xY wt t | | | d< | S )ag
Extract metadata from a PDF file.
Args:
fp: The file pointer to the PDF file.
password: The password for the PDF file, if encrypted. Defaults to an empty
string.
caching: Whether to cache the PDF structure. Defaults to True.
Returns:
Metadata of the PDF file.
r ) PDFDocumentPDFPage PDFParser)r r zD[WARNING] Metadata key "%s" could not be parsed due to exception: %sNr> )pdfminer.pdfpager r r infoupdatera r r Exceptionr r r, r r create_pages)r r r r r r r parserdocrM r" rj rk es r9
_get_metadataz PDFMinerParser._get_metadata} s> $ EDDDDDDDDD 2 k&8WEEEH " "D OOD ! ! ! ! NN$$ DAq
, ? ? B B
$FF
#&d7+?+?+D+D&E&E"F"F s A==
B:)B55B:r@ r r c # K ddl }ddlm} ddlm}mmm}m}m m
ddl m }m
} ddlm} t! |j dk rt% d n # t$ $ r t% d w xY w| 5 }
t) 5 | |
j pd
} | } t/ d d d
d
|
j pd
z }
|j |
d< G fdd| }t5 j || || | }g }t9 | D ]\ }} d d | | }|! } j" dk rP d d tG |tI |
d|iz V |% d r
|dd }|& | j" dk r: j' ( | }tG |tI |
V ddd n # 1 swxY w Y ddd dS # 1 swxY w Y dS )a
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
Raises:
ImportError: If the `pdfminer.six` or `pillow` package is not found.
Yield:
An iterator over the parsed documents.
r N)PDFLayoutAnalyzer)LAParams LTContainerLTImageLTItemLTPageLTText LTTextBox)PDFPageInterpreterPDFResourceManager)r i:>4zThis parser is tested with pdfminer.six version 20201018 or later. Remove pdfminer, and install pdfminer.six with `pip uninstall pdfminer && pip install pdfminer.six`.zMpdfminer package not found, please install it with `pip install pdfminer.six`r/ r PDFMinerr r; c @ e Zd Z dd fd
Zdfd Z xZS )*PDFMinerParser.lazy_parse..Visitorr] Nrsrcmgrr4 pagenorW laparamsOptional[LAParams]r+ Nonec P t ||| d S )N)r9 r: )r r )r r8 r9 r: r s r9 r z3PDFMinerParser.lazy_parse..Visitor.__init__ s* GG$$WVh$OOOOOrL ltpager0 c : d fd | d S )Nitemr/ r+ r< c t | r| D ]
} | n7t | r' | t |
r d d S t | r j rddlm} |
}| | }t j t
|z }d|j
d< t j | j
} t || j d S d S d S )Nr0 r ) ImageWriterrD r; )rU writeget_textr pdfminer.imagerB export_imager from_pathr rM r r r rK r )r@ childrB image_writerfilenamer@ r r- r. r1 r2 renderr tempdirtext_ios r9 rK zIPDFMinerParser.lazy_parse..Visitor.receive_layout..render sx %dK88 ;)- . . &u
. 'f 5 5 ; #MM$--// : : :%dI66 ! #MM$ / / / / / 'g 6 6 ! # 1 " F F F F F F/:{7/C/C +7+D+DT+J+J'+~d7mmh6N'O'O:=
h 7-1$($6$A$A$$G$G." ."". !+ !(
$7(,j$:R%& %&!" !" !" !" !" " "" !DrL )r@ r/ r+ r< ry )
mer> rK r- r. r1 r2 r rL rM s
@r9 receive_layoutz9PDFMinerParser.lazy_parse..Visitor.receive_layout sX ! ! ! ! ! ! ! ! ! ! ! ! !8 F6NNNNNrL )r] N)r8 r4 r9 rW r: r; r+ r< )r> r0 r+ r< )r r r r rO r )r r- r. r1 r2 r rL rM s @r9 Visitorr7 s #$37 P P P P P P P # # # # # # # # # # # # # # # #rL rP )r: rP r rv r ))pdfminerpdfminer.converterr+ pdfminer.layoutr, r- r. r/ r0 r1 r2 pdfminer.pdfinterpr3 r4 r! r rW r r2 r r get_pagesr rl r) r; r StringIOr truncateseek process_pager rg r r rX endswithr r r4 )r r@ rR r+ r, r/ r0 r3 r4 r r r r8 r rP visitor_for_all all_contentirP r document_contentr- r. r1 r2 rL rM s ` @@@@@@r9 r zPDFMinerParser.lazy_parse s OOO < < < < < <
R Q Q Q Q Q Q Q 0 0 0 0 0 08'((833!L 4 2
O <1C1E1E O %%lT]=Pb%QQE((**G* 'JPRSS$$\DM>> O K$U++
1
14 ### Q ,,T222"++--#>>++9&&$$Q'''LLOOO"%-!3LFA;4N!O!O
((.. 1#+CRC= &&x0000yH$$#'#7#<#<[#I#I !1 / = = YO O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O sC AA A7K G?J( K (J, ,K /J, 0K KKF)r rs r rt r r r r, r r r r r r )r r r+ r, )r r r+ r )r/ T)r r r r, r rs r+ rN r )
r r r r r r r staticmethodr r r) r r r s @r9 r r s 0 0d $ %:B #'*2 77;KQ,0:B :B :B :B :B :B :B :Bx \* \< , , , , ,\y y y y y y y yrL r c e Zd ZdZ ej Z d)ddedddddd* fdZd+d Z d,d-d Z
d.d$Z d/d&Z d0d'Z
d1d(Z xZS )2
PyMuPDFParsera Parse a blob from a PDF using `PyMuPDF` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'PyMuPDF' library for PDF processing and offers synchronous blob
parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pymupdf
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyMuPDFParser
parser = PyMuPDFParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# images_parser = TesseractBlobParser(),
# extract_tables="markdown",
# extract_tables_settings=None,
# text_kwargs=None,
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NFrP r6 )r r r r r extract_tablesextract_tables_settings text_kwargsr r rs r rt r r r r, r r r r rd /Union[Literal['csv', 'markdown', 'html'], None]re r+ r< c > t |dvrt d |r|dvrt d || _ || _ || _ |pi | _ |r|st }|| _ || _ || _
|| _ | | _ dS )a Initialize a parser based on PyMuPDF.
Args:
password: Optional password for opening encrypted PDFs.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from the PDF.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extract_tables: Whether to extract tables in a specific format, such as
"csv", "markdown", or "html".
extract_tables_settings: Optional dictionary of settings for customizing
table extraction.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the mode is not "single" or "page".
ValueError: If the extract_tables format is not "markdown", "html",
or "csv".
r r )markdownrI csvzmode must be markdownN)
r r rT r r r rf r r r r rd re ) r rf r r r r r r rd re r s r9 r zPyMuPDFParser.__init__` s V ) ) ) :;; ; 6n4O O O 455 5 .
&," 1- 1 . 0 0M ,#6 * ,'>$$$rL r@ r r c , | | S rx ) _lazy_parse)r r@ s r9 r zPyMuPDFParser.lazy_parse s
rL c # K ddl }|p| j }| j sNddlm}m}m}m} i ddddddddd dd
|d dd dd
|ddddddd|d|dddddddddddd| _ n # t $ r t d w xY wt j
5 | 5 }|j |j
| } n |j
|d } | j r| | j ddd d | | | z }
g } | D ]v} | | | | }
| j d k r+t+ |
t- |
d | j iz V a| |
w| j d k r8t+ | j | t- |
V ddd n # 1 swxY w Y ddd dS # 1 swxY w Y dS )!a
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
text_kwargs: Optional keyword arguments to pass to the `get_text` method.
If provided at run time, it will override the default text_kwargs.
Raises:
ImportError: If the `pypdf` package is not found.
Yield:
An iterator over the parsed documents.
r N)DEFAULT_JOIN_TOLERANCE DEFAULT_MIN_WORDS_HORIZONTALDEFAULT_MIN_WORDS_VERTICALDEFAULT_SNAP_TOLERANCEclipvertical_strategylineshorizontal_strategyvertical_lineshorizontal_linessnap_tolerancesnap_x_tolerancesnap_y_tolerancejoin_tolerancejoin_x_tolerancejoin_y_toleranceedge_min_length min_words_verticalmin_words_horizontalintersection_toleranceintersection_x_toleranceintersection_y_tolerance)text_tolerancetext_x_tolerancetext_y_tolerancestrategy add_lineszGpymupdf package not found, please install it with `pip install pymupdf`pdf)streamfiletypePyMuPDFr/ r rP r r )pymupdfrf re
pymupdf.tablern ro rp rq r2 rc _lockr datar is_encrypted authenticater _extract_metadata_get_page_contentrg r r rX numberr r r4 )r r@ rf r rn ro rp rq r[ r' r full_contentrP r s r9 rl zPyMuPDFParser._lazy_parse s ,) NNN%9)9K/
0D0 (0 *7 0
%d 0 '
0 %&<0 '0 '0 %&<0 '0 '0 &q0 )*D 0 +,H 0 -a!0" /#0$ /%0& '(()() $!%/0 0 0,2 -
!!##
y9$&',y11CC&',i%HHHC# 4$$T]333 ) ($& **355 6
"
6
6D # 5 5c4 M M S S U UHyF** &)1%7 , /D D& & %++H55559(("%)%9%>%>|%L%L!3L!A!A 5
sC A"A' 'BG0%D'G G0G G0 G G00G47G4r' pymupdf.Document pymupdf.PagerN c |j di i | j |}| || }| | }g }|r| | |r| | t || }|S )a: Get the text of the page using PyMuPDF and RapidOCR and issue a warning
if it is empty.
Args:
doc: The PyMuPDF document object.
page: The PyMuPDF page object.
blob: The blob being parsed.
Returns:
str: The text content of the page.
ry )rD rf _extract_images_from_page_extract_tables_from_pager r ) r r' rP rf rp r tables_from_pagern r s r9 r z PyMuPDFParser._get_page_content s " ' MM)LD,<)L )LMM 99#tDD 99$?? , MM* + + + , MM* + + +)&.AArL r c t i ddd|j |j t dfdj D }dD ]}|j v rj | ||< |S )zExtract metadata from the document and page.
Args:
doc: The PyMuPDF document object.
blob: The blob being parsed.
Returns:
dict: The extracted metadata.
r r/ )r= r< r? r; r[ r> c z i | ]7}t j | t t f )|j | 8S ry )rU rM r, rW r rj r' s r9
z3PyMuPDFParser._extract_metadata..4 sK !#,q/C:>>s|A rL )modDate creationDate)rl r; r rM )r r' r@ rM rj s ` r9 r z PyMuPDFParser._extract_metadata s #
) ($& "k!%#&s88
\
" - . .ACL !l1o rL c ( | j sdS ddl}| }g }|D ]+}| j r |d } |j || }t j |j t j |j |j
d } t j }
t j |
| |
j dk rt# j |
d } t) | j | j } | t1 | | | j -t4 t8 t= d| S ) a Extract images from a PDF page and get the text using images_to_text.
Args:
doc: The PyMuPDF document object.
page: The PyMuPDF page object.
Returns:
str: The extracted text from the images on the page.
r/ r Nr rv application/x-npyr r ) r r
get_imagesPixmapr r samplesr r r r r r numpyr r r r r r r r r r rK r r rB r r4 r )
r r' rP r img_listr) r7 xrefpiximager r@ r s
r9 r z'PyMuPDFParser._extract_images_from_page@ s ! 2??$$ C!
1v$gnS$//
ckBBBJJJ 2 !jll
;... ((**1Q66 ~ ((**6I "$"4"?"?"E"E F F S
'j$:RSS !''#((f)=)=>> (
rL c | j dS ddl}t |j j |fi | j pi }|r| j dk r$t d |D S | j dk r$t d |D S | j dk r$t d |D S t d
| j d dS ) zExtract tables from a PDF page.
Args:
page: The PyMuPDF page object.
Returns:
str: The extracted tables in the specified format.
Nr/ r ri c 6 g | ]}| S ry ) to_markdownr tables r9
z;PyMuPDFParser._extract_tables_from_page..| s$ )W)W)W%%*;*;*=*=)W)W)WrL rI c b g | ],}| d d d -S )F)headerindex bold_rows) to_pandasto_htmlr s r9 r z;PyMuPDFParser._extract_tables_from_page.. sR " ))11#("'&+ 2 rL rj c ` g | ]+}| d d ,S )F)r r )r to_csvr s r9 r z;PyMuPDFParser._extract_tables_from_page.. sO
" ))00#("' 1 rL zextract_tables z not implemented) rd r r r find_tablesre _JOIN_TABLESr4 rT )r rP r tables_lists r9 r z'PyMuPDFParser._extract_tables_from_pagej s= &2 %GM %d S St/K/Qr S S
"j00#(()W)W;)W)W)WXXX$..#(( &1
$--#((
&1 !Kd&9KKK rrL r )rf r r rs r rt r r r r, r r r r rd rg re r r+ r< r rx )r@ r rf r r+ r )r' r rP r rf rN r+ r, )r' r r@ r r+ r )r' r rP r r+ r, )rP r r+ r, )r r r r threadingLockr r r r rl r r r r r r s @r9 rc rc ' s 2 2l
IN E 15 $;?
#'*0 77;KQJN<@;? ;? ;? ;? ;? ;? ;? ;?z
15 _ _ _ _ _B : @(
(
(
(
T, , , , , , , ,rL rc c d e Zd ZdZ ej Z d ddedddd fdZd dZ d dZ
xZ S ) PyPDFium2Parserao Parse a blob from a PDF using `PyPDFium2` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'PyPDFium2' library for PDF processing and offers synchronous
blob parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pypdfium2
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyPDFium2Parser
parser = PyPDFium2Parser(
# password=None,
mode="page",
pages_delimiter="
",
# extract_images = True,
# images_to_text = convert_images_to_text_with_tesseract(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
FNrP r6 )r r r r r r rs r rt r r r r, r r r r r+ r< c t |dvrt d || _ |r|st }|| _ || _ || _ || _ || _ dS )uk Initialize a parser based on PyPDFium2.
Args:
password: Optional password for opening encrypted PDFs.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from the PDF.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extraction_mode: “plain” for legacy functionality, “layout” for experimental
layout mode functionality
extraction_kwargs: Optional additional parameters for the extraction
process.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the mode is not "single" or "page".
r r N)
r r rT r r r r r r r )r r r r r r r r s r9 r zPyPDFium2Parser.__init__ s L ) ) ) :;; ; , 1- 1 . 0 0M *#6
. rL r@ r r c # K ddl }n # t $ r t d w xY wt j 5 | 5 }d} |j || j d }g }ddddt | z }|j |d < t | |d
< t | D ]\ }}| } d
| }
| | | } t% | g|
} | | j d k rA| d s| d z
} t- | t/ i |d |i
V | | | j dk r8t- | j
| t/ |
V |r| n # |r| w w xY wddd n # 1 swxY w Y ddd dS # 1 swxY w Y dS )r r NzKpypdfium2 package not found, please install it with `pip install pypdfium2`T)r autoclose PyPDFium2r/ r r; r> r0 rP r r ) pypdfium2r2 r r r PdfDocumentr rl get_metadata_dictr; r r get_textpager4 get_text_range
splitlinescloser r rg r r[ r rX r r )
r r@ r r[ r r r r rP text_pagerp image_from_pager s
r9 r zPyPDFium2Parser.lazy_parse s +
" 4 + 4 +!!## 3
+y !
1+!6!6!DMT" " "J $&L %0#.(*$ $ (
(D(D(F(FGG $HL
.2[L*25j//L /-6z-B-B : :) T$($5$5$7$7 )- % 4 4 6 6 A A C C* * ")))*.*H*H*N*N#9 , -~$ $ %'' !
9..#+#4#4T#:#: 1 (D 0"*-5);%&*6%&(. %& %&*" *"# # # ) / / 9 9 9 9yH,, &)-)=)B)B<)P)P%7 %E%E
" +"((*** " +"((****+e3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+ 3
+4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 + 4 +sV #I(I GH(I(II I(I I(I I((I,/I, pypdfium2._helpers.page.PdfPagec | j sdS ddlm} t | |j f }|sdS g }|D ]}t
j }| }|j
dk rHt j || t j | d }t! | j | j } | t) || | j | t. t2 | S ) r r/ r N)r r r r r )r
pypdfium2.rawrawr get_objectsFPDF_PAGEOBJ_IMAGEr r
get_bitmapto_numpysizer r r r r r r r r rK r r r rB r r4 )
r rP pdfium_cr)
str_imagesr r r r@ text_from_images
r9 r z)PyPDFium2Parser._extract_images_from_pageQ so ! 2((((((d&&x/J.L&MMNN 2
E *,,K ''))2244H}q J{E$4$4$6$6$?$?$A$A B B B>+"6"6"8"8DWXXXD "4#5#@#@#F#F G G TO #D/4;STT
KKMMMM ''<3D3DZ3P3P'QQQrL r` )r rs r rt r r r r, r r r r r+ r< r )rP r r+ r, ) r r r r r r r r r r r r r s @r9 r r s 0 0h
IN E %0/ #'*0 77;KQ0/ 0/ 0/ 0/ 0/ 0/ 0/ 0/dM+ M+ M+ M+^ R R R R R R R RrL r c : e Zd ZdZ ddd ZddZddZddZdS )PDFPlumberParserz Parse `PDF` with `PDFPlumber`.NFrf Optional[Mapping[str, Any]]dedupers r r+ r< c z ddl }n # t $ r t d w xY w|pi | _ || _ || _ dS )zInitialize the parser.
Args:
text_kwargs: Keyword arguments to pass to ``pdfplumber.Page.extract_text()``
dedupe: Avoiding the error of duplicate characters if `dedupe=True`.
r NzEpillow package not found, please install it with `pip install pillow`)r r2 rf r r )r rf r r r s r9 r zPDFPlumberParser.__init__u sf JJJJ W
'," ,s !r@ r r c # K ddl } 5 } |j | fdj D E d{V ddd dS # 1 swxY w Y dS )Lazily parse the blob.r Nc
g | ]}t | d z | z t j j |j dz
t
j dfi fdj D S )r0 r] )r; r[ rP r> c | i | ]8}t j | t t fv *|j | 9S ry )rb rM r, rW r s r9 r z:PDFPlumberParser.lazy_parse... sH ! #CLO 4 4c
B B s|A B B BrL r ) r _process_page_contentr r r; r r r rM )r rP r@ r' r s r9 r z/PDFPlumberParser.lazy_parse.. s & % !%!;!;D!A!A"44T::"; "&*k)-$($4q$8+.sy>> %(\ rL )
pdfplumberr r r )r r@ r r[ r' s `` @r9 r zPDFPlumberParser.lazy_parse s
9!*/),,C & I'
s ,AA ArP pdfplumber.page.Pager, c | | j r$ | j di | j S |j di | j S )z)Process the page content based on dedupe.ry )r dedupe_charsr rf )r rP s r9 r z&PDFPlumberParser._process_page_content sQ ; H34$$&&3GGd6FGG G t 444#3444rL c f ddl m} | j sdS g }|j D ]}|d d j t
v r|d d dk r| t j | d|d d |d d
f|d
d | t j |d
t j
|d d
|d d d
,|d d j t v r/| |d
ut! j d t% | S )z8Extract images from page and get the text with RapidOCR.r r r/ r FilterBitsPerComponentr] 1WidthHeightLr rv r )r r r r) r r r r r frombytesr convertr r r r warningswarnr: )r rP r r) r7 s r9 r z*PDFPlumberParser._extract_images_from_page s " 2; 5 5C8}X&+/GGGx=!3499MM !OO #!$X w!7X x9P Q #H
6 6 8 8 &gcll MM
c(m&<&<&>&>bhOOOWW M( 3S]75KR
X x(-1FFF
c(m 4 4 6 67777
344440888rL )NFF)rf r r rs r rs r+ r< r )rP r r+ r, )r r r r r r r r ry rL r9 r r r sz (( 48 $ - - - - -, :5 5 5 5 9 9 9 9 9 9rL r c . e Zd ZdZ ddddd ZddZdS )AmazonTextractPDFParsera Send `PDF` files to `Amazon Textract` and parse them.
For parsing multi-page PDFs, they have to reside on S3.
The AmazonTextractPDFLoader calls the
[Amazon Textract Service](https://aws.amazon.com/textract/)
to convert PDFs into a Document structure.
Single and multi-page documents are supported with up to 3000 pages
and 512 MB of size.
For the call to be successful an AWS account is required,
similar to the
[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
requirements.
Besides the AWS configuration, it is very similar to the other PDF
loaders, while also supporting JPEG, PNG and TIFF and non-native
PDF formats.
```python
from langchain_community.document_loaders import AmazonTextractPDFLoader
loader=AmazonTextractPDFLoader("example_data/alejandro_rosalez_sample-small.jpeg")
documents = loader.load()
```
One feature is the linearization of the output.
When using the features LAYOUT, FORMS or TABLES together with Textract
```python
from langchain_community.document_loaders import AmazonTextractPDFLoader
# you can mix and match each of the features
loader=AmazonTextractPDFLoader(
"example_data/alejandro_rosalez_sample-small.jpeg",
textract_features=["TABLES", "LAYOUT"])
documents = loader.load()
```
it will generate output that formats the text in reading order and
try to output the information in a tabular structure or
output the key/value pairs with a colon (key: value).
This helps most LLMs to achieve better accuracy when
processing these texts.
``Document`` objects are returned with metadata that includes the ``source`` and
a 1-based index of the page number in ``page``. Note that ``page`` represents
the index of the result returned from Textract, not necessarily the as-written
page number in the document.
N)linearization_configtextract_featuresOptional[Sequence[int]]client
Optional[Any]r !Optional[TextLinearizationConfig]r+ r< c ddl ddlmc m} | _ || _ |fd|D | _ ng | _ ||| _ n#| j dddd | _ n # t $ r t d w xY w|s> ddl
}| d
| _ dS # t $ r t d w xY w|| _ dS ) a5 Initializes the parser.
Args:
textract_features: Features to be used for extraction, each feature
should be passed as an int that conforms to the enum
`Textract_Features`, see `amazon-textract-caller` pkg
client: boto3 textract client
linearization_config: Config to be used for linearization of the output
should be an instance of TextLinearizationConfig from
the `textractor` pkg
r Nc : g | ]} | S ry )Textract_Features)r ftcs r9 r z4AmazonTextractPDFParser.__init__.. s4 * * *01B((++* * *rL Tz# z## *)hide_figure_layout title_prefixsection_header_prefixlist_element_prefixzCould not import amazon-textract-caller or amazon-textract-textractor python package. Please install it with `pip install amazon-textract-caller` & `pip install amazon-textract-textractor`.textractzRCould not import boto3 python package. Please install it with `pip install boto3`.)
textractcaller textractor.entities.documententitiesdocumentr
textractorr r r r2 boto3r boto3_textract_client)r r r r r r
r s @r9 r z AmazonTextractPDFParser.__init__ sb & ' ' ' ' = = = = = = = = =DG (DO ,* * * *5F* * *&& *,&#/,@)),0O,S,S'+!%*/(+ -T - -) <
0
-2\\*-E-E***
!B
*0D & & &s A%A) )B B) )Cr@ r r c # l K |j r!t t |j nd}|rL|j dk rA|j r:| j t |j | j | j }nI| j | | j | j j
j | j }| j j
| }t |j D ]<\ }}t | | j |j |dz d V =dS ) zIterates over the Blob pages and returns an Iterator with a Document
for each page, like the other parsers If multi-page document, blob.path
has to be set to the S3 URI and for single page docs
the blob.data is taken
Ns3)input_documentfeaturesr )r r call_moder )configr] r; rP r )pathr r, schemenetlocr
call_textractr r as_bytesTextract_Call_Mode
FORCE_SYNCr r r r r rD r r; )r r@ url_parse_resulttextract_response_jsonr idxrP s r9 r z"AmazonTextractPDFParser.lazy_parseB sT 822 IC !]]$2K] L L$(KqAA
rL )NN)r r r r r r r+ r< r )r r r r r r ry rL r9 r r sf 0 0h 6: $=0
CG =0 =0 =0 =0 =0 =0~! ! ! ! ! !rL r c * e Zd ZdZddZdd Zdd
ZdS )DocumentIntelligenceParserzjLoads a PDF with Azure Document Intelligence
(formerly Form Recognizer) and chunks at character level.r r modelr, c J t j d || _ || _ d S )Na< langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParserand langchain_community.document_loaders.pdf.DocumentIntelligenceLoader are deprecated. Please upgrade to langchain_community.document_loaders.DocumentIntelligenceLoader for any file parsing purpose using Azure Document Intelligence service.)r r r r" )r r r" s r9 r z#DocumentIntelligenceParser.__init__j s/
rL r@ r r8 r+ r c # K |j D ]H}d d |j D }t ||j |j d }|V Id S )N c g | ] }|j
S ry )rA )r lines r9 r z=DocumentIntelligenceParser._generate_docs..x s A A A A A ArL r r )r r4 rt r r; r )r r@ r8 prA ds r9 _generate_docsz)DocumentIntelligenceParser._generate_docsv s{
Ahh A A A A ABBG $ "k M A GGGG
rL c # K | 5 }| j | j | }| }| || }|E d{V ddd dS # 1 swxY w Y dS )r N)r r begin_analyze_documentr" r8 r* )r r@ file_objpollerr8 docss r9 r z%DocumentIntelligenceParser.lazy_parse s
8[77
HMMF]]__F&&tV44DOOOOOOO
s AA77A;>A;N)r r r" r, )r@ r r8 r r+ r r )r r r r r r* r ry rL r9 r! r! f s\ A A
rL r! )r) r* r+ r, )r@ r rA r, rB r, r+ r, )rM rN r+ rN )rn ro rp r, r+ r, )Cr
__future__r rI r loggingr r r pathlibr tempfiler typingr r r r
r r r
r r r r urllib.parser r r langchain_core.documentsr )langchain_community.document_loaders.baser 1langchain_community.document_loaders.blob_loadersr 3langchain_community.document_loaders.parsers.imagesr r r r r r )textractor.data.text_linearization_configr r r r: getLoggerr r r r r r rQ rK rX rl r} r r r rc r r r r! ry rL r9 r< s . . " " " " " " ' ' ' ' ' ' " ! ! ! ! ! - - - - - - D D D D D D B B B B B B
RNNNLLLQQQQQQ999 " <
8 $ $* ! UUU , &# # # #N
2 2 2 2je
e
e
e
e
. e
e
e
PJ J J J J^ J J JZ
o o o o oN o o od VR VR VR VR VRn VR VR VRr[9 [9 [9 [9 [9~ [9 [9 [9|S S S S Sn S S Sl& & & & & & & & & & rL