dh F S r SSKJr SSKrSSKrSSKrSSKrSSKrSSKJr SSK J
r
SSK J r SSK
JrJrJrJrJrJrJrJrJrJrJr SSKJr SSKrSSKr SS K J r SS
K J r SS K!J"r" SS K#J$r$J%r% \( a SSK&r&SSK'r'SSK(r(SSK)r)SS
K*J+r+ / SQr,/ SQr- S)S jr.\R^ " \05 r1Sr2Sr3Sr4Sr51 Skr6S*S jr7S+S jr8S+S jr9SS/r:S,S jr; " S S \ 5 r< " S S \ 5 r= " S S \ 5 r> " S! S"\ 5 r? " S# S$\ 5 r@ " S% S&\ 5 rA " S' S(\ 5 rBg)-z(Module contains common parsers for PDFs. ) annotationsN)datetime)Path)TemporaryDirectory)
TYPE_CHECKINGAnyBinaryIOIterableIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)BaseBlobParser)Blob)BaseImageBlobParserRapidOCRBlobParser)TextLinearizationConfig) DCTDecodeDCT JPXDecode)
LZWDecodeLZW FlateDecodeFl
ASCII85DecodeA85ASCIIHexDecodeAHxRunLengthDecodeRLCCITTFaxDecodeCCF JBIG2Decodec SSK Jn U" 5 nSnU H= nU" U5 u pVU( d M U Vs/ sH o3S PM nnWSR U5 -
nM? U$ ! [ a [ S5 ef = fs snf )zExtract text from images with RapidOCR.
Args:
images: Images to extract text from.
Returns:
Text extracted from images.
Raises:
ImportError: If `rapidocr-onnxruntime` package is not installed.
r )RapidOCRzc`rapidocr-onnxruntime` package not found, please install it with `pip install rapidocr-onnxruntime`
)rapidocr_onnxruntimer* ImportErrorjoin)imagesr* ocrtextimgresult_s h/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_community/document_loaders/parsers/pdf.py!extract_from_images_with_rapidocrr8 @ s
1 *C
DH 6*01&$1g&F1 DIIf% %D
K
1
2s A
A/A,z
{image_text}
r- z
> sourcecreatorproducer total_pages creationdatec U( aZ U R =( d SnUS:X a UR SS5 nSU SU S3nU$ US:X a S [ R " US
S 9 S U S
3nU$ )a Format the content of the image with the source of the blob.
blob: The blob containing the image.
format::
The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(`
`)
#z markdown-img]z\\]z![z]()zhtml-imgz
![T)quotez src=]()
)r9 replacehtmlescape)blobcontentformatr9 s r7 _format_inner_imagerI i sx # ^ # ooc62G7)2fXQ/G N z
!"4;;wd#C"DF6(RVWG N c [ R U R 5 5 ( d [ S5 e[ U R SS5 [ 5 ( d [ S5 eU $ )zValidate that the metadata has all the standard keys and the page is an integer.
The standard keys are:
- source
- total_page
- creationdate
- creator
- producer
Validate that page is an integer if it is present.
z3The PDF parser must valorize the standard metadata.pager z(The PDF metadata page must be a integer.)_STD_METADATA_KEYSissubsetkeys
ValueError
isinstancegetint)metadatas r7 _validate_metadatarU sP & &x}} 7 7NOO hll61-s 3 3CDD OrJ c < 0 nSSS.nU R 5 H u p4[ U5 [ [ 4; a [ U5 nUR S5 ( a USS nUR 5 nUS; a; [ R " UR SS 5 S
5 R S 5 X' M X2; a XAX# ' XAU' M [ U[ 5 ( a UR 5 X' M [ U[ 5 ( d M XAU' M U$ ! [ a XAU' GM f = f) zPurge metadata from unwanted keys and normalize key names.
Args:
metadata: The original metadata dictionary.
Returns:
The cleaned and normalized the key format of metadata dictionary.
r< r9 )
page_count file_path/r, N)r= moddate'r+ zD:%Y%m%d%H%M%S%zT)
itemstypestrrS
startswithlowerr strptimerC isoformatrP rQ strip)rT new_metadatamap_keykvs r7 _purge_metadatari s $&L# G 73* $AA << !"A
GGI + +
$"*"3"3IIc2&(:#)C.
\'( $ O
3
ggiLO
3
O) !*
$"#Q
$s 38DDDz
c ^ SU4S jjmT" XS5 nU( d5 SnSR [ S U 5 5 nU( a [ S U- nX- nU$ )a Insert extras such as image/table in a text between two paragraphs if possible,
else at the end of the text.
Args:
extras: List of extra content (images/tables) to insert.
text_from_page: The text content from the page.
Returns:
The merged text with extras inserted.
c & > U ( a [ Hw nUR U5 nUS:w d M S nU( a T " XS U S5 nU( a XQUS - nO6SnSR [ S U 5 5 nU( a X8- nUS U U- XS - n U$ S n U$ UnU$ )NFr+ rj c U $ N xs r7 O_merge_text_and_extras.._recurs_merge_text_and_extras.. s !rJ )_PARAGRAPH_DELIMITERrfindr0 filter)
extrastext_from_pagerecursdelimpos
previous_textall_text
all_extras
str_extras _recurs_merge_text_and_extrass
r7 r =_merge_text_and_extras.._recurs_merge_text_and_extras s -$**51"9$(M (E "4C$8%)
%#0#$3G#G%'
%+[[ V1L%M
%).);J *4C 0: =t@T T !
1 .* &HrJ Tr+ rj c U $ ro rp rq s r7 rs (_merge_text_and_extras.. s !rJ rm )rx list[str]ry r_ rz boolreturn
Optional[str])r0 rw ru )rx ry r~ r r r s @r7 _merge_text_and_extrasr sr +. 8< < -VTJH
[[ V!<=
-b1J>J!. OrJ c | ^ \ rS rSrSr S S\SSSSS. S
U 4S jjjjrSS jrSS
jrS r U =r
$ ) PyPDFParser a Parse a blob from a PDF using `pypdf` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images.
It integrates the 'pypdf' library for PDF processing and offers synchronous blob
parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pypdf
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyPDFParser
parser = PyPDFParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# images_parser = TesseractBlobParser(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NrL r3 plain)modepages_delimiter
images_parserimages_inner_formatextraction_modeextraction_kwargsc > [ T U ] 5 US; a [ S5 eX l U( a U( d
[ 5 nXPl X`l Xl X0l X@l Xpl
U=( d 0 U l g)uM Initialize a parser based on PyPDF.
Args:
password: Optional password for opening encrypted PDFs.
extract_images: Whether to extract images from the PDF.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extraction_mode: “plain” for legacy functionality, “layout” extract text
in a fixed width format that closely adheres to the rendered layout in
the source pdf.
extraction_kwargs: Optional additional parameters for the extraction
process.
Raises:
ValueError: If the `mode` is not "single" or "page".
singlerL mode must be single or pageN) super__init__rP extract_imagesr r r passwordr r r r )
selfr r r r r r r r __class__s
r7 r PyPDFParser.__init__$ sh J ) ) :; ; , - . 0M *#6
. . !2!8b rJ c # J ^ ^ # SSK m SU U 4S jjnUR 5 nT R " UT R S9n[ SSSS .[
[ UR =( d 0 5 - UR [ UR 5 S
.- 5 n/ n[ UR 5 H u pxU" US 9n T R U5 n
[ U
/U 5 R 5 n T R S:X a* [# U [% UUUR& U S .- 5 S
9v Mq UR) U 5 M T R S:X a. [# T R* R- U5 [% U5 S
9v SSS5 g! [ a [ S5 ef = f! , ( d f g= f7f)
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
Raises:
ImportError: If the `pypdf` package is not found.
Yield:
An iterator over the parsed documents.
r NzE`pypdf` package not found, please install it with `pip install pypdf`rL c > TR R S5 ( a U R 5 $ U R " SSTR 0TR D6$ )z
Extract text from image given the version of pypdf.
Args:
page: The page object to extract text from.
Returns:
str: The extracted text.
3r rp ) __version__r` extract_textr r )rL pypdfr s r7 _extract_text_from_page7PyPDFParser.lazy_parse.._extract_text_from_pagem sZ ++C00((**(( $($8$8,, rJ r PyPDFr+ r; r: r= )r9 r< )rL )rL
page_label page_contentrT r )rL zpypdf.PageObjectr r_ )r r/ as_bytes_io PdfReaderr ri r dictrT r9 lenpages enumerateextract_images_from_pager rd r r rU page_labelsappendr r0 )
r rF r pdf_file_obj
pdf_reader doc_metadata single_texts page_numberrL ry images_from_pager~ r s
` @r7
lazy_parsePyPDFParser.lazy_parseW s $
<
NJ* $"MtZ 0 0 6B78 #kk#&z'7'7#8 L L%.z/?/?%@! !8d!C#'#@#@#F 1%& %' 99&"%-!3 ((3.8.D.D[.Q " !''1% &A& yyH$ !%!5!5!:!:S>S>U3V(W XH NN#89'"$**,K",,.55: OOH-22;u2M >>+*>*>*@KXD!%d&8&8&C&CD&I!J!W!WJMM+Dd>V>VW9 > !''#((f)=> (
rJ )r r r r r r r r NF)r zOptional[Union[str, bytes]]r r r Literal['single', 'page']r r_ r Optional[BaseImageBlobParser]r +Literal['text', 'markdown-img', 'html-img']r zLiteral['plain', 'layout']r Optional[dict[str, Any]]rF r r Iterator[Document])rL zpypdf._page.PageObjectr r_ ) __name__
__module__ __qualname____firstlineno____doc___DEFAULT_PAGES_DELIMITERr r r __static_attributes__
__classcell__r s @r7 r r s .d 15 $19
+1 77;KQ6=6:19-19 19
( 19
19 519 I19 419 419 19fKZ4
4
rJ r c ^ \ rS rSrSrSr SSS\SSSS. SU 4S jjjjr\SS j5 r \SS
j5 r
S SS jjr SS jr S
r
U =r$ )PDFMinerParseri a Parse a blob from a PDF using `pdfminer.six` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'pdfminer.six' library for PDF processing and offers synchronous
blob parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pdfminer.six pillow
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PDFMinerParser
parser = PDFMinerParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# extract_images = True,
# images_to_text = convert_images_to_text_with_tesseract(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
FNr r3 )r r r r r concatenate_pagesc N > [ TU ] 5 US; a [ S5 eU( a U( d
[ 5 nXl XPl X`l X l X0l X@l UbF [ R ( d S[ l [ R S5 U( a SOSU l gg)a Initialize a parser based on PDFMiner.
Args:
password: Optional password for opening encrypted PDFs.
mode: Extraction mode to use. Either "single" or "page" for page-wise
extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from PDF.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
concatenate_pages: Deprecated. If True, concatenate all PDF pages
into one a single document. Otherwise, return one document per page.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the `mode` is not "single" or "page".
Warnings:
`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'
instead.
r r NTzS`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'` instead.r rL )r r rP r r r r r r r r _warn_concatenate_pagesr r ) r r r r r r r r r s r7 r PDFMinerParser.__init__ s P ) ) :; ; - . 0M , *#6
. (!999=6= %66DI )rJ c ^ SSK Jm [ U [ 5 ( a& U R S5 ( a [ U SS SS5 $ S U 5 nS R
U4S
jU 5 5 $ ! [ a [ U 5 s $ f = f) z
Decodes a PDFDocEncoding string to Unicode.
Adds py3 compatibility to pdfminer's version.
Args:
s: The string to decode.
Returns:
str: The decoded Unicode string.
r )PDFDocEncodings Nzutf-16beignorec 3 d # U H' n[ U[ 5 ( a [ U5 OUv M) g 7fro )rQ r_ ord).0cs r7 -PDFMinerParser.decode_text..] s% CAjC 0 0CFa7s .0r+ c 3 , ># U H
nTU v M g 7fro rp )r
or s r7 r r
^ s ;d>!,ds )pdfminer.utilsr rQ bytesr` r_ r0
IndexError)sordsr s @r7 decode_textPDFMinerParser.decode_textL st 2 a ALL$=$=quj(3 3 CCD77;d;; ; q6M s #A( (B ?B c SSK Jn [ U S5 ( a U R 5 n [ U [
5 ( a# [ [
[ R U 5 5 $ [ X5 ( a [ R U R 5 $ [ U [ [ 45 ( a [ R U 5 $ [ U [ 5 ( a2 U R 5 H u p#[ R U5 X' M U $ U $ )z
Recursively resolve the metadata values.
Args:
obj: The object to resolve and decode. It can be of any type.
Returns:
The resolved and decoded object.
r ) PSLiteralresolve)pdfminer.psparserr hasattrr rQ listmapr resolve_and_decoder namer_ r r r] )r r rg rh s r7 r !PDFMinerParser.resolve_and_decodeb s 0 3 " "++-C c4 N = =sCD D
'
'!--chh7 7
c5\
*
*!--c2 2
T
"
" '::1= $J
rJ c SSK JnJnJn U" U5 nU" XrUS9n0 n UR H n
U R U
5 M U R
5 H u p [ R U 5 X' M [ [ UR U5 5 5 U S' U $ ! [ a+ n
[ R SU [ U
5 5 Sn
A
My Sn
A
ff = f)a
Extract metadata from a PDF file.
Args:
fp: The file pointer to the PDF file.
password: The password for the PDF file, if encrypted. Defaults to an empty
string.
caching: Whether to cache the PDF structure. Defaults to True.
Returns:
Metadata of the PDF file.
r ) PDFDocumentPDFPage PDFParser)r cachingzD[WARNING] Metadata key "%s" could not be parsed due to exception: %sNr< )pdfminer.pdfpager" r# r$ infoupdater] r r Exceptionr r r_ r r create_pages)r fpr r% r" r# r$ parserdocrT r' rg rh es r7
_get_metadata PDFMinerParser._get_metadata~ s $ ED 2 &WEHHD OOD ! NN$DA
, ? ? B % #&d7+?+?+D&E"F
$F
s B
C# C Cc # ^ ^^^^^^# SSK nSSKJn SSKJnJmJmJnJnJ mJ
m SSK J nJ
n SSKJn [! UR" 5 S: a [% S5 e UR' 5 n
[) 5 mU R+ U
T R, =( d S
S 9n U" 5 n [/ S S S
S
.T R1 U
T R, =( d S
S 9- 5 n
UR2 U
S' " UUUUU UU4S jSU5 n[4 R6 " 5 mU" X" X" 5 S95 n/ n[9 U 5 H u nnTR; S5 TR= S5 UR? U5 TRA 5 nURC 5 nT RD S:X a> TR; S5 TR= S5 [G U[I U
SU0- 5 S9v M URK S5 ( a USS nURM U5 M T RD S:X a0 T RN RQ U5 n[G U[I U
5 S9v SSS5 SSS5 g! [$ a [% S 5 ef = f! , ( d f N0= f! , ( d f g= f7f)a0
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
Raises:
ImportError: If the `pdfminer.six` or `pillow` package is not found.
Yield:
An iterator over the parsed documents.
r N)PDFLayoutAnalyzer)LAParams LTContainerLTImageLTItemLTPageLTText LTTextBox)PDFPageInterpreterPDFResourceManager)r# i:>4zThis parser is tested with pdfminer.six version 20201018 or later. Remove pdfminer, and install pdfminer.six with `pip uninstall pdfminer && pip install pdfminer.six`.zMpdfminer package not found, please install it with `pip install pdfminer.six`r+ r PDFMinerr r9 c ` >^ \ rS rSr S SU 4S jjjrSUUUUUUU4S jjrSrU =r$ )*PDFMinerParser.lazy_parse..Visitori c " > [ TU ] XUS9 g )N)pagenolaparams)r r )r rsrcmgrr@ rA r s r7 r 3PDFMinerParser.lazy_parse..Visitor.__init__ s G$Wh$OrJ c 6 >^ SUUUUUUUU 4S jjmT" U5 g )Nc > [ U T5 ( a U H nT " U5 M
O0[ U T 5 ( a TR U R 5 5 [ U T
5 ( a TR S5 g [ U T5 ( a T R ( a SSKJn U" T
5 nUR
U 5 n[ R " [ T
5 U- 5 nSUR S' [ T R R U5 5 R nTR [ XVT R 5 5 g g g )Nr- r ) ImageWriterr? r9 )rQ writeget_textr pdfminer.imagerF export_imager from_pathr rT r r r rI r )itemchildrF image_writerfilenamerF r r4 r5 r8 r9 renderr tempdirtext_ios r7 rP IPDFMinerParser.lazy_parse..Visitor.receive_layout..render s %dK88)- &u
*. 'f 5 5 #MM$--/ :%dI66 #MM$ / 'g 6 6 # 1 1 F/:7/C +7+D+DT+J'+~~d7mh6N'O:=
h 7-1$($6$6$A$A$$G.""., !+ !(
$7(,$:R:R%&!" 2" !rJ )rL r6 r Nonerp )
meltpagerP r4 r5 r8 r9 r rQ rR s
@r7 receive_layout9PDFMinerParser.lazy_parse..Visitor.receive_layout s ! !8 6NrJ rp )r, N)rB r; r@ rS rA zOptional[LAParams]r rT )rV r7 r rT )r r r r r rW r r )r r4 r5 r8 r9 r rQ rR s @r7 Visitorr> sP #$37 P /P P 1 P
P P # # #rJ rY )rA rL r rm r ))pdfminerpdfminer.converterr2 pdfminer.layoutr3 r4 r5 r6 r7 r8 r9 pdfminer.pdfinterpr: r; r& r# rS r r/ r r get_pagesr ri r/ r9 r StringIOr truncateseek process_pager rd r r rU endswithr r r0 )r rF r[ r2 r3 r6 r7 r: r; r# r r rB r rY visitor_for_all all_contentirL r~ document_contentr4 r5 r8 r9 rQ rR s ` @@@@@@r7 r PDFMinerParser.lazy_parse s[ <
R 08''(83!L 4
<1C1E%%lT]]=Pb%QE(*G* 'JPRS$$\DMM O K$U+4 # Q ,,T2"++-#>>+99&$$Q'LLO"%-!3LFA;4N!O
((..#+CR= &&x0% ,& yyH$#'#7#7#<#<[#I !1 / = Y 2F
2
2F1E
sN J AI J * I;5G I* I; J I''J *
I8 4I;;
J J r r r r r r F)r r r r r r r r_ r r r r r zOptional[bool])r zUnion[bytes, str]r r_ )r r r r )r+ T)r+ r r r_ r% r r dict[str, Any]r )r r r r r r r r staticmethodr r r/ r r r r s @r7 r r s 0d $ %:B #'*2 77;KQ,0:B :B :B
( :B
:B 5:B I:B *:B :Bx * < , , , ,
,\y yrJ r c ^ \ rS rSrSr\R " 5 r SSS\SSSSS. SU 4S jjjjr SS jr
S SS jjr SS
jr SS jr
SS jrSS
jrSrU =r$ )
PyMuPDFParseri( a Parse a blob from a PDF using `PyMuPDF` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'PyMuPDF' library for PDF processing and offers synchronous blob
parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pymupdf
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyMuPDFParser
parser = PyMuPDFParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# images_parser = TesseractBlobParser(),
# extract_tables="markdown",
# extract_tables_settings=None,
# text_kwargs=None,
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NrL r3 )r r r r r extract_tablesextract_tables_settingsc $ > [ T
U ] 5 US; a [ S5 eU( a US; a [ S5 eX@l XPl X0l U=( d 0 U l U( a U( d
[ 5 nX l Xpl X`l
Xl Xl g)a Initialize a parser based on PyMuPDF.
Args:
password: Optional password for opening encrypted PDFs.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from the PDF.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extract_tables: Whether to extract tables in a specific format, such as
"csv", "markdown", or "html".
extract_tables_settings: Optional dictionary of settings for customizing
table extraction.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the mode is not "single" or "page".
ValueError: If the extract_tables format is not "markdown", "html",
or "csv".
r r )markdownrD csvzmode must be markdownN)
r r rP r r r text_kwargsr r r r rp rq ) r ru r r r r r r rp rq r s r7 r PyMuPDFParser.__init__a s V ) ) :; ; n4O O 45 5 .
&," - . 0M ,#6 * ,'>$rJ c $ U R U5 $ ro ) _lazy_parse)r rF s r7 r PyMuPDFParser.lazy_parse s
rJ c # " # SSK nU=( d U R nU R ( dN SSKJnJnJnJn 0 SS_SS_SS_SS_S S_S
U_S S_S S_S
U_SS_SS_SS_SU_SU_SS_SS_SS_SSSSSS.EU l [ R UR 5 nUR c UR " U5 n OUR " USS9n U R ( a U R U R 5 SSS S .U R# X5 - n
/ n U Hj n U R% XU5 R' 5 n
U R( S :X a& [+ U
[- U
S U R. 0- 5 S 9v MY U R1 U
5 Ml U R( S :X a. [+ U R2 R5 U 5 [- U
5 S 9v SSS5 SSS5 g! [ a [ S5 ef = f! , ( d f N0= f! , ( d f g= f7f)!a Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
text_kwargs: Optional keyword arguments to pass to the `get_text` method.
If provided at run time, it will override the default text_kwargs.
Raises:
ImportError: If the `pypdf` package is not found.
Yield:
An iterator over the parsed documents.
r N)DEFAULT_JOIN_TOLERANCE DEFAULT_MIN_WORDS_HORIZONTALDEFAULT_MIN_WORDS_VERTICALDEFAULT_SNAP_TOLERANCEclipvertical_strategylineshorizontal_strategyvertical_lineshorizontal_linessnap_tolerancesnap_x_tolerancesnap_y_tolerancejoin_tolerancejoin_x_tolerancejoin_y_toleranceedge_min_length min_words_verticalmin_words_horizontalintersection_toleranceintersection_x_toleranceintersection_y_tolerance)text_tolerancetext_x_tolerancetext_y_tolerancestrategy add_lineszGpymupdf package not found, please install it with `pip install pymupdf`pdf)streamfiletypePyMuPDFr+ r rL r r )pymupdfru rq
pymupdf.tabler{ r| r} r~ r/ ro _lockr datar is_encrypted authenticater _extract_metadata_get_page_contentrd r r rU numberr r r0 )r rF ru r r{ r| r} r~ rX r- r full_contentrL r~ s r7 rx PyMuPDFParser._lazy_parse s ,) %9)9)9K// 0D0 (0 *7 0
%d 0 '
0 %&<0 '0 '0 %&<0 '0 '0 &q0 )*D 0 +,H 0 -a!0" /#0$ /%0& '(()() $!%/0,>
!!#y99$!,,y1C!,,i%HC##$$T]]3 ) ($& **35 6
" D # 5 5c M S S UHyyF* &)1%7 , /D D& %++H5 99("%)%9%9%>%>|%L!3L!A 5 $ !
-
$# !
sN HA8G <H G> D&G-G> HG**H-
G; 7G>>
H Hc UR " S0 0 U R EUED6nU R X5 nU R U5 n/ nU( a UR U5 U( a UR U5 [ Xt5 nU$ )zGet the text of the page using PyMuPDF and RapidOCR and issue a warning
if it is empty.
Args:
doc: The PyMuPDF document object.
page: The PyMuPDF page object.
blob: The blob being parsed.
Returns:
str: The text content of the page.
rp )rH ru _extract_images_from_page_extract_tables_from_pager r ) r r- rL ru ry r tables_from_pagerx r~ s r7 r PyMuPDFParser._get_page_content s| " M)LD,<,<)L )LM 99#D 99$? MM* + MM* +)&ArJ c n [ 0 SSSUR UR [ U5 S.EUR Vs0 sH< n[ UR U [
[ 45 ( d M- X1R U _M> snE5 nS H% nX1R ; d M UR U XC' M' U$ s snf )zExtract metadata from the document and page.
Args:
doc: The PyMuPDF document object.
blob: The blob being parsed.
Returns:
dict: The extracted metadata.
r r+ )r; r: r= r9 rX r< )modDate creationDate)ri r9 r rT rQ r_ rS )r r- rF rg rT s r7 r PyMuPDFParser._extract_metadata! s #
) ($& "kk!%#&s8
!\\ )!#,,q/C:> 'A||A& )
" -ALL !ll1o - s +B2
*B2
c B U R ( d gSSKnUR 5 n/ nU GHA nU R ( d M US nUR " X5 n[ R
" UR [ R S9R UR UR S5 n [ R " 5 n
U
R 5 R S:X a M [ R " X5 [" R$ " U
R' 5 SS9n [) U R R+ U 5 5 R, n UR/ [1 XU R2 5 5 GMD [4 R7 [8 R; [= SU5 5 S9$ ) zExtract images from a PDF page and get the text using images_to_text.
Args:
doc: The PyMuPDF document object.
page: The PyMuPDF page object.
Returns:
str: The extracted text from the images on the page.
r+ r Nr rm application/x-npyr r ) r r
get_imagesPixmapr r samplesr r r r r r r r numpyr r r r r r r r rI r r rH r r0 rw )
r r- rL r img_listr1 r4 xrefpiximager rF r s
r7 r 'PyMuPDFParser._extract_images_from_pageA sA !!??$C!!!1v nnS/
ckkBJJJJ 2 !jjl ((*11Q6
;.~~ ((*6I "$"4"4"?"?"E F S S
'$:R:RS# ( !''#((f)=> (
rJ c U R c gSSKn[ UR R " U40 U R
D65 nU( a U R S:X a2 [ R U Vs/ sH oDR 5 PM sn5 $ U R S:X aB [ R U Vs/ sH" nUR 5 R SSSS9PM$ sn5 $ U R S:X aA [ R U Vs/ sH! nUR 5 R SSS 9PM# sn5 $ [ S
U R S 35 egs snf s snf s snf ) zExtract tables from a PDF page.
Args:
page: The PyMuPDF page object.
Returns:
str: The extracted tables in the specified format.
Nr+ r rs rD F)headerindex bold_rowsrt )r r zextract_tables z not implemented)
rp r r table find_tablesrq _JOIN_TABLESr0 to_markdown to_pandasto_htmlto_csvrP )r rL r tables_listr s r7 r 'PyMuPDFParser._extract_tables_from_pagek sr & MM % %d Kd.J.J K
""j0#((;)W;%*;*;*=;)WXX$$.#(( &1
&1E )11#("'&+ 2
&1
$$-#(( &1
&1E )00#("' 1 &1 !%d&9&9%::JK 5 *Xs -E /(E'E) r rp rq r r r r r ru r )ru r r r r r r r r r_ r r r r rp z/Union[Literal['csv', 'markdown', 'html'], None]rq r r rT r ro )rF r ru r r r )r- pymupdf.DocumentrL pymupdf.Pageru rl r r_ )r- r rF r r r )r- r rL r r r_ )rL r r r_ )r r r r r threadingLockr r r r rx r r r r r r r s @r7 ro ro ( s3 2l
NN E 15 $;?
#'*0 77;KQJN<@;?-;? ;?
;? (
;? ;? 5;? I;? H;? ":;?
;? ;?z
15 __
. _
_B
$
: @(
#(
+7(
(
T, ,rJ ro c ^ \ rS rSrSr\R " 5 r S SS\SSS. S U 4S jjjjr S
S jr
SS jr S
r U =r
$ )PyPDFium2Parseri ao Parse a blob from a PDF using `PyPDFium2` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'PyPDFium2' library for PDF processing and offers synchronous
blob parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pypdfium2
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyPDFium2Parser
parser = PyPDFium2Parser(
# password=None,
mode="page",
pages_delimiter="
",
# extract_images = True,
# images_to_text = convert_images_to_text_with_tesseract(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NrL r3 )r r r r r c > [ TU ] 5 US; a [ S5 eXl U( a U( d
[ 5 nXPl X`l X l X0l X@l g)u Initialize a parser based on PyPDFium2.
Args:
password: Optional password for opening encrypted PDFs.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from the PDF.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extraction_mode: “plain” for legacy functionality, “layout” for experimental
layout mode functionality
extraction_kwargs: Optional additional parameters for the extraction
process.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the mode is not "single" or "page".
r r N)
r r rP r r r r r r r )r r r r r r r r s r7 r PyPDFium2Parser.__init__ sU L ) ) :; ; , - . 0M *#6
. rJ c # z # SSK n[ R UR 5 nSn UR
" X0R SS9n/ nSSSS.[ UR 5 5 - nUR US ' [ U5 US
' [ U5 H u pxUR 5 n S R U R 5 R 5 5 n
U R! 5 U R# U5 n [% U /U
5 R' 5 n UR! 5 U R( S :X a8 U R+ S 5 ( d U S -
n [- U [/ 0 UES U0E5 S
9v M UR1 U 5 M U R( S:X a. [- U R2 R U5 [/ U5 S
9v U( a UR! 5 SSS5 SSS5 g! [ a [ S5 ef = f! U( a UR! 5 f f = f! , ( d f NM= f! , ( d f g= f7f)r r NzKpypdfium2 package not found, please install it with `pip install pypdfium2`T)r autoclose PyPDFium2r+ r r9 r< r- rL r r ) pypdfium2r/ r r r PdfDocumentr ri get_metadata_dictr9 r r get_textpager0 get_text_range
splitlinescloser r rd r rd r rU r r )
r rF r rX r r r r rL text_pagery image_from_pager~ s
r7 r PyPDFium2Parser.lazy_parse s)
"
"!!#y !
1+!*!6!6!MMT"J $&L %0#.(*$ (
(D(D(FG $HL
.2[[L*25j/L /-6z-B) $($5$5$7 )- % 4 4 6 A A C* ")*.*H*H*N#9 , -~$ %' !
99.#+#4#4T#:#: (D 0"*-5);%&*6%&(. %&*"# ) / / 95 .C8 yyH, &)-)=)=)B)B<)P%7 %E
""((*g $ #
" +
v ""((* "e $# #
"sa H;G# H;H*HFG<;HH* H;#G99H;<HH
H' #H**
H84H;c U R ( d gSSKJn [ UR UR
4S95 nU( d g/ nU GH n[ R " 5 nUR 5 R 5 nUR S: a MI [ R " XeR 5 R 5 5 [ R " UR 5 SS9n[! U R R# U5 5 R$ n UR' [) XU R* 5 5 UR- 5 GM [. R1 [2 R5 U5 S9$ ) r r+ r N)rw r r r r )r
pypdfium2.rawrawr get_objectsFPDF_PAGEOBJ_IMAGEr r
get_bitmapto_numpysizer r r r r r r r r rI r r r rH r r0 )
r rL pdfium_cr1
str_imagesr r r rF text_from_images
r7 r )PyPDFium2Parser._extract_images_from_pageR s !!(d&&x/J/J.L&MN
E **,K '')224H}}q JJ{$4$4$6$?$?$A B>>+"6"6"8DWXD "4#5#5#@#@#F G T TO #D4;S;ST
KKM !''<3D3DZ3P'QQrJ rj rk )r r r r r r r r_ r r r r r rT r )rL z pypdfium2._helpers.page.PdfPager r_ )r r r r r r r r r r r r r r r s @r7 r r s 0h
NN E %0/ #'*0 77;KQ0/ 0/ 0/
( 0/
0/ 50/ I0/
0/ 0/dM+^ R RrJ r c X \ rS rSrSr S S
S jjrS S jrS S jrS S jrSr g)
PDFPlumberParseris z Parse `PDF` with `PDFPlumber`.Nc x SSK nU=( d 0 U l X l X0l g! [ a [ S5 ef = f)zInitialize the parser.
Args:
text_kwargs: Keyword arguments to pass to ``pdfplumber.Page.extract_text()``
dedupe: Avoiding the error of duplicate characters if `dedupe=True`.
r NzEpillow package not found, please install it with `pip install pillow`)r r/ ru deduper )r ru r r r s r7 r PDFPlumberParser.__init__v sE
'," ,
W
s # 9c # n # SSK nUR 5 nUR " U5 nUR VVs/ sH n[ U R U5 S- U R
U5 - [ UR UR UR S-
[ UR 5 S.40 UR Vs0 sH: n[ UR U 5 [ [ 4; d M+ XdR U _M< snD6S9PM snn Sh vN SSS5 gs snf s snnf N! , ( d f g= f7f)Lazily parse the blob.r Nr- r, )r9 rX rL r< r )
pdfplumberr r r r _process_page_contentr r r9 r r rT r^ r_ rS )r rF r rX r- rL rg s r7 r PDFPlumberParser.lazy_parse s"
9 //),C* II'& &D% !%!;!;D!A"44T:"; "&*kk)-$($4$4q$8+.syy> &)\\%1 #CLLO 4c
B /A||A .%1 $ &'
sR D5"D$A>D 7)D $D 7 D D$ D"
D$ D5D D$$
D2.D5c U R ( a* UR 5 R " S0 U R D6$ UR " S0 U R D6$ )z)Process the page content based on dedupe.rp )r dedupe_charsr ru )r rL s r7 r &PDFPlumberParser._process_page_content sF ;;$$&33Gd6F6FG G 44#3#344rJ c SSK Jn U R ( d g/ nUR GHG nUS S R [
; a US S S:X af UR
[ R " UR SUS S US S
4US R 5 5 R S 5 5 5 M UR
[ R " US R 5 [ R S 9R US S
US S S
5 5 M US S R [ ; a% UR
US R 5 5 GM1 [ R" " S5 GMJ [% U5 $ )z8Extract images from page and get the text with RapidOCR.r r r+ r FilterBitsPerComponentr, 1WidthHeightLr rm r )r r r r1 r r r r r frombytesr convertr r r r warningswarnr8 )r rL r r1 r4 s r7 r *PDFPlumberParser._extract_images_from_page sH "";;C8}X&++/GGx=!349MM !OO #!$X w!7X x9P Q #H
6 6 8 &gcl MM
c(m&<&<&>bhhOWW M( 3S]75KR
X x(--1FF
c(m 4 4 67
34+ . 188rJ )r r ru )NFF)ru zOptional[Mapping[str, Any]]r r r r r rT r )rL zpdfplumber.page.Pager r_ )
r r r r r r r r r r rp rJ r7 r r s sJ ( 48 $ -0- - -
-,:5 9rJ r c J \ rS rSrSr SSS. S S jjjrS
S jrSrg) AmazonTextractPDFParseri a Send `PDF` files to `Amazon Textract` and parse them.
For parsing multi-page PDFs, they have to reside on S3.
The AmazonTextractPDFLoader calls the
[Amazon Textract Service](https://aws.amazon.com/textract/)
to convert PDFs into a Document structure.
Single and multi-page documents are supported with up to 3000 pages
and 512 MB of size.
For the call to be successful an AWS account is required,
similar to the
[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
requirements.
Besides the AWS configuration, it is very similar to the other PDF
loaders, while also supporting JPEG, PNG and TIFF and non-native
PDF formats.
```python
from langchain_community.document_loaders import AmazonTextractPDFLoader
loader=AmazonTextractPDFLoader("example_data/alejandro_rosalez_sample-small.jpeg")
documents = loader.load()
```
One feature is the linearization of the output.
When using the features LAYOUT, FORMS or TABLES together with Textract
```python
from langchain_community.document_loaders import AmazonTextractPDFLoader
# you can mix and match each of the features
loader=AmazonTextractPDFLoader(
"example_data/alejandro_rosalez_sample-small.jpeg",
textract_features=["TABLES", "LAYOUT"])
documents = loader.load()
```
it will generate output that formats the text in reading order and
try to output the information in a tabular structure or
output the key/value pairs with a colon (key: value).
This helps most LLMs to achieve better accuracy when
processing these texts.
``Document`` objects are returned with metadata that includes the ``source`` and
a 1-based index of the page number in ``page``. Note that ``page`` represents
the index of the result returned from Textract, not necessarily the as-written
page number in the document.
N)linearization_configc SSK nSSKJs Jn X@l XPl Ub& U Vs/ sH odR
U5 PM snU l O/ U l Ub X0l O"U R
R SSSSS9U l U( d SSK nUR S 5 U l
gX l
gs snf ! [ a [ S5 ef = f! [ a [ S
5 ef = f) a Initializes the parser.
Args:
textract_features: Features to be used for extraction, each feature
should be passed as an int that conforms to the enum
`Textract_Features`, see `amazon-textract-caller` pkg
client: boto3 textract client
linearization_config: Config to be used for linearization of the output
should be an instance of TextLinearizationConfig from
the `textractor` pkg
r NTz# z## *)hide_figure_layout title_prefixsection_header_prefixlist_element_prefixzCould not import amazon-textract-caller or amazon-textract-textractor python package. Please install it with `pip install amazon-textract-caller` & `pip install amazon-textract-textractor`.textractzRCould not import boto3 python package. Please install it with `pip install boto3`.)textractcaller textractor.entities.documententitiesdocumenttc
textractorTextract_Featurestextract_featuresr r r/ boto3clientboto3_textract_client)r r r r r r fr s r7 r AmazonTextractPDFParser.__init__ s & ' = =G (O ,5F*5F((+5F*& *,&#/,@),0OO,S,S'+!%*/(+ -T -)
-2\\*-E* *0 &E* <
!B
s- B& B!B& !B& ?B? !B& &B<?Cc # # UR ( a [ [ UR 5 5 OSnU( ad UR S:X aT UR ( aC U R
R
[ UR 5 U R U R S9nO\U R
R
UR 5 U R U R
R R U R S9nU R R R U5 n[ UR 5 H8 u pV[ UR# U R$ S9UR& US- S.S9v M: g7f) zIterates over the Blob pages and returns an Iterator with a Document
for each page, like the other parsers If multi-page document, blob.path
has to be set to the S3 URI and for single page docs
the blob.data is taken
Ns3)input_documentfeaturesr )r r call_moder )configr, r9 rL r )pathr r_ schemenetlocr
call_textractr r as_bytesTextract_Call_Mode
FORCE_SYNCr r r r r rH r r9 )r rF url_parse_resulttextract_response_jsonr idxrL s r7 r "AmazonTextractPDFParser.lazy_parseC s" 8>2IC !]]$2K2K] L$(KKqA
3s E0E2)r r r r r )NN)r zOptional[Sequence[int]]r z
Optional[Any]r z!Optional[TextLinearizationConfig]r rT r )r r r r r r r r rp rJ r7 r r sN 0h 6: $=0
CG =02=0 =0
@ =0
=0~!rJ r c 6 \ rS rSrSrSS jrS S jrS
S jrSrg) DocumentIntelligenceParserig zfLoads a PDF with Azure Document Intelligence
(formerly Form Recognizer) and chunks at character level.c H [ R " S5 Xl X l g )Na< langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParserand langchain_community.document_loaders.pdf.DocumentIntelligenceLoader are deprecated. Please upgrade to langchain_community.document_loaders.DocumentIntelligenceLoader for any file parsing purpose using Azure Document Intelligence service.)r r r model)r r r- s r7 r #DocumentIntelligenceParser.__init__k s
rJ c # # UR H\ nSR UR Vs/ sH oDR PM sn5 n[ UUR
UR S.S9nUv M^ g s snf 7f)N r r )r r0 r rG r r9 r )r rF r5 plinerG ds r7 _generate_docs)DocumentIntelligenceParser._generate_docsw s` Ahh A ABG $ "kk MMA G As )A5A0
6A5c # # UR 5 nU R R U R U5 nUR 5 nU R X5 nU Sh vN SSS5 g N
! , ( d f g= f7f)r N)r r begin_analyze_documentr- r5 r4 )r rF file_objpollerr5 docss r7 r %DocumentIntelligenceParser.lazy_parse sf
8[[77
HMF]]_F&&t4DOO
s/ BA
A0 A.!A0% B.A00
A>:B)r r- N)r r r- r_ )rF r r5 r r r r ) r r r r r r r4 r r rp rJ r7 r+ r+ g s A
rJ r+ )r1 z,Sequence[Union[Iterable[np.ndarray], bytes]]r r_ )rF r rG r_ rH r_ r r_ )rT rl r rl )rx r ry r_ r r_ )Cr
__future__r rD r loggingr r r pathlibr tempfiler typingr r r r
r r r
r r r r urllib.parser r r langchain_core.documentsr )langchain_community.document_loaders.baser 1langchain_community.document_loaders.blob_loadersr 3langchain_community.document_loaders.parsers.imagesr r r r r r )textractor.data.text_linearization_configr r r r8 getLoggerr r r r r r rM rI rU ri ru r r r ro r r r r+ rp rJ r7 rH sC . " ' " - D B
Q9 " 8 >
8 $* ! U ,&#N
2je
. e
PJ^ JZ
oN od VRn VRr[9~ [9|Sn Sl& & rJ