j 2 d Z ddlmZ ddlZddlZddlZddlZddlZddlmZ ddl m
Z
ddl m Z ddl
mZmZmZmZmZmZmZmZmZmZmZ ddlmZ ddlZddlZ dd l m Z dd
l m Z dd l!m"Z" dd l#m$Z$m%Z% erddl&Z&ddl'Z'ddl(Z(ddl)Z)dd
l*m+Z+ g dZ,g dZ- d)dZ. ej^ e0 Z1dZ2dZ3dZ4dZ5h dZ6d*dZ7d+dZ8d+dZ9ddgZ:d,dZ; G d d e Z< G d d e Z= G d d e Z> G d! d"e Z? G d# d$e Z@ G d% d&e ZA G d' d(e ZBy)-z(Module contains common parsers for PDFs. ) annotationsN)datetime)Path)TemporaryDirectory)
TYPE_CHECKINGAnyBinaryIOIterableIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)BaseBlobParser)Blob)BaseImageBlobParserRapidOCRBlobParser)TextLinearizationConfig) DCTDecodeDCT JPXDecode)
LZWDecodeLZW FlateDecodeFl
ASCII85DecodeA85ASCIIHexDecodeAHxRunLengthDecodeRLCCITTFaxDecodeCCF JBIG2Decodec ddl m} | }d}| D ]6 } || \ }}|s|D cg c] }|d }}dj | z
}8 |S # t $ r t d w xY wc c}w )zExtract text from images with RapidOCR.
Args:
images: Images to extract text from.
Returns:
Text extracted from images.
Raises:
ImportError: If `rapidocr-onnxruntime` package is not installed.
r )RapidOCRzc`rapidocr-onnxruntime` package not found, please install it with `pip install rapidocr-onnxruntime`
)rapidocr_onnxruntimer* ImportErrorjoin)imagesr* ocrtextimgresult_s z/var/www/html/Fitness-lenito-AI-main/venv/lib/python3.12/site-packages/langchain_community/document_loaders/parsers/pdf.py!extract_from_images_with_rapidocrr8 @ s
1 *C
DH *01&$d1g&F1 DIIf% %D
K
1
2s A A&A#z
{image_text}
r- z
> sourcecreatorproducer total_pages creationdatec |rU| j xs d}|dk( r |j dd }d| d| d}|S |dk( r d t j |d
d | d
}|S )a Format the content of the image with the source of the blob.
blob: The blob containing the image.
format::
The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(`
`)
#z markdown-img]z\\]z![z]()zhtml-imgz
![T)quotez src=]()
)r9 replacehtmlescape)blobcontentformatr9 s r7 _format_inner_imagerI i sx # ^ # ooc62G7)2fXQ/G N z
!"4;;wd#C"DF6(RVWG N c t j | j s t d t | j dd t s t d | S )zValidate that the metadata has all the standard keys and the page is an integer.
The standard keys are:
- source
- total_page
- creationdate
- creator
- producer
Validate that page is an integer if it is present.
z3The PDF parser must valorize the standard metadata.pager z(The PDF metadata page must be a integer.)_STD_METADATA_KEYSissubsetkeys
ValueError
isinstancegetint)metadatas r7 _validate_metadatarU sJ & &x}} 7NOO hll61-s 3CDD OrJ c i }ddd}| j D ] \ }}t | t t fvr t | }|j d r|dd }|j }|dv r: t
j |j dd d
j d ||< ||v r|||| < |||< t |t r|j ||< t |t s|||< |S # t $ r |||< Y w xY w) zPurge metadata from unwanted keys and normalize key names.
Args:
metadata: The original metadata dictionary.
Returns:
The cleaned and normalized the key format of metadata dictionary.
r< r9 )
page_count file_path/r, N)r= moddate'r+ zD:%Y%m%d%H%M%S%zT)
itemstypestrrS
startswithlowerr strptimerC isoformatrP rQ strip)rT new_metadatamap_keykvs r7 _purge_metadatari s $&L# G 1 73* $AA << !"A
GGI + +
$"*"3"3IIc2&(:#)C. Q
'\'(L $ LO
3
ggiLO
3
LO) !*
$"# Q
$s +8C44DDz
c dfd | |d }|s1d}dj t d | }|r t d |z }||z }|S )a5 Insert extras such as image/table in a text between two paragraphs if possible,
else at the end of the text.
Args:
extras: List of extra content (images/tables) to insert.
text_from_page: The text content from the page.
Returns:
The merged text with extras inserted.
c | rwt D ]j }|j | }|dk7 sd }|r
| |d | d }|r |||d z }n3d}dj t d | }|r||z }|d | |z ||d z } |S d }|S |}|S )NFr+ rj c | S N xs r7 zO_merge_text_and_extras.._recurs_merge_text_and_extras.. s !rJ )_PARAGRAPH_DELIMITERrfindr0 filter)
extrastext_from_pagerecursdelimpos
previous_textall_text
all_extras
str_extras _recurs_merge_text_and_extrass
r7 r z=_merge_text_and_extras.._recurs_merge_text_and_extras s -$**51"9$(M (E "N4C$8%)
%#0>#$3G#G%'
%+[[ V1L%M
%).);J *4C 0: =st@T T !
1 .* &HrJ Tr+ rj c | S ro rp rq s r7 rs z(_merge_text_and_extras.. s !rJ rm )rw list[str]rx r_ ry boolreturn
Optional[str])r0 rv rt )rw rx r} r~ r r s @r7 _merge_text_and_extrasr sv +. 8< < -V^TJH
[[ V!<=
-b1J>J!J. OrJ c h e Zd ZdZ d
deddddd d fdZd dZd
d Z xZS ) PyPDFParsera Parse a blob from a PDF using `pypdf` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images.
It integrates the 'pypdf' library for PDF processing and offers synchronous blob
parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pypdf
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyPDFParser
parser = PyPDFParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# images_parser = TesseractBlobParser(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NrL r3 plain)modepages_delimiter
images_parserimages_inner_formatextraction_modeextraction_kwargsc t | |dvr t d || _ |r |s
t }|| _ || _ || _ || _ || _ || _
|xs i | _ y)u Initialize a parser based on PyPDF.
Args:
password: Optional password for opening encrypted PDFs.
extract_images: Whether to extract images from the PDF.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extraction_mode: “plain” for legacy functionality, “layout” extract text
in a fixed width format that closely adheres to the rendered layout in
the source pdf.
extraction_kwargs: Optional additional parameters for the extraction
process.
Raises:
ValueError: If the `mode` is not "single" or "page".
singlerL mode must be single or pageN) super__init__rP extract_imagesr r r passwordr r r r )
selfr r r r r r r r __class__s
r7 r zPyPDFParser.__init__$ sv J ) ) :; ; , - . 0M *#6
. . !2!8b rJ c # @ K ddl d fd }|j 5 } j | j }t dddd t
t |j xs i z |j t |j d
z }g }t |j D ] \ }} || } j | }
t |
g| j } j dk( r,t# | t% |||j& | d z
v|j) | j dk( r1t# j* j- | t% |
ddd y# t $ r t d w xY w# 1 sw Y yxY ww)m
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
Raises:
ImportError: If the `pypdf` package is not found.
Yield:
An iterator over the parsed documents.
r NzE`pypdf` package not found, please install it with `pip install pypdf`rL c j j d r| j S | j ddj ij S )z
Extract text from image given the version of pypdf.
Args:
page: The page object to extract text from.
Returns:
str: The extracted text.
3r rp ) __version__r` extract_textr r )rL pypdfr s r7 _extract_text_from_pagez7PyPDFParser.lazy_parse.._extract_text_from_pagem sY ++C0((**(t(( $($8$8,, rJ r PyPDFr+ r; r: r= )r9 r< )rL )rL
page_label page_contentrT r )rL zpypdf.PageObjectr r_ )r r/ as_bytes_io PdfReaderr ri r dictrT r9 lenpages enumerateextract_images_from_pager rd r r rU page_labelsappendr r0 )
r rF r pdf_file_obj
pdf_reader doc_metadata single_texts page_numberrL rx images_from_pager} r s
` @r7
lazy_parsezPyPDFParser.lazy_parseW s $
<(
NJ* $"MtZ 0 0 6B78 #kk#&z'7'7#8 L L%.z/?/?%@! T!8d!C#'#@#@#F 1%& %' 99&"%-!3 ((3.8.D.D[.Q " !''1% &A& yyH$ !%!5!5!:!:S>S>U3V(W XH NN#89'"$**,K",,.55: OOH-22;u2M >>+*>*>*@KXD!%d&8&8&C&CD&I!J!W!WJMM+D*d>V>VW9 > !''#((f)=> (
rJ NF)r zOptional[Union[str, bytes]]r r r Literal['single', 'page']r r_ r Optional[BaseImageBlobParser]r +Literal['text', 'markdown-img', 'html-img']r zLiteral['plain', 'layout']r Optional[dict[str, Any]]rF r r Iterator[Document])rL zpypdf._page.PageObjectr r_ ) __name__
__module__ __qualname____doc___DEFAULT_PAGES_DELIMITERr r r
__classcell__r s @r7 r r s .d 15 $19
+1 77;KQ6=6:19-19 19
( 19
19 519 I19 419 419fKZ4
rJ r c e Zd ZdZdZ d ddedddd d
fdZedd Zedd Z d dd
Z
dd Z xZ S )PDFMinerParsera Parse a blob from a PDF using `pdfminer.six` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'pdfminer.six' library for PDF processing and offers synchronous
blob parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pdfminer.six pillow
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PDFMinerParser
parser = PDFMinerParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# extract_images = True,
# images_to_text = convert_images_to_text_with_tesseract(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
FNr r3 )r r r r r concatenate_pagesc , t | |dvr t d |r |s
t }|| _ || _ || _ || _ || _ || _ |`)
concatenate_pages: Deprecated. If True, concatenate all PDF pages
into one a single document. Otherwise, return one document per page.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the `mode` is not "single" or "page".
Warnings:
`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'
instead.
r r NTzS`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'` instead.r rL )r r rP r r r r r r r r _warn_concatenate_pagesr r ) r r r r r r r r r s r7 r zPDFMinerParser.__init__ s P ) ) :; ; - . 0M , *#6
. (!999=6= %66DI )rJ c ddl m t | t r!| j d rt | dd dd S d | D }d j
fd
|D S # t $ r t | cY S w xY w) z
Decodes a PDFDocEncoding string to Unicode.
Adds py3 compatibility to pdfminer's version.
Args:
s: The string to decode.
Returns:
str: The decoded Unicode string.
r )PDFDocEncodings Nzutf-16beignorec 3 V K | ]! }t |t r t | n| # y wro )rQ r_ ord).0cs r7 z-PDFMinerParser.decode_text..] s# CAjC 0CFa7s ')r+ c 3 ( K | ] }| y wro rp )r or s r7 r z-PDFMinerParser.decode_text..^ s ;d>!,ds )pdfminer.utilsr rQ bytesr` r_ r0
IndexError)sordsr s @r7 decode_textzPDFMinerParser.decode_textL sn 2 a ALL$=quj(3 3 CCD77;d;; ; q6M s "A A43A4c ddl m} t | d r| j } t | t
r#t t
t j | S t | | r t j | j S t | t t f rt j | S t | t r2| j D ] \ }}t j | | |< | S | S )z
Recursively resolve the metadata values.
Args:
obj: The object to resolve and decode. It can be of any type.
Returns:
The resolved and decoded object.
r ) PSLiteralresolve)pdfminer.psparserr hasattrr rQ listmapr resolve_and_decoder namer_ r r r] )r r rg rh s r7 r z!PDFMinerParser.resolve_and_decodeb s 0 3 "++-C c4 N = =sCD D
Y
'!--chh7 7
c5\
*!--c2 2
T
" 1'::1=A $J
rJ c ddl m}m}m} || } |||| }i } |j D ] }
| j |
| j
D ] \ } } t j | | | < t t |j | | d< | S # t $ r*}
t j d| t |
Y d}
~
wd}
~
ww xY w)ag
Extract metadata from a PDF file.
Args:
fp: The file pointer to the PDF file.
password: The password for the PDF file, if encrypted. Defaults to an empty
string.
caching: Whether to cache the PDF structure. Defaults to True.
Returns:
Metadata of the PDF file.
r ) PDFDocumentPDFPage PDFParser)r cachingzD[WARNING] Metadata key "%s" could not be parsed due to exception: %sNr< )pdfminer.pdfpager r r infoupdater] r r Exceptionr r r_ r r create_pages)r fpr r r r r parserdocrT r rg rh es r7
_get_metadataz PDFMinerParser._get_metadata~ s $ ED 2 &8WEHHD OOD ! NN$DAq
, ? ? B % #&d7+?+?+D&E"F
$F
s B C$ C Cc # K ddl }ddlm} ddlm}mmm}m}m m
ddl m }m
} ddlm} t! |j" dk r t% d |j' 5 }
t) 5 | j+ |
j, xs d
} | } t/ d d d
d
j1 |
j, xs d
z }
|j2 |
d< G fdd| }t5 j6 || || | }g }t9 | D ] \ }}j; d j= d |j? | jA }|jC } jD dk( r@j; d j= d tG |tI |
d|iz |jK d r|dd }|jM | jD dk( r3 jN jQ | }tG |tI |
ddd ddd y# t$ $ r t% d w xY w# 1 sw Y *xY w# 1 sw Y yxY ww)a
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
Raises:
ImportError: If the `pdfminer.six` or `pillow` package is not found.
Yield:
An iterator over the parsed documents.
r N)PDFLayoutAnalyzer)LAParams LTContainerLTImageLTItemLTPageLTText LTTextBox)PDFPageInterpreterPDFResourceManager)r i:>4zThis parser is tested with pdfminer.six version 20201018 or later. Remove pdfminer, and install pdfminer.six with `pip uninstall pdfminer && pip install pdfminer.six`.zMpdfminer package not found, please install it with `pip install pdfminer.six`r+ r PDFMinerr r9 c N e Zd Z d d fd
Zdfd Z xZS )*PDFMinerParser.lazy_parse..Visitorc * t | ||| y )N)pagenolaparams)r r )r rsrcmgrr1 r2 r s r7 r z3PDFMinerParser.lazy_parse..Visitor.__init__ s G$WVh$OrJ c 2 d fd | y )Nc J t | r| D ]
} | n+t | r j | j t |
rj d y t | r j rddlm} |
}|j
| }t j t
|z }d|j d< t j j | j }j t || j y y y )Nr- r ) ImageWriterr? r9 )rQ writeget_textr pdfminer.imager6 export_imager from_pathr rT r r r rI r )itemchildr6 image_writerfilenamerF r r% r& r) r* renderr tempdirtext_ios r7 r@ zIPDFMinerParser.lazy_parse..Visitor.receive_layout..render s %dK8)- &u
*. 'f 5 #MM$--/ :%dI6 #MM$ / 'g 6 # 1 1 F/:7/C +7+D+DT+J'+~~d7mh6N'O:=
h 7-1$($6$6$A$A$$G.""., !+ !(
$7(,j$:R:R%&!" 2" !rJ )r< r' r Nonerp )
meltpager@ r% r& r) r* r rA rB s
@r7 receive_layoutz9PDFMinerParser.lazy_parse..Visitor.receive_layout s ! !8 6NrJ )r, N)r3 r, r1 rS r2 zOptional[LAParams]r rC )rE r( r rC )r r r r rF r )r r% r& r) r* r rA rB s @r7 Visitorr/ sE #$37 P /P P 1 P
P # #rJ rG )r2 rL r rm r ))pdfminerpdfminer.converterr# pdfminer.layoutr$ r% r& r' r( r) r* pdfminer.pdfinterpr+ r, r r rS r r/ r r get_pagesr ri r! r9 r StringIOr truncateseek process_pager rd r r rU endswithr r r0 )r rF rI r# r$ r' r( r+ r, r r r r3 r rG visitor_for_all all_contentirL r} document_contentr% r& r) r* rA rB s ` @@@@@@r7 r zPDFMinerParser.lazy_parse sY <
R 08''(83!L 4
<1C1E%%lT]]=Pb%QE(*G* 'JPRS$$\DMM O K$U+4 # Q ,,T2"++-#>>+99&$$Q'LLO"%-!3LFA;4N!O
((.#+CR= &&x0% ,& yyH$#'#7#7#<#<[#I !1 / = Y 2F
2
2F1E
sN I;A
I I;) I/4GI#:I/ I; I I;#I, (I//I84I;F)r r r r r r r r_ r r r r r zOptional[bool])r zUnion[bytes, str]r r_ )r r r r )r+ T)r r r r_ r r r dict[str, Any]r )
r r r r r r r staticmethodr r r! r r r s @r7 r r s 0d $ %:B #'*2 77;KQ,0:B :B :B
( :B
:B 5:B I:B *:Bx * < , , , ,
,\yrJ r c e Zd ZdZ ej
Z d
ddeddddd d fdZddZ d ddZ
dd Z dd
Z dd Z
dd Z xZS )
PyMuPDFParsera Parse a blob from a PDF using `PyMuPDF` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'PyMuPDF' library for PDF processing and offers synchronous blob
parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pymupdf
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyMuPDFParser
parser = PyMuPDFParser(
# password = None,
mode = "single",
pages_delimiter = "
",
# images_parser = TesseractBlobParser(),
# extract_tables="markdown",
# extract_tables_settings=None,
# text_kwargs=None,
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NrL r3 )r r r r r extract_tablesextract_tables_settingsc t
| |dvr t d |r|dvr t d || _ || _ || _ |xs i | _ |r |s
t }|| _ || _ || _
|| _ | | _ y)a Initialize a parser based on PyMuPDF.
Args:
password: Optional password for opening encrypted PDFs.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from the PDF.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extract_tables: Whether to extract tables in a specific format, such as
"csv", "markdown", or "html".
extract_tables_settings: Optional dictionary of settings for customizing
table extraction.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the mode is not "single" or "page".
ValueError: If the extract_tables format is not "markdown", "html",
or "csv".
r r )markdownrD csvzmode must be markdownN)
r r rP r r r text_kwargsr r r r r\ r] ) r ra r r r r r r r\ r] r s r7 r zPyMuPDFParser.__init__a s V ) ) :; ; n4O O 45 5 .
&," - . 0M ,#6 * ,'>$rJ c $ | j | S ro ) _lazy_parse)r rF s r7 r zPyMuPDFParser.lazy_parse s
rJ c # K ddl }|xs | j }| j sNddlm}m}m}m} i ddddddddd dd
|d dd dd
|ddddddd|d|dddddddddddd| _ t j 5 |j 5 }|j |j | } n |j |d } | j r| j | j ddd d | j# | | z }
g } | D ]k } | j% | | | j' }
| j( d k( r(t+ |
t- |
d | j. iz [| j1 |
m | j( d k( r1t+ | j2 j5 | t- |
ddd ddd y# t $ r t d w xY w# 1 sw Y *xY w# 1 sw Y yxY ww)!a
Lazily parse the blob.
Insert image, if possible, between two paragraphs.
In this way, a paragraph can be continued on the next page.
Args:
blob: The blob to parse.
text_kwargs: Optional keyword arguments to pass to the `get_text` method.
If provided at run time, it will override the default text_kwargs.
Raises:
ImportError: If the `pypdf` package is not found.
Yield:
An iterator over the parsed documents.
r N)DEFAULT_JOIN_TOLERANCE DEFAULT_MIN_WORDS_HORIZONTALDEFAULT_MIN_WORDS_VERTICALDEFAULT_SNAP_TOLERANCEclipvertical_strategylineshorizontal_strategyvertical_lineshorizontal_linessnap_tolerancesnap_x_tolerancesnap_y_tolerancejoin_tolerancejoin_x_tolerancejoin_y_toleranceedge_min_length min_words_verticalmin_words_horizontalintersection_toleranceintersection_x_toleranceintersection_y_tolerance)text_tolerancetext_x_tolerancetext_y_tolerancestrategy add_lineszGpymupdf package not found, please install it with `pip install pymupdf`pdf)streamfiletypePyMuPDFr+ r rL r r )pymupdfra r]
pymupdf.tablere rf rg rh r/ r[ _lockr datar is_encrypted authenticater _extract_metadata_get_page_contentrd r r rU numberr r r0 )r rF ra r re rf rg rh rX r r full_contentrL r} s r7 rc zPyMuPDFParser._lazy_parse s ,) %9)9)9K// 0D0 (0 *7 0
%d 0 '
0 %&<0 '0 '0 %&<0 '0 '0 &q0 )*D 0 +,H 0 -a!0" /#0$ /%0& '(()() $!%/0,>
!!#y99$&',,y1C&',,i%HC##$$T]]3 ) ($& **35 6
" D # 5 5c4 M S S UHyyF* &)1%7 , /D D& %++H5 99("%)%9%9%>%>|%L!3L!A 5 $ !
-
$# !
sN G:A.G
2G:G.D&G"9G. G:
G G:"G+ 'G..G73G:c |j di i | j |}| j || }| j | }g }|r|j | |r|j | t || }|S )a: Get the text of the page using PyMuPDF and RapidOCR and issue a warning
if it is empty.
Args:
doc: The PyMuPDF document object.
page: The PyMuPDF page object.
blob: The blob being parsed.
Returns:
str: The text content of the page.
rp )r8 ra _extract_images_from_page_extract_tables_from_pager r ) r r rL ra rx r tables_from_pagerw r} s r7 r z PyMuPDFParser._get_page_content s " ' M)LD,<,<)L )LM 99#tD 99$? MM* + MM* +)&.ArJ c X t i ddd|j |j t | d|j D ci c]5 }t |j | t
t f r||j | 7 c} }dD ]# }||j v s|j | ||< % |S c c}w )zExtract metadata from the document and page.
Args:
doc: The PyMuPDF document object.
blob: The blob being parsed.
Returns:
dict: The extracted metadata.
r r+ )r; r: r= r9 rX r< )modDate creationDate)ri r9 r rT rQ r_ rS )r r rF rg rT s r7 r z PyMuPDFParser._extract_metadata! s #
) ($& "kk!%#&s8
!\\ )!#,,q/C:> s||A& )
" -ACLL !ll1o - s :B'
c 4 | j syddl}|j }g }|D ]= }| j s|d } |j || }t j
|j t j j |j |j d } t j }
|
j j dk( rt j |
| t# j$ |
j' d } t) | j j+ | j, } |j/ t1 | | | j2 @ t4 j7 t8 j; t= d| S ) a Extract images from a PDF page and get the text using images_to_text.
Args:
doc: The PyMuPDF document object.
page: The PyMuPDF page object.
Returns:
str: The extracted text from the images on the page.
r+ r Nr rm application/x-npyr r ) r r
get_imagesPixmapr r samplesr r r r r r r r numpyr r r r r r r r rI r r rH r r0 rv )
r r rL r img_listr1 r4 xrefpiximager rF r s
r7 r z'PyMuPDFParser._extract_images_from_pageA sF !!??$C!!1v$gnnS$/
ckkBJJJJ 2 !jjl ((*11Q6
;.~~ ((*6I "$"4"4"?"?"E F S S
'j$:R:RS# ( !''#((f)=> (
rJ c
| j yddl}t |j j |fi | j
}|r| j dk( r1t j |D cg c] }|j c} S | j dk( rCt j |D cg c]$ }|j j ddd & c} S | j dk( rBt j |D cg c]# }|j j dd % c} S t d
| j d yc c}w c c}w c c}w ) zExtract tables from a PDF page.
Args:
page: The PyMuPDF page object.
Returns:
str: The extracted tables in the specified format.
Nr+ r r_ rD F)headerindex bold_rowsr` )r r zextract_tables z not implemented)
r\ r r table find_tablesr] _JOIN_TABLESr0 to_markdown to_pandasto_htmlto_csvrP )r rL r tables_listr s r7 r z'PyMuPDFParser._extract_tables_from_pagek sv & %GMM % %d Kd.J.J K
""j0#((;)W;%%*;*;*=;)WXX$$.#(( &1
&1E )11#("'&+ 2
&1
$$-#(( &1
&1E )00#("' 1 &1 !%d&9&9%::JK 5 *Xs &E&)E8(E r )ra r r r r r r r r r_ r r r r r\ z/Union[Literal['csv', 'markdown', 'html'], None]r] r r rC r ro )rF r ra r r r )r pymupdf.DocumentrL pymupdf.Pagera rX r r_ )r r rF r r r )r r rL r r r_ )rL r r r_ )r r r r threadingLockr r r r rc r r r r r r s @r7 r[ r[ ( s+ 2l
INN E 15 $;?
#'*0 77;KQJN<@;?-;? ;?
;? (
;? ;? 5;? I;? H;? ":;?
;?z
15 __
. _
_B
$
: @(
#(
+7(
(
T,rJ r[ c e Zd ZdZ ej
Z d ddeddd d
fdZd dZ d dZ
xZ S )
PyPDFium2Parserao Parse a blob from a PDF using `PyPDFium2` library.
This class provides methods to parse a blob from a PDF document, supporting various
configurations such as handling password-protected PDFs, extracting images, and
defining extraction mode.
It integrates the 'PyPDFium2' library for PDF processing and offers synchronous
blob parsing.
Examples:
Setup:
.. code-block:: bash
pip install -U langchain-community pypdfium2
Load a blob from a PDF file:
.. code-block:: python
from langchain_core.documents.base import Blob
blob = Blob.from_path("./example_data/layout-parser-paper.pdf")
Instantiate the parser:
.. code-block:: python
from langchain_community.document_loaders.parsers import PyPDFium2Parser
parser = PyPDFium2Parser(
# password=None,
mode="page",
pages_delimiter="
",
# extract_images = True,
# images_to_text = convert_images_to_text_with_tesseract(),
)
Lazily parse the blob:
.. code-block:: python
docs = []
docs_lazy = parser.lazy_parse(blob)
for doc in docs_lazy:
docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)
NrL r3 )r r r r r c t | |dvr t d || _ |r |s
t }|| _ || _ || _ || _ || _ y)uk Initialize a parser based on PyPDFium2.
Args:
password: Optional password for opening encrypted PDFs.
mode: The extraction mode, either "single" for the entire document or "page"
for page-wise extraction.
pages_delimiter: A string delimiter to separate pages in single-mode
extraction.
extract_images: Whether to extract images from the PDF.
images_parser: Optional image blob parser.
images_inner_format: The format for the parsed output.
- "text" = return the content as is
- "markdown-img" = wrap the content into an image markdown link, w/ link
pointing to (`![body)(#)`]
- "html-img" = wrap the content as the `alt` text of an tag and link to
(``)
extraction_mode: “plain” for legacy functionality, “layout” for experimental
layout mode functionality
extraction_kwargs: Optional additional parameters for the extraction
process.
Returns:
This method does not directly return data. Use the `parse` or `lazy_parse`
methods to retrieve parsed documents with content and metadata.
Raises:
ValueError: If the mode is not "single" or "page".
r r N)
r r rP r r r r r r r )r r r r r r r r s r7 r zPyPDFium2Parser.__init__ sa L ) ) :; ; , - . 0M *#6
. rJ c # T K ddl }t j 5 |j 5 }d} |j
|| j d }g }ddddt |j z }|j |d < t | |d
< t | D ] \ }}|j } d j | j j }
| j! | j# | } t% | g|
j' } |j! | j( d k( r5| j+ d s| d z
} t- | t/ i |d |i
|j1 | | j( dk( r1t- | j2 j | t/ |
|r|j! ddd ddd y# t $ r t d w xY w# |r|j! w w xY w# 1 sw Y BxY w# 1 sw Y yxY ww)r r NzKpypdfium2 package not found, please install it with `pip install pypdfium2`T)r autoclose PyPDFium2r+ r r9 r< r- rL r r ) pypdfium2r/ r r r PdfDocumentr ri get_metadata_dictr9 r r get_textpager0 get_text_range
splitlinescloser r rd r rR r rU r r )
r rF r rX r r r r rL text_pagerx image_from_pager} s
r7 r zPyPDFium2Parser.lazy_parse s-
"
"!!#y !
1+!6!6!6!DMMT"J $&L %0#.(*$ (
(D(D(FG $HL
.2[[L*25j/L /-6z-B) T$($5$5$7 )- % 4 4 6 A A C* ")*.*H*H*N#9 , -~$ %' !
99.#+#4#4T#: (D 0"*-5);%&*6%&(. %&*"# ) / / 95 .C8 yyH, &)-)=)=)B)B<)P%7 %E
""((*g $ #
" +
v ""((* "e $# #
"sa H(G H(H HFG8<HH H( G55H(8H
HH H H%!H(c | j syddlm} t |j |j
f }|syg }|D ] }t
j }|j j }|j dk rFt j ||j j t j |j d }t! | j j# | j$ } |j' t) || | j* |j- t. j1 t2 j5 | S ) r r+ r N)rv rv r r r )r
pypdfium2.rawrawr get_objectsFPDF_PAGEOBJ_IMAGEr r
get_bitmapto_numpysizer r r r r r r r r rI r r r rH r r0 )
r rL pdfium_cr1
str_imagesr r r rF text_from_images
r7 r z)PyPDFium2Parser._extract_images_from_pageR s! !!(d&&x/J/J.L&MN
E **,K '')224H}}q JJ{E$4$4$6$?$?$A B>>+"6"6"8DWXD "4#5#5#@#@#F G T TO #D/4;S;ST
KKM !''<3D3DZ3P'QQrJ rW )r r r r r r r r_ r r r r r rC r )rL z pypdfium2._helpers.page.PdfPager r_ ) r r r r r r r r r r r r r s @r7 r r s 0h
INN E %0/ #'*0 77;KQ0/ 0/ 0/
( 0/
0/ 50/ I0/
0/dM+^ RrJ r c F e Zd ZdZ d ddZd dZd
dZd
dZy) PDFPlumberParserz Parse `PDF` with `PDFPlumber`.Nc p ddl }|xs i | _ || _ || _ y# t $ r t d w xY w)zInitialize the parser.
Args:
text_kwargs: Keyword arguments to pass to ``pdfplumber.Page.extract_text()``
dedupe: Avoiding the error of duplicate characters if `dedupe=True`.
r NzEpillow package not found, please install it with `pip install pillow`)r r/ ra deduper )r ra r r r s r7 r zPDFPlumberParser.__init__v sI
'," ,
W
s 5c # \ K ddl }|j 5 } |j | }|j D cg c] }t | j | dz | j
| z t |j |j |j dz
t |j dfi |j D ci c]6 }t |j | t t fv r||j | 8 c} c}}E d{ ddd yc c}w c c}}w 7 # 1 sw Y yxY ww)Lazily parse the blob.r Nr- r, )r9 rX rL r< r )
pdfplumberr r r r _process_page_contentr r r9 r r rT r^ r_ rS )r rF r rX r rL rg s r7 r zPDFPlumberParser.lazy_parse s&
9!*//),C* II'& &D% !%!;!;D!A"44T:"; "&*kk)-$($4$4q$8+.syy> &)\\%1 #CLLO 4c
B s||A .%1 $ &'
sL D,"D A>D7;D 2 D>D D D
D,DD D)%D,c | j r* |j j di | j S |j di | j S )z)Process the page content based on dedupe.rp )r dedupe_charsr ra )r rL s r7 r z&PDFPlumberParser._process_page_content sJ ;;34$$&33Gd6F6FG G t 44#3#344rJ c ddl m} | j syg }|j D ]> }|d d j t
v r|d d dk( rd|j
t j |j d|d d |d d
f|d j j d |j
t j |d j t j j |d d
|d d d
|d d j t v r$|j
|d j *t! j" d A t% | S )z8Extract images from page and get the text with RapidOCR.r r r+ r FilterBitsPerComponentr, 1WidthHeightLr rm r )r r r r1 r r r r r frombytesr convertr r r r warningswarnr8 )r rL r r1 r4 s r7 r z*PDFPlumberParser._extract_images_from_page sH "";;C8}X&++/GGx=!349MM !OO #!$X w!7X x9P Q #H
6 6 8 &gcl MM
c(m&<&<&>bhhOWW M( 3S]75KR
X x(--1FF
c(m 4 4 67
34+ . 188rJ )NFF)ra zOptional[Mapping[str, Any]]r r r r r rC r )rL zpdfplumber.page.Pager r_ )r r r r r r r r rp rJ r7 r r s sJ ( 48 $ -0- - -
-,:5 9rJ r c : e Zd ZdZ ddd ddZddZy) AmazonTextractPDFParsera Send `PDF` files to `Amazon Textract` and parse them.
For parsing multi-page PDFs, they have to reside on S3.
The AmazonTextractPDFLoader calls the
[Amazon Textract Service](https://aws.amazon.com/textract/)
to convert PDFs into a Document structure.
Single and multi-page documents are supported with up to 3000 pages
and 512 MB of size.
For the call to be successful an AWS account is required,
similar to the
[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
requirements.
Besides the AWS configuration, it is very similar to the other PDF
loaders, while also supporting JPEG, PNG and TIFF and non-native
PDF formats.
```python
from langchain_community.document_loaders import AmazonTextractPDFLoader
loader=AmazonTextractPDFLoader("example_data/alejandro_rosalez_sample-small.jpeg")
documents = loader.load()
```
One feature is the linearization of the output.
When using the features LAYOUT, FORMS or TABLES together with Textract
```python
from langchain_community.document_loaders import AmazonTextractPDFLoader
# you can mix and match each of the features
loader=AmazonTextractPDFLoader(
"example_data/alejandro_rosalez_sample-small.jpeg",
textract_features=["TABLES", "LAYOUT"])
documents = loader.load()
```
it will generate output that formats the text in reading order and
try to output the information in a tabular structure or
output the key/value pairs with a colon (key: value).
This helps most LLMs to achieve better accuracy when
processing these texts.
``Document`` objects are returned with metadata that includes the ``source`` and
a 1-based index of the page number in ``page``. Note that ``page`` represents
the index of the result returned from Textract, not necessarily the as-written
page number in the document.
N)linearization_configc ddl }ddlmc m} || _ || _ |%|D cg c] }|j
| c}| _ ng | _ ||| _ n$| j
j dddd | _ |s ddl }|j d | _
y|| _
yc c}w # t $ r t d w xY w# t $ r t d
w xY w) a5 Initializes the parser.
Args:
textract_features: Features to be used for extraction, each feature
should be passed as an int that conforms to the enum
`Textract_Features`, see `amazon-textract-caller` pkg
client: boto3 textract client
linearization_config: Config to be used for linearization of the output
should be an instance of TextLinearizationConfig from
the `textractor` pkg
r NTz# z## *)hide_figure_layout title_prefixsection_header_prefixlist_element_prefixzCould not import amazon-textract-caller or amazon-textract-textractor python package. Please install it with `pip install amazon-textract-caller` & `pip install amazon-textract-textractor`.textractzRCould not import boto3 python package. Please install it with `pip install boto3`.)textractcaller textractor.entities.documententitiesdocumenttc
textractorTextract_Featurestextract_featuresr r r/ boto3clientboto3_textract_client)r r r r r r fr s r7 r z AmazonTextractPDFParser.__init__ s & ' = =DG (DO ,5F*5FB((+5F*& *,&#/,@),0OO,S,S'+!%*/(+ -T -)
-2\\*-E* *0D &E* <
!B
s' !B$ B >B$ <B< B$ $B9<Cc # K |j r t t |j nd}|ra|j dk( rR|j rF| j
j
t |j | j | j }n_| j
j
|j | j | j
j j | j }| j j j | }t |j D ]> \ }}t |j# | j$ |j& |dz d @ yw) zIterates over the Blob pages and returns an Iterator with a Document
for each page, like the other parsers If multi-page document, blob.path
has to be set to the S3 URI and for single page docs
the blob.data is taken
Ns3)input_documentfeaturesr )r r call_moder )configr, r9 rL r )pathr r_ schemenetlocr
call_textractr r as_bytesTextract_Call_Mode
FORCE_SYNCr r r r r r8 r r9 )r rF url_parse_resulttextract_response_jsonr idxrL s r7 r z"AmazonTextractPDFParser.lazy_parseC s% 8>2IC !]]$2K2K] L$(KKqA
3s E+E-)NN)r zOptional[Sequence[int]]r z
Optional[Any]r z!Optional[TextLinearizationConfig]r rC r )r r r r r r rp rJ r7 r r sN 0h 6: $=0
CG =02=0 =0
@ =0
=0~!rJ r c ( e Zd ZdZddZddZddZy) DocumentIntelligenceParserzjLoads a PDF with Azure Document Intelligence
(formerly Form Recognizer) and chunks at character level.c J t j d || _ || _ y )Na< langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParserand langchain_community.document_loaders.pdf.DocumentIntelligenceLoader are deprecated. Please upgrade to langchain_community.document_loaders.DocumentIntelligenceLoader for any file parsing purpose using Azure Document Intelligence service.)r r r model)r r r s r7 r z#DocumentIntelligenceParser.__init__k s#
rJ c # K |j D ]] }dj |j D cg c] }|j c} }t ||j
|j d }| _ y c c}w w)N r r )r r0 rk rG r r9 r )r rF r5 plinerG ds r7 _generate_docsz)DocumentIntelligenceParser._generate_docsw sc Ahh A ABG $ "kk MMA G As )A5A0
7A5c # K |j 5 }| j j | j | }|j }| j || }|E d{ ddd y7
# 1 sw Y yxY ww)r N)r r begin_analyze_documentr r5 r )r rF file_objpollerr5 docss r7 r z%DocumentIntelligenceParser.lazy_parse sj
8[[77
HMF]]_F&&tV4DOO
s/ A=AA1!A/"A1& A=/A11A:6A=N)r r r r_ )rF r r5 r r r r )r r r r r r r rp rJ r7 r r g s A
rJ r )r1 z,Sequence[Union[Iterable[np.ndarray], bytes]]r r_ )rF r rG r_ rH r_ r r_ )rT rX r rX )rw r rx r_ r r_ )Cr
__future__r rD r loggingr r r pathlibr tempfiler typingr r r r
r r r
r r r r urllib.parser r r langchain_core.documentsr )langchain_community.document_loaders.baser 1langchain_community.document_loaders.blob_loadersr 3langchain_community.document_loaders.parsers.imagesr r r r r r )textractor.data.text_linearization_configr r r r8 getLoggerr r r r r r rM rI rU ri rt r r r r[ r r r r rp rJ r7 r! sE . " ' " - D B
Q9 " 8 >
8 $* ! U ,&#N
2je
. e
PJ^ JZ
oN od VRn VRr[9~ [9|Sn Sl& & rJ