jQ,dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZej<eZ Gdd eZ!Gd d eZ"y) z:Pebblo's safe dataloader is a wrapper for document loadersN)version)AnyDictIterableIteratorListOptional)Document) BaseLoader) BATCH_SIZE_BYTESPLUGIN_VERSIONApp FrameworkIndexedDocumentPebbloLoaderAPIWrappergenerate_size_based_batches get_full_pathget_loader_full_pathget_loader_type get_runtimeget_source_sizeceZdZUdZdZeed< dddddeded ed ed e ed ed e ededefdZ de e fdZ d dZdee fdZed dZdefdZde efdZdede e fdZde e fdZde dede fdZdeddfdZy)!PebbloSafeLoaderzkPebblo Safe Loader class is a wrapper around document loaders enabling the data to be scrutinized. F_discover_sentNlocal)classifier_locationanonymize_snippetslangchain_loadernameowner descriptionapi_key load_semanticclassifier_urlrrcr|rt|ts td||_tt j |_||_tjjdxs||_ ||_ ||_ t|j|_g|_g|_tt%|jj'ddj'dd} t)| |_t-|j|_t0|_| |j|j*d|j.dkDrdt|j.ini|_|j7|_t;|||| |_|j<j?|j8y) NzMust specify a valid name.PEBBLO_LOAD_SEMANTIC.'r)loader source_path source_typesource_path_size)r"rr$r) isinstancestr NameErrorapp_nameuuiduuid4load_idr*osenvirongetr#r r!rr+docs docs_with_idtypesplitrr,rr-r batch_sizeloader_details_get_app_detailsappr pb_clientsend_loader_discover) selfrrr r!r"r#r$rr loader_names u/var/www/html/Fitness-lenito-AI-main/venv/lib/python3.12/site-packages/langchain_community/document_loaders/pebblo.py__init__zPebbloSafeLoader.__init__%sn:dC089 9 4::<( & ZZ^^,BCT} &/ <$& 35$t{{+,2237;AA#FqI *;7 /0@0@ A*!++++  ((1,$S)>)>%?@  ((*/ 3)1   ++DHH5returncx|jj|_|j|jS)zxLoad Documents. Returns: list: Documents fetched from load method of the wrapped `loader`. )r*loadr8classify_in_batches)rBs rDrIzPebbloSafeLoader.loadVs. KK$$&    "yyrFct|j|j}g}t|}t |D]\}}||dz k(}||_|j |_|jj|j |j|j|}|j||jr|j|}n|j}|j|||_y)z Classify documents in batches. This is to avoid API timeouts when sending large number of documents. Batches are generated based on the page_content size. ) loading_endN)rr8r<len enumerate _index_docsr9r@classify_documentsr?r=_add_pebblo_specific_metadatar#_add_semantic_to_docs _unindex_docsextend) rBbatchesprocessed_docs total_batchesibatch is_last_batchclassified_docsbatch_processed_docss rDrJz$PebbloSafeLoader.classify_in_batchesas )D IIt) *,G !'*HAu"#}q'8"8MDI $ 0 0 2D "nn??!!##) @O  . . ?!!'+'A'A/'R$'+'9'9';$  ! !"6 7+"# rFc#K |jj} t|}t|f|_ |j|_ |jj|j|j|j }|j#||j$r|j'||_ n|j)|_ |jd#t$rI}|jjjd}t j |t||d}~wwxYw#t$r g|_ YywxYww)zLoad documents in lazy fashion. Raises: NotImplementedError: raised when lazy_load id not implemented within wrapped loader. Yields: list: Documents from loader's lazy loading. z does not implement lazy_load()Nr)r* lazy_loadNotImplementedError __class____name__loggererrornext StopIterationr8listrPr9r@rQr?r=rRr#rSrT)rB doc_iteratorexcerr_strdocclassified_docs rDr_zPebbloSafeLoader.lazy_loads, 8;;002L  <(cV DI $ 0 0 2D !^^>>!!488T-@-@N  . .~ >!! 66~F  ..0 ))A, ! # 8..7788WXG LL !%g.C 7 8!   sKEC'E D<Bz!PebbloSafeLoader._get_app_detailss\ )] 7**((LL)$* 56   rFc t|jDcgc])\}}tddt|i|j +}}}|Scc}}w)z Indexes the documents and returns a list of IndexedDocument objects. Returns: List[IndexedDocument]: A list of IndexedDocument objects with unique IDs. pb_id)rOr8rr/dict)rBrYrkr9s rDrPzPebbloSafeLoader._index_docssU$DII. .3  7#a& 7CHHJ 7.    s.A r\cZ|jDcic].}|jt|j|j0}}|j D]-}|j d}||vs|j|||/|j Dcgc]}|}}|Scc}wcc}w)aF Adds semantic metadata to the given list of documents. Args: classified_docs (Dict): A dictionary of dictionaries containing the classified documents with pb_id as key. Returns: List[Document]: A list of Document objects with added semantic metadata.  page_contentmetadatarw)r9rwr r|r}valuesr7_add_semantic_to_doc)rBr\rk indexed_docsrldoc_idsemantic_metadata_docss rDrSz&PebbloSafeLoader._add_semantic_to_docss(( ( IIxS-=-= U U(  .446N#''0F%)),v*>O7 2>1D1D1F!G1F##1F!G%% "Hs 3B# B(ct|jDcgc]&\}}t|j|j(}}}|Scc}}w)z Converts a list of IndexedDocument objects to a list of Document objects. Returns: List[Document]: A list of Document objects. r{)rOr9r r|r})rBrYrkr8s rDrTzPebbloSafeLoader._unindex_docssP$D$5$56 63 #"2"2S\\ J6    s+A rkrlct|jdij|jd<t|jdij|jd<|S)a4 Adds semantic metadata to the given document in-place. Args: doc (Document): A Document object. classified_doc (dict): A dictionary containing the classified document. Returns: Document: The Document object with added semantic metadata. entitiespebblo_semantic_entitiestopicspebblo_semantic_topics)rgr7keysr})rBrkrls rDrz%PebbloSafeLoader._add_semantic_to_docsg48   z2 . 3 3 54  /026   x , 1 1 32  -. rFc |jD]}|j}|jjjdk(r)t |j d|j|d<n8t |j d|j d|j|d<|j |jij dd|d<y)z*Add Pebblo specific metadata to documents.SharePointLoadersource full_path pb_checksumN) r9r}r*rarbrr7r+rw)rBr\rk doc_metadatas rDrRz.PebbloSafeLoader._add_pebblo_specific_metadatas$$C<rrPrrSrTryrrRrxrFrDrrsD!ND !%#(,/6$+#(/6$/6/6 /6  /6 # /6/6! /6!/6!/6b d8n #@ 8H- D""#, T/2 &T&d8n&2 tH~ $8&TdrFrceZdZdZddddddeedeedeeedeeee fdeeeee fd df d Z d e e fd Z d ee fd Zy) PebbloTextLoaderz Loader for text data. Since PebbloSafeLoader is a wrapper around document loaders, this loader is used to load text data directly into Documents. N)ridsr} metadatastextsrrr}rrGcJ||_||_||_||_||_y)a Args: texts: Iterable of text data. source: Source of the text data. Optional. Defaults to None. ids: List of unique identifiers for each text. Optional. Defaults to None. metadata: Metadata for all texts. Optional. Defaults to None. metadatas: List of metadata for each text. Optional. Defaults to None. N)rrrr}r)rBrrrr}rs rDrEzPebbloTextLoader.__init__s'*    "rFc#Kt|jD]\}}d}|jxsi}|jrE|t |jkr-|j|r|j |j||j r'|t |j kr|j |}t|||yw)zi Lazy load text data into Documents. Returns: Iterator of Documents N)idr|r})rOrr}rrNupdaterr )rBrYtext_idr}s rDr_zPebbloTextLoader.lazy_load9s!,GAtC}}*H~~!c$..&9"9dnnQ>Oq 12xxADHH -hhqkcxH H-sCCcVg}|jD]}|j||S)z` Load text data into Documents. Returns: List of Documents )r_append)rB documentsrks rDrIzPebbloTextLoader.loadIs. >>#C   S !$rF)rbrrrrr/r rrrrErr r_rIrxrFrDrrs!%#'-148#}# # d3i # 4S>* #Dc3h01# #6I8H-I d8n rFr)#rloggingr5r2importlib.metadatartypingrrrrrr langchain_core.documentsr )langchain_community.document_loaders.baser $langchain_community.utilities.pebblor r rrrrrrrrrr getLoggerrbrcrrrxrFrDrsc@ &@@-@       8 $uzup=z=rF