dhc*SSKJr SSKrSSKrSSKrSSKrSSKrSSKJr SSK J r SSK J r J r JrJrJr SSKJr SSKJr SSKJr SS KJr SS KJrJr SS KJr SS KJr \R@"\!5r"S r#Sr$Sr%Sr&/SQr'/SQr(S/r)/SQr*\'\(\)\*S.r+"SS\,\5r-"SS\5r."SS\5r/"SS\5r0"SS\5r1"S S!\5r2S-S"jr3S.S#jr4S/S$jr5S0S%jr6S1S&jr7S2S3S'jjr8S4S(jr9S5S)jr:S6S*jr;"S+S,\5rr8cP[R5HupX;dM Us $ g)zReturn loader type among, file, dir or in-memory. Args: loader (str): Name of the loader, whose type is to be resolved. Returns: str: One of the loader type among, file/dir/in-memory. unsupported)LOADER_TYPE_MAPPINGitems)loader loader_typeloaderss r9get_loader_typer|s+!4 9 9 ;   !< r8cSSKJnJnJnJn Sn[ U[ 5(d[RS5 U$URnSU;a\[ X5(aSURSUR3nGO[ X5(aSURSUR3nGOS U;a'US nU(aS U;aUS nU(aUSU3nGOmS U;aUS nGO`S U;aUS nGOSS U;a7US nU(a)[ U[5(a[U5S:aUSnGO[ X5(aSnGO[ X5(aSUR3nOUR R"S:XaUR%S5(aUR%S5n SU 3nOUR%S5(a8UR%S/5n SR'U V s/sH n SU S3PM sn 5nOMUR%S5(a7UR%S/5n SR'U V s/sH n SU S3PM sn 5n[+[-U55$s sn fs sn f![(a N*f=f)zReturn an absolute source path of source of loader based on the keys present in Document. Args: loader (BaseLoader): Langchain document loader, derived from Baseloader. r)r%rr&rrnzGloader is not derived from BaseLoader, source location will be unknown!bucketzgc://rlzs3://sourcechannelrG file_path web_pathsr+z notiondb://r' folder_idz+https://drive.google.com/drive/u/2/folders/file_idsz, z https://drive.google.com/file/d/z/view document_idsz#https://docs.google.com/document/d/z/edit)$langchain_community.document_loadersr%rr&r isinstancerloggererror__dict__r~blobkeyrblen database_id __class__r0getjoin Exceptionrtr=)ryr%rr&rlocation loader_dictrrrrfile_idrdoc_ids r9get_loader_full_pathrsnH fj ) ) U //K/ { "&00"6==/6;;-@F11"6==/6::,?  $"8,HI4%i0"*1WI6H { ""6*H K '";/H K '#K0IZ 488S^a=O$Q<  0 0"H  / /$V%7%7$89H    & &*= ={++'OOK8 H T,,&??:r:99(0'/G;7)5I'/ 00*~rB 99'3&2F>fXUK&2 X ''!    sm2I"<,I"*+I" I"$ I"1;I".I"I""AI")7I" I/I"77I".I=I" I"" I/.I/c [5n[SURSS5S9n[R"5n[ UR [RSURSS5URUR[5URSS5URS S5S 9nS UR;aS Ul S Ul [RSU35 [RSU35 X4$)zFetch the current Framework and Runtime details. Returns: Tuple[Framework, Runtime]: Framework and Runtime for the current app instance. langchainlibrary_versionN)rTrUPWDrKrmrPruntime_version)rFrGrKrLrMrJrNrODarwindesktopzMac OSXz framework zruntime )r rRrrKunamerBnoderLenvironsystemrUget_iprErPrdebug) runtime_envr\rrPs r9 get_runtimers *+K +//2CT"JI NN E ZZ ZZ Y7 <<== 8I6$):IF G7::  # LL:i[)* LL8G9%&  r8cSSKnUR5nURU5nU$![a URS5nU$f=f)z>Fetch local runtime ip address. Returns: str: IP address rN localhost)socket gethostname gethostbynamer)rrF public_ips r9rr.sY     D6((.   6((5 6s)AAc2/n/nSnUHrn[URRS55nXa:aURU/5 M@XF-U:aURU5 /nSnURU5 XF- nMt U(aURU5 U$)a Generate batches of documents based on page_content size. Args: docs: List of documents to be batched. max_batch_size: Maximum size of each batch in bytes. Defaults to 100*1024(100KB) Returns: List[List[Document]]: List of batches of documents rutf-8)r page_contentencodeappend)rcmax_batch_sizebatches current_batchcurrent_batch_sizedocdoc_sizes r9generate_size_based_batchesr>s%'G$&MC,,33G<=  $ NNC5 !!,~=}- " %&"   %  * !&}% Nr8cSSKn[R"U5RnUR U5R nU$![ a SnU$f=f)zoFetch owner of local file path. Args: file_path (str): Local file path. Returns: str: Name of owner. rNrm)pwdrLstatst_uidgetpwuidpw_namer)rrfile_owner_uidfile_owner_names r9get_file_owner_from_pathrfsV$+22,,~6>>  $# $s?A AAcU(dgSn[RRU5(a![RRU5nU$[RR U5(aSn[R "U5Hvup4nUHjn[RR X65n[RRU5(aMHU[RRU5- nMl Mx UnU$)zFetch size of source path. Source can be a directory or a file. Args: source_path (str): Local path of data source. Returns: int: Source size in bytes. r)rLrGisfilegetsizeisdirwalkrislink) source_pathsize total_sizedirpath_ filenamesffps r9get_source_sizerys  D ww~~k""ww{+ K { # # %'WW[%9 !G WW\\'-ww~~b))"''//""55J&:  Kr8c>URS5n[U5nU$)zCalculate the content size in bytes: - Encode the string to bytes using a specific encoding (e.g., UTF-8) - Get the length of the encoded bytes. Args: data (str): Data string. Returns: int: Size of string in bytes. r)rr)dataencoded_contentrs r9calculate_content_sizers!kk'*O  D Kr8cJ^\rSrSr%SrS\S'SrS\S'S\S'S\S 'S rS \S 'SU4S jjrSSjr SSSjjr SSjr SSSjjr SSjr \SSSjj5r\S Sj5r\S!Sj5rSrU=r$)"PebbloLoaderAPIWrapperizWrapper for Pebblo Loader API.rIapi_keyrDr=riclassifier_url cloud_urlFrfrjc >[USSS5US'[USS[5US'[USS[5US'[TU]"S 0UD6 g) z%Validate that api key in environment.rPEBBLO_API_KEYrHrPEBBLO_CLASSIFIER_URLrPEBBLO_CLOUD_URLNr/)r _DEFAULT_CLASSIFIER_URL_DEFAULT_PEBBLO_CLOUD_URLsuper__init__)selfkwargsrs r9rPebbloLoaderAPIWrapper.__init__sh0 I/ y$8 $&=?V$  3 K!35N { "6"r8cTSnURSS9nURS:XaKUR5nUR[R R 3nURSXTU5nUR(aURSS9nU(aB[R"UR5RS5nURSU05 URS[05 UR[R R 3nURSXtU5ngg) zk Send app discovery request to Pebblo server & cloud. Args: app (App): App instance to be discovered. NT exclude_unsetrDPOST cloud_requestpebblo_server_versionpebblo_client_version)rdri _make_headersrr-r6value make_requestrjsonloadstextrupdatePLUGIN_VERSIONr) rapp pebblo_resppayloadheadersapp_discover_urlrpebblo_cloud_urlrs r9send_loader_discover+PebbloLoaderAPIWrapper.send_loader_discovers (((.  # #w .((*G&&'(B(B(H(H'IJ ++F4DwWK <<((t(z3An Exception caught in classify_documents: local %sNrjz pebblo-cloudz4API key is missing for sending docs to Pebblo cloud.)rrprepare_docs_for_classificationbuild_classification_payloadrirrr-r5rrrrrrrrwarningrupdate_doc_datapopsend_docs_to_pebblo_cloud NameError)r docs_with_idrrergrrhrcsource_aggregate_sizerclassified_docsr load_doc_urlrclassified_doces r9classify_documents)PebbloLoaderAPIWrapper.classify_documentss"%((; / < &*&J&J ~' #33 ~5J   # #w .((*G"11263D3D3J3J2KLL Y"//L7C *.**[5E5E*F*J*J6SU*V'..+G4nE+W <<''72$$WV_oF KK,d 3  * *7 3   % % 7 NNQ RRS S! YTVWXX YsA(F F/F**F/cURSS9nUR[RR3nUR SX2U5ng![ a n[RSU5 SnAgSnAff=f)zi Send documents to Pebblo cloud. Args: payload (dict): The payload containing documents to be sent. Trrz3An Exception caught in classify_documents: cloud %sN) rrr-r5rrrrr)rrrrrr s r9r0PebbloLoaderAPIWrapper.send_docs_to_pebblo_cloudsx$$4$8"nn-f.?.?.E.E-FG U!!&*:WMA U NNPRS T T UsA A7A22A7cSSS.nU(aEUR(aURSUR05 U$[RS5 U$)z Generate headers for the request. args: cloud_request (bool): flag indicating whether the request is for Pebblo cloud. returns: dict: Headers for the request. zapplication/json)Acceptz Content-Typez x-api-keyz,API key is missing for Pebblo cloud request.)rrrr)rrrs r9r$PebbloLoaderAPIWrapper._make_headers(sM).  || T\\:;MNr8c URURU[URUSUURUR S. nUSLaSUS'SU;aUUSS'[ S 0UD6RSS9nU$) a Build the payload for document classification. Args: app (App): App instance. docs (List[dict]): List of documents to be classified. loader_details (dict): Loader details. source_owner (str): Owner of the source. source_aggregate_size (int): Aggregate size of the source. loading_end (bool): Boolean indicating the halt of data loading by loader. Returns: dict: Payload for document classification. false) rTrYrcr]r[rergrhrirjTtruergrerrr/)rTrYrr[rirjr`rd)rrrcrerhrrgrs r9r3PebbloLoaderAPIWrapper.build_classification_payload?s0HHYY,{{,"(#'#;#;"&"9"9 #  $ %+GM "7*)()*AB..%%D%9r8c H[XX#US9n[RSUURR[ [ URR (aURR O/55[ UR55 UR[R:a$[RSUR35 U$UR[R:a$[RSUR35 U$UR[R:wa"[RSUR35 U$![a [RSU5 g[a n[RSU5 SnAgSnAff=f) a Make a request to the Pebblo API Args: method (str): HTTP method (GET, POST, PUT, DELETE, etc.). url (str): URL for the request. headers (dict): Headers for the request. payload (Optional[dict]): Payload for the request (for POST, PUT, etc.). timeout (int): Timeout for the request in seconds. Returns: Optional[Response]: Response object if the request is successful. )methodurlrrtimeoutz5Request: method %s, url %s, len %s response status %szPebblo Server: Error z$Pebblo received an invalid payload: z-Pebblo returned an unexpected response code: zUnable to reach server %sz'An Exception caught in make_request: %sN)rrrrr=rbody status_coderINTERNAL_SERVER_ERRORr BAD_REQUESTrOKrr)rrrrrresponser s r9r#PebbloLoaderAPIWrapper.make_requestks`* IwH LLG  $$C1A1A1F1F((--BOPH(()  ##z'G'GG!6x7K7K6LMNO%%)?)??!Ehmm_UVO %%6C++,. O = NN6 < I NNDa H H Is,CEAEAE F!8 F!FF!c B/nSnUVs/sHoUR5PM nnSnUGHmnURS05nURS/5n USS:Xa[URSUS55n O+[URS URSU555n URS [U 55n URS [ U 55n [ URS 55n [ U 5nXN- nURS S5=(d SnURU U UURS05RS5U S.U (aSU 0O0EU bSU 0O0E5 USS:XdGMMU(aGMWURS5US'SnGMp X44$s snf)a< Prepare documents for classification. Args: docs_with_id (List[IndexedDocument]): List of documents to be classified. source_path (str): Source path of the documents. loader_details (dict): Contains loader info. Returns: Tuple[List[dict], int]: Documents and the aggregate size of the source. rFmetadataauthorized_identitiesryr(rrrsrYrrr>N last_modified)rrr>r$ file_ownersource_path_sizesource_full_urlT)rdrrtrrr=rr)rrrercrr doc_contentsource_path_update doc_metadatadoc_authorized_identitiesdoc_source_pathdoc_source_ownerdoc_source_sizerpage_content_sizers r9r6PebbloLoaderAPIWrapper.prepare_docs_for_classifications$ !-9:\cxxz\ :"C77:r2L(4(8(89PRT(U %h'+=="/ $$X~m/LM##0 $$#$((;?# ,//1/B  +..v7WXOsww~67L 6| D  ! 6 !WWWd+0qF KK'#2#%(WWZ%<%@%@%Q"2 512KL+6,_= (x(,>>**0<0@0@AR0S}-%)"]^**c;sFc  UH}nURUS05nURURS5URS5URS05URS05S.5 URS5 M g) z Update the document data with classified information. Args: docs (List[dict]): List of document data to be updated. classified_docs (dict): The dictionary containing classified documents. r> pb_checksumloader_source_pathentitiestopics)r2r3r4r5rN)rrr)rcrdoc_dataclassified_datas r9r&PebbloLoaderAPIWrapper.update_doc_datasH-11(72CRHO OO#2#6#6}#E*9*=*=>R*S / 3 3J C-11(B?   LL r8r/)rr)rrWreturnNone)F) rList[IndexedDocument]rrWrerdrgrfr9rd)rrdr9r:)rrfr9rd)rrWrc List[dict]rerdrhr=rintrgrfr9rd)N) rr=rr=rrdrzOptional[dict]rr=r9zOptional[Response])rr;rr=rerdr9zTuple[List[dict], int])rcr<rrdr9r:)r0r1r2r3r4r@rirjrrr rrr staticmethodrrrr7 __classcell__)rs@r9rrs( "&&I!!&!$$U #NH" <+<< <  <  <| U.* ** *  * # ** *X #' // // /  /  //bD++D+D+D+ D+D+L  r8r)rGr=r9r=)ryr=r9r=)ryrr9r=)r9zTuple[Framework, Runtime])r9r=)r)rczList[Document]rr=r9zList[List[Document]])rr=r9r=)rr=r9r=)rr=r9r=)= __future__rrloggingrLrorKenumrhttprtypingrrrr r langchain_core.documentsr langchain_core.envr langchain_core.utilsr pydanticrrequestsrrrequests.exceptionsr)langchain_community.document_loaders.baser getLoggerr0rrrrBATCH_SIZE_BYTES file_loader dir_loader in_memory cloud_folderrwr=r-r;rBrRrWr`rtr|rrrrrrrrr/r8r9rSs;"  33-65&0@   8 $11         -S$-%h%5i52$ $+)+*V)V2. E(P> "1;% %*-%%P&2  W YW r8