o gY @sddlZejdddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZdd lmZmZdd lmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-e.Z/e0dZ1ej2dZ3dZ4dZ5dZ6dZ7gdZ8GdddeZ9GdddeZ:GdddeZ:dd d!gd"d#ee9d$d%dd d!gddddZ;Gd&d'd'Z .d@d/e=d0e"d1e?d2e e=e ffd3d4Z@d5e e=d(e=d*ed2e e=e ffd6d7ZAd5e e=d(e=d*ed+e"d2df d8d9ZBeCd:krAd;ZDe"d<EZFe@eDeFd=d>ej2dZ3eFj0d?eDd?ZGdSdS)AN.)ThreadPoolExecutor)DictListOptionalUnionAny)ChatPromptTemplate)%SentenceTransformersTokenTextSplitter)JsonOutputParser) ChatOpenAI) BaseModelField) OPENAI_EFOPENAI_MODEL_MINI)Project)ChromaDB) ClientConfig)LoadPDF) LoadWordDoc)DynamoDB) SitemapScrape)S3BucketStorageget_default_bucket_names3xcap_s3_storage).pdf.docxz.pptxi皙?z_+_) comapny_infocompany_financialindustryotherwebsitec@sNeZdZUdZedddZeed<edddZeed<edddZ eed <d S) FinancialInfoz)Model for financial document information..z Document Typetitle document_typezStart Date YYYY-MM-DD start_datezEnd Date YYYY-MM-DDend_dateN) __name__ __module__ __qualname____doc__rr'str__annotations__r(r)r0r0=/var/www/html/XCapMarket/utils/document_loader/DocUploader.pyr$/s r$c@&eZdZUdZedddZeed<dSDocClassificationz"Model for document classification..zClassification of the documentr%classificationNr*r+r,r-rr5r.r/r0r0r0r1r47 r4c@r2r3r6r0r0r0r1r4>r7z'What is the legal name of the business?company_alt_names)question attributeaHWhat type of financial document is this: tax, income_statement, balance_sheet, cash_flow, Other? If other, explain it. It can be multiple typesWhat is the date range of the document? Start date and end date return a JSON format: document_type, start_date: YYYY-MM-DD, end_date: YYYY-MM-DDcompany_financialspydantic_object)r9r: output_parserc@s\eZdZdZ d,dededeefddZdedee e ee ffd d Z d edefd d Z de efddZ    d-de ee fdeedeedeedeede eeff ddZd.dededefddZeddZd/ddZd e ee fddfd!d"Zd#e edefd$d%Zd#ee eefdee eeffd&d'Zd/d(d)Zd/d*d+ZdS)0 DocUploaderz2Upload and process documents to various databases.Nprojectclientdoc_pathcCstttjddd|_t|_t|jt|_ t |_ d|_ ||_ ||_d|_g|_d|_g|_||_d|_d|_d|_d|_d|_d|_d|_d|_|rQ|jj|_|ritj||_||j|_|||_dSdS)zInitialize the DocUploader. Args: project: Project object containing project information client: ClientConfig object with client-specific settings doc_path: Path to the document to be processed OPENAI_API_KEYr)modelapi_key temperatureTN)r rosenvironllm35DEFAULT_CHUNK_SIZE chunk_sizeintDEFAULT_CHUNK_OVERLAP_RATIO chunk_overlaprdb supportedr@rAcustomer_chroma_collectionchunks doc_summarychunks_summary file_pathdoc_type file_namedoc_classificationdocument_text_chunked chroma_dbpathbasenameclassify_documentget_document_text)selfr@rArBr0r0r1__init__bs>  zDocUploader.__init__returncCs||}|drt|}d|_|j S|ds|dr)t||j}d|_|j Sd|_td|td|dS) zGet the document text in chunks. Args: doc_path: Path to the document Returns: List of document chunks or None if document type is not supported rpdf_docrz.docword_docFzUnsupported document type: N) lowerendswithrrWrr@rQ XCM_loggerwarningrZ)r`rBlower_doc_path doc_loaderr0r0r1r_s   zDocUploader.get_document_textfilenamecCsRtgd}ttd}||jB|B}|dtdt||d}|dS)zClassify the document based on its filename. Args: filename: Name of the document file Returns: Classification of the document ))systemz2You are a financial analyst able to classify items)rlzGYou will choose between the following classes: {classification_choices})rlau Here are some examples of the classification: "TTM through March 2024" -> "company_financial" "questionnaire" -> "company_info" "23611D Remodeling in the US Industry Report -IBIS" -> "industry" "2023 P and L-Bal Sheet" -> "company_financial" "2024 YTD comparison"" -> "company_financial" )humanz%The document file name is: {filename})rmzJReturn the answer in JSON format as only the classification: {JSON_format}r< )classification_choicesrork JSON_formatr5) r from_messagesr r4rJinvokejoinDOCUMENT_CLASSIFICATIONSget_format_instructions)r`rkpromptparserchain chain_outputr0r0r1r^s   zDocUploader.classify_documentcCst}|jj|jtd}g}d}|jdurgSg}g}g}|jD]S}t|j|jd} | |d} t | D]-\} } | || } | |}d|vrN|d|d<| | | || | | | q5td|t|j|j|d7}q |j|||d|S) z_Upload the document to ChromaDB. Returns: List of document chunks embedding_functionN)rLrOtext table_namez.Uploaded chunk %s of %s File: %s to ChromaDBids documents metadatas)r chroma_clientget_or_create_collectionrRrrZr rLrO split_text enumerate unique_idstandardize_metadataappendrginfolenrXupsert)r`chromadb collectionrS chunk_number unique_idsdocuments_chunksrchunksplitter text_chunksi text_chunkrmetadatar0r0r1upload_to_chromadbsL          zDocUploader.upload_to_chromadbrrXrWrY chunk_typecCsN|r|n|j}|r |n|j}|r|n|j}|r|n|d}|jj||||dS)aVStandardize the metadata for document chunks. Args: chunk: Document chunk file_name: Name of the file doc_type: Type of document doc_classification: Classification of document chunk_type: Type of chunk Returns: Standardized metadata dictionary type) project_idrXr'rYr)rXrWrYr@r)r`rrXrWrYrr0r0r1rs z DocUploader.standardize_metadatarrrcCs t|jj|jt|t|gS)zCreate a unique ID for the document chunk. Args: chunk_number: Number of the chunk i: Index within the chunk Returns: Unique ID string )DELIMITER_TEXTrsr@rrXr.)r`rrr0r0r1r3s zDocUploader.unique_idcCsddd|DS)z5Remove non-printable characters from the text fields.rGcss|] }|r|VqdSN) isprintable).0cr0r0r1 Dsz-DocUploader._sanitize_text..)rs)r}r0r0r1_sanitize_textAszDocUploader._sanitize_textcstdjjtjjddd}fdd|jD}t}|D].}ddl}| dfd d |D}| j j |td |d td |d q"j j j jj d d |jd|jdS)z,Upload the website to ChromaDB and DynamoDB.z Uploading website %s to ChromaDBT) summarize exclude_blogsc s6g|]\}}jj|d|d|ddqS)rnr}summary)rurlr}r)r@rrrs)rritemr`r0r1 Msz1DocUploader.vectorize_website..rNg?cs*i|]\}}|t|tr|n|qSr0) isinstancer.r)rkvrr0r1 \sz1DocUploader.vectorize_website..zUploaded website %s to DynamoDBrimages)rrr}r)rgrr@ company_urlr urls_scrapeditemsrtimesleepupload_to_dynamodbrP web_pagesr image_urlsupload_website_to_vectorDB)r`scrapeitems_to_uploadrPrr item_cleanr0rr1vectorize_websiteFs4      zDocUploader.vectorize_websitewebsite_scrapedc Cst}|jj|jtd}d|_g}g}g}|D]1\}}|ddus)|ddkr*q||} |j||dddd} | | | | | |dq|j |||d dS) zzUpload the website to VectorDB. Args: website_scraped: Dictionary of scraped website content rzWebsiterNrGr#r})rWrYrr) rrrrRrrXrrrrr) r`rrrrr document_textrrrrr0r0r1ros6    z&DocUploader.upload_website_to_vectorDBrScsV||IdH}t|dkr|d}n d|}||IdH}||_||_||fS)zSummarize the document chunks. Args: chunks: List of document chunks Returns: Tuple of (chunks_summary, doc_summary) Nr|rrn)summarize_chunkrrsrTrU)r`rSrUrTr0r0r1summarize_chunkss   zDocUploader.summarize_chunkscsntgd}||jB}t|tr(|dd|DIdH}dd|D}|S|d|iIdH}|j}|S)zSummarize a chunk or list of chunks. Args: chunks: Single chunk or list of chunks to summarize Returns: Summarized content ))rlza You are skilled investment banker and expert in summarizing information as a financial analyst. )rlzSummarize the following text:)rmzThe following text:)rmz{chunk}cSsg|]}d|iqS)rr0)rrr0r0r1rsz/DocUploader.summarize_chunk..NcSsg|]}|jqSr0)content)r chunk_outputr0r0r1rsr)r rqrJrlistabatchainvoker)r`rSrvrxryoutputr0r0r1rs  zDocUploader.summarize_chunkc Cs<t}||j|jj|j|j|j|j|j |j |j ddS)z Upload the document to DynamoDB.)rrBrYrXrWrTrUrSN) rr project_docsr@rrVrYrXrWrTrUrS)r`rPr0r0r1rszDocUploader.upload_to_dynamodbc Cst|jdus |jtvr dSt|jdurdSt|jD]}gd}d}|ddur@|dt|}||jB|dB}n t|}||jB}tdt |j dD]B}|j ||d}d |}|ddur|| |d |||d d }n | |d ||d }t|d ddur|}qS|j}qSt|j|d } | durg} | |t|j|d | q|jdS)z-Process follow-up questions for the document.N))rlzYYou are a skilled knowledge worker. Answer the questions asked and return only the answer)rmzThe questions is {question})rmzBYou have access to the following chunk of the document: {document})rmz]You have access to the previous answer to the question as well: {previous_answer}. Update it.rGr>)rmz>Your output should be in the following format: {output_format}rz r9)r9documentprevious_answer output_format)r9rrrr:)rYrtfollow_up_questions_to_askgetrr rqrJrangerrSrsrrrugetattrrr@setattrupdate_project_in_db) r`r9prompt_messagesrrvrxrrryattribute_to_updater0r0r1follow_up_questionss^         zDocUploader.follow_up_questionsr)NNNN)r)rbN)r*r+r,r-rrrr.rarrrr_r^rrrMr staticmethodrrrtuplerrrrrr0r0r0r1r?_sV "/18   #  )) "r? bucket_namerBr@ client_configcCszB td|dd|d|}t|||}td|j||_|t | |j| td|dWdSt yVtj d|d d YdSw) Nz Uploading z to ChromaDBzs3:///zDocument classification: z Uploaded z to DynamoDBzError processing document: T)exc_info)printr?rgrrYrrSrasynciorunrr Exceptionerror)rrBr@r doc_uploaderr0r0r1r8s   rFrrAprocess_on_threadsrbcCs`|}|jd|d}tj|d}ttd}|t|}|r*t|t||ddiSt|t|S)aUpload documents to ChromaDB. Args: project_id: ID of the project client: ClientConfig object process_on_threads: Whether to process documents on threads Returns: Dictionary with task information or success message r)r)rmessagezFiles uploaded successfully.) rArcheck_project_in_dbrrlist_files_in_folderrprocess_docs_on_threadsdoc_upload_create_task)rrArr folder_namer@ s3_storagefilesr0r0r1doc_uploader_callLs     rrc Cs^ddlm}ddlm}g}|D]}||j|||j}||q||j|j}||dS)aCreate Celery tasks for document uploading. Args: files: List of file paths bucket_name: Name of the S3 bucket project: Project object client_config: ClientConfig object Returns: Dictionary with task IDs r)process_uploaded_doc)r) file_tasksr#)tasks.presentation_tasksrrdelayrrAr) rrr@rcelery_vectorize_websiterfile file_taskwebsite_vector_taskr0r0r1rhs     rcshttdtddfdd|D}|D]}|qWddS1s-wYdS)zProcess documents on threads for concurrent uploads. Args: files: List of file paths bucket_name: Name of the S3 bucket project: Project object client_config: ClientConfig object r|r) max_workersc sg|] }t|qSr0)submitr)rrBrrexecutorr@r0r1rsz+process_docs_on_threads..N)rminrH cpu_countresult)rrr@rfuturesfuturer0rr1rs  "r__main__z8east-cobb-lifestyle-8318100a-2160-4435-bdf9-ab16b055e2cesunbeltT)rr)F)Hsysr\rrloggingrHconcurrent.futuresrtypingrrrrrboto3langchain.promptsr langchain.text_splitterr langchain_core.output_parsersr langchain_openair pydanticr rconfigs.configrr+services.ppt_generator.data_classes.projectrutils.chroma_dbrutils.client_checkr utils.document_loader.pdf_loaderr!utils.document_loader.word_loaderrutils.dynamo_dbrutils.webscrape.sitemap_scraperutils.s3_storagerr getLoggerrgrArrIrVALID_DOCUMENT_EXTENSIONSrKrNrrtr$r4rr?r.rboolrrrr*rget_client_configrrr0r0r0r1s                  \