o rgnF@sddlZejdddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z ddlmZddlmZddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZdd lm Z ddl!m"Z"ddl#m$Z%e&Z'eee(deddddZ)ee(deddddZ*eee(dddddZ+GdddZ,GdddZ-e.dkre/dZ0ed1Z2e-e0e2Z3e34dZ5e6e5dSdS)N.) load_dotenv) AgentExecutorcreate_tool_calling_agent)ChatPromptTemplate)tool) ChatOpenAI) OPENAI_EFOPENAI_MODEL_4OOPENAI_MODEL_MINI)Project)ChromaDB) ClientConfig) DocUploaderDOCUMENT_CLASSIFICATIONS) ConvertTable)PerplexityResearch) ResearcherOPENAI_API_KEYg?)api_keymodel temperaturerequest_timeout max_retriesg?)rrrrrc@seZdZdZdZdZdedefddZdd Z dd ed e j e e de fddZdd efddZddZddZdedefddZdedefddZd S) RetrieveFromVectorDBz A class to retrieve data from the Chroma database. This class provides methods to interact with a vector database collection using a specified project ID and client Chroma collection. x< project_idclient_chroma_collectioncCs(||_t}|jj|td|_d|_dS)z&Retrieve data from the chroma database)embedding_functionN)rr chroma_clientget_collectionr vectordb_collectionagent_executor)selfrr chroma_dbr(=/var/www/html/XCapMarket/utils/researcher/chroma_search_v2.py__init__Gs  zRetrieveFromVectorDB.__init__cCs |jjd|jid}t|dkS)'Check if the project contains documentsrwherer)r$queryrlen)r&idsr(r(r)project_contain_docsXs z)RetrieveFromVectorDB.project_contain_docsNrr.exclude_doc_types n_resultsc Csjzt|d}|dur g}|jj|||g|d}|WSty4}z t|gWYd}~Sd}~ww)zQuery the chroma databaserN)r- query_textsr3)maxr$r.create_where_filter Exceptionloggingerror)r&r.r2r3itemser(r(r)r.]s  zRetrieveFromVectorDB.querycCsL|dus t|dkrd|jiSt|tr|g}dd|jidd|iigi}|S)z$Create a where filter for the query.Nrrz$anddoc_classificationz$nin)r/r isinstancestr)r&r2r-r(r(r)r6ss  z(RetrieveFromVectorDB.create_where_filtercCsfg}t|ddD]&\}}|dddvr%||d}|d|q ||dd|q |S)z.Return the documents or tables from the items. metadatasr table_nameNNzTable below is in markdown: documents) enumerategetget_table_from_dynamoappend)r&r: docs_strsidoctabler(r(r)return_docs_or_tablessz*RetrieveFromVectorDB.return_docs_or_tablescCst|}t|d}|S)zGet the table from dynamo db. table_str)rget_table_from_dynamodbdecode_table_text)r&r@rKtable_markdownr(r(r)rFs z*RetrieveFromVectorDB.get_table_from_dynamodoc_strquestionc st}|rtdntdztgd}|tB}|j||ddddIdH}|jWSt yU}zd dl }| t|t j |d d WYd}~d Sd}~ww) zProcess a single document.z%Loop is running before the async callz,Loop is not running, starting the async call)systema You are an Investment Banking managing director, creating a detail overview of a company for a buyer. You strive to provide data-points when answering the question. You are not providing an opinion but rather a factual answer and current analysis. You want to make your answer specific to the company and quantitative, if possible, in nature. When comparing company trends to industry trends and you don't have enough information use [XXX] as a placeholder. If you don't have enough information, do not try to answer the question and only return 'Documents don't contain enough information'. )rTz Answer the question based on the information provided and append to the answer if relevant information is provided. You want to understand the previous answer before adding to it. )humanz-The question you are answering is: {question})rUz)Information you have access to: {doc_str}rRrQr )rtimeoutNrTexc_infozszBRetrieveFromVectorDB.answer_question_from_docs..z cSsg|]}|jqSr()rb)rlresultr(r(r)rns)rr_r`batchjoin)r&rkrRrgrh batch_outputr(rmr)answer_question_from_docssz.RetrieveFromVectorDB.answer_question_from_docs)Nr)N)__name__ __module__ __qualname____doc__ async_timeoutasync_timeout_internalr>r*r1typingLiteraltuplerintr.listr6rLrFrjrsr(r(r(r)r<s0    Crc @seZdZdZdZddedefddZd d Ze e  dd e d e de de de f ddZ e e de de fddZe e dde de de fddZde de fddZdS)InternalSearchz/Class for the primary research for the project.rNrBprojectclientcCs&||_||_||_|durtddS)NzClient cannot be empty.)rr main_question ValueError)r&rrr.r(r(r)r*s zInternalSearch.__init__cCs:t|jj|jjd}|jjd|jjid}t|ddkS)r+rr rr,r0r)rrrrr'r$rEr/)r&retrieve_from_vector_dbr0r(r(r)r1 s z#InternalSearch.project_contain_docsrr.r3rr returnc Cs|durdS|dur dS|durdSzt||d}||j||d}|||WStyO}zddl}|t|tj |dd WYd}~d Sd}~ww) a Search the internal database for information Args: query (str): Question to search for in the database which should be at minimum of 20 words project_id (str): Project to search in client_chroma_collection (str): Collection to search in n_results (int, optional): Number of document chunks to search for. Defaults to 5. Returns: str: Answer to the question NzProject ID cannot be empty.z)Client chroma collection cannot be empty.zQuery cannot be empty.r)r3rTrY?This search was unsuccessful, you can try a different question.) rrLr.rsr7rcrdr^rer9)r.r3rr rrkr;rcr(r(r)search_internals. zInternalSearch.search_internalrRcCs*tgd}|tB}|d|i}|jS)ze Taking a question, generate the relevant sub-questions to answer the main question. ))rTau You are an investment banking managing director. You are doing a deep dive into a company and need to answer the question provided. You have access to an internal database where you can ask natural language questions. Take the question and a sub-question that needs to be answered to get a comprehensive answer. )rUz The main question is: {question})rUzYour output should be a sub-question that assists in answering the main question. It will be passed to another LLM like you. So provide the necessary details.rR)rr_llm4o_sub_queryinvokerb)rRrgrhrir(r(r)question_generationAs z"InternalSearch.question_generationCYou are a research expert specifically for investment banking needspersonac Csrz!t}dg}|r|d|f|d|f||}||WSty8}z t|WYd}~dSd}~ww)a5Search the web for information. Args: question (str): The question to search for. persona (str, optional): The persona to use. Defaults to None. Returns: answer (dict): The answer to the question. answer: str, citations: list(url) )rTz!You are a helpful research agent.rTrUNr)rrGask_perplexity pretty_printr7r^)rRrperplexity_researchmessagesresponser;r(r(r)simple_web_search_s  z InternalSearch.simple_web_searchc Cs|j|jg}tgd}t|t|d}t||dd|_d}|dkrbz|j|d |j j |j j |j j|j d}|d WSty]}z|d7}t|td WYd }~nd }~ww|dks"d S) z=Breakdown the main question and answer it with relevant docs.))rTacYou are an Investment Banking managing director. You are going to be asked a high-level question about a company. You should aim to answer the question comprehensively and provide a detailed answer. You want to make sure to provide a detailed answer with data points and comparisons to the relevant industry. )rTaYou have access to tools. You should use them to answer the question. Before breaking down the question further, always try to answer the question with the internal search tool. If the answer is not sufficient, then breakdown the question to get sub-questions and a research plan. Rely on the internal search tool to get information and then only augment the information with external sources using the simple web search tool.)rUa Breakdown the slide into the questions you want the slide to answer and then use the tools to answer those questions. Ensure the questions are answered, otherwise invoke the tools again to answer the questions for a maximum of 4 times. If there is a statistic about the company, you want to make sure to compare it against the industry overall or an average using the Researcher tool. For financial questions, return the numerical value and the source of the information.)rUz1The company we are researching is: {company_name})rUzSome tools are going to ask for a project Id: {project_id} and a client chroma collection: {client_chroma_collection}. Make sure to pass it in.)rUzGYou have the following information about the company: {company_project})rUz2The question you are answering is: {main_question}) placeholderz{agent_scratchpad})toolsllmpromptT)agentrverboser )r company_namerr company_projectoutputz retrying...NzFailed to get an answer)rrrr_rllm35rr%rrqrcompany_alt_namesrrr'stringify_company_infor7r^)r&rRrrgrretry agent_outputr;r(r(r)break_down_question_and_answer{sD (  z-InternalSearch.break_down_question_and_answerrA)rNN)r)rtrurvrwrxr rr*r1 staticmethodrr>r}rrrrr(r(r(r)rsF  *r__main__ sonivate_xcmxcmz&How much much has the company raised? )7syspathrGr[r8osrzdotenvrlangchain.agentsrrlangchain.promptsrlangchain.toolsrlangchain_openairconfigs.configr r r +services.ppt_generator.data_classes.projectr utils.chroma_dbr utils.client_checkr!utils.document_loader.DocUploaderrr!utils.document_loader.word_loaderrutils.researcher.perplexityrutils.researcher.researcher_v2r ResearcherV2 getLoggerregetenvrr`rrrrtcheck_project_in_dbprget_client_configcustomersearchr search_answerr^r(r(r(r)st            AO