tjN|ddlmZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZej Z!d Z"d%dZ#d&dZ$d'dZ%e e&ee&e'e(e)ffZ*GddeZ+GddeZ,GddeZ-d(d"Z.d)d$Z/dS)*) annotationsN)sha1)Thread)AnyDictIterableListMappingOptionalTupleUnionDocument) Embeddings) VectorStore) BaseSettingsSettingsConfigDict) TypedDict)maximal_marginal_relevanceFsstrargsrreturnboolc|D] }||vrdS dS)z Check if a string has multiple substrings. Args: s: The string to check *args: The substrings to check for in the string Returns: bool: True if all substrings are present in the string, False otherwise FT)rras h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_community/vectorstores/starrocks.pyhas_mul_sub_strrs- A::55  4Nonec6trt|dSdS)z[ Print a debug message if DEBUG is True. Args: s: The message to print N)DEBUGprint)rs r debug_outputr%&s%   ar connectionqueryList[dict[str, Any]]c\|}|||j}g}|D]A}i}t |D]\}}||d} ||| <||Bt |||S)z Get a named result from a query. Args: connection: The connection to the database query: The query to execute Returns: List[dict[str, Any]]: The result of the query r)cursorexecute descriptionfetchall enumerateappendr%close) r&r'r*columnsresultvalueridxdatumks rget_named_resultr80s   F NN5 G F"" #E**  JC QAAaDD a LLNNN Mr cBeZdZUded<ded<ded<ded<d ed <d S) QueryResultzList[List[str]]idsz List[Any] embeddingsList[Document] documentszOptional[List[Metadata]] metadataszOptional[List[float]] distancesN)__name__ __module__ __qualname____annotations__rr rr:r:LsN''''$$$$$$r r:ceZdZUdZdZded<dZded<dZded <d Zded <d d dddZ ded<dZ ded<dZ ded<d!dZ e ddddZd S)"StarRocksSettingsaStarRocks client configuration. Attribute: StarRocks_host (str) : An URL to connect to MyScale backend. Defaults to 'localhost'. StarRocks_port (int) : URL port to connect with HTTP. Defaults to 8443. username (str) : Username to login. Defaults to None. password (str) : Password to login. Defaults to None. database (str) : Database name to find the table. Defaults to 'default'. table (str) : Table name to operate on. Defaults to 'vector_table'. column_map (Dict) : Column type map to project column name onto langchain semantics. Must have keys: `text`, `id`, `vector`, must be same size to number of columns. For example: .. code-block:: python { 'id': 'text_id', 'embedding': 'text_embedding', 'document': 'text_plain', 'metadata': 'metadata_dictionary_in_json', } Defaults to identity map. localhostrhostiF#intportrootusernamepasswordiddocument embeddingmetadata)rOrPrQrRzDict[str, str] column_mapdefaultdatabase langchaintableitemrrc"t||SN)getattr)selfrXs r __getitem__zStarRocksSettings.__getitem__stT"""r z.envutf-8 starrocks_ignore)env_fileenv_file_encoding env_prefixextraN)rXrrr)rArBrC__doc__rHrDrJrLrNrSrUrWr]r model_configrr rrFrFTs6DDHH  ""JHE####&%! LLLr rFceZdZdZ dAdBfd ZdCdZedDdZdEdZdFdZ dGdHdZ e dIdJd"Z dKd#Z dAdLd)Z dMdNd.Z dMdOd/Z dMdPd1ZdQd2ZedKd3Z dRdSd;Z dTdUd@ZxZS)V StarRocksa`StarRocks` vector store. You need a `pymysql` python package, and a valid account to connect to StarRocks. Right now StarRocks has only implemented `cosine_similarity` function to compute distance between two vectors. And there is no vector inside right now, so we have to iterate all vectors and compute spatial distance. For more information, please visit [StarRocks official site](https://www.starrocks.io/) [StarRocks github](https://github.com/StarRocks/starrocks) NrQrconfigOptional[StarRocksSettings]kwargsrrr!c P ddl}n#t$rtdwxYw ddlm}||_n#t$r d|_YnwxYwt |||_nt|_|jsJ|jjr |jj sJ|jj r|jj r |jj sJdD]}||jj vsJt|d}d|jj d |jj d |jj d d |jj d d |jj dd|jj dd |_||_d|_d|_||_d|_t+|j|jd|jj|jj |jj|jj|jj d||_t+|jt5|j|jdS)zStarRocks Wrapper to LangChain embedding_function (Embeddings): config (StarRocksSettings): Configuration to StarRocks Client rNzVCould not import pymysql python package. Please install it with `pip install pymysql`.)tqdmc|SrZr)xrks rz$StarRocks.__init__..sQr )rOrQrPrRtestzCREATE TABLE IF NOT EXISTS .z( rOz string, rPrQz array, rRzf string ) ENGINE = OLAP PRIMARY KEY(id) DISTRIBUTED BY HASH(id) PROPERTIES ("replication_num" = "1")\)rs'DESC)rHrJuserrNrUr)pymysql ImportErrorrmpgbarsuper__init__rirFrHrJrSrUrWlen embed_queryschemadimBS must_escapeembedding_function dist_orderr%connectrLrNr&r8) r\rQrirkrwrmr7r __class__s rr{zStarRocks.__init__s  NNNN   @    / ! ! ! ! ! !DJJ / / /..DJJJ /    DKK+--DK{{{4DK$4444{%T$+*>T4;CTTTT< / /A ......)''//00 K037;3D [D! [J' [K(   [J'  &"+ T[!!!*'/ !!%[)[)       T[!!!$+66666s" 4A A r3rcFdfd|DS)NrMc3DK|]}|jvr j|n|VdSrZ)rr).0cr\s r z'StarRocks.escape_str..s>VV1!t/?*?*?$'1QVVVVVVr )join)r\r3s` r escape_strzStarRocks.escape_strs+wwVVVVPUVVVVVVr c|jSrZ)rr\s rr<zStarRocks.embeddingss &&r transacr column_names Iterable[str]c d|}t|jjdg}|D]J}dfdt |D}|d|dKdjjdjjd|dd|d }|S) N,rQcg|]C\}}|kr&dt|dndt|DS)rtz array)rr)rr5_nembed_tuple_indexr\s r z/StarRocks._build_insert_sql..sn "b"3338DOOCGG4477775CGG55 r ()z1 INSERT INTO rrz)) VALUES z ) rtupleindexrirSr.r/rUrW)r\rrks_datani_strrs` @r_build_insert_sqlzStarRocks._build_insert_sqls XXl # #!,//55 K "; /   # #A &/q\\   A LLQ " " " "[),0K,=@B%   r cz|||}t|t|j|dSrZ)rr%r8r&)r\rr _insert_querys r_insertzStarRocks._inserts=..w EE ]###-88888r textsr?Optional[List[dict]] batch_sizerIr;Optional[Iterable[str]] List[str]c |p d|D}|jj}g}|d||d||d|jt |i}|p d|D}t t j|||d<tt|t|z dksJt| \} } d} | t| d t| D]} t| | |jjd|jksJ|| t||krD| r| t#|j|| g } | g}t|dkr,| r| ||| d |DS#t($rG} t*d t/| dt1| dgcYd} ~ Sd} ~ wwxYw)aInsert more texts through the embeddings and add to the VectorStore. Args: texts: Iterable of strings to add to the VectorStore. ids: Optional list of ids to associate with the texts. batch_size: Batch size of insertion metadata: Optional column data to be inserted Returns: List of ids from adding the texts into the VectorStore. cvg|]6}t|d7S)r^)rencode hexdigest)rts rrz'StarRocks.add_texts..s8IIIad188G,,--7799IIIr rOrPrQcg|]}iSrr)r_s rrz'StarRocks.add_texts.. s!4!4!4"!4!4!4r rRrNzInserting data...)desctotal)targetrcg|]}|Srr)ris rrz'StarRocks.add_texts..7s###!A###r   )rirSrembed_documentslistmapjsondumpsr|setzipitemsryrrr/rrrstart Exceptionloggererrortyper)r\rr?rr;rkcolmap_rrkeysvaluesrves r add_textszStarRocks.add_textssj*III5III+( DM3 J  K $"9"I"I$u++"V"V 4!4!4e!4!4!4 ,/ I,F,F WZ()3w<<#l"3"33449999L..001 f AZZV #6c)nn  ! !$**T[%;K%HIIJKKtxWWWWq!!!w<<:--!dl'4IIIAGGIII G7||aFFHHH Wd+++##s### #    LLS477SSCFFSSS T T TIIIIII sD*H IGAJ>>>>> *+d22B ,BBt{/@BBU dou 5 5 A AA P7PPP6PPP PE @%@@@@ @EE *+d22 r q_emb List[float]topk where_str Optional[str]c|dtt|}|rd|}nd}d|jjdd|jjdd|d |jjd d |jjd d |jjd |jjd|d|jd|d}t||S)NrzWHERE rMz> SELECT id as id, rPz as document, rRzB as metadata, cosine_similarity_norm(array[z], rQz) as dist, z as embedding FROM rrz z ORDER BY dist z LIMIT ) rrrrirSrUrWrr%)r\rrr q_emb_strrs r_build_query_sqlzStarRocks._build_query_sqlss$HHSe__--  ,,,III ' 3  ' 3   6?   ' 4  ' 4  +&  *.):     ?      U r r'r7r=c R|j|j|||fi|S)aPerform a similarity search with StarRocks Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of Documents )similarity_search_by_vectorrr})r\r'r7rrks rsimilarity_searchzStarRocks.similarity_searchs@&0t/  # / / 6 69  HN   r c "|||} tj|}fd|DS#t$rG}tdt |dt|dgcYd}~Sd}~wwxYw)aPerform a similarity search with StarRocks by vectors Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of (Document, similarity) c g|]Q}t|jjdtj|jjdRS)rPrR page_contentrRrrirSrloadsrr4r\s rrz9StarRocks.similarity_search_by_vector..sh  !"4;#9*#E!F!Z$+*@*L(MNNr rrrN)rr8r&rrrrr)r\rQr7rrkrq_rrs` rrz%StarRocks.similarity_search_by_vectors.%%iI>> "4?E::C       LLS477SSCFFSSS T T TIIIIII s"= B.sv    %&t{'=j'I%J!%Adk.DZ.P,Q!R!RfI    r rrrN) rrr}r8r&rrrrr)r\r'r7rrkrrs` r'similarity_search_with_relevance_scoresz1StarRocks.similarity_search_with_relevance_scoress&%%  # / / 6 69       *$/5AA        LLS477SSCFFSSS T T TIIIIII s A B$.s&&&Q4&&&r cdg|],}tj|jjd-S)rQrrrirSrs rrzEStarRocks.max_marginal_relevance_search_by_vector..s?GH 1T[3K@ABBr c@g|]}|jjdS)rPrrs rrzEStarRocks.max_marginal_relevance_search_by_vector.. s(JJJq/ ;<JJJr cdg|],}tj|jjd-S)rRrrs rrzEStarRocks.max_marginal_relevance_search_by_vector.. s2VVVQtz!DK$::$F"GHHVVVr cg|] }|d S)rrr s rrzEStarRocks.max_marginal_relevance_search_by_vector.. s...Qqy...r )r;r<r>r?r@)dtyper<)r7rc"g|] \}}|v | Srr)rrr4 mmr_selecteds rrzEStarRocks.max_marginal_relevance_search_by_vector..s'UUU$!Q1 CTCTACTCTCTr ) rr8r&r:rnparrayfloat32_results_to_docsr.) r\rQr7rrrkrrresults candidatesselected_resultsrs ` @r'max_marginal_relevance_search_by_vectorz1StarRocks.max_marginal_relevance_search_by_vectors+%%i$??t66&&#&&&LOKJJJcJJJVVVVRUVVV..#...   2 HYbj 1 1 1 L !#    &g.. UUUU)J*?*?UUUr filterOptional[Dict[str, str]]where_documentc |jtd|j|}|||||||S)NzBFor MMR search, you must specify an embedding function oncreation.)rrr!)r< ValueErrorr}r) r\r'r7rrrr!rkrQs rmax_marginal_relevance_searchz'StarRocks.max_marginal_relevance_searchsg ? "T O//66 ;;  #) <   r rZ)rQrrirjrkrrr!)r3rrr)rr)rrrrrr)rrrrrr!)NrN) rrr?rrrIr;rrkrrr)NNNr)rrrQrr?rrirjrrrrIrkrrrh)rr)rrrrIrrrr)rN) r'rr7rIrrrkrrr=) rQrr7rIrrrkrrr=) r'rr7rIrrrkrrr)rr!)rrr) rQrr7rIrrIrr rkrrr )rrrNN)r'rr7rIrrIrr rr r!r rkrrr=)rArBrCrer{rpropertyr<rrr classmethodrrrrrrrrrr$ __classcell__)rs@rrhrhs+  "/3B7B7B7B7B7B7B7HWWWW'''X'49999+/'+ 77777r 59.2,0[86IM6BF     4#' #####LBF#####J    222X2       F +/37         r rhrr=c4dt|DS)Ncg|]\}}|Srr)rdocrs rrz$_results_to_docs..6s C C CFCC C C Cr )_results_to_docs_and_scoresrs rrr5s C C9'BB C C CCr rc\dt|d|d|dDS)Nc\g|])}t|d|dpi|df*S)rrrrr)rr2s rrz/_results_to_docs_and_scores..:sJ     vay6!9? C C CVAYO   r r>r?r@)rr,s rr+r+9sE   K K K      r )rrrrrr)rrrr!)r&rr'rrr()rrrr=)rrrr)0 __future__rrlogginghashlibr threadingrtypingrrrr r r r r numpyrlangchain_core.documentsrlangchain_core.embeddingsrlangchain_core.vectorstoresrpydantic_settingsrrtyping_extensionsr&langchain_community.vectorstores.utilsr getLoggerrr#rr%r8rrIr rMetadatar:rFrhrr+rr rr=s5"""""" MMMMMMMMMMMMMMMMMMMM------000000333333>>>>>>>>''''''MMMMMM          2 3c3t344 5%%%%%)%%%33333 333lh h h h h h h h V DDDDr