tjcdZddlmZddlZddlZddlZddlZddlmZm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZddlmZejeZGd d eZd ZGd d ZdS)z,Module providing Infinispan as a VectorStore) annotationsN)AnyIterableListOptionalTupleTypeUnioncast)Response)Document) Embeddings) VectorStoreceZdZdZ dBdCd ZdDd ZdEdZdFdZdGdZdHdZ dIdJdZ dHdZ dHdZ dKdZ dHd ZdHd!Z dBdLd)Z dMdNd.Z dMdOd0Z dMdPd2Z dMdQd3ZdRd5ZdSd8ZdTd9Ze dUdVdAZdS)W InfinispanVSa`Infinispan` VectorStore interface. This class exposes the method to present Infinispan as a VectorStore. It relies on the Infinispan class (below) which takes care of the REST interface with the server. Example: ... code-block:: python from langchain_community.vectorstores import InfinispanVS from mymodels import RGBEmbeddings ... vectorDb = InfinispanVS.from_documents(docs, embedding=RGBEmbeddings(), output_fields=["texture", "color"], lambda_key=lambda text,meta: str(meta["_key"]), lambda_content=lambda item: item["color"]) or an empty InfinispanVS instance can be created if preliminary setup is required before populating the store ... code-block:: python from langchain_community.vectorstores import InfinispanVS from mymodels import RGBEmbeddings ... ispnVS = InfinispanVS() # configure Infinispan here # i.e. create cache and schema # then populate the store vectorDb = InfinispanVS.from_documents(docs, embedding=RGBEmbeddings(), output_fields: ["texture", "color"], lambda_key: lambda text,meta: str(meta["_key"]), lambda_content: lambda item: item["color"]) N embeddingOptional[Embeddings]idsOptional[List[str]]kwargsrc jtdi|_|_tjdd_tjdd_|_jdd_jdkr!jdd_ntj dtjd d_ j dkr!jd d_ ntj d tjd fd _ jdfd_jd_|_dS)at Parameters ---------- cache_name: str Embeddings cache name. Default "vector" entity_name: str Protobuf entity name for the embeddings. Default "vector" text_field: str Protobuf field name for text. Default "text" vector_field: str Protobuf field name for vector. Default "vector" lambda_content: lambda Lambda returning the content part of an item. Default returns text_field lambda_metadata: lambda Lambda returning the metadata part of an item. Default returns items fields excepts text_field, vector_field, _type output_fields: List[str] List of fields to be returned from item, if None return all fields. Default None kwargs: Any Rest of arguments passed to Infinispan. See docs cache_namevector entity_name textfield text_fieldtextz9`textfield` is deprecated. Please use `text_field` param. vectorfield vector_fieldz=`vectorfield` is deprecated. Please use `vector_field` param.lambda_contentc.|SN)_default_contentitemselfs k/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_community/vectorstores/infinispanvs.pyz'InfinispanVS.__init__..is4+@+@+F+Flambda_metadatac.|Sr#)_default_metadatar%s r(r)z'InfinispanVS.__init__..lsD,B,B4,H,Hr* output_fieldsN) Infinispanispn_configurationstrget _cache_name _entity_name _embedding _textfieldwarningswarnDeprecationWarning _vectorfield _to_content _to_metadata_output_fields_ids)r'rrrs` r(__init__zInfinispanVS.__init__8s6(((( $t266|XNNOO 3 7 7 x P PQQ#-11+rBB ?b "155lFKKDOO MK"   !/33M2FF   " " $ 3 7 7 Q QD   MO"    .22 FFFF  !/33 HHHH  #155oFF r*r&dictreturnct|}||jd||jd|dd|S)N_type)rBpopr<r8)r'r&metas r(r-zInfinispanVS._default_metadataqsSDzz "D))) $''' $ r*dict[str, Any]c6||jSr#)r4r8)r'r&s r(r$zInfinispanVS._default_contentxsxx(((r*templ dimensionintr3cd}||j||jfz}d}|D]!\}}t|tr|d|zdzt |zdzz }nt|t r|d|zdzt |zdzz }nt|t r|d|zdzt |zdzz }nzt|tr|d|zdzt |zdzz }nFt|tr|d |zdzt |zdzz }ntd |z|d z }#|d z }|S) NzV /** * @Indexed */ message %s { /** * @Vector(dimension=%d) */ repeated float %s = 1; zoptional string z = z; zoptional int64 zoptional double zoptional bytes zoptional bool zEUnable to build proto schema for metadata. Unhandled type for field: z} ) r6r<items isinstancer3rLfloatbytesbool Exception)r'rJrKmetadata_proto_tplmetadata_protoidxfvs r(schema_builderzInfinispanVS.schema_builder{s ,     /   KKMM  DAq!S!! "4q"85"@3s88"Ke"SSAs## "3a"7%"?#c(("JU"RRAu%% "4q"85"@3s88"Ke"SSAu%% "3a"7%"?#c(("JU"RRAt$$ "2Q"6">S"IE"QQ1345 1HCC%r*protor cH|j|jdz|S)zDeploy the schema for the vector db Args: proto(str): protobuf schema Returns: An http Response containing the result of the operation .proto)r1 schema_postr6)r'r\s r( schema_createzInfinispanVS.schema_creates$y$$T%6%A5IIIr*cF|j|jdzS)zDelete the schema for the vector db Returns: An http Response containing the result of the operation r^)r1 schema_deleter6r's r(rbzInfinispanVS.schema_deletes" y&&t'88'CDDDr*rconfigch|dkr d|jzdz}|j|j|S)zCreate the cache for the vector db Args: config(str): configuration of the cache. Returns: An http Response containing the result of the operation raZ { "distributed-cache": { "owners": "2", "mode": "SYNC", "statistics": true, "encoding": { "media-type": "application/x-protostream" }, "indexing": { "enabled": true, "storage": "filesystem", "startup-mode": "AUTO", "indexing-mode": "AUTO", "indexed-entities": [ "z" ] } } } )r6r1 cache_postr5)r'rds r( cache_createzInfinispanVS.cache_createsK R<< #!$"# 2y##D$4f===r*c@|j|jS)zDelete the cache for the vector db Returns: An http Response containing the result of the operation )r1 cache_deleter5rcs r(rizInfinispanVS.cache_delete y%%d&6777r*c@|j|jS)zClear the cache for the vector db Returns: An http Response containing the result of the operation )r1 cache_clearr5rcs r(rlzInfinispanVS.cache_clear y$$T%5666r*rTc@|j|jS)zOChecks if the cache exists Returns: true if exists )r1 cache_existsr5rcs r(rozInfinispanVS.cache_existsrjr*c@|j|jS)zClear the index for the vector db Returns: An http Response containing the result of the operation )r1 index_clearr5rcs r(cache_index_clearzInfinispanVS.cache_index_clearrmr*c@|j|jS)z{Rebuild the for the vector db Returns: An http Response containing the result of the operation )r1 index_reindexr5rcs r(cache_index_reindexz InfinispanVS.cache_index_reindexs y&&t'7888r*texts Iterable[str] metadatasOptional[List[dict]] last_vectorOptional[List[float]] List[str]c $g}t|}|r||j|}|r|||s d|D}|jp d|D}tt |||} | D]u\} } } d|j|j| i} | | tj | }|j | ||j|| v|S)Ncg|]}iSr/r/.0_s r( z*InfinispanVS.add_texts..s++++++r*cNg|]"}ttj#Sr/)r3uuiduuid4rs r(rz*InfinispanVS.add_texts..s&===!C --===r*rE)listrFr7embed_documentsappendr@zipr6r<updatejsondumpsr1putr5)r'rvrxrzrresulttexts_lembedsr data_inputmetadataembedkeydatadata_strs r( add_textszInfinispanVS.add_textss'u++   KKMMM0099  ' MM+ & & & ,++U+++Ii===u===#i5566 $.   HeST.0A5ID KK ! ! !z$''H IMM#x)9 : : : MM#     r*querykList[Document]c H|||}d|DS)z"Return docs most similar to query.)rrcg|]\}}|Sr/r/rdocrs r(rz2InfinispanVS.similarity_search..s,,,Q,,,r*)similarity_search_with_score)r'rrr documentss r(similarity_searchzInfinispanVS.similarity_searchs055EQ5GG ,,),,,,r*List[Tuple[Document, float]]c h|j|}|||}|S)aPerform a search on a query string and return results with score. Args: query (str): The text being searched. k (int, optional): The amount of results to return. Defaults to 4. Returns: List[Tuple[Document, float]] )rr)r7 embed_query&similarity_search_with_score_by_vector)r'rrrrrs r(rz)InfinispanVS.similarity_search_with_scores8++E22??%ST?UU r* List[float]c F|||}d|DS)Ncg|]\}}|Sr/r/rs r(rz.+s&&&Q&&&r*)r)r'rrrress r(similarity_search_by_vectorz(InfinispanVS.similarity_search_by_vector's-99)QGG&&#&&&&r*c|jAd|jzdz|jzdztj|zdzt |z}nud}|jddD] }|dz|zd z}|dz|jdz}|d z|jzdz|jzdztj|zdzt |z}|j||j}tj |j }| |S) a$Return docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. Returns: List of pair (Documents, score) most similar to the query vector. Nzselect v, score(v) from z v where v.z <-> ~zselect zv.,z, score(v) from ) r?r6r<rrr3r1 req_queryr5loadsr_query_result_to_docs)r'rr query_str query_projfield query_resrs r(rz3InfinispanVS.similarity_search_with_score_by_vector-si   &*#$ #$  *Y'' (  a&& I#J,SbS1 = ='$.6< #d*T-@-DDJ$%#$  # $   *Y'' (a&& I'' 43CDD IN++))&111r*rc$g}|dD]}|dpi|j d}nfd|jD}t||||}||df|S)Nhitshit*c<i|]}||Sr/)r4)rrrs r( z6InfinispanVS._query_result_to_docs..as%KKK#swws||KKKr*) page_contentrzscore())r?r r=r>r)r'rrrowentityrrs @r(rz"InfinispanVS._query_result_to_docsXs &> 4 4Ce*"C"*SKKKKt7JKKK!--f55**622C   c3y>2 3 3 3 3r*rNonec|||}||}|j|jjjks Jd t j|jdJ| sO| }|j|jjjks Jd | dSdS)Nz)Unable to create schema. Already exists? errorz(Unable to create cache. Already exists? ) r[r` status_coder1CodesOKrrrrorgrr)r'rrKschemaoutputs r( configurezInfinispanVS.configureis$$Xy99##F++!TY_%7777 7877 (z&+&&w/777  "" %&&((F%);;;;:<;; ,  " " $ $ $ $ $ % %r*cV||dSr#)rbrircs r( config_clearzInfinispanVS.config_clearzs,  r*TclsType[InfinispanVS]r clear_oldOptional[bool] auto_configc |d||d|}|rt|pgdkr|r|||t|dz } ttt |}||dt| nA|r|||t|dz } |r|||| |S)aReturn VectorStore initialized from texts and embeddings. In addition to parameters described by the super method, this implementation provides other configuration params if different configuration from default is needed. Parameters ---------- ids : List[str] Additional list of keys associated to the embedding. If not provided UUIDs will be generated clear_old : bool Whether old data must be deleted. Default True auto_config: bool Whether to do a complete server setup (caches, protobuf definition...). Default True kwargs: Any Rest of arguments passed to InfinispanVS. See docs)rrrrO)rr/) lenrrr rrBrrlr) rrvrrxrrrr infinispanvsvecs r( from_textszInfinispanVS.from_texts~s:sBYCBB6BB  ?3yB//!33 ,))+++''c%jj1n(=>>CT$Z33I  " "9Q<S : : : : +((***''c%jj1n(=>>C  A  " "5)C " @ @ @r*)NN)rrrrrr)r&rBrCrB)r&rHrCr)rJrBrKrLrCr3)r\r3rCr )rCr )r)rdr3rCr )rCrT) rvrwrxryrzr{rrrCr|)r)rr3rrLrrrCr)rr3rrLrrrCr)rrrrLrrrCr)rrrrLrCr)rrHrCr)rrBrKrLrCr)rCr)NNTT)rrrvr|rrrxryrrrrrrrrrCr)__name__ __module__ __qualname____doc__rAr-r$r[r`rbrgrirlrorrrurrrrrrrr classmethodrr/r*r(rrs5""L+/#'77777r))))####JJJJJEEEE!>!>!>!>!>F88887777888877779999+/-1 6$%-----$%"01'''''01)2)2)2)2)2V"%%%%" +/#'$(&*))))[)))r*r c eZdZdZddddgdddddf d4dZd5d6dZ d5d7dZ d5d7d Zd8d#Zd8d$Z d9d%Z d:d(Z d;d*Z dd.Zd?d0Zd=d1Zd=d2Zd3S)@r0a>Helper class for `Infinispan` REST interface. This class exposes the Infinispan operations needed to create and set up a vector db. You need a running Infinispan (15+) server without authentication. You can easily start one, see: https://github.com/rigazilla/infinispan-vector#run-infinispan httprz127.0.0.1:11222z/rest/v2/cachesz/rest/v2/schemasTrr3userpasswordhostsr| cache_url schema_urluse_post_for_queryrThttp2verifyrrc  T ddl} n#t$rtdwxYw| j|_| |_||_||_||_|d|_|jdz|jz|_ ||_ ||_ ||_ ||_ |jrm|jrf|jdkr"| |j|j} n|j|jf} | |j |j | | |_dS| |j |j | |_dS) aK Parameters ---------- schema: str Schema for HTTP request: "http" or "https". Default "http" user, password: str User and password if auth is required. Default None hosts: List[str] List of server addresses. Default ["127.0.0.1:11222"] cache_url: str URL endpoint for cache API. Default "/rest/v2/caches" schema_url: str URL endpoint for schema API. Default "/rest/v2/schemas" use_post_for_query: bool Whether POST method should be used for query. Default True http2: bool Whether HTTP/2 protocol should be used. `pip install "httpx[http2]"` is needed for HTTP/2. Default True verify: bool Whether TLS certificate must be verified. Default True rNzCould not import httpx python package. Please install it with `pip install httpx`or `pip install "httpx[http2]"` if you need HTTP/2.z://r)usernamer)rhttp1authr)rrr)httpx ImportErrorcodesrr2_schema_user _password_host _default_node _cache_url _schema_url_use_post_for_query_http2 DigestAuthClient_h2c) r'rrrrrrrrrrrrs r(rAzInfinispan.__init__seF  LLLL   F  [ $  !1X !\E1DJ>#%#5  : $. |v%%AFAQAQ!Z$.BRBB DN3 k+o %DIII k+o%DIIIs!FrrlocalrCr cl|jr||||S||||S)aRequest a query Args: query(str): query requested cache_name(str): name of the target cache local(boolean): whether the query is local to clustered Returns: An http Response containing the result set or errors )r _query_post _query_get)r'rrrs r(rzInfinispan.req_querys=  # >##E:u== =uj%888r*rc|j|jzdz|zdzt|z}d|i}tj|}|j||ddit}|S)N/z?action=search&local=r Content-Typeapplication/jsoncontentheaderstimeout)rrr3rrrpost REST_TIMEOUT)r'rrrapi_urlr data_jsonresponses r(rzInfinispan._query_posts  o   &  & %jj   #Jt$$ 9>> #%78 "   r*c|j|jzdz|zdz|zdzt|z}|j|t }|S)Nrz?action=search&query=z&local=r)rrr3rr4r)r'rrrrrs r(rzInfinispan._query_get'sw  o   &  &      %jj  9==,=??r*rrc|j|jzdz|zdz|z}|j||ddit}|S)aPost an entry Args: key(str): key of the entry data(str): content of the entry in json format cache_name(str): target cache Returns: An http Response containing the result of the operation rrrrrrrrrr'rrrrrs r(rzInfinispan.post7s[$t6> #%78 "   r*c|j|jzdz|zdz|z}|j||ddit}|S)aPut an entry Args: key(str): key of the entry data(str): content of the entry in json format cache_name(str): target cache Returns: An http Response containing the result of the operation rrrr)rrrrrr s r(rzInfinispan.putIs[$t6>'5,>OOr*rdcz|j|jzdz|z}|j||ddit}|S)zCreate a cache Args: name(str): name of the cache. config(str): configuration of the cache. Returns: An http Response containing the result of the operation rrrrr )r'rrdrrs r(rfzInfinispan.cache_postusP$t6> #%78 "   r*cr|j|jzdz|z}|j|t}|S)zDelete a schema Args: name(str): name of the schema. Returns: An http Response containing the result of the operation rr )rrrdeleterr'rrrs r(rbzInfinispan.schema_deletes=$t'77#=D9##G\#BBr*cr|j|jzdz|z}|j|t}|S)zDelete a cache Args: name(str): name of the cache. Returns: An http Response containing the result of the operation rr )rrrrrrs r(rizInfinispan.cache_deletes<$t6>'<>@@r*c\|j|jzdz|zdz}||S)zCheck if a cache exists Args: cache_name(str): name of the cache. Returns: True if cache exists rr)rrresource_existsr'rrs r(rozInfinispan.cache_existss;   03 6 Co U ##G,,,r*rcn|j|t}|j|jjkS)zCheck if a resource exists Args: api_url(str): url of the resource. Returns: true if resource exists r )rheadrrrr)r'rrs r(rzInfinispan.resource_existss-9>>'<>@@#tz}44r*ct|j|jzdz|zdz}|j|tS)zClear an index on a cache Args: cache_name(str): name of the cache. Returns: An http Response containing the result of the operation rz/search/indexes?action=clearr r rs r(rqzInfinispan.index_clearsO  o   -  - y~~g|~<<C,>C $$        "             - - - -5555==== ======r*r0)r __future__rrloggingrr9typingrrrrrr r r rr langchain_core.documentsr langchain_core.embeddingsrlangchain_core.vectorstoresr getLoggerrloggerrrr0r/r*r(r(sM22""""""  JJJJJJJJJJJJJJJJJJJJ------000000333333  8 $ $UUUUU;UUUp  o=o=o=o=o=o=o=o=o=o=r*