tjr,PdZddlZddlmZddlmZmZmZmZddl m Z ddl m Z m Z mZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lmZmZdd lm Z ddl!m"Z"e dZ#e deZ$dee#dee#ge$fdee#fdZ%GddeZ&dS)zEnsemble Retriever. Ensemble retriever that ensemble the results of multiple retrievers by using weighted Reciprocal Rank Fusion. N) defaultdict)CallableHashableIterableIterator)chain)AnyTypeVarcast)#AsyncCallbackManagerForRetrieverRunCallbackManagerForRetrieverRun)Document) BaseRetriever RetrieverLike)RunnableConfig) ensure_config patch_config)ConfigurableFieldSpecget_unique_config_specs)model_validator)overrideTH)bounditerablekeyreturnc#Kt}|D]*}||x}|vr|||V+dS)a Yield unique elements of an iterable based on a key function. Args: iterable: The iterable to filter. key: A function that returns a hashable key for each element. Yields: Unique elements of the iterable based on the key function. N)setadd)rrseeneks c/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_classic/retrievers/ensemble.py unique_by_keyr%$sZ 55D QKA $ $ HHQKKKGGGc .eZdZUdZeeed<eeed<dZe ed<dZ e dzed<e dee fd Zed ed ee efdefd Ze dde dedzdedeefdZe dde dedzdedeefdZde dedeefdZde dedeefdZddde dededzdeefdZddde dededzdeefdZdeeedeefdZdS)EnsembleRetrieveraJRetriever that ensembles the multiple retrievers. It uses a rank fusion. Args: retrievers: A list of retrievers to ensemble. weights: A list of weights corresponding to the retrievers. Defaults to equal weighting for all retrievers. c: A constant added to the rank, controlling the balance between the importance of high-ranked items and the consideration given to lower-ranked items. id_key: The key in the document's metadata used to determine unique documents. If not specified, page_content is used. retrieversweights<cNid_keyrc>td|jDS)z+List configurable fields for this runnable.c3.K|]}|jD]}|VdSN) config_specs).0 retrieverspecs r$ z1EnsembleRetriever.config_specs..LsJ' ' i>T' ' 6:D' ' ' ' ' ' ' r&)rr))selfs r$r1zEnsembleRetriever.config_specsIs4'' ' "&/' ' '    r&before)modevaluesc|d}|s#t|d}d|z g|z|d<|S|d}t|t|kr2dt|dt|d}t|td|Dsd}t||S) Nr*r)z7Length of weights must match number of retrievers (got z weights for z retrievers).c3"K|] }|dkV dS)rN)r2ws r$r5z1EnsembleRetriever._set_weights..bs&**Q1q5******r&z7At least one ensemble weight must be greater than zero.)getlen ValueErrorany)clsr9r* n_retrieversr)msgs r$ _set_weightszEnsembleRetriever._set_weightsPs**Y'' vl344L!"\!1 2\ AF9 ML) w<<3z?? * *RG RR36z??RRR S// !**'***** "KCS// ! r&inputconfigkwargsc (ddlm}t|}||dd|dd|dg|j|di|j}|jd|fd |d p|i|} | ||| }|j |fi||S#t$r}| |d}~wwxYw) Nr)CallbackManager callbacksverboseFtagsmetadatarMinheritable_tags local_tagsinheritable_metadatalocal_metadatanamerun_name run_managerrH) langchain_core.callbacksrKr configurer?rNrOon_retriever_startget_name rank_fusionon_retriever_end Exceptionon_retriever_error) r6rGrHrIrKcallback_managerrXresultr"s r$invokezEnsembleRetriever.invokehsc =<<<<<v&&*44 JJ{ # # JJy%00#ZZ33y!'J!;!;=5  :&9    J'':4==??    %%eV%TTF )K (     M     * *1 - - -  sC,, D6D  Dc \Kddlm}t|}||dd|dd|dg|j|di|j}|jd|fd |d p|i|d{V} | ||| d{V}|j |fi|d{V|S#t$r!}| |d{Vd}~wwxYw) Nr)AsyncCallbackManagerrLrMFrNrOrPrUrVrW) rYrerrZr?rNrOr[r\ arank_fusionr^r_r`) r6rGrHrIrerarXrbr"s r$ainvokezEnsembleRetriever.ainvokes BAAAAAv&&/99 JJ{ # # JJy%00#ZZ33y!'J!;!;=:  @,?    J'':4==??           ,,'-F/+.       M   0033 3 3 3 3 3 3 3  s D D+ D&&D+queryrXc.|||S)zGet the relevant documents for a given query. Args: query: The query to search for. run_manager: The callback handler to use. Returns: A list of reranked documents. )r]r6rhrXs r$_get_relevant_documentsz)EnsembleRetriever._get_relevant_documentss {333r&c>K|||d{VS)zAsynchronously get the relevant documents for a given query. Args: query: The query to search for. run_manager: The callback handler to use. Returns: A list of reranked documents. N)rfrjs r$_aget_relevant_documentsz*EnsembleRetriever._aget_relevant_documentss0 &&uk:::::::::r&)rHcfdt|jD}tt|D]}d||D||<||S)akRank fusion. Retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. run_manager: The callback handler to use. config: Optional configuration for the retrievers. Returns: A list of reranked documents. c g|]D\}}|td|dzES retriever_r;)tag)rL)rcr get_childr2ir3rHrhrXs r$ z1EnsembleRetriever.rank_fusion..sx    9   )338LQU8L8L3MM      r&cxg|]7}t|trttd|n|8S)str page_content) isinstancerxrr r2docs r$rvz1EnsembleRetriever.rank_fusion..sS!!!**++  A!!)!,!!!N1   ,,^<<.sx   !Ay!! "-"7"7.(sH!!!3=S(2K2KTc****QT!!!r&)asynciogatherr~r)rr@rrs ``` r$rfzEnsembleRetriever.arank_fusions* '~      %.do$>$>          s>**++  A!!)!,!!!N1   ,,^<<\}}j|jn|j jxx||j zz z cc<?Vtj |}tt|fddfd  S) aPerform weighted Reciprocal Rank Fusion on multiple rank lists. You can find more details about RRF here: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf. Args: doc_lists: A list of rank lists, where each rank list contains unique items. Returns: The final aggregated list of items sorted by their weighted RRF scores in descending order. zz.Vs'{*$$dk2r&TcPj|jn|jjSr0r)r} rrf_scorer6s r$rz.]s'I$(K$7  S\$+=Vr&)reverser)r@r*rArfloatzipr~r-rzrOr,r from_iterablesortedr%) r6rrEdoc_listweightrankr}all_docsrs ` @r$rz*EnsembleRetriever.weighted_reciprocal_rank0sR y>>S.. . .PCS// !'2%&8&8 #It|E J J J . . Hf&xq999 . . c ;.(( \$+6  tdf}- . .&y11       r&r0) __name__ __module__ __qualname____doc__listr__annotations__rr,intr-rxpropertyrr1r classmethoddictr rFrrrrcrgr rkr rmr]rfrr=r&r$r(r(5s  ]#### %[ AsKKKFC$J  d#89   X _(###$sCx.S[$#,)-"""%" " h """X"H)-&&&%& & h &&&X&P444 4 h 4444$;;9 ; h ;;;;.)- (=(=(=(=4(= % (= h (=(=(=(=^)- *=*=*=*=9*= % *= h *=*=*=*=X0 X'0  h0 0 0 0 0 0 r&r()'rr collectionsrcollections.abcrrrr itertoolsrtypingr r r rYr r langchain_core.documentsrlangchain_core.retrieversrrlangchain_core.runnablesrlangchain_core.runnables.configrrlangchain_core.runnables.utilsrrpydanticrtyping_extensionsrrrr%r(r=r&r$rs ######BBBBBBBBBBBB .-----BBBBBBBB333333GGGGGGGG%$$$$$&&&&&& GCLL GCx   HQKhsAv.>8A;"k k k k k k k k k k r&