AhYHBSrSSKJr SSKrSSKJrJr SSKJr SSK J r SSK J r J r Jr SSKJr SS KJr SS KJr SS KJr SS KJr \R2"\5r"S S\\ 5r"SS\5r"SS5r"SS\\5r "SS\\5r!"SS\\5r"g)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Sequence)Enum)AnyOptionalUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)Chainc\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSrSrSrSrSrg) EvaluatorTypezThe types of the evaluators.qacot_qa context_qapairwise_string score_stringlabeled_pairwise_stringlabeled_score_string trajectorycriterialabeled_criteriastring_distance exact_match regex_matchpairwise_string_distanceembedding_distancepairwise_embedding_distance json_validity json_equalityjson_edit_distancejson_schema_validationN)__name__ __module__ __qualname____firstlineno____doc__QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCE EXACT_MATCH REGEX_MATCHPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATION__static_attributes__r'S/var/www/html/shao/venv/lib/python3.13/site-packages/langchain/evaluation/schema.pyrrs& B F%JS'O!L7H1@#VH<)7'OPKIKN9=-M"?;#M.#M=-T5IrBrc6\rSrSrSr\\SSj55rSrg) LLMEvalChainKz,A base class for evaluators that use an LLM.c g)z#Create a new evaluator from an LLM.Nr')clsllmkwargss rCfrom_llmLLMEvalChain.from_llmNrBr'N)rIr rJrreturnrE) r(r)r*r+r, classmethodrrKrAr'rBrCrErEKs622rBrEc\rSrSrSr\S Sj5r\S Sj5r\S Sj5r\S Sj5r S S Sjjr S r g)_EvalArgsMixinTz(Mixin for checking evaluation arguments.cgz2Whether this evaluator requires a reference label.Fr'selfs rCrequires_reference!_EvalArgsMixin.requires_referenceWrBcg)0Whether this evaluator requires an input string.Fr'rUs rCrequires_input_EvalArgsMixin.requires_input\rYrBc6SURRS3$)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__r(rUs rC_skip_input_warning"_EvalArgsMixin._skip_input_warningas$DNN$;$;#<^^,,--GHCS/ !  d&9&9 ))a 8  " "y'8^^,,--JKCS/ !  )@)@ --! <*A rBr'rNboolrNstr)NN)rk Optional[str]rlrtrNNone) r(r)r*r+r,propertyrWr\rbrernrAr'rBrCrQrQTs2 VV  $( $= ==  ==rBrQc\rSrSrSr\S Sj5r\S Sj5r\SSS.SSjj5r SSS.SSjjr SSS.SS jjr SSS.SS jjr S r g)StringEvaluatorz_Grade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.c.URR$)zThe name of the evaluation.r`rUs rCevaluation_nameStringEvaluator.evaluation_names~~&&&rBcgrTr'rUs rCrW"StringEvaluator.requires_referencerYrBNrkinputc g)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr'rV predictionrkrrJs rC_evaluate_strings!StringEvaluator._evaluate_stringsrMrBcT# [SUR4UUUS.UD6IShvN $N7f)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrrkr)rrrs rC_aevaluate_strings"StringEvaluator._aevaluate_stringssD,%   " " "       s (&(c LURX#S9 UR"SUUUS.UD6$)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rkrlrr')rnrrs rCevaluate_strings StringEvaluator.evaluate_stringss@$ ##i#F%% !    rBch# URX#S9 UR"SUUUS.UD6IShvN $N7f)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rrNr')rnrrs rCaevaluate_strings!StringEvaluator.aevaluate_stringssM$ ##i#F,, !      s )202r'rrrp) rzUnion[str, Any]rkOptional[Union[str, Any]]rrrJrrNdict) rrsrkrtrrtrJrrNr)r(r)r*r+r,rvr{rWrrrrrrAr'rBrCrxrxs@ '' 04+/  $ -  )       604+/  $ -  )      F$(#   !        <$(#   !         rBrxc\rSrSrSr\SSS.S Sjj5rSSS.S SjjrSSS.S SjjrSSS.S Sjjr S r g) PairwiseStringEvaluatorizDCompare the output of two models (or two outputs of the same model).Nrc g)Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr'rVr prediction_brkrrJs rC_evaluate_string_pairs.PairwiseStringEvaluator._evaluate_string_pairsrMrBcV# [SUR4UUUUS.UD6IShvN $N7f)Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nrrrkr)rrrs rC_aevaluate_string_pairs/PairwiseStringEvaluator._aevaluate_string_pairssG(%   ' ' "%      )')c NURX4S9 UR"SUUUUS.UD6$)rrrr')rnrrs rCevaluate_string_pairs-PairwiseStringEvaluator.evaluate_string_pairs8sC( ##i#F** !%     rBcj# URX4S9 UR"SUUUUS.UD6IShvN $N7f)rrrNr')rnrrs rCaevaluate_string_pairs.PairwiseStringEvaluator.aevaluate_string_pairsUsP( ##i#F11 !%       *313r') rrsrrsrkrtrrtrJrrNr) r(r)r*r+r,rrrrrrAr'rBrCrrs=N $(#     !        4$(#     !       F$(#     !       D$(#     !        rBrc\rSrSrSr\S Sj5r\SS.S Sjj5rSS.S Sjjr SS.S Sjjr SS.S S jjr S r g) AgentTrajectoryEvaluatorisz,Interface for evaluating agent trajectories.cg)r[Tr'rUs rCr\'AgentTrajectoryEvaluator.requires_inputvsrBN)rkc g)NEvaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nr'rVragent_trajectoryrrkrJs rC_evaluate_agent_trajectory3AgentTrajectoryEvaluator._evaluate_agent_trajectory{rMrBcV# [SUR4UUUUS.UD6IShvN $N7f)]Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)rrrkr)rrrs rC_aevaluate_agent_trajectory4AgentTrajectoryEvaluator._aevaluate_agent_trajectorysG*%   + + "-      rc NURXCS9 UR"SUUUUS.UD6$)rrrrrrkr')rnrrs rCevaluate_agent_trajectory2AgentTrajectoryEvaluator.evaluate_agent_trajectorysC* ##i#F.. !-     rBcj# URXCS9 UR"SUUUUS.UD6IShvN $N7f)rrrNr')rnrrs rCaevaluate_agent_trajectory3AgentTrajectoryEvaluator.aevaluate_agent_trajectorysP* ##i#F55 !-       rr'rp) rrsrz!Sequence[tuple[AgentAction, str]]rrsrkrtrJrrNr) r(r)r*r+r,rvr\rrrrrrAr'rBrCrrssE6 $(   <    !      8$(   <    !     J$(   <    !     H$(   <    !      rBr)#r, __future__rloggingabcrrcollections.abcrenumrtypingrr r warningsr langchain_core.agentsr langchain_core.language_modelsr langchain_core.runnables.configrlangchain.chains.baser getLoggerr(loggerrsrrErQrxrrr'rBrCrs9"#$''-<;'   8 $3JC3Jl2520=0=fw ncw to nco dx ~sx rB