Ah4zSrSSKJr SSKJrJrJrJr SSKJ r SSK J r SSK J r SSKJr SSKJrJr SS KJrJr SS KJrJrJr SS KJr SS KJr SS KJ r J!r! SSK"J#r$ \\\\/\\\\%44r&\\\\\\/\\\\%44r'"SS\5r("SS\(5r)\\&\\!4r*\\ \+\(4r,"SS\5r-g)z!Configuration for run evaluators.)Sequence)AnyCallableOptionalUnion) Embeddings)BaseLanguageModel)BasePromptTemplate) RunEvaluator)EvaluationResultEvaluationResults)ExampleRun) BaseModel ConfigDictField) CRITERIA_TYPE)EmbeddingDistance) EvaluatorTypeStringEvaluator)StringDistancec<\rSrSr%Sr\\S'S\\\ 44Sjr Sr g) EvalConfig!zConfiguration for a given run evaluator. Parameters ---------- evaluator_type : EvaluatorType The type of evaluator to use. Methods ------- get_kwargs() Get the keyword arguments for the evaluator configuration. evaluator_typereturnc>0nUHup#US:XdUcMX1U'M U$)zGet the keyword arguments for the load_evaluator call. Returns ------- Dict[str, Any] The keyword arguments for the load_evaluator call. r)selfkwargsfieldvals Y/var/www/html/shao/venv/lib/python3.13/site-packages/langchain/smith/evaluation/config.py get_kwargsEvalConfig.get_kwargs2s3JE((CK5M rN) __name__ __module__ __qualname____firstlineno____doc__r__annotations__dictstrrr$__static_attributes__rr&r#rr!s# "!DcNr&rc^\rSrSr%SrSr\\\S'Sr \\\S'Sr \\\S'S\ \\ 44U4Sjjr S rU=r$) SingleKeyEvalConfigCzBConfiguration for a run evaluator that only requires a single key.N reference_keyprediction_key input_keyrcZ>[TU]5nSHnURUS5 M U$)N)r3r4r5)superr$pop)rr key __class__s r#r$SingleKeyEvalConfig.get_kwargsQs.#%CC JJsD !D r&r)r'r(r)r*r+r3rr.r,r4r5r-rr$r/ __classcell__r:s@r#r1r1Cs\L#'M8C='@$(NHSM( $Ix}#BDcNr&r1c\rSrSr%Sr\"\S9r\\\ \ 4\ S'Sr \ \\ \ S'Sr\ \\\ S'Sr\ \\ S'Sr\ \\ S 'Sr\ \\ S 'Sr\ \\ S '\"S S 9r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r "SS\!5r""S S!\5r#"S"S#\5r$"S$S%\5r%"S&S'\%5r&S(r'g)) RunEvalConfig]aConfiguration for a run evaluation. Parameters ---------- evaluators : List[Union[EvaluatorType, EvalConfig, RunEvaluator, Callable]] Configurations for which evaluators to apply to the dataset run. Each can be the string of an :class:`EvaluatorType `, such as EvaluatorType.QA, the evaluator type string ("qa"), or a configuration for a given evaluator (e.g., :class:`RunEvalConfig.QA `). custom_evaluators : Optional[List[Union[RunEvaluator, StringEvaluator]]] Custom evaluators to apply to the dataset run. reference_key : Optional[str] The key in the dataset run to use as the reference string. If not provided, it will be inferred automatically. prediction_key : Optional[str] The key from the traced run's outputs dictionary to use to represent the prediction. If not provided, it will be inferred automatically. input_key : Optional[str] The key from the traced run's inputs dictionary to use to represent the input. If not provided, it will be inferred automatically. eval_llm : Optional[BaseLanguageModel] The language model to pass to any evaluators that use a language model. )default_factory evaluatorsNcustom_evaluatorsbatch_evaluatorsr3r4r5eval_llmTarbitrary_types_allowedc^\rSrSr%SrSr\\\S'Sr \\ \S'\ Rr \ \S'S S\\S\SS4U4S jjjrS rU=r$) RunEvalConfig.CriteriazConfiguration for a reference-free criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Ncriteriallmrr rc *>[TU]"SSU0UD6 gNrKrr7__init__rrKr r:s r#rPRunEvalConfig.Criteria.__init__ G  9h 9& 9r&rN)r'r(r)r*r+rKrrr,rLr rCRITERIArrrPr/r<r=s@r#CriteriarIsi -1(=)0+/X' (/(5(>(> >15 :}- : :  : :r&rVc^\rSrSr%SrSr\\\S'Sr \\ \S'\ Rr \ \S'S S\\S\SS4U4S jjjrS rU=r$) RunEvalConfig.LabeledCriteriazConfiguration for a labeled (with references) criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. NrKrLrr rc *>[TU]"SSU0UD6 grNrOrQs r#rP&RunEvalConfig.LabeledCriteria.__init__rSr&rrT)r'r(r)r*r+rKrrr,rLr rLABELED_CRITERIArrrPr/r<r=s@r#LabeledCriteriarXsi -1(=)0+/X' (/(5(F(F F15 :}- : :  : :r&r]cp\rSrSr%Sr\R r\\S'Sr \ \ \S'Sr \ \ \S'\"SS9rS rg) RunEvalConfig.EmbeddingDistanceaConfiguration for an embedding distance evaluator. Parameters ---------- embeddings : Optional[Embeddings] The embeddings to use for computing the distance. distance_metric : Optional[EmbeddingDistanceEnum] The distance metric to use for computing the distance. rN embeddingsdistance_metricTrFr)r'r(r)r*r+rEMBEDDING_DISTANCErr,rarrrbEmbeddingDistanceEnumr model_configr/rr&r#rr_sE )6(H(H H+/ HZ(/;?"78?!$( r&rc`\rSrSr%Sr\R r\\S'Sr \ \ \S'Sr \ \S'Srg) RunEvalConfig.StringDistancezConfiguration for a string distance evaluator. Parameters ---------- distance : Optional[StringDistanceEnum] The string distance metric to use. rNdistanceTnormalize_scorer)r'r(r)r*r+rSTRING_DISTANCErr,rirStringDistanceEnumrjboolr/rr&r#rrgs@ )6(E(E E15(-.5 !%$ Nr&rcd\rSrSr%Sr\R r\\S'Sr \ \ \S'Sr \ \ \S'Srg)RunEvalConfig.QAzConfiguration for a QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNrLpromptr)r'r(r)r*r+rQArr,rLrr rqr r/rr&r#rrros9 )6(8(8 8+/X' (//3+,3r&rrcd\rSrSr%Sr\R r\\S'Sr \ \ \S'Sr \ \ \S'Srg)RunEvalConfig.ContextQAi Configuration for a context-based QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNrLrqrr'r(r)r*r+r CONTEXT_QArr,rLrr rqr r/rr&r# ContextQArt 9 )6(@(@ @+/X' (//3+,3r&rxcd\rSrSr%Sr\R r\\S'Sr \ \ \S'Sr \ \ \S'Srg)RunEvalConfig.CoTQAirurNrLrqrrvrr&r#CoTQAr{ryr&r|c<\rSrSr%Sr\R r\\S'Sr g)RunEvalConfig.JsonValidityi*zDConfiguration for a json validity evaluator. Parameters ---------- rrN) r'r(r)r*r+r JSON_VALIDITYrr,r/rr&r# JsonValidityr~* )6(C(C Cr&rc<\rSrSr%Sr\R r\\S'Sr g)#RunEvalConfig.JsonEqualityEvaluatori3zDConfiguration for a json equality evaluator. Parameters ---------- rrN) r'r(r)r*r+r JSON_EQUALITYrr,r/rr&r#JsonEqualityEvaluatorr3rr&rcf\rSrSr%Sr\R r\\S'Sr \ \S'Sr \ \S'Sr \ \S'Sr g ) RunEvalConfig.ExactMatchi<a0Configuration for an exact match string evaluator. Parameters ---------- ignore_case : bool Whether to ignore case when comparing strings. ignore_punctuation : bool Whether to ignore punctuation when comparing strings. ignore_numbers : bool Whether to ignore numbers when comparing strings. rF ignore_caseignore_punctuationignore_numbersrN)r'r(r)r*r+r EXACT_MATCHrr,rrmrrr/rr&r# ExactMatchr<s8 )6(A(A A! T!#(D($$r&rcJ\rSrSr%Sr\R r\\S'Sr \ \S'Sr g)RunEvalConfig.RegexMatchiNzConfiguration for a regex match string evaluator. Parameters ---------- flags : int The flags to pass to the regex. Example: re.IGNORECASE. rrflagsrN) r'r(r)r*r+r REGEX_MATCHrr,rintr/rr&r# RegexMatchrNs" )6(A(A Asr&rc ^\rSrSr%Sr\R r\\S'Sr \ \ \S'Sr \ \ \S'Sr\ \\S'Sr\ \\S'S S\ \ S\ \S \S S4U4S jjjrS rU=r$)RunEvalConfig.ScoreStringiZajConfiguration for a score string evaluator. This is like the criteria evaluator but it is configured by default to return a score on the scale from 1-10. It is recommended to normalize these scores by setting `normalize_by` to 10. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. normalize_by: Optional[int] = None If you want to normalize the score, the denominator to use. If not provided, the score will be between 1 and 10 (by default). prompt : Optional[BasePromptTemplate] rNrKrL normalize_byrqr rc *>[TU]"SXS.UD6 g)N)rKrrrO)rrKrr r:s r#rP"RunEvalConfig.ScoreString.__init__us G  Th TV Tr&r)NN)r'r(r)r*r+r SCORE_STRINGrr,rKrrrLr rfloatrqr rrPr/r<r=s@r# ScoreStringrZs ()6(B(B B,0(=)0+/X' (/(, huo,/3+,315,0 U}- U#5/ U  U   U Ur&rc8\rSrSr%\R r\\S'Srg) RunEvalConfig.LabeledScoreStringi}rrN) r'r(r)r*rLABELED_SCORE_STRINGrr,r/rr&r#LabeledScoreStringr}s(5(J(J Jr&rr)(r'r(r)r*r+rlistrBrSINGLE_EVAL_CONFIG_TYPECUSTOM_EVALUATOR_TYPEr,rCrrDBATCH_EVALUATOR_LIKEr3r.r4r5rEr rrer1rVr]rrrrrxr|rrrrrrrr/rr&r#r?r?]sF d#  # ! # $ X@Dx%: ;<C8=Aht$89:A $(M8C='@$(NHSM( $Ix}#B,0Hh()0H $L:&:.:-:, / *N,N, 4 44'4 4#4 D*DD D%(%$ ( !U)!UFK[Kr&r?N).r+collections.abcrtypingrrrrlangchain_core.embeddingsrlangchain_core.language_modelsr langchain_core.promptsr langsmithr langsmith.evaluation.evaluatorr r langsmith.schemasrrpydanticrrr(langchain.evaluation.criteria.eval_chainr,langchain.evaluation.embedding_distance.baserrdlangchain.evaluation.schemarr)langchain.evaluation.string_distance.baserrlr-RUN_EVALUATOR_LIKErrr1rr.rr?rr&r#rs'$110<5"N*11BG(7  -t 346  c]HXg./0 -t 346 D*,0,OP sJ >?aKIaKr&