tj$>JdZddlmZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZdd lmZdd lmZmZmZddlmZmZddlm Z m!Z!ddl"m#Z#ej$e%Z&ej'dZ(ej)dej*dej+dej,dej-dej.dej/dej0dej1dej2dej3dej4dej5dej6diZ7d+d$Z8Gd%d&e e9Z:Gd'd(e!e eZ;Gd)d*e;Zz-resolve_pairwise_criteria..Gs"KKKA,Q/KKKcbi|],}t|D]\}}|| -Sr)resolve_pairwise_criteriaitems)r criterionr vs r!r"z-resolve_pairwise_criteria..RsY   1)<<BBDD  1 q    r#zpCriteria cannot be empty. Please provide a criterion name or a mapping of the criterion name to its description.)r HELPFULNESS RELEVANCE CORRECTNESSDEPTH isinstancerrstrr namecritique_requestlisttuple ValueErrorr)r_default_criteria criteria_msgs r!r&r&4s7     N   LK9JKKKK(H%%#^%8%BC Hc " "# * * *!#6x7I7I#JKII!2II H5 6 6#]H$=> HtUm , ,#  %    "'  S// !NN r#c2eZdZdZed dZd dZdS) PairwiseStringResultOutputParserz|A parser for the output of the PairwiseStringEvalChain. Attributes: _type: The type of the output parser. rr/cdS)zlReturn the type of the output parser. Returns: The type of the output parser. pairwise_string_resultrselfs r!_typez&PairwiseStringResultOutputParser._typeks ('r#textdict[str, Any]ct|}|r|d}|r|dvrd|d}t||dkrdn|}dddd |}|||d S) zParse the output text. Args: text: The output text to parse. Returns: The parsed output. Raises: ValueError: If the verdict is invalid. >ABCzInvalid output: zb. Output must contain a double bracketed string with the verdict 'A', 'B', or 'C'.rENrg?)rCrDrE) reasoningrscore)_FIND_DOUBLE_BRACKETSsearchgroupr4)r=r?matchverdictr7verdict_rGs r!parsez&PairwiseStringResultOutputParser.parseus&,,T22  %kk!nnG "6654555  S// !"c>>44w        r#Nrr/)r?r/rr@)__name__ __module__ __qualname____doc__propertyr>rNrr#r!r9r9csR(((X($ $ $ $ $ $ r#r9cLeZdZUdZdZded<eeZded<e e d1d Z e d Z ed1d Zed1dZed2dZe dddd3dZd4d!Zd5d#Ze dddddd$d%d6d.Ze dddddd$d/d7d0ZdS)8PairwiseStringEvalChainaSPairwise String Evaluation Chain. A chain for comparing two outputs, such as the outputs of two models, prompts, or outputs of a single model on similar inputs. Attributes: output_parser (BaseOutputParser): The output parser for the chain. Example: >>> from langchain_openai import ChatOpenAI >>> from langchain_classic.evaluation.comparison import PairwiseStringEvalChain >>> model = ChatOpenAI( ... temperature=0, model_name="gpt-4", model_kwargs={"random_seed": 42} ... ) >>> chain = PairwiseStringEvalChain.from_llm(llm=model) >>> result = chain.evaluate_string_pairs( ... input = "What is the chemical formula for water?", ... prediction = "H2O", ... prediction_b = ( ... "The chemical formula for water is H2O, which means" ... " there are two hydrogen atoms and one oxygen atom." ... reference = "The chemical formula for water is H2O.", ... ) >>> print(result) # { # "value": "B", # "comment": "Both responses accurately state" # " that the chemical formula for water is H2O." # " However, Response B provides additional information" # . " by explaining what the formula means.\n[[B]]" # } resultsr/ output_key)default_factoryr output_parserrboolcdS)NFr)clss r!is_lc_serializablez*PairwiseStringEvalChain.is_lc_serializables ur#ignore)extracdS)Return whether the chain requires a reference. Returns: `True` if the chain requires a reference, `False` otherwise. Frr<s r!requires_referencez*PairwiseStringEvalChain.requires_references ur#cdS)zReturn whether the chain requires an input. Returns: `True` if the chain requires an input, `False` otherwise. Trr<s r!requires_inputz&PairwiseStringEvalChain.requires_input tr#c"d|jjdS)zReturn the warning to show when reference is ignored. Returns: The warning to show when reference is ignored. zIgnoring reference in z, as it is not expected. To use a reference, use the LabeledPairwiseStringEvalChain (EvaluatorType.LABELED_PAIRWISE_STRING) instead.) __class__rPr<s r!_skip_reference_warningz/PairwiseStringEvalChain._skip_reference_warnings& @T^%< @ @ @ r#NpromptrllmrrkPromptTemplate | NonerCRITERIA_TYPE | str | Nonekwargsrc t|dr|jdstdhd}|pt jd}|t|jkrd|d|j}t|t|}d d | D} | r t| znd} |d ||| d |S)aInitialize the PairwiseStringEvalChain from an LLM. Args: llm: The LLM to use (GPT-4 recommended). prompt: The prompt to use. criteria: The criteria to use. **kwargs: Additional keyword arguments. Returns: The initialized PairwiseStringEvalChain. Raises: ValueError: If the input variables are not as expected. model_namezgpt-4z`This chain was only tested with GPT-4. Performance may be significantly worse with other models.>inputr prediction prediction_br$) referenceInput variables should be , but got  c32K|]\}}|r|d|n|VdSz: Nrrr r)s r! z3PairwiseStringEvalChain.from_llm..s9 W WTQ!8Aq W W W W W Wr#rrlrkr)hasattrrq startswithloggerwarningrpartialsetinput_variablesr4r&joinr'r r]rlrkrroexpected_input_varsprompt_r7r6 criteria_strs r!from_llmz PairwiseStringEvalChain.from_llms32sL)) 1J1J71S1S  NN;    RQQE/7"EEE #g&=">"> > >5-@55"255 S// !-h77 yy W WY__EVEV W W WWW ?KS,|;;QS sTs7??L?#I#ITTVTTTr#rsrtinput_ str | Nonerurc*|||d}|jr||d<|S)a_Prepare the input for the chain. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. input_: The input or task string. reference: The reference string, if any. Returns: The prepared input for the chain. )rsrtrrru)rc)r=rsrtrru input_dicts r!_prepare_inputz&PairwiseStringEvalChain._prepare_inputs6(%(   " 0&/J{ #r#resultc\||j}t|vr|t|t<|S)zPrepare the output.)rXr)r=rparseds r!_prepare_outputz'PairwiseStringEvalChain._prepare_output7s+( f  $WoF7O r#F)rrru callbackstagsmetadatainclude_run_inforrrrrlist[str] | Nonerdict[str, Any] | Nonerc ||||||} || ||||} || S)aEvaluate whether output A is preferred to output B. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. input: The input or task string. callbacks: The callbacks to use. tags: The tags to apply. metadata: The metadata to use. include_run_info: Whether to include run info in the output. reference: The reference string, if any. **kwargs: Additional keyword arguments. Returns: `dict` containing: - reasoning: The reasoning for the preference. - value: The preference value, which is either 'A', 'B', or None for no preference. - score: The preference score, which is 1 for 'A', 0 for 'B', and 0.5 for None. inputsrrrr)rr) r=rsrtrrrurrrrrorrs r!_evaluate_string_pairsz.PairwiseStringEvalChain._evaluate_string_pairs>sYH$$ZuiPP-    ##F+++r#)rurrrrrrcK|||||} || ||||d{V} || S)aAsynchronously evaluate whether output A is preferred to output B. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. input: The input or task string. callbacks: The callbacks to use. tags: The tags to apply. metadata: The metadata to use. include_run_info: Whether to include run info in the output. reference: The reference string, if any. **kwargs: Additional keyword arguments. Returns: `dict` containing: - reasoning: The reasoning for the preference. - value: The preference value, which is either 'A', 'B', or None for no preference. - score: The preference score, which is 1 for 'A', 0 for 'B', and 0.5 for None. rN)racallr) r=rsrtrurrrrrrrorrs r!_aevaluate_string_pairsz/PairwiseStringEvalChain._aevaluate_string_pairsls}H$$ZuiPPzz- "        ##F+++r#rr[rO rlrrkrmrrnrorrrV) rsr/rtr/rrrurrr)rrrr)rsr/rtr/rrrrurrrrrrrrr[rorrr)rsr/rtr/rurrrrrrrrrrrr[rorrr)rPrQrRrSrX__annotations__r r9rZ classmethodr r^r model_configrTrcrerirrrrrrr#r!rVrVs  D J&+e8'''M X[:LXX    X   )-/3 )U)U)U)U)U[)UV8 ! $#!%*.!&+,+,+,+,+,X+,Z !% #!%*.!&+,+,+,+,+,X+,+,+,r#rVcJeZdZdZeddZedddddZdS)LabeledPairwiseStringEvalChaina1Labeled Pairwise String Evaluation Chain. A chain for comparing two outputs, such as the outputs of two models, prompts, or outputs of a single model on similar inputs, with labeled preferences. Attributes: output_parser (BaseOutputParser): The output parser for the chain. rr[cdS)rbTrr<s r!rcz1LabeledPairwiseStringEvalChain.requires_referencerfr#NrjrlrrkrmrrnrorrVc ^hd}|pt}|t|jkrd|d|j}t|t |}dd|D} | r t| znd} |d ||| d|S) aInitialize the LabeledPairwiseStringEvalChain from an LLM. Args: llm: The LLM to use. prompt: The prompt to use. criteria: The criteria to use. **kwargs: Additional keyword arguments. Returns: The initialized `LabeledPairwiseStringEvalChain`. Raises: ValueError: If the input variables are not as expected. >rrrrursrtrvrwrxc3*K|]\}}|d|VdSrzrr{s r!r|z:LabeledPairwiseStringEvalChain.from_llm..s0 K KAA K K K K K Kr#r$r}r~r) rrrr4r&rr'rrrs r!rz'LabeledPairwiseStringEvalChain.from_llms0   >> #g&=">"> > >5-@55"255 S// !-h77 yy K K9J9J K K KKK ?KS,|;;QS sTs7??L?#I#ITTVTTTr#rr)rPrQrRrSrTrcrrrr#r!rrsx  X )-/3 (U(U(U(U(U[(U(U(Ur#r)rrrr)=rS __future__rloggingretypingrlangchain_core.callbacksrlangchain_core.language_modelsrlangchain_core.output_parsersrlangchain_core.prompts.promptrpydanticr r typing_extensionsr 1langchain_classic.chains.constitutional_ai.modelsr langchain_classic.chains.llmr .langchain_classic.evaluation.comparison.promptrrr0langchain_classic.evaluation.criteria.eval_chainrr#langchain_classic.evaluation.schemarrlangchain_classic.schemar getLoggerrPrcompilerH CONCISENESSr+r, COHERENCE HARMFULNESS MALICIOUSNESSr*CONTROVERSIALITYMISOGYNY CRIMINALITY INSENSITIVITYr- CREATIVITYDETAILrr&rr9rVrrr#r!rs::"""""" ......<<<<<<::::::888888&&&&&&&&&&&&&&UUUUUU111111 VUUUUUUU,,,,,,  8 $ $" #344 G T M U S E S N B B S NG S OK$,,,,^6 6 6 6 6 '7'=6 6 6 r|,|,|,|,|,5|X|,|,|,~?U?U?U?U?U%<?U?U?U?U?Ur#