AhySrSSKJr SSKrSSKrSSKrSSKrSSKrSSK r SSK J r J r SSK J r JrJrJrJrJr SSKJr SSKJr SSKJr SS KJrJr SS KJrJr SS KJ r J!r!J"r" SS KJ#r$ SS KJ%r& SSK'J(r(J)r) SSK*J+r+ SSK,J-r- SSK.J/r/J0r0 SSK1J2r2J3r3 SSK1J4r5 SSK6J7r7J8r8 SSK9J:r:J;r;Jr> SSK?J@r@ SSKAJBrB SSKCJDrD SSKEJFrF SSKGJHrH SSKIJJrJJKrKJLrL SSKMJNrO SS KPJ#rQ SSKPJRrRJSrS \ (aSSKTrU\R"\W5rX\\/\\F\ 44\\\Y/\4\ \F4rZ\\/\\F\ 44\4r["SS\\5r]"S S!\Y5r^"S"S#\Y5r_SMSNS$jjr`SOS%jra"S&S'\D5rbSPS(jrcSQS)jrdSRS*jreSSS+jrfSTS,jrgSUS-jrhSVS.jriSWS/jrjSXS0jrkSYS1jrlSZS2jrmSSSSS3.S[S4jjrnSSSS5.S\S6jjroSS7.S]S8jjrpSSSS5.S^S9jjrqSSSS5.S\S:jjrrSS7.S]S;jjrsS_S`S<jjrt"S=S>\DS?S@9ru\R"SASB55rwSaSCjrxSbSDjrySErzSSSFSSS?SSG.ScSHjjr{SSSFSSS?SSG.ScSIjjr|SJr}\}\|l\}RSKSL5\{lg)dz>Utilities for running language models or Chains over datasets.) annotationsN)datetimetimezone) TYPE_CHECKINGAnyCallableOptionalUnioncast)warn_deprecated) Callbacks)BaseLanguageModel) BaseMessagemessages_from_dict) ChatResult LLMResult)RunnableRunnableConfigRunnableLambdaconfig)utils)EvaluatorCallbackHandlerwait_for_all_evaluators)LangChainTracer)Client) get_git_infoget_langchain_env_var_metadata)EvaluationResult RunEvaluator) run_evaluator) as_runnableis_traceable_function)DatasetDataTypeExampleRun TracerSession)LangSmithError) HTTPError) TypedDict)Chain)load_evaluator) EvaluatorTypePairwiseStringEvaluatorStringEvaluator) evaluation)name_generationprogressc\rSrSrSrSrg)InputFormatErrorKz(Raised when the input format is invalid.N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__r7_/var/www/html/shao/venv/lib/python3.13/site-packages/langchain/smith/evaluation/runner_utils.pyr5r5Ks2r>r5c0\rSrSrSrSSjrSSjrSrg) TestResultRz1A dictionary of the results of a single test run.cUR5nURVs/sH#nURS5(dUS;dM!UPM% nnURSS9R USS9$s snf)zReturn quantiles for the feedback scores. This method calculates and prints the quantiles for the feedback scores across all feedback keys. Returns: A DataFrame containing the quantiles for each feedback key. )inputs.outputs. reference>inputoutputall)include)axis) to_dataframecolumns startswithdescribedrop)selfdfcolto_drops r?get_aggregate_feedback!TestResult.get_aggregate_feedbackUsy   zz !~~BCC)) !  {{5{)..wQ.??  s A'A'c SSKn/n/nUSR5GHwupgUSnURS5n [ U [ 5(a(U R5V V s0sH upSU 3U _M n n n O U c0n OSU 0n 0USR5V V s0sH upS U 3U _M sn n EU En S U;a[[ US [ 5(a;U R US R5V V s0sH upS U 3U _M sn n 5 OUS U S 'U R 0UVs0sHnS UR3UR_M snEURS 5USURS5S.E5 URU 5 URU5 GMz UR"XTS9$![anSn[U5UeSnAff=fs sn n fs sn n fs sn n fs snf)z#Convert the results to a dataframe.rNzfPandas is required to convert the results to a dataframe. to install pandas, run `pip install pandas`.resultsfeedbackrHrErGrDrFz reference.z feedback.Errorexecution_timerun_id)errorr\r])index) pandas ImportErroritemsget isinstancedictupdatekeyscoreappend DataFrame)rRpdemsgindicesrecords example_idresultrZoutput_kvrHrfs r?rMTestResult.to_dataframejs * "&y/"7"7"9 Jj)Hjj*G'4((8? HHQC.!+H"G,06w0E0E0GH0GWQC=!#0GHAf$f[1488HH9? 9L9R9R9TU9T:aS)1,9TU&,K%8AkN HH=EFX155'*AGG3XF#ZZ0&,-=&>$jj2   NN1  NN: &=#:@||G33S *@ c" )  *II V Gs/F)!G G .G #G ) G3GGr7N)return pd.DataFrame)r8r9r:r;r<rVrMr=r7r>r?rArARs;@ @*-4r>rAc:^\rSrSrSrSU4SjjrSSjrSrU=r$) EvalErrorz"Your architecture raised an error.c *>[TU]"SSU0UD6 g)Nr[r7)super__init__)rRr[kwargs __class__s r?rEvalError.__init__s /u//r>cTX$![anSUS3n[U5UeSnAff=f)Nz%'EvalError' object has no attribute '')KeyErrorAttributeError)rRnamerlrms r? __getattr__EvalError.__getattr__s: -:  -9$qAC %1 , -s '"'r7)r[ BaseExceptionrrrxNone)rstrrxr) r8r9r:r;r<rrr= __classcell__)rs@r?r{r{s,0--r>r{c^^ ^ ^ ^ [U[5(abUmTRRnURb7TRRRnSUSUSUS3n[ U5eU4Sj$[U[ 5(aU$[U[5(aUm U 4Sj$[U5(a[U5(a[[[U55m U 4Sj$U"5n[[U5m [U[ 5(aU$[[[U55(a[[[U55m U 4S j$[U[5(dU 4S j$T $U$![aP [[U5n[R"U5n[ R#SU5 [%U5m U 4S js$f=f) zForgive the user if they pass in a chain without memory instead of a chain factory. It's a common mistake. Raise a more helpful error message as well.a$Cannot directly evaluate a chain with stateful memory. To evaluate this chain, pass in a chain constructor that initializes fresh memory each time it is called. This will safegaurd against information leakage between dataset examples. For example: def chain_constructor(): new_memory = z(...) return z*(memory=new_memory, ...) run_on_dataset("z", chain_constructor, ...)c>T$Nr7)chainsr?(_wrap_in_chain_factory..sur>c>T$rr7)lcfsr?rrssr>c>T$rr7 runnable_sr?rr9r>z'Wrapping function %s as RunnableLambda.c>T$rr7)wrappedsr?rrs7r>c>T$rr7rsr?rrrr>c>[T5$r)r) constructorsr?rrs >+6r>)rdr,rr8memory ValueErrorrrcallabler#r"r r TypeErrorinspect signatureloggerinfor) llm_or_chain_factory dataset_name chain_class memory_classrm_model user_funcsigrrrrrs @@@@@r?_wrap_in_chain_factoryrs &..$oo..  & & 2 <<11::L$%1>2)]##/.0J L S/ !&(9::##&11"$%% !5 6 6#D3G$HII$ $ #)+F8%9: f/ 0 0M h!7 8 8#D6$:;I$ $&(++6 6 ) #X';_get_prompt..s> (91Jq#  (9z,Expected list of strings for 'prompts', got rKc3@# UHn[U[5v M g7frrrs r?rrs.S7az!S/A/A7rz)LLM Run expects string prompt input. Got z5LLM Run expects 'prompt' or 'prompts' in inputs. Got rz)LLM Run expects single prompt input. Got z prompts.) r5rdrtyper8listrIlennextitervalues)inputsrmrprompt_s r? _get_promptrs +s##G6&*C006tF8 (.y(9> ; ; VI./889; #3' '# V tFMMO,- gs # #iG  & &3.S7.S+S+SG=fXFC"3' 'EfXNs## 7|qqz 5c'l^9 MC 3 r>c$\rSrSr%SrS\S'Srg)ChatModelInputizJInput for a chat model. Parameters: messages: List of chat messages. zlist[BaseMessage]messagesr7Nr8r9r:r;r<__annotations__r=r7r>r?rrs  r>rcU(d Sn[U5eUR5nSU;aURS5US'O4[U5S:Xa%[ [ UR 555US'SU;acUSn[U[5(a[SU55(aU/n[U5S:Xa[US5US'U$Sn[U5eSU3n[U5e) zGet Chat Messages from inputs. Args: inputs: The input dictionary. Returns: A list of chat messages. Raises: InputFormatError: If the input format is invalid. rrrGrKc3@# UHn[U[5v M g7fr)rdrers r?r _get_messages..5s2 )5AJq$  rrzGBatch messages not supported. Please provide a single list of messages.zMChat Run expects single List[dict] or List[List[dict]] 'messages' input. Got ) r5copypoprrrrrdrrIr)rrm input_copy raw_messagess r? _get_messagesr s +s##JV(nnZ8 7 V "4 #89 7*!'* lD ) )c2 )52 / / )>L |  !"4\!_"EJw  , #3' ' h  3 r>cU(a{U"UR=(d 05n[U[5(dJ[U[5(a[ SU55(dSUS[ U5S3n[ U5egg[UR=(d 05 g![ aP [UR=(d 05 g![ a!nSURS3n[ U5UeSnAff=ff=f)Nc3@# UHn[U[5v M g7frrdrrrms r?r>_validate_example_inputs_for_language_model..RsILSJsK00LrzWhen using an input_mapper to prepare dataset example inputs for an LLM or chat model, the output must a single string or a list of chat messages. Got: of type .zvExample inputs do not match language model input format. Expected a dictionary with messages or a single prompt. Got: z Please update your dataset OR provide an input_mapper to convert the example.inputs to a compatible format for the llm or chat model you wish to evaluate.) rrdrrrIrr5rr) first_example input_mapper prompt_inputrmerr2s r?+_validate_example_inputs_for_language_modelrJs#M$8$8$>B? ,,, |T * *ILIII'yl1C0DAG  #3' 'J- 6  ,,2 3 6 6m228b9# 6*112GG's+5 6 6s*B## C=.C C9C44C99C=cpU(aU"UR=(d 05n[UR5RU5n[ U[ 5(dSUS[ U5S3n[U5eU(a+SURSUR53n[U5egURn[UR5RU5n[U5S:Xa[UR5S:XagU(a+SURSUR53n[U5eg) zB? 5++,77 E ,--&yl1C0DAG  #3' ' $//0 8I8I8K7LN #3' ' %++ 5++,77 E |  !c%*:*:&;q&@  #--./$))+, . #3' 'r>c[U[5(a [X5 gU"5n[U[5(a [ XU5 g[U[ 5(a[ RSU5 gg)z9Validate that the example inputs are valid for the model.z Skipping input validation for %sN)rdrrr,rrrdebug)examplerrrs r?_validate_example_inputsrs[ &(9::3GJ$& eU # # .w| L x ( ( LL;U C)r>c jU(a[U[5(aSupESnOOSnU"5n[U[5(a UROSn[U[5(a UROSn[ UUUUSR (a[USR 5OSUU5nU$SnU$)z >&0 #JHH(*E-7u-E-E))4J/9%/G/G%++TK-   )1!)<)cSnUR(a1URnU(aX!;a[RSUU5 U$U(a[U5S:XaUSnU$Ub%[U5S:a[RSU5 U$)NzZInput key %s not in chain's specified input keys %s. Evaluation behavior may be undefined.rKrzChain expects multiple input keys: %s, Evaluator is likely to fail. Evaluation behavior may be undefined. Specify an input_key in the RunEvalConfig to avoid this warning.) input_keyrwarningr)rrrs r?_determine_input_keyrsI $$ )5 NN8    J1,qM    C Oa$7 P    r>cSnUR(a1URnU(aX!;a[RSUU5 U$U(a[U5S:XaUSnU$Ub%[U5S:a[RSU5 U$)Nz`Prediction key %s not in chain's specified output keys %s. Evaluation behavior may be undefined.rKrzChain expects multiple output keys: %s, Evaluation behavior may be undefined. Specify a prediction_key in the RunEvalConfig to avoid this warning.)prediction_keyrrr)rrrs r?_determine_prediction_keyrsN .. >< NN8    [)Q.$Q   S%5%9 ;   r>cUR(a-URnU(aX!;aSUSU3n[U5eU$U(a%[U5S:Xa[[ U55nU$SnU$)NzReference key z! not in Dataset example outputs: rK) reference_keyrrrr)rexample_outputsrrms r?_determine_reference_keyrs,, }C 0%%4$57 S/ !  S1Q6T/23   r>c [U[5(aU$[U[[45(a6[U[5(d [U5n[ XS9nUR n O[U[ R5(aSU0UR5En [ UR40U D6nURR n [U[ R5(a?UR=(d UnUR=(d UnUR=(d UnO4[U5(a [U5$S[!U53n [#U 5e[U[$5(aNUR&(aUcSU SUS3n [#U 5e[(R*R-UUUUUUU /S9n U $[U[.5(aSU S 3n [1U 5eSU S 3n [1U 5e) N)rrzUnknown evaluator type: zPMust specify reference_key in smith_eval.RunEvalConfig to use evaluator of type z) with dataset with multiple output keys: r)rrrtagszRun evaluator for z is not implemented. PairwiseStringEvaluators compare the outputs of two different models rather than the output of a single model. Did you mean to use a StringEvaluator instead? See: https://python.langchain.com/docs/guides/evaluation/string/z is not implemented)rdr r.rr-valuesmith_eval_config EvalConfig get_kwargsevaluator_typeSingleKeyEvalConfigrrrrrun_evaluator_decrrr0requires_reference smith_evalStringRunEvaluatorChainfrom_run_and_data_typer/NotImplementedError) eval_configeval_llmrrrrrr evaluator_ eval_type_tagrrmr!s r?_construct_run_evaluatorrs+|,,+ s344+}55' 4K#K> #)) K!2!=!= > >>[%;%;%=>#K$>$>I&I #2288 k#4#H#H I I#--:I(77I>N'55FM +   --(k):(;<o*o..  ( (]-B&&3_577F6GqJ  S/ !"::QQ   )'R ,  J 7 8 8 0Q Q "#&&#=/1DE!#&&r>cL[X5n[X5n[X5nXEU4$r)rrr)rrrrrrrs r? _get_keysrTs- %V8I.vCN,VEM m 33r>c /nSupxn UR(d2UR(a2[SUR55(a[UUUU5upxn URH0n [ U UR UUUU UU5n UR U 5 M2 UR=(d /n U Hn [U [5(aUR U 5 M+[U [5(a3UR [RRU UUUUU S95 Ms[U 5(aUR [U 55 MSU S3n[U5e U$)z Load run evaluators from a configuration. Args: config: Configuration for the run evaluators. Returns: A list of run evaluators. NNNc3@# UHn[U[5v M g7fr)rdr0)rrls r?r'_load_run_evaluators..usQ8P1 1o..8Pr)rrrzUnsupported custom evaluator: z+. Expected RunEvaluator or StringEvaluator.) evaluatorscustom_evaluatorsanyrrrrirdr r0r r r rr r)rrrrrrrrrrrr!rcustom_evaluatorrms r?rr`st"N/?,I}   Q8P8PQ Q Q3<     4 0 = (( 0  OO         m,)006B- & 5 5  ! !"2 3 (/ : :  ! !22II$'#1"/ J & ' '  ! !"34D"E F11A0BC=> S/ !+.. r>r callbacksrmetadatac `# UbU"U5n[U[5(d,[U[5(aJ[SU55(a3UR U[ UU=(d /U=(d 0S9S9IShvN $SUS3n[ U5e[U5nUR U[ UU=(d /U=(d 0S9S9IShvN n U $NVN![ aI [U5n UR"S0U DS[ UU=(d /U=(d 0S90D6IShvN n U $f=f7f) aAsynchronously run the language model. Args: llm: The language model to run. inputs: The input dictionary. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. input_mapper: Optional function to map inputs to the expected format. Returns: The LLMResult or ChatResult. Raises: ValueError: If the LLM type is unsupported. InputFormatError: If the input format is invalid. Nc3@# UHn[U[5v M g7frrrs r?r_arun_llm..OD.9C CCD.CA D+!D$"D+'D.*D++D.rrr c # UcUOU"U5n[U[5(a[U[5(aw[U5S:XahUR(aW[ [ UR555nURU[UU=(d /U=(d 0S9S9IShvN nU$[U=(d /UU=(d 0S9n URXiS9IShvN nU$N<N7f)z%Run a chain asynchronously on inputs.NrKr%rrrr ) rdr,rerrrrrr'r rrrrrr inputs_valrHrunnable_configs r? _arun_chainr3s%,f,v2FG5%  w % % LA    4()*}} !#ZR!R%   M )^  }}W}EE M Fs$B(C+*C'+5C+ C)!C+)C+)rc  # [U[5(aSOSnSn[U[5(a?[UUR=(d 0USUSUUR S5S9IShvN nOEU"5n[ UUR=(d 0USUSUUR S5S9IShvN nUnU$NNN ![ aAn[RSUURURU5 [US 9nSnAU$SnAff=f7f) atAsynchronously run the Chain or language model. Args: example: The example to run. llm_or_chain_factory: The Chain or language model constructor to run. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. input_mapper: Optional function to map the input to the expected format. Returns: A list of outputs. LLMr,Nrrr rz*%s failed for example %s with inputs %s %sr[) rdrr+rrcr3 Exceptionrridr{) rrrr chain_or_llmrqrHrrls r?_arun_llm_or_chainr:s(02CDD'F$ *,= > > )$$"F^ -)J/ !F)*E&$"F^ -)J/ F M9 $ 9  JJ NN  # M$sTDAC-B<.AC3B>4C:D<C>C D  6DDD  Dc  UbU"U5n[U[5(d,[U[5(aD[SU55(a-UR U[ UU=(d /U=(d 0S9S9nU$SUS3n[ U5e[U5n UR U [ UU=(d /U=(d 0S9S9nU$![ a5 [U5n UR"S0U DS[ X%=(d 0S90D6nU$f=f) a Run the language model on the example. Args: llm: The language model to run. inputs: The input dictionary. callbacks: The callbacks to use during the run. tags: Optional tags to add to the run. input_mapper: function to map to the inputs dictionary from an Example Returns: The LLMResult or ChatResult. Raises: ValueError: If the LLM type is unsupported. InputFormatError: If the input format is invalid. c3@# UHn[U[5v M g7frrrs r?r_run_llm..gr$rr%rz'Input mapper returned invalid format: r&r)rr r7) rdrrrIinvokerr5rr) rrrrrr r(r)rm llm_promptsr*s r?_run_llmr@Js22)&1 (# . . )4 0 0Orr/s r? _run_chainrBs%,f,v2FG5%  w % % LA    4()* !#ZR!R  M )^  g> Mr>c [U[5(aSOSnSn[U[5(a7[UUR=(d 0USUSUUR S5S9nO=U"5n[ UUR=(d 0USUSUUR S5S9nUnU$![ aWn[U5Rn [RSUURURU U5 [US 9nSnAU$SnAff=f) a~ Run the Chain or language model synchronously. Args: example: The example to run. llm_or_chain_factory: The Chain or language model constructor to run. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. Returns: Union[List[dict], List[str], List[LLMResult], List[ChatResult]]: The outputs of the model or chain. r5r,Nrrr r,zC%s failed for example %s with inputs %s Error Type: %s, Message: %sr6) rdrr@rrcrBr7rr8rrr8r{) rrrrr9rqrHrrl error_types r?_run_llm_or_chainrEs*02CDD'F$ *,= > >"$$"{#F^)J/ F)*E$"{#F^)J/ F M  $!W%%  R  JJ NN   # M $sB B** D 4A DD c P[X!5nURUS9n[URURUS95n U (dSUS3n [ U 5eU V s/sH!oR (dMU R PM# n n U (a [U 5OSn U (aU R5OSnU=(d 0n[5nU(a0UESU0EnXS'URUURU(aSU0O0US9nUR SUR3-n[#SUSUSUSUR 3SS9 UUX4$s sn f![[ [4aJnS [U5;ae[R"5nS US US US 3nSUSU3n [ U 5UeSnAff=f)N)r) dataset_idas_ofzDataset z has no example rows.gitdataset_versionr)reference_dataset_id project_extrar zalready exists z+ run_on_dataset( ... project_name="z - z", # Update since z already exists ) z Test project z/ already exists. Please use a different name: z/compare?selectedSessions=z)View the evaluation results for project 'z' at: z View all tests for Dataset z at: T)flush)r read_datasetr list_examplesr8r modified_atmax isoformatrcreate_projectr*r)ruuiduuid4urlprint)clientrr project_nameproject_metadatarrJ wrapped_modeldatasetrrmexrPmax_modified_atinferred_versiongit_infoprojectrluid example_msgcomparison_urls r?_prepare_eval_runres++?NM!!|!2>>HKG+6c+&4O6E0024%+1r>  " x   /?*+'' !(,064.b% ( *[[%? |#LLN 3L>B !&&2^6'++ H  '7 44YH, z> 2% CF * jjl .C5(:<.I L>*- ! o1$%s&E1E1AEF%AF  F%c8\rSrSr%SrS\S'S\S'S\S'S rg ) _RowResulti*z5A dictionary of the results for a single example row.z Optional[list[EvaluationResult]]rZzOptional[float]r\ Optional[str]r]r7Nrr7r>r?rgrg*s?..## r>rgF)totalc\rSrSr%SrS\S'S\S'S\S'S \S 'S \S 'S rS\S'SSjrSSjrSSjr SSjr SSSjjr \ SSSjj5r Srg )_DatasetRunContaineri2z3A container to help manage the state of a eval run.rrXr(raMCFr[ list[Example]rzlist[RunnableConfig]configsNz6Optional[list[smith_eval_config.BATCH_EVALUATOR_LIKE]]batch_evaluatorsc v0n[URU5GHupE[[UR [ UR 5055nURUR S/5UR S5UR S5S.U[ UR 5'[U[5(a&URU[ UR 5S'OXS[ UR 5S'UR(dMURU[ UR 5S'GM U$)NrZr\r])rGrZr\r]r[rHrF) ziprr rgrcrr8rrdr{r[r)rR batch_resultsall_eval_resultsrYrrH row_results r?_merge_test_outputs(_DatasetRunContainer._merge_test_outputs=s "4==-@OGj*:*>*>s7::PR*STJ &NN:r:",..1A"B$..2 (GC O $ &),,4:LLGJJ(15;GJJ(28?GJJ(5 Ar>c URnU(d/$URVs/sHo1[UR5PM nn/n[R R 5nUHnU"X@R5n[U[5(aUR5nUR[[U55 UR"URR40UDSURRS.D6 M SSS5 U$s snf![ a# ["R%S['U55 Mf=f!,(df  U$=f)N)r] project_idz Error running batch evaluator %s)rorrr8 concurrentfuturesThreadPoolExecutorrdrrerir submitrXcreate_feedbackrar7r exceptionrepr) rRrunsrr runs_listaggregate_feedbackexecutor evaluatorrqs r?_run_batch_evaluators*_DatasetRunContainer._run_batch_evaluatorsSs** I:>--H-w#gjj/*- H    2 2 4' &y--@F!&*:;;!'&--d4.@AOO 33  $#'<c6[RS5 [5 UR5up#SnUR(a&[RS5 UR U5nUR X5n[URRUUS9$)Nz#Waiting for evaluators to complete.zRunning session evaluators.)rYrYaggregate_metrics) rrrrrorrurArar)rRrrrsrrrYs r?_collect_test_results*_DatasetRunContainer._collect_test_resultss  9:!%)%:%:%<"!  KK5 6!%!;!;H!E **=K**0  r>cURU5nU(aUR5n[U5 UR RURR[R"[R5S9 U$![an[R SUSS9 SnANwSnAff=f![a n[R SUSS9 SnAU$SnAff=f)Nz&Failed to print aggregate feedback: %sT)exc_info)rzFailed to close project: %s)rrV_display_aggregate_resultsr7rrrXupdate_projectrar8rnowrutc)rRrrverboserY agg_feedbackrls r?finish_DatasetRunContainer.finishs ,,];  Y&==? *<8 J KK & & !hll3 '   Y EqSW X Y J LL6DL I Js/BAB4 B1B,,B14 C>CCc hU=(d [R"5nU (aU (d0n U RSU 05 [UUUUU UU S9uppU=(d /nU RR S5=(d 0R 5HunnURSUSU35 M SU RS0nU (aU US'[U5n [U UUUR=(d [R5n[USX5 [R"[!U55nUVs/sHLn[#[%U R&UUR(S9[+U=(d /UUR(SS 9U/UUUS 9PMN nnU"UU U UUU(aUR,S 9$SS 9$s snf) N revision_id)rZrrJrIzgit:=rJr)rYrXrp)rrXrpmax_concurrency)rrrr )rXrar[rrnro)r2 random_namerfrer rcrbrirrrr%kvrr3ProgressBarCallbackrrrrr8rro)clsrXrrrYr1rrconcurrency_levelrZrrJr[rar\rrsrt run_metadatar progress_barrrns r?prepare_DatasetRunContainer.prepares$D'B'B'D ##%  # #]K$@ A4E   -+5 1 zr%%))%06B==?DAq KK$qc1# '@)7+;+;r?rkrk2sF= N  !!OSLS0  ,"2G: D   *  ,:>$(8H H H 5 H $ H 7 H "H 6H H 3H #H 8H  H H r>rkcSSKJn U"5nU"5SL=(a S[[U55;$![a gf=f)Nr) get_ipythonzmqshellF)IPython.core.getipythonrrrra)rress r?_is_jupyter_environmentrsB7m}D(IZ3tCy>-II s36 AAc[5(aSSKJnJn U"U"S55 U"U5 gUR SSS9n[ S5 [ U5 g)Nr)HTMLdisplayz

Experiment Results:

c US$)Nz.2fr7)xs r?r,_display_aggregate_results..s aW:r>right) float_formatjustifyz Experiment Results:)rIPython.displayrr to_stringrW)aggregate_resultsrrformatted_strings r?rrsU  1345!",66-7  &' r>aThe input_mapper argument is deprecated and will be removed in a future release. Please add a RunnableLambda to your chain to map inputs to the expected format instead. Example: def construct_chain(): my_chain = ... input_mapper = {'other_key': 'MyOtherInput', 'my_input_key': x} return input_mapper | my_chain run_on_dataset(..., llm_or_chain_factory=construct_chain) (See https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.base.RunnableLambda.html)r)r1rJrrYrZrrc# U RSS5n U (a[S[SS9 U c[5R S5n U RSS5n U (a [SSSS9 U (a[SS U R 5S 3SS 9 U=(d [ 5n[RUUUUUU U UUU US 9 n [R"U RS R S5/[[R"[U R U S9U R"U R5Q76IShvN nU R%XS9$N7f)Nr0.0.305Tmessagependingrr0.1.9qThe tags argument is deprecated and will be removed in a future release. Please specify project_metadata instead.PThe following arguments are deprecated and will be removed in a future release: rrremovalrZrrJrrrrr)rr _INPUT_MAPPER_DEP_WARNINGrrcrrrkrrunnable_utilsgather_with_concurrencyrnmap functoolspartialr:r[rr)rXrrr1rJrrYrZrrrrr containerrrs r?arun_on_datasetr'sa::nd3L +DdS46::=I ::fd #D  U   4{{}oQ    vxF$,, )'- I)@@!  !23    "%.%<%<)          M   M  ;; sD4E 6E 7E c `U RSS5n U (a[S[SS9 U RSS5n U (a [SSSS9 U c[5R S5n U (a[SS U R 5S 3SS 9 U=(d [ 5n[RUUUUUU U UUU US 9 n US :XaH[U RU R5VVs/sHup[UUU RU S9PM nnnO~[R"U RS 5n[!UR#[$R&"[U RU S9U RU R55nSSS5 U R)WUS9$s snnf!,(df  N$=f)NrrTrrrrrrrrrrrr)rr rrrcrrrkrrqrrnrEr[r2get_executor_for_configrrrrr)rXrrr1rJrrYrZrrrrrrrrrrrs r?run_on_datasetrhs::nd3L +DdS ::fd #D  U  46::=I   4{{}oQ    vxF$,, )'- IA$'y'9'99;L;L#M $N %.%<%<)   $N   4 4Y5F5Fq5I Jh  %%)-6-D-D%1 &&%% MK   M7  ;;/ K Js&!F.AF F-a1 Run the Chain or language model on a dataset and store traces to the specified project name. Args: dataset_name: Name of the dataset to run the chain on. llm_or_chain_factory: Language model or Chain constructor to run over the dataset. The Chain constructor is used to permit independent calls on each example without carrying over state. evaluation: Configuration for evaluators to run on the results of the chain concurrency_level: The number of async tasks to run concurrently. project_name: Name of the project to store the traces in. Defaults to {dataset_name}-{chain class name}-{datetime}. project_metadata: Optional metadata to add to the project. Useful for storing information the test variant. (prompt version, model version, etc.) client: LangSmith client to use to access the dataset and to log feedback and run traces. verbose: Whether to print progress. tags: Tags to add to each run in the project. revision_id: Optional revision identifier to assign this test run to track the performance of different versions of your system. Returns: A dictionary containing the run's project name and the resulting model outputs. For the (usually faster) async version of this function, see :func:`arun_on_dataset`. Examples -------- .. code-block:: python from langsmith import Client from langchain_openai import ChatOpenAI from langchain.chains import LLMChain from langchain.smith import smith_eval.RunEvalConfig, run_on_dataset # Chains may have memory. Passing in a constructor function lets the # evaluation framework avoid cross-contamination between runs. def construct_chain(): llm = ChatOpenAI(temperature=0) chain = LLMChain.from_string( llm, "What's the answer to {your_input_key}" ) return chain # Load off-the-shelf evaluators via config or the EvaluatorType (string or enum) evaluation_config = smith_eval.RunEvalConfig( evaluators=[ "qa", # "Correctness" against a reference answer "embedding_distance", smith_eval.RunEvalConfig.Criteria("helpfulness"), smith_eval.RunEvalConfig.Criteria({ "fifth-grader-score": "Do you have to be smarter than a fifth grader to answer this question?" }), ] ) client = Client() run_on_dataset( client, dataset_name="", llm_or_chain_factory=construct_chain, evaluation=evaluation_config, ) You can also create custom evaluators by subclassing the :class:`StringEvaluator ` or LangSmith's `RunEvaluator` classes. .. code-block:: python from typing import Optional from langchain.evaluation import StringEvaluator class MyStringEvaluator(StringEvaluator): @property def requires_input(self) -> bool: return False @property def requires_reference(self) -> bool: return True @property def evaluation_name(self) -> str: return "exact_match" def _evaluate_strings(self, prediction, reference=None, input=None, **kwargs) -> dict: return {"score": prediction == reference} evaluation_config = smith_eval.RunEvalConfig( custom_evaluators = [MyStringEvaluator()], ) run_on_dataset( client, dataset_name="", llm_or_chain_factory=construct_chain, evaluation=evaluation_config, ) zrun_on_dataset(zawait arun_on_dataset()z )rrrrrxrl)rdict[str, Any]rxr)rrrxre)rr&rrrxr)rr&rr,rrrxr)rr&rrlrrrxr) rrlrrmr1rrr%rxzOptional[list[RunEvaluator]])rsmith_eval.RunEvalConfigrrrxrh)rrrrrxrh)rrrrrxrh)rzYUnion[smith_eval_config.SINGLE_EVAL_CONFIG_TYPE, smith_eval_config.CUSTOM_EVALUATOR_TYPE]rzOptional[BaseLanguageModel]rrrr%rrrrhrrhrrhrxr ) rrrrrrrrrxz2tuple[Optional[str], Optional[str], Optional[str]])rrrrrr%rrrrrrrxzlist[RunEvaluator])rrrrrrrr rrr rrxUnion[str, BaseMessage])rzUnion[Chain, Runnable]rrrr rrrrr rrxzUnion[dict, str]) rr&rrrrlrrrxz'Union[dict, str, LLMResult, ChatResult])rrrrrr rrrrr rrxrr)rXrrrrrrYrrZrrrrJzOptional[Union[str, datetime]]rxz1tuple[MCF, TracerSession, Dataset, list[Example]])rxr)rryrxr)rXzOptional[Client]rrrrr1rrJrrrrYrhrZrrrrrhrrrxr)r< __future__rconcurrent.futuresry dataclassesrrloggingrTrrtypingrrrr r r langchain_core._apir langchain_core.callbacksr langchain_core.language_modelsrlangchain_core.messagesrrlangchain_core.outputsrrlangchain_core.runnablesrrrrr2rr!langchain_core.tracers.evaluationrr langchain_core.tracers.langchainrlangsmith.clientr langsmith.envrrlangsmith.evaluationrr r!r langsmith.run_helpersr"r#langsmith.schemasr$r%r&r'r(langsmith.utilsr)requestsr*typing_extensionsr+langchain.chains.baser,langchain.evaluation.loadingr-langchain.evaluation.schemar.r/r0langchain.smithr1r langchain.smith.evaluationrr2r3r`rk getLoggerr8rrerrlr7r5rAr{rrrrrrrrrrrrrrr+r3r:r@rBrErerg dataclassrkrrrrr_RUN_ON_DATASET_DOCSTRINGreplacer7r>r?rs)D" '0.<C8MM><=#FELL*''7 5B@   8 $ Ruh' '( dVS[    HRuh// 02C CD3y3E4E4P - - ': 0: :  : z. b Y & T 6 61 6  6F%(%( %(2%( %(PD DD2D D(3  " < $#6 $$4 $(&BB * B  BB)B!BB"BBJ 4 $ 4# 4% 4) 4 8 4A $AAA) A $ A % AAV!%48)-A A A  A  A 2 A'AAR!%48)-! !! !!  ! 2 !'!!R59 5 5 5 5 2 5 - 5@!%48)-? ? ??  ? 2 ?'??N!%48)-! !! !!  ! 2 !'!!R59 8 8 8 8 2 8 - 8@26 $6:<5 <5<51<5 <5 / <5  <54<57<5~% G G G T  :,6:6:"&15!%>< ><><1>< 3 >< 4 ><>< ></><><><><>