dhk S SK r S SKrS SKrS SKrS SKrS SKrS SKJr S SKJ r S SK
J r J r J
r
JrJrJr S SKJrJr S SKJr S SKJr S SKJr S S KJrJ r S S
K J r J r J r J!r!J"r"J#r# \ RH " \%5 r&S \ 4S jr'S \
\( 4S
jr)S \
\( 4S jr* SS\(S\ S\ S \+4S jjr,S\(S\(S \ 4S jr- " S S5 r. " S S\ \5 r/g) N)deepcopy)Path)AnyDictListOptionalSequenceUnion) AgentAction AgentFinish)BaseCallbackHandler)Document) LLMResult)get_from_dict_or_env guard_import)BaseMetadataCallbackHandler flatten_dict hash_string
import_pandas import_spacyimport_textstatreturnc [ S5 $ )zKImport the mlflow python package and raise an error if it is not installed.mlflow)r e/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_community/callbacks/mlflow_callback.py
import_mlflowr s
!!r c
/ SQ$ )z!Get the metrics to log to MLFlow.)stepstartsendserrorstext_ctr chain_starts
chain_ends
llm_startsllm_ends llm_streams tool_starts tool_ends
agent_endsretriever_startsretriever_endsr r r r mlflow_callback_metricsr/ " s r c
/ SQ$ )z.Get the text complexity metrics from textstat.)flesch_reading_easeflesch_kincaid_grade
smog_indexcoleman_liau_indexautomated_readability_index dale_chall_readability_scoredifficult_wordslinsear_write_formula gunning_fogfernandez_huertaszigriszt_pazosgutierrez_polinicrawfordgulpease_indexosmanr r r r get_text_complexity_metricsr@ 7 s r textnlptextstatc x 0 nUbL [ 5 Vs0 sH oD[ X$5 " U 5 _M nnUR SU05 UR U5 Ub` [ 5 nU" U 5 nUR R USSSS9nUR R USSSS9n UU S.n
UR U
5 U$ s snf )aq Analyze text using textstat and spacy.
Parameters:
text (str): The text to analyze.
nlp (spacy.lang): The spacy language model to use for visualization.
textstat: The textstat library to use for complexity metrics calculation.
Returns:
(dict): A dictionary containing the complexity metrics and visualization
files serialized to HTML string.
text_complexity_metricsdepFT)stylejupyterpageent)dependency_treeentities)r@ getattrupdater displacyrender) rA rB rC respkeyrE spacydocdep_outent_outtext_visualizationss r analyze_textrX M s D9T9V#
9V#'- -9V #
.0GHI +,
$i..''5%d'S..''5%d'S '
'( K+#
s B7prompt
generationc \ U R SS5 nUR SS5 nSU SU S3$ )zConstruct an html element from a prompt and a generation.
Parameters:
prompt (str): The prompt.
generation (str): The generation.
Returns:
(str): The html string.
z
z
z>:
z"
)replace)rY rZ formatted_promptformatted_generations r )construct_html_from_prompt_and_generationr` w sO ~~dF3%--dF; - . /
r c < \ rS rSrSrS\4S jr S S\S\\\4 S\ \ S S4S
jjr
S S jr S \S
\ S S4S jr
S!S\\\\ 4 \\\4 4 S\ \ S S4S jjrS\\\4 S\S S4S jrS\S\S S4S jrS\S\S S4S jrS\S\S S4S jrS\S S4S jrS \S S4S jrS rg)" MlflowLogger ar Callback Handler that logs metrics and artifacts to mlflow server.
Parameters:
name (str): Name of the run.
experiment (str): Name of the experiment.
tags (dict): Tags to be attached for the run.
tracking_uri (str): MLflow tracking server uri.
This handler implements the helper functions to initialize,
log metrics and artifacts to the mlflow server.
kwargsc [ 5 U l S[ R ; ay U R R S5 U R R
R R 5 U l U R R U R 5 U l
O[ USSS5 nU R R U5 UR S5 =n( a5 U R R U5 R R U l Ov[ USS5 nU R R! U5 U l
U R b U R R U l O U R R# U5 U l U R% US US
UR SS 5 5 UR S S5 U l g ) NDATABRICKS_RUNTIME_VERSION
databricks tracking_uriMLFLOW_TRACKING_URI run_idexperiment_nameMLFLOW_EXPERIMENT_NAMErun_namerun_tags
artifacts_dir)r r osenvironset_tracking_uritrackingfluent_get_experiment_id mlf_expidget_experimentmlf_expr getget_runinfo
experiment_idget_experiment_by_namecreate_experiment start_rundir)selfrd rh rk rl s r __init__MlflowLogger.__init__ s[ #o '2:: 5 KK ( ( 6 ![[ 1 1 8 8 K K MDN ;;55dnnEDL/(=r L
KK ( ( 6 H--v-!%!4!4V!}}T""NN6#9#9FMM#IQO CRy5(++**,77 8 C XX__F r c 8 U R R 5 g)zTo finish the run.N)r end_runr s r
finish_runMlflowLogger.finish_run s r rR valuec L U R R XU R S9 g)z To log metric to mlflow server.rk N)r
log_metricrk )r rR r s r metricMlflowLogger.metric s s$++ >r datar c J U R R XR S9 g)z%To log all metrics in the input dict.r N)r log_metricsrk )r r r s r metricsMlflowLogger.metrics s
[[ 9r filenamec U R R U[ R R U R
U S35 U R S9 g)z,To log the input data as json file artifact.z.jsonr N)r log_dictrq pathr r rk )r r r s r jsonfMlflowLogger.jsonf > "'',,txxH:U); "'',,txxH:T):;DKK
r r c J U R R XR S9 g)z/To upload the file from given path as artifact.r N)r log_artifactrk )r r s r artifactMlflowLogger.artifact s kk :r chainc b U R R R USU R S9 g )Nzlangchain-modelr )r langchain log_modelrk )r r s r langchain_artifact MlflowLogger.langchain_artifact s& ''/@'Ur )r ry rw r rk Nr N)r )__name__
__module__ __qualname____firstlineno____doc__r r strr r r r floatr r
intr r r r rA r r __static_attributes__r r r rb rb sG
3 3B HL #CH~ 7?}
& ?# ?e ? ?
TU:$sEz*DcN:;:CKC=:
:
$sCx.
C
D
8# 8# 8$ 8
;S ;T ;V V Vr rb c ^ \ rS rSrSr S1S\\ S\\ S\\ S\\ S\\ S \S
S4U 4S jjjrS2S jr S
\\\
4 S\ \ S\
S
S4S jr S\S\
S
S4S jr
S\S\
S
S4S jrS\S\
S
S4S jrS
\\\
4 S\\\
4 S\
S
S4S jrS\\\\
4 \\ \ 4 S\
S
S4S jrS\S\
S
S4S jrS
\\\
4 S \S\
S
S4S jrS \
S\
S
S4S jrS\S\
S
S4S jrS!\S\
S
S4S" jrS#\S\
S
S4S$ jrS%\ S\
S
\
4S& jr S
\\\
4 S'\S\
S
\
4S( jr S)\ \ S\
S
\
4S* jr!S\S\
S
\
4S+ jr"S
\
4S, jr#S
\$4S- jr%S3S.\
S#\$S
S4S/ jjr&S0r'U =r($ )4MlflowCallbackHandler a' Callback Handler that logs metrics and artifacts to mlflow server.
Parameters:
name (str): Name of the run.
experiment (str): Name of the experiment.
tags (dict): Tags to be attached for the run.
tracking_uri (str): MLflow tracking server uri.
This handler will utilize the associated callback method called and formats
the input of each callback function with metadata regarding the state of LLM run,
and adds the response to the list of records for both the {method}_records and
action. It then logs the response to mlflow server.
Nr
experimentr rh rk rp r c > [ 5 [ 5 [ T
U ]
5 Xl X l U=( d 0 U l X@l XPl X`l [ R " 5 U l [ U R U R
U R U R U R U R S9U l / U l SU l [# 5 n UR% S5 U l [1 5 U l [5 5 V s0 sH oS_M sn U l / / / / / / / / / / / / / S.
U l g! [& a [( R+ S5 Nff = f! [, a) n[( R+ UR. 5 SnANSnAff = f! [, a0 n[( R+ UR. 5 SU l SnANSnAff = fs sn f )z Initialize callback handler.)rh rl rn ro rk rp Nen_core_web_smzfRun `python -m spacy download en_core_web_sm` to download en_core_web_sm model for text visualization.r )
on_llm_start_recordson_llm_token_recordson_llm_end_recordson_chain_start_recordson_chain_end_recordson_tool_start_recordson_tool_end_recordson_text_recordson_agent_finish_recordson_agent_action_recordson_retriever_start_recordson_retriever_end_recordsaction_records) r r superr r r r rh rk rp tempfileTemporaryDirectorytemp_dirrb mlflgr rB r loadOSErrorloggerwarning ImportErrormsgr rC r/ r records) r r r r rh rk rp rS erR __class__s r r MlflowCallbackHandler.__init__ s
$JB ( * 335
! ** OOYYYY;; ,,
%' NE
::&67 ! + -DM
+B*CD*C3Q*CD %'$&"$&($&%'#% !')')*,(* (
O
" NN155 ! ! " ! NN155 ! DMM ! EsH
E D0 ,F G0 EE
F FF
G&G Gc U R R 5 H u pSU R U' M U R R 5 H u p/ U R U' M g )Nr )r itemsr )r r vs r _reset MlflowCallbackHandler._resetO sN LL&&(DA DLLO )LL&&(DA DLLO )r
serializedpromptsrd c U R S== S-
ss' U R S== S-
ss' U R S== S-
ss' U R S n0 nUR SS05 UR [ U5 5 UR U R 5 U R R U R U R S S9 [ U5 Hr u pg[ U5 nXxS' U R S R U5 U R S
R U5 U R R US U S U 35 Mt g
)zRun when LLM starts.r r' r! action on_llm_startr rY r r
llm_start__prompt_N) r rN r r enumerater r appendr ) r r r rd r' rQ idxrY prompt_resps r r "MlflowCallbackHandler.on_llm_startU s
V ! \"a'" X !# \\,/
! X~./ L,- DLL!
4< JJ [Jzl(3%*P Q .r tokenc U R S== S-
ss' U R S== S-
ss' U R S n0 nUR SUS.5 UR U R 5 U R R U R U R S S9 U R S R U5 U R S R U5 U R R US U 35 g
) z#Run when LLM generates a new token.r r r) on_llm_new_token)r r r r r llm_new_tokens_Nr rN r r r r )r r rd r) rQ s r r &MlflowCallbackHandler.on_llm_new_tokenm s V ! ]#q(#ll=1 ! 1EBC DLL!
4<?r responsec
& U R S== S-
ss' U R S== S-
ss' U R S== S-
ss' U R S n0 nUR SS05 UR [ UR =( d 0 5 5 UR U R 5 U R R U R U R S S9 UR
GH n[
U5 GH u pg[ U5 nUR [ UR 5 5 5 UR [ UR U R U R S95 S U; a8 UR S 5 n U R R U U R S S9 U R S
R U5 U R S R U5 U R R! US U S
U 35 SU; a7 US n
U R R# U
S[% UR 5 - 5 SU; d GMS US n U R R# U S[% UR 5 - 5 GM GM g)zRun when LLM ends running.r r r( r" r
on_llm_endr )rB rC rE r r llm_end_ _generation_rK zdep-rL zent-N)r rN r
llm_outputr generationsr r dictrX rA rB rC popr r r r r ) r r rd r( rQ r r rZ generation_respcomplexity_metricsrK rL s r r
MlflowCallbackHandler.on_llm_end~ s$ V ! Z A% V ! <<
+ ! X|,- L!4!4!:;< DLL!
4<#J>41qc1#J>#JKK
%
% ((F#CF5CJF#CDK f+K d^
*8 -.55jA %&--j9
|L>%BC $K#Cs G
Goutputsc N U R S== S-
ss' U R S== S-
ss' U R S== S-
ss' U R S n0 n[ U[ 5 ( a9 SR UR 5 VVs/ sH u pVU SU 3PM snn5 nO@[ U[
5 ( a SR [
[ U5 5 nO [ U5 nUR SUS.5 UR U R 5 U R R U R U R S S 9 U R S
R U5 U R S R U5 U R R US U 35 g
s snnf )z Run when chain ends running.r r r& r" r! r" on_chain_end)r r* r r r
chain_end_N)
r r$ r r r r% mapr rN r r r r )r r* rd r& rQ r r chain_outputs r r, "MlflowCallbackHandler.on_chain_end sP
V ! \"a'" V ! \\,/
! gt $ $ 88GMMO$LODAs!A3ZO$LML
&
& 88CW$56L w u pgU[ U[
5 ( d [
U5 OSR S U 5 5 _M@ snnS.PMw nnnnUR SUS .5 UR U R 5 U R R U R U R S S
9 U R S R U5 U R S R U5 U R R US
U 35 gs snnf s snnnf )z Run when Retriever ends running.r r r. r" r! c 3 6 # U H n[ U5 v M g 7fr )r ).0xs r 9MlflowCallbackHandler.on_retriever_end.. s %8ac!ffas ) page_contentmetadataon_retriever_end)r rX r r r retriever_end_N)
r r_ r` r r$ r% r r rN r r r r ) r rX rd r. rQ rT r r retriever_documentss r ra &MlflowCallbackHandler.on_retriever_endr s
V ! %&!+& V ! &67 ! !
! !$ 0 0 !$ 2 2 4
!5 )!T 2 2 A XX%8a%8 89
!5
!
1@STU DLL!
4<CMlflowCallbackHandler._create_session_analysis_df.. s CGr r )axis prompt_steprK rL )token_usage_total_tokenstoken_usage_prompt_tokens token_usage_completion_tokensrA output_stepr: )r rA c & [ U S U S 5 $ )NrY r: )r` )rows r rl rm s AH
s8}r chat_html) r DataFramer columnsr applydropnarenamerC r@ rB concat) r pdon_llm_start_records_dfon_llm_end_records_dfllm_input_columnsllm_input_prompts_dfcomplexity_metrics_columnsvisualizations_columnstoken_usage_columnsr\ llm_outputs_dfsession_analysis_dfs r _create_session_analysis_df1MlflowCallbackHandler._create_session_analysis_df s
_"$,,t||
x ,
%
K( #"5
s ;E3E3c 2 [ U R S 5 $ )Nr )boolr r s r _contain_llm_records*MlflowCallbackHandler._contain_llm_records s DLL!7899r langchain_assetc [ 5 nU R R SUR U R S 5 5 U R 5 ( a U R
5 nUR S5 nUR SSSS9nU R R SUR U5 5 U R R SR UR 5 5 S5 U( a S[ [ U5 5 ; a U R R U5 OV[ [ U R R" S 5 5 n UR% U5 U R R' U5 U( a+ U R R7 5 U R9 5 g g ! [( a UR+ U5 U R R' U5 Nl! [, a$ [/ S
5 [0 R2 " 5 N[4 a$ [/ S
5 [0 R2 " 5 Nf = f[4 a# [/ S
5 [0 R2 " 5 Nf = f) Nr rv r\ rj T)regexsession_analysisz langchain.chains.llm.LLMChainz
model.jsonzCould not save model.) r r r rw r r r r r] r r tolistr typer r r r saver
ValueError
save_agentAttributeErrorprint traceback print_excNotImplementedErrorr r )r r rF r} r rv langchain_asset_paths r
flush_tracker#MlflowCallbackHandler.flush_tracker s
_
)2<< EU8V+WX $ $ & &"&"B"B"D +// Q1R S JJOOBGGI$4$4$6 7 E /#d?6K2LL
--o>'*4
0B0BL+Q'R$#(()=>JJ''(<=" JJ ! ! # KKM ! "
'223GH
++,@A) 56!++- . 56!++- + 12 '')s< ,F2 2
I4=,G++*II4)II4I,I43I4)
r rp r r r r rB r rk r r rC rh )zlangchainrun-%r NNNrj r )NF))r r r r r r r r r r r r r r r r
BaseExceptionr r r
r, r2 r6 r< r@ rC r rH r rN rU r r ra rf r r r r r
__classcell__)r s @r r r s /$/ #&* $ E
smE
SME
tn E
sm E
E
E
E
E
N! R sCx.R379RHKR
R0@c @S @T @",U9 ,U ,U ,U\$- $3 $4 $
D sCx.D26sCx.DLOD
D:: T#s(^S$s) ;<:HK:
:4$M $S $T $
< sCx.<58k >S >S >,FcNF F F
F0"BH%"B "B
"BH$
$ $ $
;#S ;#z:d :&S & &RV & &r r )NN)0loggingrq r r r r copyr pathlibr typingr r r r r r
langchain_core.agentsr r langchain_core.callbacksr
langchain_core.documentsr langchain_core.outputsr langchain_core.utilsr r #langchain_community.callbacks.utilsr r r r r r getLoggerr r r r r/ r@ r rX r` rb r r r r r s
= = : 8 - , C
8 $"s "
c *T#Y 0 '
' ' '
'Tc s s ,iV iVXH 79L Hr