Ch:SSKJr SSKrSSKrSSKrSSKJrJr SSKr SSK r SSK r SSK J r Jr SSKJr SSKJr \(a SSK Jr SSKJr \R,"\5r"S S \5rg) ) annotationsN) TYPE_CHECKINGCallable)average_precision_score ndcg_score)SentenceEvaluator)cos_sim)Tensor)SentenceTransformerc ^\rSrSrSrSSS\SSSSS4 SU4S jjjrSSS jjrSS jrSS jr SS jr SSSjjr Sr Sr U=r$)RerankingEvaluatorat This class evaluates a SentenceTransformer model for the task of re-ranking. Given a query and a list of documents, it computes the score [query, doc_i] for all possible documents and sorts them in decreasing order. Then, MRR@10, NDCG@10 and MAP is compute to measure the quality of the ranking. Args: samples (list): A list of dictionaries, where each dictionary represents a sample and has the following keys: - 'query': The search query. - 'positive': A list of positive (relevant) documents. - 'negative': A list of negative (irrelevant) documents. at_k (int, optional): Only consider the top k most similar documents to each query for the evaluation. Defaults to 10. name (str, optional): Name of the evaluator. Defaults to "". write_csv (bool, optional): Write results to CSV file. Defaults to True. similarity_fct (Callable[[torch.Tensor, torch.Tensor], torch.Tensor], optional): Similarity function between sentence embeddings. By default, cosine similarity. Defaults to cos_sim. batch_size (int, optional): Batch size to compute sentence embeddings. Defaults to 64. show_progress_bar (bool, optional): Show progress bar when computing embeddings. Defaults to False. use_batched_encoding (bool, optional): Whether or not to encode queries and documents in batches for greater speed, or 1-by-1 to save memory. Defaults to True. truncate_dim (Optional[int], optional): The dimension to truncate sentence embeddings to. `None` uses the model's current truncation dimension. Defaults to None. mrr_at_k (Optional[int], optional): Deprecated parameter. Please use `at_k` instead. Defaults to None. Example: :: from sentence_transformers import SentenceTransformer from sentence_transformers.evaluation import RerankingEvaluator from datasets import load_dataset # Load a model model = SentenceTransformer("all-MiniLM-L6-v2") # Load a dataset with queries, positives, and negatives eval_dataset = load_dataset("microsoft/ms_marco", "v1.1", split="validation") samples = [ { "query": sample["query"], "positive": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if is_selected], "negative": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if not is_selected], } for sample in eval_dataset ] # Initialize the evaluator reranking_evaluator = RerankingEvaluator( samples=samples, name="ms-marco-dev", ) results = reranking_evaluator(model) ''' RerankingEvaluator: Evaluating the model on the ms-marco-dev dataset: Queries: 9706 Positives: Min 1.0, Mean 1.1, Max 5.0 Negatives: Min 1.0, Mean 7.1, Max 9.0 MAP: 56.07 MRR@10: 56.70 NDCG@10: 67.08 ''' print(reranking_evaluator.primary_metric) # => ms-marco-dev_ndcg@10 print(results[reranking_evaluator.primary_metric]) # => 0.6708042171399308 T@FNc >[T U]5 XlX0lU b [R SU S35 XlOX lXPlX`lXpl Xl Xl [UR[5(a([URR55UlURV s/sH-n [!U S5S:dM[!U S5S:dM+U PM/ sn UlSU(aSU-OS-S UR S 3-UlS S S SUR 3SUR 3/UlX@lSUR 3Ulgs sn f)Nz?The `mrr_at_k` parameter has been deprecated; please use `at_k=z ` instead.positivernegativer _rz _results_@z.csvepochstepsMAPMRR@NDCG@ndcg@)super__init__samplesnameloggerwarningat_ksimilarity_fct batch_sizeshow_progress_baruse_batched_encoding truncate_dim isinstancedictlistvalueslencsv_file csv_headers write_csvprimary_metric) selfrr"rr/r#r$r%r&r'mrr_at_ksample __class__s k/var/www/html/shao/venv/lib/python3.13/site-packages/sentence_transformers/evaluation/RerankingEvaluator.pyrRerankingEvaluator.__init__YsO     NN\]e\ffpq r II,$!2$8!( dllD ) ) 3 3 56DL"& !-vVJ5G1H11LFQTU[\fUgQhklQlF  -dd KPZ[_[d[dZeeiNjj   499+ DII;   # %dii[1 s8EE%Ec`US:waUS:XaSU3nO SUSUS3nOSnURbUSURS 3- n[RS URS US 35 UR U5nUS nUSnUSn UR V s/sHn [ U S5PM n n UR V s/sHn [ U S5PM n n [RS[ UR 5S[R"U 5SS[R"U 5SS[R"U 5SS[R"U 5SS[R"U 5SS[R"U 5S35 [RSUS-S35 [RSURSUS-S35 [RSURSU S-S35 UbUR(a[RRX R 5n [RR#U 5n[%U SU(aSOSS S!9n[&R("U5nU(dUR+UR,5 UR+X4XxU /5 SSS5 S US"UR3US#UR3U 0nUR/UUR5nUR1UUX45 U$s sn fs sn f!,(df  Nk=f)$a Evaluates the model on the dataset and returns the evaluation metrics. Args: model (SentenceTransformer): The SentenceTransformer model to evaluate. output_path (str, optional): The output path to write the results. Defaults to None. epoch (int, optional): The current epoch number. Defaults to -1. steps (int, optional): The current step number. Defaults to -1. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. z after epoch z in epoch z after z stepsrNz (truncated to )z0RerankingEvaluator: Evaluating the model on the z dataset:mapmrrndcgrrz Queries: z Positives: Min z.1fz, Mean z, Max z Negatives: Min zMAP: dz.2frz: rawzutf-8)newlinemodeencodingzmrr@r)r'r inforcompute_metricesrr,npminmeanmaxr"r/ospathjoinr-isfileopencsvwriterwriterowr.prefix_name_to_metrics store_metrics_in_model_card_data)r1model output_pathrrout_txtscoresmean_apmean_mrr mean_ndcgr3 num_positives num_negativescsv_pathoutput_file_existsfrPmetricss r5__call__RerankingEvaluator.__call__s B;{)%1&ugWUG6BG    ( ):):(;1= =G FtyykQYZaYbbcde&&u--%=6N @D||L|VVJ/0| L?C||L|VVJ/0| L DLL)**=bff]>STW=XX_`b`g`ghu`vwz_{|BCECICIJWCXY\B]]pqsqwqwxEqFGJpKKRSUSZSZ[hSijmRnntuwu{u{|IuJKNtO P   eGcM#./0 d499+R3s';<= eDII;bS(=>?  "t~~ww||K?H!#!9 h8JPS^efjkA)OOD$4$45w) LM g 7499+ DII;   --gtyyA --eWeK9MLgfsL:L-A L L-chUR(aURU5$URU5$)z Computes the evaluation metrics for the given model. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. )r&compute_metrices_batchedcompute_metrices_individual)r1rTs r5rE#RerankingEvaluator.compute_metricess7((  ) )% 0 11%8 c v/n/n/nURUURVs/sHoUSPM snSURS9n/nURH+nURUS5 URUS5 M- URXSURS9nSupURGHBn Xin U S- n [ U S5n [ U S5nXX-U-nXU-- n U S:XdUS:XaMGUR X5n[ UR 5S:aUSn[R"U*5nUR5R5nS/U -S/U--nSn[USUR5HunnUU(dMSUS-- n O URU5 UR[U/U/URS 95 UR[UU55 GME [ R""U5n[ R""U5n[ R""U5nUUUS .$s snf) a Computes the evaluation metrics in a batched way, by batching all queries and all documents together. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. queryencode_fn_namer%rrdocument)rrrkr;r<r=) embed_inputsrr%extendr,r#shapetorchargsortcputolist enumerater"appendrrrFrH)r1rTall_mrr_scoresall_ndcg_scores all_ap_scoresr3all_query_embsall_docs all_docs_embs query_idxdocs_idxinstance query_embnum_posnum_negdocs_emb pred_scorespred_scores_argsort is_relevant mrr_scorerankindexrXrYrZs r5rd+RerankingEvaluator.compute_metrices_batchedsT ** +/<< 8<G_< 8""44 + llF OOF:. / OOF:. /#)) J$J`J`* #  H&1I NI(:./G(:./G$0BW0LMH ') )H!|w!|--iBK;$$%))!n "'-- "= %//+224K#-1#-7KI()*GGO, xCCo 9s H6 c /n/n/n[R"URUR(+SS9GHxnUSn[US5n[US5n[ U5S:Xd[ U5S:XaMEXx-n S/[ U5-S/[ U5--n UR X/SSS 9n UR XS SS 9n UR X5n [ U R5S:aU Sn [R"U *5nU R5R5n Sn[USUR5HunnU U(dMSUS-- n O URU5 UR[U /U /URS 95 UR[!X55 GM{ ["R$"U5n["R$"U5n["R$"U5nUUUS .$) a Computes the evaluation metrics individually by embedding every (query, positive, negative) tuple individually. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. Samples)disabledescrirrrrmFrjrlrnrp)tqdmrr%r*r,rqr#rsrtrurvrwrxr"ryrrrFrH)r1rTrzr{r|rrirrdocsrrrrrrrrrXrYrZs r5re.RerankingEvaluator.compute_metrices_individual!s  $,,D*GGO, xCCrgc Uc URnO%US:Xa URnOUS:Xa URnW"U4URUSURS.UD6$)NrirlT)r$r%convert_to_tensorr')encode encode_queryencode_documentr$r')r1rT sentencesrkr%kwargs encode_fns r5rqRerankingEvaluator.embed_inputsXsn  ! I w &**I z )--I  /"**     rgcZSUR0nURbURUS'U$)Nr"r')r"r')r1 config_dicts r5get_config_dict"RerankingEvaluator.get_config_dictos2tyy)    (*.*;*;K 'rg) r"r$r-r.rr0rr%r#r'r&r/)rz list[dict[str, str | list[str]]]r"intrstrr/boolr#z4Callable[[torch.Tensor, torch.Tensor], torch.Tensor]r$rr%rr&rr' int | Noner2r)Nr8r8) rTr rU str | Nonerrrrreturnzdict[str, float])rTr )NN) rTr rzstr | list[str] | np.ndarrayrkrr%z bool | Nonerr )__name__ __module__ __qualname____firstlineno____doc__r rrarErdrerqr__static_attributes__ __classcell__)r4s@r5r r s$=DOV"'%)#'#.21.2.2 .2  .2 M .2.2 .2#.2!.2.2.2bik=(=7A=QT=be= =~ GDR5Dv&*)-  " 0 #  '    .rgr ) __future__rrOloggingrJtypingrrnumpyrFrtrsklearn.metricsrr2sentence_transformers.evaluation.SentenceEvaluatorrsentence_transformers.utilr r )sentence_transformers.SentenceTransformerr getLoggerrr r rgr5rsP"  * ?P.M   8 $Z*Zrg