tjddlmZddlZddlZddlZddlZddlmZmZddl Z ddl Z ddl m Z ddlm Z ddlmZejeZerddlmZdd lmZdd lmZ d>d?d=ZdS)@) annotationsN) TYPE_CHECKINGLiteral)trange)tqdm)is_datasets_availableDataset) CrossEncoder)SentenceTransformertopFtriplet @Tdatasetr modelr anchor_column_name str | Nonepositive_column_namecorpuslist[str] | None cross_encoderCrossEncoder | None range_minint range_max int | None max_score float | None min_scoreabsolute_marginrelative_margin num_negativessampling_strategyLiteral['random', 'top']query_prompt_name query_promptcorpus_prompt_name corpus_promptinclude_positivesbool output_format=Literal['triplet', 'n-tuple', 'labeled-pair', 'labeled-list'] output_scores batch_sizefaiss_batch_size use_faissuse_multi_processlist[str] | boolverbose cache_folder as_triplets bool | Nonemarginreturnc *5 mnopqrsttstdt|dkrtd|dkrtd|d|r( ddl}n"#t$r}td|d}~wwxYwddlm}|j} |r|| vr| d}|r|| vr| d }|s$|s"t| d krtd |$|rd nd }t d|d|rN|dks | ||| dkrt d|d krt dd }|dkrt dd }d}|dvrtd|d| |} t d| dt| | |}!t!|!}" d krtd dd}#|x| | | | dzz|"z}n| z|"z}|rG||"zdkr>t#|dddkr d|"z }d |d!}#|rt%d"|d|rt%d#|d$|dkrtd%|d||krtd&|d'|d|# ||z kr!td( d)||z d*|d+|d,|# i}$t||}%t||tdu}&|&sttt&tzd-t+D}'|%mtt&|%}%d.t+|%D}(t|%})t/j|)d/}*|jn|)tmkr%|r#t%d0|)d1tmd2|"d kr(t7j|!}+t%d3|+d4d5d},d}-|rt;j|d6|jj pd}.tCj"tG|.|||%f$d78%}/tCj"tG|.||f$d78%}0t:j&'|d9|/d:}1t:j&'|d;|0d:}2t:j&(|1r1t7j)|1}-|rt%d<|1d=|-j*d>t:j&(|2r1t7j)|2},|rt%d?|2d=|,j*d>|,|-|r.|+tY|tZrdn|@}3nd}3|,|.|3|ddd||A},|-|/|%|3|ddd||A}-|r|0|3|rt:j&(|1s)t7j1|1|-|rt%dB|1t:j&(|2s)t7j1|2|,|rt%dC|2|rz|2|3}4 |4}5d|5_5d|5_6|7|4|5D}4n#tp$rYnwxYw|49|,g}6g}7tudt|-|dEFD]V}8|-|8|8|z}9|4;|9||"zG\}:o|6<|:|7<oWt/j=t7j>|6dH?n}:t/j=t7j>|7dH?nong}6g}7t/j@|,},t||"zt};tudt|-|dIFD]z}8|B|-|8|8|z|,?n}|>dkrt/jD|:|:F|:Ed|>ftdL gd K}:t/jDooFoEd|>fd/gd KodMt|)DrtmtD]>\}?}@|(|?}A|'|@r|Avr!r|A<|'|@?dNrDpgtgmt|)D]N}BtJfdOr|BDmJ|%|Bgp|BzOnfdPrDr|-pfdQt|)D}-|,t/jDrK}C|L|-|C?ns~-~C~,|>| | |7|r.|+tY|tZrdn|@}3nd}3tt+odRtoSD]z\}B}D|%|B}?fdT|DD}Et|E}F|Ntt|?g|Fz|E|d|3U?n}G|G|:|B<{|Nttmt|d|3U?ns|r|0|3tdL |:od/k<|sAt/jOorfdVt|)D}HtdL |:|H<|:P}F| | t/jQ|)sjsjRW}Id}Jt|)D]5}Kt/jAs|J|Jp|Kz|I|K<|Jp|Kz }J6| |:| z|IS|:Ed d jTk}LtdL |:|L<|LUV}M|Mr|M|M|Fz dX|$dY<|F|Mz}F| |I|IW| zz }N|:|NS|:Ed d jTk}LtdL |:|L<|LUV}M|Mr|M|M|Fz dX|$dZ<|F|Mz}F|L|:|k}LtdL |:|L<|LUV}M|Mr |M|M|Fz dX|$d[<| L|:| k}LtdL |:|L<|LUV}M|Mr |M|M|Fz dX|$d\<t/jC|:|d J\q}Oo|*|Ofo|rodd|dfoqdd|dfq| dkroddd foqddd fqn| d]kroEd qXUd z }P|PY ^}P fd_|PD}Qo|*|Qfoq|*|QfqqZd d`\q}Oo|*|Ofo|rt%da|d krOt/jDopfdbt|)Dot/jDpqfdct|)DqqtdL k}Rt/j[o}St/j[o}To|Roq|Rqd}Ut|)D]x}Kt/j\|KSp|K |S|U|Up|Kz<r|KS d jT|T|U|Up|Kz<|Up|Kz }Uy|S|R}S|T|RrsS d jT|R}V|g|gddgi}W|rg|Wde<t|Sro|VqD]\}X}Y}Z}[}\|W|<|%|X|W|<|Y|Wdd<|Z|rA|Wde<|[V|\Vg|Vqz }]|RP}^n|dfkr|g|gi}W|rg|Wdg<ng|Wdh<d}Jt|)D]l}Ar|AD]}Y|W|<|%|A|W|<|Y|r6s|JV}[|Wdg<|[n|Wdh<d |Jd z }Jto|Aq|AD]\}Z}_|_tdL kr|W|<|%|A|W|<|Z|r.|Wdg<|_V|Wdh<dnt/jDpqfdit|)DqqtdL k}Rq|RqsS d jT|Rqz }]|) zt|z}^n|d krt/jDopfdjt|)Dot/jDpqfdkt|)DqqtdL k]d K}Rq|Rqo|Ro|mfdlt+|RD|tfdmt+|RDifdnt+ojTd oD}W|rFt/jDs|Rd/qgd KK|Wde<q^qsS d jT|R^qz }]|REd}^n0|dpkr)t/jDopfdqt|)Dot/jDpqfdrt|)DqqtdL k}R|mfdst+|RD|tfdtt+t|RoDi}W|r qsfdut+|RD|Wde<ndv|RD|Wdw<q|RqsS d jT|Rqz }]|REd}^|j_|W}`|rtdx}ady}bt%|a`dzd{d|d}t%|a`d~|bts|btqddt.jfdt.jafdt.jbfddfddfddfddfddffD]r\}c}dt%|a`|cc|b|ds|b|dq|b|d|]sdY| fdZ| fd[|fd\| ffD]C\}e}f|e|$vr:|$|ed}g|$|ed}ht%d|gdd|hdd|ed|fd D|^t|`z }i|idkrdg}j|dkr|j<d| |j<dY| |j<dZ||j<d[t|jd kr*d'|jdd/dz|jd/z}kn|jd}k|i|^z }lt%d|id|ldd|kdt|jd krdndd |`S)u+I Add hard negatives to a dataset of (anchor, positive) pairs to create (anchor, positive, negative) triplets or (anchor, positive, negative_1, ..., negative_n) tuples. Hard negative mining is a technique to improve the quality of a dataset by adding hard negatives, which are texts that may appear similar to the anchor, but are not. Using hard negatives can improve the performance of models trained on the dataset. This function uses a SentenceTransformer model to embed the inputs in the dataset, and then finds the closest matches to each anchor input in the dataset. It then samples negatives from the closest matches, optionally using a CrossEncoder model to rescore the candidates. Supports prompt formatting for models that expect specific instruction-style input. You can influence the candidate negative selection in various ways: - **range_min**: Minimum rank of the closest matches to consider as negatives: useful to skip the most similar texts to avoid marking texts as negative that are actually positives. - **range_max**: Maximum rank of the closest matches to consider as negatives: useful to limit the number of candidates to sample negatives from. A lower value makes processing faster, but may result in less candidate negatives that satisfy the margin or max_score conditions. - **max_score**: Maximum score to consider as a negative: useful to skip candidates that are too similar to the anchor. - **min_score**: Minimum score to consider as a negative: useful to skip candidates that are too dissimilar to the anchor. - **absolute_margin**: Absolute margin for hard negative mining: useful to skip candidate negatives whose similarity to the anchor is within a certain margin of the positive pair. A value of 0 can be used to enforce that the negative is always further away from the anchor than the positive. - **relative_margin**: Relative margin for hard negative mining: useful to skip candidate negatives whose similarity to the anchor is within a certain margin of the positive pair. A value of 0.05 keeps a negative only if its similarity to the anchor is at least 5% of the positive score magnitude below the positive. - **sampling_strategy**: Sampling strategy for negatives: "top" or "random". "top" will always sample the top n candidates as negatives, while "random" will sample n negatives randomly from the candidates that satisfy the margin or max_score conditions. .. tip:: The excellent `NV-Retriever paper `_ is a great resource for understanding the details of hard negative mining and how to use it effectively. Notably, it reaches the strongest performance using these settings:: dataset = mine_hard_negatives( dataset=dataset, model=model, relative_margin=0.05, # keep negatives at least 5% of the positive score magnitude below the positive num_negatives=num_negatives, # 10 or less is recommended sampling_strategy="top", # "top" means that we sample the top candidates as negatives batch_size=batch_size, # Adjust as needed use_faiss=True, # Optional: Use faiss/faiss-gpu for faster similarity search ) This corresponds with the `TopK-PercPos (95%)` mining method. Example: >>> from sentence_transformers.util import mine_hard_negatives >>> from sentence_transformers import SentenceTransformer >>> from datasets import load_dataset >>> # Load a Sentence Transformer model >>> model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") >>> >>> # Load a dataset to mine hard negatives from >>> dataset = load_dataset("sentence-transformers/natural-questions", split="train") >>> dataset Dataset({ features: ['query', 'answer'], num_rows: 100231 }) >>> dataset = mine_hard_negatives( ... dataset=dataset, ... model=model, ... range_min=10, ... range_max=50, ... max_score=0.8, ... relative_margin=0.05, ... num_negatives=5, ... sampling_strategy="random", ... batch_size=128, ... use_faiss=True, ... ) Batches: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 588/588 [00:32<00:00, 18.07it/s] Batches: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 784/784 [00:08<00:00, 96.41it/s] Querying FAISS index: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:06<00:00, 1.06it/s] Negative candidates mined, preparing dataset... Metric Positive Negative Difference Count 100,231 487,865 Mean 0.6866 0.4194 0.2752 Median 0.7010 0.4102 0.2760 Std 0.1125 0.0719 0.1136 Min 0.0303 0.1702 0.0209 25% 0.6221 0.3672 0.1899 50% 0.7010 0.4102 0.2760 75% 0.7667 0.4647 0.3590 Max 0.9584 0.7621 0.7073 Skipped 427,503 potential negatives (8.36%) due to the relative_margin of 0.05. Skipped 978 potential negatives (0.02%) due to the max_score of 0.8. Could not find enough negatives for 13290 samples (2.65%). Consider adjusting the range_max, range_min, relative_margin and max_score parameters if you'd like to find more valid negatives. >>> dataset Dataset({ features: ['query', 'answer', 'negative'], num_rows: 487865 }) >>> dataset[0] { 'query': 'when did richmond last play in a preliminary final', 'answer': "Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved since 1995. A series of close losses hampered the Tigers throughout the middle of the season, including a 5-point loss to the Western Bulldogs, 2-point loss to Fremantle, and a 3-point loss to the Giants. Richmond ended the season strongly with convincing victories over Fremantle and St Kilda in the final two rounds, elevating the club to 3rd on the ladder. Richmond's first final of the season against the Cats at the MCG attracted a record qualifying final crowd of 95,028; the Tigers won by 51 points. Having advanced to the first preliminary finals for the first time since 2001, Richmond defeated Greater Western Sydney by 36 points in front of a crowd of 94,258 to progress to the Grand Final against Adelaide, their first Grand Final appearance since 1982. The attendance was 100,021, the largest crowd to a grand final since 1986. The Crows led at quarter time and led by as many as 13, but the Tigers took over the game as it progressed and scored seven straight goals at one point. They eventually would win by 48 points – 16.12 (108) to Adelaide's 8.12 (60) – to end their 37-year flag drought.[22] Dustin Martin also became the first player to win a Premiership medal, the Brownlow Medal and the Norm Smith Medal in the same season, while Damien Hardwick was named AFL Coaches Association Coach of the Year. Richmond's jump from 13th to premiers also marked the biggest jump from one AFL season to the next.", 'negative': "2018 NRL Grand Final The 2018 NRL Grand Final was the conclusive and premiership-deciding game of the 2018 National Rugby League season and was played on Sunday September 30 at Sydney's ANZ Stadium.[1] The match was contested between minor premiers the Sydney Roosters and defending premiers the Melbourne Storm. In front of a crowd of 82,688, Sydney won the match 21–6 to claim their 14th premiership title and their first since 2013. Roosters five-eighth Luke Keary was awarded the Clive Churchill Medal as the game's official man of the match." } >>> # To include similarity scores, use output_scores=True >>> dataset_with_scores = mine_hard_negatives( ... dataset=dataset, ... model=model, ... output_scores=True, ... # ... other parameters ... ) >>> dataset_with_scores Dataset({ features: ['query', 'answer', 'negative', 'scores'], num_rows: 487865 }) >>> dataset.push_to_hub("natural-questions-hard-negatives", "triplet-all") Args: dataset (Dataset): A dataset containing (anchor, positive) pairs. model (SentenceTransformer): A SentenceTransformer model to use for embedding the inputs. anchor_column_name (str, optional): The column name in `dataset` that contains the anchor/query. Defaults to None, in which case the first column in `dataset` will be used. positive_column_name (str, optional): The column name in `dataset` that contains the positive candidates. Defaults to None, in which case the second column in `dataset` will be used. corpus (List[str], optional): A list containing documents as strings that will be used as candidate negatives in addition to the second column in `dataset`. Defaults to None, in which case the second column in `dataset` will exclusively be used as the negative candidate corpus. cross_encoder (CrossEncoder, optional): A CrossEncoder model to use for rescoring the candidates. Defaults to None. range_min (int): Minimum rank of the closest matches to consider as negatives. Must be non-negative. Defaults to 0. range_max (int, optional): Maximum rank of the closest matches to consider as negatives. Must leave at least `num_negatives` candidates after `range_min`. Defaults to None. max_score (float, optional): Maximum score to consider as a negative. Defaults to None. min_score (float, optional): Minimum score to consider as a negative. Defaults to None. absolute_margin (float, optional): Absolute margin for hard negative mining, i.e. the minimum distance between the positive similarity and the negative similarity. Defaults to None. relative_margin (float, optional): Relative margin for hard negative mining, i.e. the minimum distance between the positive similarity and the negative similarity as a fraction of the positive score magnitude. A value of 0.05 discards negatives whose similarity is greater than ``positive_score - abs(positive_score) * 0.05``. Defaults to None. num_negatives (int): Number of negatives to sample. Must be at least 1, and must fit in the candidate window left by `range_min` and `range_max`. Defaults to 3. sampling_strategy (Literal["random", "top"]): Sampling strategy for negatives: "top" or "random". Defaults to "top". query_prompt_name (Optional[str], optional): The name of a predefined prompt to use when encoding the first/anchor dataset column. It must match a key in the ``model.prompts`` dictionary, which can be set during model initialization or loaded from the model configuration. For example, if ``query_prompt_name="query"`` and the model prompts dictionary includes {"query": "query: "}, then the input "What is the capital of France?" is transformed into: "query: What is the capital of France?" before encoding. This is useful for models that were trained or fine-tuned with specific prompt formats. Ignored if ``query_prompt`` is provided. Defaults to None. query_prompt (Optional[str], optional): A raw prompt string to prepend directly to the first/anchor dataset column during encoding. For instance, `query_prompt="query: "` transforms the input "What is the capital of France?" into: "query: What is the capital of France?". Use this to override the prompt logic entirely and supply your own prefix. This takes precedence over ``query_prompt_name``. Defaults to None. corpus_prompt_name (Optional[str], optional): The name of a predefined prompt to use when encoding the corpus. See ``query_prompt_name`` for more information. Defaults to None. corpus_prompt (Optional[str], optional): A raw prompt string to prepend directly to the corpus during encoding. See ``query_prompt`` for more information. Defaults to None. include_positives (bool): Whether to include the positives in the negative candidates. Setting this to True is primarily useful for creating Reranking evaluation datasets for CrossEncoder models, where it can be useful to get a full ranking (including the positives) from a first-stage retrieval model. Defaults to False. output_format (Literal["triplet", "n-tuple", "labeled-pair", "labeled-list"]): Output format for the `datasets.Dataset`. When ``output_scores=False`` (default), options are: - "triplet": (anchor, positive, negative) triplets, i.e. 3 columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.CachedMultipleNegativesRankingLoss`. - "n-tuple": (anchor, positive, negative_1, ..., negative_n) tuples, i.e. 2 + num_negatives columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.CachedMultipleNegativesRankingLoss`. - "labeled-pair": (anchor, document, label) tuples with a label of 0 for negative and 1 for positive, i.e. 3 columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.BinaryCrossEntropyLoss`. - "labeled-list": (anchor, [doc1, doc2, ..., docN], [label1, label2, ..., labelN]) tuples with labels of 0 for negative and 1 for positive, i.e. 3 columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.LambdaLoss`. Defaults to "triplet". See ``output_scores`` for the output formats when ``output_scores=True``. output_scores (bool): Whether to include similarity scores in the output dataset. When True, adds score fields to the output: - For "triplet" format: adds `scores` column with query-positive and query-negative similarity scores, for 4 columns total. - For "n-tuple" format: adds `scores` column with a list of similarity scores for the query-positive and each of the query-negative pairs, for 3 + num_negatives columns total. Useful for e.g. :class:`~sentence_transformers.sparse_encoder.losses.SparseMarginMSELoss`. - For "labeled-pair" format: replaces the `label` column with a `score` column. Labels are binary (1 for positive, 0 for negative), but scores contain the actual similarity scores computed by the model or cross_encoder. The output has 3 columns. - For "labeled-list" format: replaces the `labels` column with a `scores` column. Labels are binary (1 for positive, 0 for negative), but scores contain the actual similarity scores computed by the model or cross_encoder. The output has 3 columns. Defaults to False. batch_size (int): Batch size for encoding the dataset. Defaults to 32. faiss_batch_size (int): Batch size of queries for the top-k similarity search, both with FAISS and without (``use_faiss=False``), where it bounds the size of the intermediate similarity matrix to ``faiss_batch_size * len(corpus)``. Defaults to 16384. use_faiss (bool): Whether to use FAISS for similarity search. May be recommended for large datasets. Requires the ``faiss-cpu`` or ``faiss-gpu`` package. On GPU, FAISS can retrieve at most 2048 documents per query, so an automatically derived ``range_max`` is capped to fit. Defaults to False. use_multi_process (bool | List[str], optional): Whether to use multi-GPU/CPU processing. If True, uses all GPUs if CUDA is available, and 4 CPU processes if it's not available. You can also pass a list of PyTorch devices like ["cuda:0", "cuda:1", ...] or ["cpu", "cpu", "cpu", "cpu"]. verbose (bool): Whether to print statistics and logging. Defaults to True. cache_folder (str, optional): Directory path for caching embeddings. If provided, the function will save ``query_embeddings_{hash}.npy`` and ``corpus_embeddings_{hash}.npy`` under this folder after the first run, and on subsequent calls will load from these files if they exist to avoid recomputation. The hashes are computed based on the model name, the query/corpus prompts, and the queries/corpus, so changing the prompt arguments invalidates the cache rather than silently returning embeddings from a different prompt. Defaults to None. as_triplets (bool, optional): Deprecated. Use `output_format` instead. Defaults to None. margin (float, optional): Deprecated. Use `absolute_margin` or `relative_margin` instead. Defaults to None. Returns: Dataset: A dataset containing the specified output format. If `output_scores=False` (default), the formats are: - "triplet": (anchor, positive, negative) - "n-tuple": (anchor, positive, negative_1, ..., negative_n) - "labeled-pair": (anchor, document, label) - "labeled-list": (anchor, [documents], [labels]) And if `output_scores=True`, the formats are: - "triplet": (anchor, positive, negative, [scores]) - "n-tuple": (anchor, positive, negative_1, ..., negative_n, [scores]) - "labeled-pair": (anchor, document, score) - "labeled-list": (anchor, [documents], [scores]) zGPlease install `datasets` to use this function: `pip install datasets`.rz9The dataset is empty. Please provide a non-empty dataset.z1faiss_batch_size must be a positive integer, got .NzlPlease install `faiss` to use this function with `use_faiss=True`: `pip install faiss-cpu` (or `faiss-gpu`).r rz)Dataset must contain exactly two columns.rn-tuplezrThe `as_triplets` parameter is deprecated. Use the `output_format` parameter instead. Setting `output_format` to `z`.rzWhen using `include_positives=True`, updating `range_min`, `range_max`, `max_score`, `margin`, or `sampling_strategy` from the default values may still discard the positive values.z~When using `include_positives=True`, `output_format` will be set to `"n-tuple"` to ensure that the ranking order is preserved.zn-tuple-scoresz`output_format="n-tuple-scores"` is deprecated. Use `output_format="n-tuple"` with `output_scores=True` instead for equivalent output formats.T)rr? labeled-pair labeled-listzInvalid output_format: zI. Must be one of 'triplet', 'n-tuple', 'labeled-pair', or 'labeled-list'.zThe `margin` parameter is deprecated. Use the `absolute_margin` and/or `relative_margin` parameter instead. Setting `absolute_margin` to `z4num_negatives must be at least 1, got num_negatives= i get_num_gpuscdS)NrrFg/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/sentence_transformers/util/hard_negatives.pyz%mine_hard_negatives..tsarGz# Note that range_max was capped to z because FAISS on GPU can only retrieve up to 2048 documents per query, some of which are reserved for the positives. Passing use_faiss=False lifts that cap, although it can cost a lot more memory.zUsing FAISS on GPU, we can only retrieve up to 2048 documents per query, some of which are reserved for the positives. Setting range_max to zSetting range_max to z" based on the provided parameters.z.range_min must be non-negative, got range_min=z8range_min must be smaller than range_max, got range_min=z and range_max=z Cannot mine z negatives per query: only z6 candidates remain after skipping the first range_min=z of range_max=zJ. Consider decreasing num_negatives or range_min, or increasing range_max.ci|]\}}|| SrFrF).0idxtexts rH z'mine_hard_negatives..s??? T$???rGci|]\}}|| SrFrF)rKrLquerys rHrNz'mine_hard_negatives..sCCC*#u5#CCCrGzFound z unique queries out of z total queries.zFound an average of z.3fz positives per query.)exist_okF)usedforsecurityquery_embeddings_z.npycorpus_embeddings_z%[Cache] Loaded query embeddings from z (shape=)z&[Cache] Loaded corpus embeddings from )target_devices)poolr1normalize_embeddingsconvert_to_numpyshow_progress_bar prompt_namepromptz"[Cache] Saved query embeddings to z#[Cache] Saved corpus embeddings to )cozQuerying FAISS index)desck)axiszComputing similarity scores)radim)rcinfcg|]}gSrFrF)rK_s rH z'mine_hard_negatives../s555q555rGc,g|]}t|SrF)len)rKps rHrgz'mine_hard_negatives..6s444a3q66444rGc g|] }| SrFrF)rKdoc_idxrs rHrgz'mine_hard_negatives..<sOOOg&/OOOrGc<g|]}tj|S))device)torchtensor)rKrjrns rHrgz'mine_hard_negatives..?s(QQQ1 Qv666QQQrGcDg|]}t|D]}|SrF)range)rKrLrf n_positivess rHrgz'mine_hard_negatives..Bs6(j(j(jRWXcdgXhRiRi(j(jQ(j(j(j(jrGzRescoring with CrossEncoder)r_totalc g|] }| SrFrF)rK_idxrs rHrgz'mine_hard_negatives..Vs!I!I!I4&,!I!I!IrG)r1convert_to_tensorrXcRg|]#}tj||$SrF)roisin)rKq_idxindicespositive_indicess rHrgz'mine_hard_negatives..ps/ _ _ _UUZ(8(? @ @ _ _ _rG)rndtype)skippedratior#r$r r"random)mincVg|]%}tjt|&S)r`)rsamplerr)rKoptionsr%s rHrgz'mine_hard_negatives..s.aaa'v}U7^^}EEEaaarG)rc descendingz/Negative candidates mined, preparing dataset...cTg|]$}||d%SrrepeatrKrLr{rss rHrgz'mine_hard_negatives..2bbb#WS\00S1A1EEbbbrGcTg|]$}||d%SrrrKrLrsnegative_scoress rHrgz'mine_hard_negatives..5$r$r$rZ]_S%9%@%@SAQST%U%U$r$r$rrGnegativescoresr@scorelabelcTg|]$}||d%Srrrs rHrgz'mine_hard_negatives..%rrGcTg|]$}||d%Srrrs rHrgz'mine_hard_negatives..-rrGcTg|]$}||d%Srrrs rHrgz'mine_hard_negatives...rrGc*g|]\}}||SrFrF)rKrLkeep all_queriess rHrgz'mine_hard_negatives..7s( f f fic4ae fS!1 f f frGc*g|]\}}||SrFrF)rKrLr positivess rHrgz'mine_hard_negatives..8s'"f"f"fic4ae"f9S>"f"f"frGc:i|]\}}d|fd|DS) negative_c g|] }| SrFrF)rKneg_idxrs rHrgz2mine_hard_negatives...:s!M!M!Mg&/!M!M!MrGrF)rKi neg_indicesrs rHrNz'mine_hard_negatives..9sJ"A{ A!M!M!M!M!M!M!MrG)startrAcTg|]$}||d%Srrrs rHrgz'mine_hard_negatives..IrrGcTg|]$}||d%Srrrs rHrgz'mine_hard_negatives..JrrGcNg|]!\}}||"SrF)any)rKrLkeep_rowrs rHrgz'mine_hard_negatives..Os4 t t tmc8emeqeqeses tS!1 t t trGcg|]B\}\}}||gfdt||DzCS)c*g|]\}}||SrFrF)rKrindexrs rHrgz2mine_hard_negatives...Qs'#h#h#hkdEcg#hF5M#h#h#hrG)rzip)rKrLr indices_rowrrs rHrgz'mine_hard_negatives..Psl###0C0(K<<>>#3 #h#h#h#hSS^E_E_#h#h#hh###rGcg|]U\}}||g||zVSrF)ritemtolist)rKrLrrpositive_scoress rHrgz'mine_hard_negatives..Xsj&&&!C<<>>& %**,,-0DX0N0U0U0W0WW&&&rGcdg|]-}|dgdgt|zz.S)rr)rsum)rKrs rHrgz'mine_hard_negatives..^s>%r%r%rHckcococqcq%rqcQC#h--,?&?%r%r%rrGlabelsz{:<6} {:>14} {:>14} {:>14}cjt|tjr|dn|dS)Nz.4f,) isinstanceroTensorr)values rHrIz%mine_hard_negatives..is1:eU\;Z;Z"lUZZ\\"7"7"7ch`l`lrGMetricPositiveNegative DifferenceCountmeanmedianstdrcx|dkrtj|ntdS)Nrrd)numelrorfloatrs rHrIz%mine_hard_negatives..ws1 8J8J59V#4#4#4PUV[P\P\rGz25%c|dkr(tj|dntdS)Nrg?qrdrroquantilerrs rHrIz%mine_hard_negatives..x?V\\^^^_M_M_5>&,,..D#I#I#I#IejkpeqeqrGz50%c|dkr(tj|dntdS)Nrg?rrdrrs rHrIz%mine_hard_negatives..ys?FLLNN]^L^L^5>&,,..C#H#H#H#HdijodpdprGz75%c|dkr(tj|dntdS)Nrg?rrdrrs rHrIz%mine_hard_negatives..zrrGmaxcx|dkrtj|ntdS)Nrz-inf)rrorrrs rHrIz%mine_hard_negatives..{s1 8J8J59V#4#4#4PUV\P]P]rGr~rzSkipped rz potential negatives (z.2%z ) due to the z of rrz, z and z$Could not find enough negatives for z samples (z). Consider adjusting the z parametersz, if you'd like to find more valid negatives.)dr ImportErrorri ValueErrorfaissdatasetsr column_namesloggerwarninglist to_pandasgroupbycountto_dictvaluesrgetattrprintdictfromkeys enumeratecopyroarange unsqueezernnprosmakedirsmodel_card_data base_modelhashlibsha256reprencode hexdigestpathjoinexistsloadshapestart_multi_process_poolrr-encode_document encode_querystop_multi_process_poolsave IndexFlatIPget_embedding_dimensionGpuMultipleClonerOptionsshard useFloat16index_cpu_to_all_gpus Exceptionaddrsearchappend from_numpy concatenateto as_tensorr similaritytopkcatsizenew_fullrrrrextendrsimilarity_pairwiserpredictstackremptyr}rTrrabsisinfclampsort empty_likerpallflatten from_dictformatrr capitalize)urrrrrrrrr r"r#r$r%r&r(r)r*r+r,r.r0r1r2r3r4r6r7r8r:rexcr columnspositives_per_query max_positivesfaiss_cap_note log_countersqueriesseparate_corpus corpus_idx queries_idx n_queries batch_idxavg_positives_per_querycorpus_embeddingsquery_embeddings model_name query_hash corpus_hashquery_cache_filecorpus_cache_filerXrr^ scores_list indices_listr query_chunkrra chunk_scores chunk_indicesmissingrPpositive query_idxrLpositive_embeddings candidate_idxcandidate_passagesnum_candidates pred_scores positive_maskmax_positive_scorespositive_score_idxrzremoved_indices num_skippedrelative_threshold local_indices num_options sampled_idxindices_to_keepanchor_indices pos_indices start_idxpositive_triplet_scores dataset_data anchor_idx positive_idx negative_idx pos_score neg_scoredifference_scoresmaximum_possible_samplesnegative_scoreoutput_dataset row_format formattermetricfunction param_name param_valuer~rmissing_samples solutionsconsiderationsmissing_samples_ratiorrnr{rsrr|rrsu ` ` @@@@@@@@rHmine_hard_negativesr_s p ! " "ecddd 7||qTUUU1`M]```aaa  LLLL   <   !     "G (!37!B!B$QZ *#7w#F#F&qz F&:Fs7||q?P?PDEEE%0? i  =+8 = = =   & NN$$! E)) NNe    I % % NNQ   &M((( J   "  RRR ~m ~ ~ ~     A-< A A A   ##$677==??GGIIJ^_ffhh+,,Mq`P]```aaaN  &/*EI^!]R%78=HII"M1MAI  ]2T999wunii88::Q>> =0 O)OOO ]PY]]]  Y W)WWW X X X1}}V)VVVWWWI T& T T7@ T TCQ T T   y9,,, F= F FYQZEZ F F,5 F FEN F F5C F F   L7-.//GW1233ID(O $-- 233 4 4F@?Yv->->???J,,..K4==))**GCC '0B0BCCCKG I Y''11"55I \FC $$$$$ ZyZZ[9I9IZZZ[[[q"$'*=">"> W%<WWWWXXXv L40000*5; ^ *l,=wG H H O O Q Q!    )++ n *m-?H I I P P R R!    )++  7<< 6Z*6Z6Z6Z[[GLL7]K7]7]7]^^ 7>>* + + s!w'788  sq>NqqXhXnqqqrrr 7>>+ , , v "(9 : :  vt?PttZkZqtttuuu $4$<  11'12CT'J'JattPa2DDD  $ % 5 5%%)!%"&.$!6 ! !   #$11%%)!%"&-# 2     0  ) )$ / / / Qw~~.// O G$&6 7 7 7 OM;KMMNNNw~~/00 Q G%'8 9 9 9 QO,Q#G#G#GHHKKFSS  !O,=>>  M)3v;; 7 73/002BIfggg / /A ++, Q;;YQ8QTYZ_T`T`S`(a(abhijjjFi'*:*:GLLOOW;UWY*Z*Z [abcccG 65E)$4$4555{I66EEx& h '7 'B B B Y ' . .z(/C D D D44#3444KIKY>>OOOO9I#9NOOOPPPGCL>K,<<====QQQQ@PQQQ((j(j(j(ji8H8H(j(j(jk+EI6F,G,G,N,N,P,PQ//0@BUVVYYZ`aaO #'BiF[   99'12CT'J'JattPa:DDD"&y'9'9@]ehipeqeq"r"r"r & & CCLE!I!I!I!I=!I!I!I  !344N'//S%>13EFFGG%"& 0 bjj  &F3KK'// [),, - -!" 0  "V**   8  1 1$ 7 7 7 #5\\MF7b= . _ _ _ _ _eT]N^N^ _ _ _  "'u }\\^^N"o&A$k)OIT_bpTp2q2q ./+-  &"57J7N7N7P7PSb7b!b $'9'@'@QQR'S'S'UUO',U||mF? #)--//4466K .>IT_bpTp2q2q ./+- 9,#(<<-%))++0022  &$~5))L %  9,#(<<-%))++0022  &$~5))L % &+Z)%K%K%K"O]i./G9!!!YZZ-()!!!YZZ-8E!!!!!^m^+,)!!!^m^*;< h & &ll1oo(=(=(?(?(C(CA(F(FF !''M':: aaaaU`aaa )[01))[*@A)8)=)=!PT)=)U)U&)]23A ?@@@ !!)bbbbbQVW`QaQabbbcc)$r$r$r$r$rafgpaqaq$r$r$rss*eEll]:)'22&w// /*)/: 9%% , ,EINV[I\I\IcIcE"MJJN9y;u3E'EE F!'..}a@@B  I E0B$BB C U+ +II'8&7#2"8"8"J"J"L_"]  "   (%'L "LO ,g7NP_M M  T T HJ lIy + , 3 3GJ4G H H H - . 5 5f\6J K K K  $ + +F<,@ A A A TX&--y~~/?/?AQAQ.RSSS3oE#2#8#8#:#:  . ( (  "   '$&L ! !$&L !y)) 4 4I 0 ; ( ( /077 8JKKK1299&:NOOO 4 /0B C H H J JI )00;;;; )00333"a'""03GI4FXaHb0c0c 4 4, n!eEll]22/077 8JKKK1299&:NOOO 4 )001D1D1F1FGGGG )003333 4 )$r$r$r$r$rafgpaqaq$r$r$rss)eEll]:)/:+22=!DDFWZii#,}#[>[4\4\###   s&&&&&%.%?%?&&&L " " &s%rP_%r%r%rL ")/:+22=!DDFWZii#2#7#7#:#: &W&|44N?1 ll  j*j,OOPPP    #o..// #o..//      UZ u| $ EI  \\ ] qq r pp q qq r ]] ^ !    FH !!%%''Ihh7788Ihh7788Ihh'899::      0  0 ) $ ) $ (   #J \))&z29=$Z09vwvvvvvvXbvvhsvvv3S5H5HH Q  $ I1}}  ---*  !2333*  !2333$  ---9~~!!!%9SbS>!:!:W!DyQS}!T!*1$36N$N ! UUURgoUU+9UULOPYNN]^L^L^SSdfUUU    s*&A++ B 5BB 9^== _  _ )NNNNrNNNNNrrNNNNFrFrrFFTNNN)rjsl"""""" )))))))) """"""......  8 $ $U      FFFFFFTTTTTT &*'+#)- ""$($(27$(#%) $#S\!*/##;OOOOOOOrG