o sh@sddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZmZmZmZmZddlZddlZddlZddlmZmZddlmZmZddlmZddlmZe e!Z"dee#ej$efd efd d Z%ded efd d Z&dee#ej$efd efddZ'deded efddZ(dee#ej$efdee#ej$efd efddZ)deded efddZ*dee#ej$efdee#ej$efd efddZ+deded efddZ,dee#ej$efdee#ej$efd efddZ-dee#ej$efdee#ej$effddZ.dee#ej$efdee#ej$efd efdd Z/dee#ej$efdee#ej$effd!d"Z0d#ed$ed efd%d&Z1d'ed efd(d)Z2ed'ej$d*ee3d ej$fd+d,Z4ed'ejd*ee3d ejfd-d,Z4d'eej$ejfd*ee3d eej$ejffd.d,Z4d/d0d1d2d3d4e)fd5e e5d6e6d7e3d8e3d9e3d:e3d;e3dZ8d1d2d3d4e)fd'ed8e3d9e3d:e3d;e3ddRe5d efdSdTZ? U C V /dwd'eejej$fdWe7dXe3d7e3d6e6d e e e3f dYdZZ@Gd[d\d\eZAeejBfd]d^ZC    /dxd_e5d`eee6e5fdaee5dbee5dce6d e6f dddeZD  /dyd_e5dfe5d`eee6e5fdaee5dbee5dce6d ee5fdgdhZE  /dyd_e5die5d`eee6e5fdaee5dbee5dce6d ee5fdjdkZFdldmZGd ednfdodpZHd e6fdqdrZId e6fdsdtZJd e6fdudvZKdS)zN)contextmanager) AnyCallableDictListLiteralOptionalTypeUnionoverload)hf_hub_downloadsnapshot_download)Tensordevice)tqdm)is_torch_npu_availableareturncCst|ts t|}|S)z Converts the input `a` to a PyTorch tensor if it is not already a tensor. Args: a (Union[list, np.ndarray, Tensor]): The input array or tensor. Returns: Tensor: The converted tensor. ) isinstancertorchtensorrrX/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/sentence_transformers/util.py_convert_to_tensors rcCs|dkr |d}|S)z If the tensor `a` is 1-dimensional, it is unsqueezed to add a batch dimension. Args: a (Tensor): The input tensor. Returns: Tensor: The tensor with a batch dimension. r)dim unsqueezerrrr_convert_to_batch%s rcCst|}t|}|S)z Converts the input data to a tensor with a batch dimension. Args: a (Union[list, np.ndarray, Tensor]): The input data to be converted. Returns: Tensor: The converted tensor with a batch dimension. )rrrrrr_convert_to_batch_tensor4s rbcCs t||S) Computes the cosine similarity between two tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = cos_sim(a[i], b[j]) )cos_simrr rrrpytorch_cos_simCs r$cCs4t|}t|}t|}t|}t||ddS)r!rr)rnormalize_embeddingsrmm transpose)rr a_normb_normrrrr"Qs  r"cCs"t|}t|}tt|t|S)a Computes the pairwise cosine similarity cos_sim(a[i], b[i]). Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = cos_sim(a[i], b[i]) )rpairwise_dot_scorer%r#rrrpairwise_cos_simd r+cCs$t|}t|}t||ddS)a Computes the dot-product dot_prod(a[i], b[j]) for all i and j. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = dot_prod(a[i], b[j]) rr)rrr&r'r#rrr dot_scoreus r-cCs t|}t|}||jddS)a Computes the pairwise dot-product dot_prod(a[i], b[i]). Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = dot_prod(a[i], b[i]) r)rsumr#rrrr*s r*cC"t|}t|}tj||dd S)a: Computes the manhattan similarity (i.e., negative distance) between two tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = -manhattan_distance(a[i], b[j]) g?prrcdistr#rrr manhattan_simr,r6cCs*t|}t|}tjt||dd S)a< Computes the manhattan similarity (i.e., negative distance) between pairs of tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = -manhattan_distance(a[i], b[i]) r.r/)rrr0absr#rrrpairwise_manhattan_sims r8cCr1)a: Computes the euclidean similarity (i.e., negative distance) between two tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = -euclidean_distance(a[i], b[j]) g@r2r4r#rrr euclidean_simr,r9cCs.t|}t|}ttj||ddd S)a: Computes the euclidean distance (i.e., negative distance) between pairs of tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = -euclidean_distance(a[i], b[i]) r.r/)rrsqrtr0r#rrrpairwise_euclidean_sims r<xyc Cst|}t|}tj|ddd\}}tj|ddd\}}tj|d|dddd}|||||}|||||}tj|d|ddddd} tj|d|ddddd} || | }|| | }tjtj||fdddd} t| S)aP Computes the absolute normalized angle distance. See :class:`~sentence_transformers.losses.AnglELoss` or https://arxiv.org/abs/2309.12871v1 for more information. Args: x (Tensor): The first tensor. y (Tensor): The second tensor. Returns: Tensor: Vector with res[i] = angle_sim(a[i], b[i]) r:rr/T)rkeepdimg?)rrchunkr0concatr7) r=r>rr cdzreimdzdw norm_anglerrrpairwise_angle_sims      rJ embeddingscCstjjj|dddS)z Normalizes the embeddings matrix, so that each sentence embedding has unit length. Args: embeddings (Tensor): The input embeddings matrix. Returns: Tensor: The normalized embeddings matrix. r:r)r3r)rnn functional normalize)rKrrrr%s r% truncate_dimcCdSNrrKrOrrrtruncate_embeddings rScCrPrQrrRrrrrSrTcCs|dd|fS)a Truncates the embeddings matrix. Args: embeddings (Union[np.ndarray, torch.Tensor]): Embeddings to truncate. truncate_dim (Optional[int]): The dimension to truncate sentence embeddings to. `None` does no truncation. Example: >>> from sentence_transformers import SentenceTransformer >>> from sentence_transformers.util import truncate_embeddings >>> model = SentenceTransformer("tomaarsen/mpnet-base-nli-matryoshka") >>> embeddings = model.encode(["It's so nice outside!", "Today is a beautiful day.", "He drove to work earlier"]) >>> embeddings.shape (3, 768) >>> model.similarity(embeddings, embeddings) tensor([[1.0000, 0.8100, 0.1426], [0.8100, 1.0000, 0.2121], [0.1426, 0.2121, 1.0000]]) >>> truncated_embeddings = truncate_embeddings(embeddings, 128) >>> truncated_embeddings.shape >>> model.similarity(truncated_embeddings, truncated_embeddings) tensor([[1.0000, 0.8092, 0.1987], [0.8092, 1.0000, 0.2716], [0.1987, 0.2716, 1.0000]]) Returns: Union[np.ndarray, torch.Tensor]: Truncated embeddings. .NrrRrrrrSsF iii d sentencesshow_progress_bar batch_sizequery_chunk_sizecorpus_chunk_size max_pairstop_kscore_functionc Cs&|j|||dd} t| |||||dS)a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: model (SentenceTransformer): SentenceTransformer model for embedding computation sentences (List[str]): A list of strings (texts or sentences) show_progress_bar (bool, optional): Plotting of a progress bar. Defaults to False. batch_size (int, optional): Number of texts that are encoded simultaneously by the model. Defaults to 32. query_chunk_size (int, optional): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). Defaults to 5000. corpus_chunk_size (int, optional): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). Defaults to 100000. max_pairs (int, optional): Maximal number of text pairs returned. Defaults to 500000. top_k (int, optional): For each sentence, we retrieve up to top_k other sentences. Defaults to 100. score_function (Callable[[Tensor, Tensor], Tensor], optional): Function for computing scores. By default, cosine similarity. Defaults to cos_sim. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] T)rXrYconvert_to_tensor)rZr[r\r]r^)encodeparaphrase_mining_embeddings) modelrWrXrYrZr[r\r]r^rKrrrparaphrase_mining4srcc Cs|d7}t}d}d}tdt||D]} tdt||D]v} ||| | ||| | |} tj| t|t| ddddd\} } | } | } tt| D]=}t | |D]4\}}| |}| |}||kr| |||kr| | ||||f|d7}||kr| }|d}q^qVqqt }g}| s| \}}}t||g\}}||kr||f|vr|||f||||g| rt|dddd }|S) a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: embeddings (Tensor): A tensor with the embeddings query_chunk_size (int): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). corpus_chunk_size (int): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). max_pairs (int): Maximal number of text pairs returned. top_k (int): For each sentence, we retrieve up to top_k other sentences score_function (Callable[[Tensor, Tensor], Tensor]): Function for computing scores. By default, cosine similarity. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] rr.rTFrlargestsortedcS|dS)Nrrr=rrrz.paraphrase_mining_embeddings..keyreverse)queue PriorityQueuerangelenrtopkmincputolist enumerateputgetsetemptyrfaddappend)rKrZr[r\r]r^pairs min_score num_addedcorpus_start_idxquery_start_idxscoresscores_top_k_valuesscores_top_k_idx query_itr top_k_idx corpus_itrijentry added_pairs pairs_listscoresorted_isorted_jrrrraasP   racOst|i|S)z8This function is deprecated. Use semantic_search instead)semantic_search)argskwargsrrrinformation_retrievalsr query_embeddingscorpus_embeddingsc Cst|tjtjfrt|}n t|trt|}t|j dkr%| d}t|tjtjfr4t|}n t|tr>t|}|j |j krJ| |j }ddt t|D}t dt||D]y}t dt||D]n}|||||||||} tj| t|t| ddddd\} } | } | } t t| D]5} t| | | | D])\} }|| }|| }t|||krt||||fqt||||fqqqgq]t t|D]+}t t||D]}|||\}}||d|||<qt||d d dd ||<q|S) a This function performs a cosine similarity search between a list of query embeddings and a list of corpus embeddings. It can be used for Information Retrieval / Semantic Search for corpora up to about 1 Million entries. Args: query_embeddings (Tensor): A 2 dimensional tensor with the query embeddings. corpus_embeddings (Tensor): A 2 dimensional tensor with the corpus embeddings. query_chunk_size (int, optional): Process 100 queries simultaneously. Increasing that value increases the speed, but requires more memory. Defaults to 100. corpus_chunk_size (int, optional): Scans the corpus 100k entries at a time. Increasing that value increases the speed, but requires more memory. Defaults to 500000. top_k (int, optional): Retrieve top k matching entries. Defaults to 10. score_function (Callable[[Tensor, Tensor], Tensor], optional): Function for computing scores. By default, cosine similarity. Returns: List[List[Dict[str, Union[int, float]]]]: A list with one entry for each query. Each entry is a list of dictionaries with the keys 'corpus_id' and 'score', sorted by decreasing cosine similarity scores. rrcSsg|]}gqSrr).0_rrr sz#semantic_search..TFrd) corpus_idrcSrg)Nrrrhrrrrirjz!semantic_search..rk)rnpndarraygenericr from_numpyliststackrqshaperrtorprrrsrtruzipheapqheappush heappushpoprf)rrrZr[r]r^queries_result_listrr cos_scorescos_scores_top_k_valuescos_scores_top_k_idxr sub_corpus_idrrquery_iddoc_itrrrrrsT            rurlpathc Cstj|dkrtjtj|ddtj|dd}|jdkr1td||jt j d| dS|d }t |d 6}|j d }|durIt|nd}td |dd }|jddD]}|rh|t|||qXWdn1sswYt|||dS)a Downloads a URL to a given path on disk. Args: url (str): The URL to download. path (str): The path to save the downloaded file. Raises: requests.HTTPError: If the HTTP request returns a non-200 status code. Returns: None T)exist_ok)streamz1Exception when trying to download {}. Response {})fileN_partwbzContent-LengthB)unittotal unit_scale) chunk_size)osrdirnamemakedirsrequestsrx status_codeprintformatsysstderrraise_for_statusopenheadersintr iter_contentupdaterqwriterenameclose) rrreqdownload_filepath file_binarycontent_lengthrprogressr@rrrhttp_gets*     rbatch target_devicecCs.|D]}t||tr|||||<q|S)au Send a PyTorch batch (i.e., a dictionary of string keys to Tensors) to a device (e.g. "cpu", "cuda", "mps"). Args: batch (Dict[str, Tensor]): The batch to send to the device. target_device (torch.device): The target device (e.g. "cpu", "cuda", "mps"). Returns: Dict[str, Tensor]: The batch with tensors sent to the target device. )rrr)rrrlrrrbatch_to_device$s  rcCs4|jj}|dus|tjjkr|jjS|d|jjS)a Gives a full name (package_name.class_name) for a class / object in Python. Will be used to load the correct classes from JSON files Args: o: The object for which to get the full name. Returns: str: The full name of the object. Example: >>> from sentence_transformers.losses import MultipleNegativesRankingLoss >>> from sentence_transformers import SentenceTransformer >>> from sentence_transformers.util import fullname >>> model = SentenceTransformer('all-MiniLM-L6-v2') >>> loss = MultipleNegativesRankingLoss(model) >>> fullname(loss) 'sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss' N.) __class__ __module__str__name__)omodulerrrfullname5sr dotted_pathcCsz |dd\}}Wntyd|}t|wzt|}Wnty/t|}Ynwzt||WStyGd||f}t|w)a: Import a dotted module path and return the attribute/class designated by the last name in the path. Raise ImportError if the import failed. Args: dotted_path (str): The dotted module path. Returns: Any: The attribute/class designated by the last name in the path. Raises: ImportError: If the import failed. Example: >>> import_from_string('sentence_transformers.losses.MultipleNegativesRankingLoss') rrz"%s doesn't look like a module pathz2Module "%s" does not define a "%s" attribute/class)rsplit ValueError ImportError importlib import_module ExceptiongetattrAttributeError)r module_path class_namemsgrrrrimport_from_stringQs"     r?r thresholdmin_community_sizecCsXt|tjs t|}tj||jd}t|}g}t|t|}ttd|dt|}t t dt||d| dD]}|||||j }|jj dvr||k} | d} | |k} | s_q:| | } || }| } |j| d d \} }t| |D]\}}||d |qyq:|j|d d \}} t t|D]L}||d |kr||j|d d \}}|d |kr|t|krtd|t|}||j|d d \}}|d |kr|t|ks||||kqq:t|d dd d}g}t}t|D]'\}}g}|D] }||vr ||qt||kr ||||qt|ddd d}|S)a Function for Fast Community Detection. Finds in the embeddings all communities, i.e. embeddings that are close (closer than threshold). Returns only communities that are larger than min_community_size. The communities are returned in decreasing order. The first element in each list is the central point in the community. Args: embeddings (torch.Tensor or numpy.ndarray): The input embeddings. threshold (float): The threshold for determining if two embeddings are close. Defaults to 0.75. min_community_size (int): The minimum size of a community to be considered. Defaults to 10. batch_size (int): The batch size for computing cosine similarity scores. Defaults to 1024. show_progress_bar (bool): Whether to show a progress bar during computation. Defaults to False. Returns: List[List[int]]: A list of communities, where each community is represented as a list of indices. )rr:2rzFinding clusters)descdisable)cudanpurT)kreNr.cSt|SrQrqrhrrrrirjz%community_detection..rkcSrrQrrhrrrrirj)rrrrrr%rsrqmaxrrpTtyper0anyrrrr|rurfryrvr)rKrrrYrXextracted_communities sort_max_size start_idxrthreshold_maskrow_wise_countlarge_enough_maskrr top_k_indicescountindices top_k_valuesr top_val_large top_idx_largeunique_communities extracted_ids cluster_id communitynon_overlapped_communityidxrrrcommunity_detectionusd          rcs6eZdZdZfddZdeddffdd ZZS) disabled_tqdmz Class to override `disable` argument in case progress bars are globally disabled. Taken from https://github.com/tqdm/tqdm/issues/619#issuecomment-619639324. csd|d<tj|i|dS)NTr)super__init__)selfrrrrrrszdisabled_tqdm.__init__attrrNcs2z t|WdSty|dkrYdSw)zBFix for https://github.com/huggingface/huggingface_hub/issues/1603_lockN)r __delattr__r)rrrrrrs zdisabled_tqdm.__delattr__)rr __qualname____doc__rrr __classcell__rrrrrs rc cs:tjjj}t|z dVWt|dSt|w)z A context manager that will prevent any logging messages triggered during the body from being processed. Args: highest_level: the maximum logging level allowed. N)loggingrootmanagerr) highest_levelprevious_levelrrrdisable_loggings  rmodel_name_or_pathtoken cache_folderrevisionlocal_files_onlyc Cstt|d||||dS)a} Checks if the given model name or path corresponds to a SentenceTransformer model. Args: model_name_or_path (str): The name or path of the model. token (Optional[Union[bool, str]]): The token to be used for authentication. Defaults to None. cache_folder (Optional[str]): The folder to cache the model files. Defaults to None. revision (Optional[str]): The revision of the model. Defaults to None. local_files_only (bool): Whether to only use local files for the model. Defaults to False. Returns: bool: True if the model is a SentenceTransformer model, False otherwise. z modules.json)r#r$)boolload_file_path)r r!r"r#r$rrris_sentence_transformer_modelsr'filenamec CsLtj||}tj|r|Sz t|||d|||dWSty%YdSw)a Loads a file from a local or remote location. Args: model_name_or_path (str): The model name or path. filename (str): The name of the file to load. token (Optional[Union[bool, str]]): The token to access the remote file (if applicable). cache_folder (Optional[str]): The folder to cache the downloaded file (if applicable). revision (Optional[str], optional): The revision of the file (if applicable). Defaults to None. local_files_only (bool, optional): Whether to only consider local files. Defaults to False. Returns: Optional[str]: The path to the loaded file, or None if the file could not be found or loaded. sentence-transformers)r(r# library_namer! cache_dirr$N)rrjoinexistsr r)r r(r!r"r#r$ file_pathrrrr&(s   r& directoryc Cstj||}tj|r|S|||dd|||td}z tdi|}Wnty:d|d<tdi|}Ynwtj||S)af Loads the directory path for a given model name or path. Args: model_name_or_path (str): The name or path of the model. directory (str): The directory to load. token (Optional[Union[bool, str]]): The token for authentication. cache_folder (Optional[str]): The folder to cache the downloaded files. revision (Optional[str], optional): The revision of the model. Defaults to None. local_files_only (bool, optional): Whether to only use local files. Defaults to False. Returns: Optional[str]: The directory path if it exists, otherwise None. z/**r))repo_idr#allow_patternsr*r!r+r$ tqdm_classTr$Nr)rrr,r-rr r) r r/r!r"r#r$dir_pathdownload_kwargs repo_pathrrr load_dir_pathRs&   r6cstfdd}|S)Ncsn|dd}|rd|vrtd||d<t|dkr,g|ddd|ddR}|g|Ri|S)N repo_namer0zfProviding a `repo_name` keyword argument to `save_to_hub` is deprecated, please use `repo_id` instead.r:)poploggerwarningrq)rrrr7funcrrwrappers   "z+save_to_hub_args_decorator..wrapper) functoolswraps)r<r=rr;rsave_to_hub_args_decoratorsr@)mpsrrhpurtcCsZtjrdStjjrdStrdStjddur+ddl mm }|r+dSdS) aH Returns the name of the device where this module is running on. It's a simple implementation that doesn't cover cases when more powerful GPUs are available and not a primary device ('cuda:0') or MPS device is available, but not configured properly. Returns: str: Device name, like 'cuda' or 'cpu' rrArhabana_frameworksNrrBrt) rr is_availablebackendsrArrutil find_spechabana_frameworks.torch.hpurB)hthpurrrget_device_names  rJcCtjdduS)z> Returns True if the accelerate library is available. accelerateNrrFrGrrrris_accelerate_availablerNcCrK)z< Returns True if the datasets library is available. datasetsNrMrrrris_datasets_availablerOrQcCs totS)zd Returns True if we have the required dependencies for training Sentence Transformer models )rNrQrrrris_training_availables rR)rrrF)NNNF)NF)Lr>rrrrrnr contextlibrtypingrrrrrrr r r numpyrrrhuggingface_hubr r rrtqdm.autonotebookr transformersr getLoggerrr9rrrrrr$r"r+r-r*r6r8r9r<rJr%rrSrr%floatrcrarrrrrrrrCRITICALrr'r&r6r@rJrNrQrRrrrrsx ,   ...*.*"   %  / &I P&%&  k % / /