o sh@sddlZddlZddlmZmZmZmZmZmZddl m m Z ddl mZm Z ddlmZddlmZddlmZGdddZGd d d e jZdS) N)AnyDictIterableListOptionalUnion)Tensornn)SentenceTransformer)CachedGISTEmbedLoss)"CachedMultipleNegativesRankingLossc@sTeZdZd ddZd ddZdedefdd Zd eeefdeeeffd d Z dS)ForwardDecoratorreturnNcCs"||_d|_g|_d|_d|_dSNr)fndimcache cache_dimidx)selfrri/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/sentence_transformers/losses/MatryoshkaLoss.py__init__s  zForwardDecorator.__init__cCs||_d|_dSr)rr)rrrrrset_dims zForwardDecorator.set_dimtensorcCsP|jd}|j|krtd|jd||dd|jf}tj|ddd}|S)Nz Dimension zL in matryoshka_dims cannot be greater than the model's embedding dimension: .)pr)shaper ValueErrorF normalize)rr tensor_dimrrrshrinks  zForwardDecorator.shrinkfeaturescCsx|jdus |j|jkr||}|j||j|_n|j|j}||d|d<||d|d<|jd7_|S)Ntoken_embeddingssentence_embedding)rrrrappendrr#)rr$outputrrr__call__$s    zForwardDecorator.__call__)rN) __name__ __module__ __qualname__rrrr#rstrr*rrrrr s   & r c seZdZ  ddedejdeedeee e efdeddf fd d Z d e e eefd edefd dZde eeffddZedefddZZS)MatryoshkaLossNrmodellossmatryoshka_dimsmatryoshka_weightsn_dims_per_steprcst||_||_t|trtjdddt|tr#tjddd|dur.dgt |}t ||}t t |ddd d \|_ |_ ||_dS) a The MatryoshkaLoss can be seen as a loss *modifier* that allows you to use other loss functions at various different embedding dimensions. This is useful for when you want to train a model where users have the option to lower the embedding dimension to improve their embedding comparison speed and costs. Args: model: SentenceTransformer model loss: The loss function to be used, e.g. :class:`MultipleNegativesRankingLoss`, :class:`CoSENTLoss`, etc. matryoshka_dims: A list of embedding dimensions to be used for the loss function, e.g. [768, 512, 256, 128, 64]. matryoshka_weights: A list of weights to be used for the loss function, e.g. [1, 1, 1, 1, 1]. If None, then the weights will be set to 1 for all dimensions. n_dims_per_step: The number of dimensions to use per step. If -1, then all dimensions are used. If > 0, then a random sample of n_dims_per_step dimensions are used per step. The default value is -1. References: - The concept was introduced in this paper: https://arxiv.org/abs/2205.13147 - `Matryoshka Embeddings <../../examples/training/matryoshka/README.html>`_ Requirements: 1. The base loss cannot be :class:`CachedMultipleNegativesRankingLoss` or :class:`CachedGISTEmbedLoss`. Relations: - :class:`Matryoshka2dLoss` uses this loss in combination with :class:`AdaptiveLayerLoss` which allows for layer reduction for faster inference. Input: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | any | any | +---------------------------------------+--------+ Example: :: from sentence_transformers import SentenceTransformer, losses, InputExample from torch.utils.data import DataLoader model = SentenceTransformer("microsoft/mpnet-base") train_examples = [ InputExample(texts=['Anchor 1', 'Positive 1']), InputExample(texts=['Anchor 2', 'Positive 2']), ] train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=32) train_loss = losses.MultipleNegativesRankingLoss(model=model) train_loss = losses.MatryoshkaLoss(model, train_loss, [768, 512, 256, 128, 64]) model.fit( [(train_dataloader, train_loss)], epochs=10, ) zIMatryoshkaLoss is not compatible with CachedMultipleNegativesRankingLoss.r) stacklevelz:MatryoshkaLoss is not compatible with CachedGISTEmbedLoss.Nr'cSs|dSrr)xrrrsz)MatryoshkaLoss.__init__..T)keyreverse)superrr0r1 isinstancer warningswarnr lenzipsortedr2r3r4)rr0r1r2r3r4 dims_weights __class__rrr4s A    zMatryoshkaLoss.__init__sentence_featureslabelsc Cs|jj}zIt|}||j_tt|j}|jdkr'|jt|kr't||j}d}|D]}|j|}|j |} | ||| | ||7}q+W||j_|S||j_w)Nrg) r0forwardr ranger>r2r4randomsampler3rr1) rrDrEoriginal_forwarddecorated_forward dim_indicesr1rrweightrrrrFs"    zMatryoshkaLoss.forwardcCs|jjj|j|j|jdS)N)r1r2r3r4)r1rCr+r2r3r4rrrrget_config_dicts zMatryoshkaLoss.get_config_dictcCsdS)Na @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } rrNrrrcitationszMatryoshkaLoss.citation)Nr)r+r,r-r r Modulerintrrfloatrrrr.rrFrrOpropertyrP __classcell__rrrBrr/3s("Pr/)rHr<typingrrrrrrtorch.nn.functionalr functionalr torchrsentence_transformersr 0sentence_transformers.losses.CachedGISTEmbedLossr ?sentence_transformers.losses.CachedMultipleNegativesRankingLossr r rQr/rrrrs    &