Ch10SSKJr SSKrSSKrSSKJr SSKJr SSKrSSKJ r J r SSK J r SSKJr SSKJr SS KJr SS KJr SS KJr "S S 5r"SS5r"SS\ R45rg)) annotationsN)Iterable)Any)Tensornn) functional)SentenceTransformer)CachedGISTEmbedLoss)"CachedMultipleNegativesRankingLoss)+CachedMultipleNegativesSymmetricRankingLoss) TransformercT\rSrSrSrS SjrS SjrS SjrSSjrSSjr SSjr S r g )TransformerDecoratorz Decorator that caches the embeddings of all layers of the transformer. When `layer_idx` is set, it returns the cached embeddings of that layer instead. This is meant to override the forward function of the Transformer. cbXlX l/Ul/Ul/UlSUlSUlgNr) transformeroriginal_forward embeddingslast_embeddingsfeatures layer_idxcall_idx)selfrrs f/var/www/html/shao/venv/lib/python3.13/site-packages/sentence_transformers/losses/AdaptiveLayerLoss.py__init__TransformerDecorator.__init__s2& 0/1-/13  cXlSUlgr)rr)rrs r set_layer_idx"TransformerDecorator.set_layer_idx&s" rc[R"URVs/sHoURPM snSS9$s snf)Ndim)torchconcatrr)r embeddings rget_layer_embeddings)TransformerDecorator.get_layer_embeddings*s1||X9t~~6X^_``Xs<cURcURU5nU$URU5nU=RS- slU$)Nr#)rcall_grow_cachecall_use_cacherrroutputs r__call__TransformerDecorator.__call__-sH >> !))(3F ((2F MMQ M rcbURRRRnSURRRlUR U5n[ US5S- UlURRUSSS5 URRUS5 URRUR5VVs0sHupEUS;dM XE_M snn5 X RRRlU(aUS U$s snnf)z Temporarily sets the output_hidden_states to True, runs the model, and then restores the original setting. Use the all_layer_embeddings to get the embeddings of all layers. Tall_layer_embeddingsr#token_embeddings)r3r5) r auto_modelconfigoutput_hidden_statesrlen num_layersrappendrritems)rroriginal_output_hidden_statesr/keyvalues rr,$TransformerDecorator.call_grow_cache5s  )-(8(8(C(C(J(J(_(_%BF##**?&&x0f%;<=A v&<=aCD ##F+=$>? *0,,. t.JCCGss$-- .s0BDOOTXTaTaDbcgcqcqDrssr)rrrrrr:rrN)rr returnNonerCrDrCr)rCdict[str, Tensor]rrGrCrG) __name__ __module__ __qualname____firstlineno____doc__rr r)r0r,r-__static_attributes__rrrrs'a4trrc6\rSrSrSrSSjrS SjrS SjrSrg) ForwardDecoratorSz Decorator that caches the embeddings after all modules (e.g. pooling) of the model. Required to get the embeddings after all modules for the KL-divergence loss. This is meant to override the forward function of the SentenceTransformer. cXl/Ulg)N)fnr)rrTs rrForwardDecorator.__init__[srcdURU5nURRUS5 U$)Nsentence_embedding)rTrr;r.s rr0ForwardDecorator.__call___s-" v&:;< rcR[R"URSS9n/UlU$)Nrr$)r&r'r)rrs rget_embeddingsForwardDecorator.get_embeddingsds#\\$//q9 r)rrTNrErHrF) rIrJrKrLrMrr0rZrNrOrrrQrQSs rrQc^\rSrSrSSU4SjjjrS SjrS Sjr\S Sj5rSr U=r $) AdaptiveLayerLossjcZ>[TU]5 XlX lX0lX@lXPlX`lXpl[URS[5(de[U[[[45(a.[R"SUR R"S3SS9 gg)a The AdaptiveLayerLoss can be seen as a loss *modifier* that allows you to use other loss functions at non-final layers of the Sentence Transformer model. This is useful for when you want to train a model where users have the option to lower the number of layers used to improve their inference speed and memory usage. Args: model: SentenceTransformer model loss: The loss function to be used, e.g. :class:`MultipleNegativesRankingLoss`, :class:`CoSENTLoss`, etc. n_layers_per_step: The number of layers to use per step. If -1, then all layers are used. If > 0, then a random sample of `n_layers_per_step` layers are used per step, separate from the final layer, which is always used. The 2DMSE paper uses `n_layers_per_step=1`. The default value is 1. last_layer_weight: The weight to use for the loss of the final layer. Increase this to focus more on the performance when using all layers. The default value is 1.0. prior_layers_weight: The weight to use for the loss of the prior layers. Increase this to focus more on the performance when using fewer layers. The default value is 1.0. kl_div_weight: The weight to use for the KL-divergence loss that is used to make the prior layers match that of the last layer. Increase this to focus more on the performance when using fewer layers. The default value is 1.0. kl_temperature: The temperature to use for the KL-divergence loss. If 0, then the KL-divergence loss is not used. The default value is 1.0. References: - The concept was inspired by the 2DMSE paper: https://arxiv.org/abs/2402.14776 - `Adaptive Layers <../../../examples/sentence_transformer/training/adaptive_layer/README.html>`_ Requirements: 1. The base loss cannot be :class:`CachedMultipleNegativesRankingLoss`, :class:`CachedMultipleNegativesSymmetricRankingLoss`, or :class:`CachedGISTEmbedLoss`. Inputs: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | any | any | +---------------------------------------+--------+ Relations: - :class:`Matryoshka2dLoss` uses this loss in combination with :class:`MatryoshkaLoss` which allows for output dimensionality reduction for faster downstream tasks (e.g. retrieval). Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.MultipleNegativesRankingLoss(model=model) loss = losses.AdaptiveLayerLoss(model, loss) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() rz&MatryoshkaLoss is not compatible with .) stacklevelN)superrmodellossn_layers_per_steplast_layer_weightprior_layers_weight kl_div_weightkl_temperature isinstancer r r r warningswarn __class__rI) rrdrerfrgrhrirjrns rrAdaptiveLayerLoss.__init__ksf   !2!2#6 *,$**Q-5555   /1\^q r   MMB4>>CZCZB[[\]jk l  rc6URSRn[URSU5nX@RSlURRn[U5nX`RlUR X5UR -nUR S:a1UR5n[R"XR - SS9nURn [U S- 5n URS:a3URU S- :a [R"XR5n U Hn URU 5 UR X5n X|SU -- [!U 5- UR"--nUR S:dMZUR5n [R$"[R&"XR - SS9WSS9nX~UR -UR(--nM X0RSlXPRlU$)Nrr4r$r# batchmean) reduction)rdforwardrrQrergrjrZFsoftmaxr:rangerfrandomsampler r9rhkl_div log_softmaxri)rsentence_featureslabelsoriginal_transformer_forwardtransformer_decoratorrforward_decoratorrefinal_embeddingsr: layer_indicesr layer_lossr kl_div_losss rrsAdaptiveLayerLoss.forwards'+zz!}'<'<$ 4TZZ]D` a 5 1  ::--,-=>.  yy*3d6L6LL    "0??A  yy)9*Q"MM-9O9OPM'I ! / / :#4=JI 6]9KKdNfNfffD""Q&.==? hhMM*/B/B"BK$) D,?,??$BTBTTT'"!= 1 -  rcURRRURURUR UR URS.$)N)rerfrgrhrirj)rernrIrfrgrhrirjrs rget_config_dict!AdaptiveLayerLoss.get_config_dictsLII''00!%!7!7!%!7!7#'#;#;!//"11   rcg)Nz @misc{li20242d, title={2D Matryoshka Sentence Embeddings}, author={Xianming Li and Zongxi Li and Jing Li and Haoran Xie and Qing Li}, year={2024}, eprint={2402.14776}, archivePrefix={arXiv}, primaryClass={cs.CL} } rOrs rcitationAdaptiveLayerLoss.citations r)rirjrgrerdrfrh)r#?rrg333333?)rdr rez nn.ModulerfintrgfloatrhrrirrjrrCrD)r{zIterable[dict[str, Tensor]]r|rrCr)rCzdict[str, Any])rCstr) rIrJrKrLrrsrpropertyrrN __classcell__)rns@rr]r]js "##&%(" #`m"`m`m `m ! `m # `m`m`m `m`mD-^   rr]) __future__rrwrlcollections.abcrtypingrr&rrtorch.nnrrtsentence_transformersr 0sentence_transformers.losses.CachedGISTEmbedLossr ?sentence_transformers.losses.CachedMultipleNegativesRankingLossr Hsentence_transformers.losses.CachedMultipleNegativesSymmetricRankingLossr sentence_transformers.modelsr rrQModuler]rOrrrsY" $ $5Pn5;t;t|.g gr