o sh]@sddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZmZmZmZddlmZmZmZmZddlmZddlmZdd lmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4ddl5m6Z6m7Z7m8Z8ddl9m:Z:m;Z;mZ>m?Z?e@eAZBerddlCmDZDGdddeZEdS)N) nullcontext) TYPE_CHECKINGAnyCallableDictListOptionalTupleUnion)nn) BatchSampler ConcatDataset DataLoaderSubsetRandomSampler)EvalPredictionPreTrainedTokenizerBaseTrainerTrainerCallback) DataCollator) WandbCallback)TRAINING_ARGS_NAME)EvalLoopOutput) ParallelMode)SentenceTransformerDataCollator)SentenceEvaluatorSequentialEvaluator) CoSENTLoss)ModelCardCallback) Transformer)DefaultBatchSamplerGroupByLabelBatchSamplerNoDuplicatesBatchSamplerProportionalBatchSamplerRoundRobinBatchSampler) BatchSamplersMultiDatasetBatchSamplers$SentenceTransformerTrainingArguments)disable_loggingis_datasets_availableis_training_available)Dataset DatasetDictSentenceTransformercs\eZdZdZ             dZdeddedeedd eedffd eedd eedffd eee j eee j fe dge j j feee dge j j fffd eee ee fd eedeeee fdee gdfdee egefdeeedee jje jjjfdee e je jge jfddffdd Zd[d\fdd Zd e j j ddde j j fddZd ee dge j j fe j j fddde j j fddZd]ddZ d^ddd!eeee jeffd"edee jee jeeeffffd#d$Z d!eeee jeffdeeeee jfee jffd%d&Z!   'd_d eedeedffd(eeed)edeee"fffd*d+ Z#   'd_d,e$d-ed.eed(eeed)ede%f fd/d0 Z&d`fd1d2 Z'd[d3dd4eedefd5d6Z(  dd3dd7e)d8ed9eeed:ee j*de+f d;d<Z,  =dad3e-d>ee+d:ee j*d?ee)de+f d@dAZ.de$fdBdCZ/d[d edDde$fdEdFZ0dGdde$fdHdIZ1ddJeeddfdKdLZ2dMeddfdNdOZ3         dbdPeedQeedReeeedfdSeedTeedUeeeedfdVeeeedfd3eeeedfdWeeeedfddfdXdYZ4Z5S)cSentenceTransformerTraineru SentenceTransformerTrainer is a simple but feature-complete training and eval loop for PyTorch based on the 🤗 Transformers :class:`~transformers.Trainer`. This trainer integrates support for various :class:`transformers.TrainerCallback` subclasses, such as: - :class:`~transformers.integrations.WandbCallback` to automatically log training metrics to W&B if `wandb` is installed - :class:`~transformers.integrations.TensorBoardCallback` to log training metrics to TensorBoard if `tensorboard` is accessible. - :class:`~transformers.integrations.CodeCarbonCallback` to track the carbon emissions of your model during training if `codecarbon` is installed. - Note: These carbon emissions will be included in your automatically generated model card. See the Transformers `Callbacks `_ documentation for more information on the integrated callbacks and how to write your own callbacks. Args: model (:class:`~sentence_transformers.SentenceTransformer`, *optional*): The model to train, evaluate or use for predictions. If not provided, a `model_init` must be passed. args (:class:`~sentence_transformers.training_args.SentenceTransformerTrainingArguments`, *optional*): The arguments to tweak for training. Will default to a basic instance of :class:`~sentence_transformers.training_args.SentenceTransformerTrainingArguments` with the `output_dir` set to a directory named *tmp_trainer* in the current directory if not provided. train_dataset (Union[:class:`datasets.Dataset`, :class:`datasets.DatasetDict`, Dict[str, :class:`datasets.Dataset`]], *optional*): The dataset to use for training. Must have a format accepted by your loss function, see `Training Overview > Dataset Format <../../../docs/sentence_transformer/training_overview.html#dataset-format>`_. eval_dataset (Union[:class:`datasets.Dataset`, :class:`datasets.DatasetDict`, Dict[str, :class:`datasets.Dataset`]], *optional*): The dataset to use for evaluation. Must have a format accepted by your loss function, see `Training Overview > Dataset Format <../../../docs/sentence_transformer/training_overview.html#dataset-format>`_. loss (Optional[Union[:class:`torch.nn.Module`, Dict[str, :class:`torch.nn.Module`], Callable[[:class:`~sentence_transformers.SentenceTransformer`], :class:`torch.nn.Module`], Dict[str, Callable[[:class:`~sentence_transformers.SentenceTransformer`]]]], *optional*): The loss function to use for training. Can either be a loss class instance, a dictionary mapping dataset names to loss class instances, a function that returns a loss class instance given a model, or a dictionary mapping dataset names to functions that return a loss class instance given a model. In practice, the latter two are primarily used for hyper-parameter optimization. Will default to :class:`~sentence_transformers.losses.CoSENTLoss` if no ``loss`` is provided. evaluator (Union[:class:`~sentence_transformers.evaluation.SentenceEvaluator`, List[:class:`~sentence_transformers.evaluation.SentenceEvaluator`]], *optional*): The evaluator instance for useful evaluation metrics during training. You can use an ``evaluator`` with or without an ``eval_dataset``, and vice versa. Generally, the metrics that an ``evaluator`` returns are more useful than the loss value returned from the ``eval_dataset``. A list of evaluators will be wrapped in a :class:`~sentence_transformers.evaluation.SequentialEvaluator` to run them sequentially. callbacks (List of [:class:`transformers.TrainerCallback`], *optional*): A list of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](callback). If you want to remove one of the default callbacks used, use the [`Trainer.remove_callback`] method. optimizers (`Tuple[:class:`torch.optim.Optimizer`, :class:`torch.optim.lr_scheduler.LambdaLR`]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of :class:`torch.optim.AdamW` on your model and a scheduler given by :func:`transformers.get_linear_schedule_with_warmup` controlled by `args`. Important attributes: - **model** -- Always points to the core model. If using a transformers model, it will be a [`PreTrainedModel`] subclass. - **model_wrapped** -- Always points to the most external model in case one or more other modules wrap the original model. This is the model that should be used for the forward pass. For example, under `DeepSpeed`, the inner model is wrapped in `DeepSpeed` and then again in `torch.nn.DistributedDataParallel`. If the inner model hasn't been wrapped, then `self.model_wrapped` is the same as `self.model`. - **is_model_parallel** -- Whether or not a model has been switched to a model parallel mode (different from data parallelism, this means some of the model layers are split on different GPUs). - **place_model_on_device** -- Whether or not to automatically place the model on the device - it will be set to `False` if model parallel or deepspeed is used, or if the default `TrainingArguments.place_model_on_device` is overridden to return `False` . - **is_in_train** -- Whether or not a model is currently running `train` (e.g. when `evaluate` is called while in `train`) NNNmodelr-args train_datasetr*r+ eval_datasetloss evaluator data_collator tokenizer model_initcompute_metrics callbacks optimizerspreprocess_logits_for_metricsreturnc ststd|durd}td|dt|d}n t|ts%tddur9| dur5| _ntd| durCt dt | _td d }|j r[jjs[j|j |durhtjtrhj}|durrtjd }t|trt|tst|}t|trt|tst|}tjjrdn|||||| | | | | d d _td djjDrtj dd|durtdt!j"}t|tr,fdd|#D_$t%ddg||gD]A\}}|durqt|tstd|dt&|'t&|'}r*td|dt(|dt)|dkr!dndd|d qn*|_$|durBt|t+sBt,|}|_-t.|}/||0j1j2j3j"dS) NzTo train a SentenceTransformer model, you need to install the `accelerate` and `datasets` modules. You can do so with the `train` extra: pip install -U "sentence-transformers[train]" tmp_trainerz1No `TrainingArguments` passed, using `output_dir=z`.) output_dirzEPlease use `TrainingArguments` imported from `sentence_transformers`.z<`Trainer` requires either a `model` or `model_init` argumentz`Trainer` requires either a `model` or `model_init` argument, but not both. `model_init` will overwrite your model when calling the `train` method. This will become a fatal error in the next release.unused) tokenize_fn) r0r1r6r2r3r7r8r9r:r;r<TcSsg|]}t|tqS) isinstancer).0callbackrBrB[/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/sentence_transformers/trainer.py sz7SentenceTransformerTrainer.__init__.. WANDB_PROJECTzsentence-transformersz@No `loss` passed, using `losses.CoSENTLoss` as a default option.csi|] \}}||qSrB) prepare_loss)rD dataset_nameloss_fnr0selfrBrF sz7SentenceTransformerTrainer.__init__..trainevalz,If the provided `loss` is a dict, then the `z"_dataset` must be a `DatasetDict`.z:If the provided `loss` is a dict, then all keys from the `z;_dataset` dictionary must occur in `loss` also. Currently, z occursz in `z_dataset` but not in `loss`.)4r) RuntimeErrorloggerinfor&rC ValueErrorr8call_model_initwarningswarn FutureWarningto_dict hub_model_idmodel_card_datamodel_id set_model_idr7rrtokenizedictr+r*super__init__can_return_lossanycallback_handlerr:osenviron setdefaultrr0itemsr4zipsetkeyssortedlenrIrrr5r add_callback on_init_endr1statecontrol)rMr0r1r2r3r4r5r6r7r8r9r:r;r<r?default_args_dictrJdatasetmissingmodel_card_callback __class__rLrFrdss             z#SentenceTransformerTrainer.__init__cstj|d}t|ds|St|jtr<|jD] \}}t|tjj s,|||j|<qt|dr9| |j||_q|St|jtjj sL|||_|St|jdrZ| |j||_|S)N)trialr4r0) rcrXhasattrrCr4rbrktorchr Moduleoverride_model_in_loss)rMr{r0keyrKryrBrFrXs"      z*SentenceTransformerTrainer.call_model_initcCs^ddlm}|D]"\}}|dkrt||r||_q t|tjjr,t||| ||q |S)Nrr,r0) sentence_transformersr-named_childrenrCr0r}r r~setattrr)rMr4r0r-namechildrBrBrFr s z1SentenceTransformerTrainer.override_model_in_losscCs*t|tjjr ||jS|||jSN)rCr}r r~todevice)rMr4r0rBrBrFrIs z'SentenceTransformerTrainer.prepare_loss dataset_dictcCs:|D]\}}d|jvr|d|gt|||<q|S)NrJ)rk column_names add_columnrp)rMrrrvrBrBrFadd_dataset_name_columns  z2SentenceTransformerTrainer.add_dataset_name_columnFinputsreturn_outputsc Cs~|dd}||\}}|j}t|tr|r||}|jjtjkr2t |dr2t |dr2| ||}|||}|r=|ifS|S)a$ Computes the loss for the SentenceTransformer model. It uses ``self.loss`` to compute the loss, which can be a single loss function or a dictionary of loss functions for different datasets. If the loss is a dictionary, the dataset name is expected to be passed in the inputs under the key "dataset_name". This is done automatically in the ``add_dataset_name_column`` method. Note that even if ``return_outputs = True``, the outputs will be empty, as the SentenceTransformers losses do not return outputs. Args: model (SentenceTransformer): The SentenceTransformer model. inputs (Dict[str, Union[torch.Tensor, Any]]): The input data for the model. return_outputs (bool, optional): Whether to return the outputs along with the loss. Defaults to False. Returns: Union[torch.Tensor, Tuple[torch.Tensor, Dict[str, Any]]]: The computed loss. If `return_outputs` is True, returns a tuple of loss and outputs. Otherwise, returns only the loss. rJNmoduler0) popcollect_featuresr4rCrbr1 parallel_moder NOT_PARALLELr|r) rMr0rrrJfeatureslabelsrKr4rBrBrF compute_loss#s   z'SentenceTransformerTrainer.compute_losscsg}|D]>}|dr|dtd n|dr$|dtd n|dr3|dtd nq|fdd |Dq|d d}||fS) aTurn the inputs from the dataloader into the separate model inputs & the labels. Example:: >>> list(inputs.keys()) ['return_loss', 'label', 'sentence_0_input_ids', 'sentence_0_token_type_ids', 'sentence_0_attention_mask', 'sentence_1_input_ids', 'sentence_1_token_type_ids', 'sentence_1_attention_mask'] >>> features, labels = self.collect_features(inputs) >>> len(features) 2 >>> list(features[0].keys()) ['input_ids', 'token_type_ids', 'attention_mask'] >>> list(features[1].keys()) ['input_ids', 'token_type_ids', 'attention_mask'] >>> torch.equal(labels, inputs["label"]) True _input_idsN input_ids_sentence_embeddingsentence_embedding _pixel_values pixel_valuescs,i|]\}}|r|td|qSr) startswithrp)rDrvalueprefixrBrFrNqs,z?SentenceTransformerTrainer.collect_features..label)endswithrpappendrkget)rMrrcolumnrrBrrFrRs    z+SentenceTransformerTrainer.collect_featuresrP ignore_keysmetric_key_prefixcsB|dur|n|j}t|trt|jtr||}t|||Sr)r3rCr+r4rbrrcevaluate)rMr3rrryrBrFrus z#SentenceTransformerTrainer.evaluate dataloader descriptionprediction_loss_onlyc stj|||||d}|jdur|S|jr4t|jtr4|dr4|ddt|j dkr2d}n|S| r;t nt t j||j}Wdn1sQwYt|ts_d|i}t| D]}||ds{||||d|<qe|j||S)N)rrrrreval_rrPr5_)rcevaluation_loopr5 is_in_trainrCr3rbrlistrnis_local_process_zerorr'loggingINFOr0rmetricsupdate) rMrrrrroutputevaluator_metricsrryrBrFrs0   z*SentenceTransformerTrainer.evaluation_loopc st|jdtstddSz|jj}r'|ddd}|jj t |Wn t y1Ynw|j}|jdj |_zt W|j}||_||jd_ S|j}||_||jd_ w)NrzLCould not load best model, as the model is not a `transformers`-based model.-rQ)rCr0rrUrVrsbest_model_checkpointrsplitr^set_best_model_stepint Exception auto_modelrc_load_best_model)rM checkpointstep full_modelloaded_auto_modelryrBrFrs*    z+SentenceTransformerTrainer._load_best_modelrvrJcCs@t|jddh@}rtd|r|dnddt|ddS)N return_lossrJz/The following column names are invalid in your  rSz dataset: zH. Avoid using these column names, as they are reserved for internal use.)rmrrWr)rMrvrJoverlaprBrBrFvalidate_column_namess  z0SentenceTransformerTrainer.validate_column_names batch_size drop_lastvalid_label_columns generatorcCsn|jjtjkrt|||||dS|jjtjkrt||||dS|jjtjkr5tt t t ||d||dSdS)N)rvrrrr)rvrrr)r)rr) r1 batch_samplerr$ NO_DUPLICATESr!GROUP_BY_LABELr BATCH_SAMPLERrrrangerp)rMrvrrrrrBrBrFget_batch_samplers,z,SentenceTransformerTrainer.get_batch_samplerrbatch_samplersseedcCs@|jjtjkrt||||dS|jjtjkrt||||dSdS)Nrvrrr)r1multi_dataset_batch_samplerr% ROUND_ROBINr# PROPORTIONALr")rMrvrrrrBrBrFget_multi_dataset_batch_samplersz:SentenceTransformerTrainer.get_multi_dataset_batch_samplercs,jdur tdj}jtjjrjjt|t r]| D] \}}j ||dq'tj t r>|}fdd|D}t|}j||jjd}n |j|jjjjjd}jjjjjjjj|d}d j_jt|fi|_jS) a@ Returns the training [`~torch.utils.data.DataLoader`]. Will use no sampler if `train_dataset` does not implement `__len__`, a random sampler (adapted to distributed training if necessary) otherwise. Subclass and override this method if you want to inject some custom behavior. Nz+Trainer: training requires a train_dataset.rJc *g|]}j|jjjjjdqSrrrrrr1per_device_train_batch_sizedataloader_drop_lastrrDrvr6rrMrBrFrGzCSentenceTransformerTrainer.get_train_dataloader..rr collate_fn num_workers pin_memorypersistent_workersprefetch_factorrF)r2rWr6r} Generatorr1r manual_seedrCr+rkrr4rbrvaluesr rrtrain_batch_sizerrdataloader_num_workersdataloader_pin_memorydataloader_persistent_workersdataloader_prefetch_factor accelerator even_batchespreparer_train_dataloader)rMr2rJrvrrdataloader_paramsrBrrFget_train_dataloadersP        z/SentenceTransformerTrainer.get_train_dataloader)r*Ncs |durjdurjdurtgStd|dur|nj}jtjjr1 jjt |t r`t j t rA|}fdd|D}t|}j||jjd}nj|jjjjjd}jjjjjjjj|d}dj_jt|fi|S) a Returns the evaluation [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: eval_dataset (`torch.utils.data.Dataset`, *optional*): If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. Nz-Trainer: evaluation requires an eval_dataset.c rr)rr1per_device_eval_batch_sizerrrrrBrFrGcrzBSentenceTransformerTrainer.get_eval_dataloader..rrrT)r3r5rrWr6r}rr1rrrCr+r4rbrrr rrrrrrrrrrrr)rMr3rrrrBrrFget_eval_dataloaderHsL       z.SentenceTransformerTrainer.get_eval_dataloader test_datasetcsjtjjrjjt|trQ|D] \}}j ||dqtj t r2 |}fdd| D}t| }j||jjd}n |j|jjjjjd}jjjjjjjj|d}dj_jt|fi|_jS)a Returns the training [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: test_dataset (`torch.utils.data.Dataset`, *optional*): The test dataset to use. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. rc rrrrrrBrFrGrzBSentenceTransformerTrainer.get_test_dataloader..rrrF)r6r}rr1rrrCr+rkrr4rbrrr rrrrrrrrrrrrrr)rMrrJrvrrrrBrrFget_test_dataloadersJ        z.SentenceTransformerTrainer.get_test_dataloaderr?cCsx|dur|n|jj}tj|ddtd||jj||jjd|j dur.|j |t |jtj |tdS)NT)exist_okzSaving model checkpoint to )safe_serialization)r1r?rhmakedirsrUrVr0save_pretrainedsave_safetensorsr7r}savepathjoinr)rMr? state_dictrBrBrF_saves  z SentenceTransformerTrainer._savecheckpoint_pathcCs(ddlm}||}|j|dS)Nrr,)rr-r0load_state_dictr)rMrr- loaded_modelrBrBrF_load_from_checkpoints z0SentenceTransformerTrainer._load_from_checkpointlanguagelicensetags model_namefinetuned_fromtasks dataset_tags dataset_argsc KsZ|sdS|r|jj||r|jj||r!|jj||jj|jj|ddS)N)r) is_world_process_zeror0r^ set_language set_licenseadd_tags_create_model_cardr1r?) rMr r r rrrrrvrkwargsrBrBrFcreate_model_cards z,SentenceTransformerTrainer.create_model_card) NNNNNNNNNNNr/Nr)r=r-)rr+r=r+)F)NNrP)r=N)Nr) NNNNNNNNN)6__name__ __module__ __qualname____doc__rr&r rstrr r~rr}rrrrrrr optim Optimizer lr_schedulerLambdaLRTensorrdrXrrIrrboolrrfloatrrrrrrrrr rr rrrrrr r __classcell__rBrBryrFr.-sTG  ~    / %   +   $ FED   r.)FrrhrY contextlibrtypingrrrrrrr r r}r torch.utils.datar r rr transformersrrrrtransformers.data.data_collatorrtransformers.integrationsrtransformers.trainerrtransformers.trainer_utilsrtransformers.training_argsr#sentence_transformers.data_collatorr sentence_transformers.evaluationrr'sentence_transformers.losses.CoSENTLossr sentence_transformers.model_cardr(sentence_transformers.models.Transformerrsentence_transformers.samplerrr r!r"r##sentence_transformers.training_argsr$r%r&sentence_transformers.utilr'r(r)datasetsr*r+ getLoggerrrU)sentence_transformers.SentenceTransformerr-r.rBrBrBrFs8 (