tj; d Z ddlZddlZddlZddlmZ ddlZddlmZ ddl m
Z
m Z m Z ddl
mZ ddlmZ ej e Ze G d d
Z G d d Ze G d
de Z G d d Z G d de Z G d de Z G d de Z G d de Z G d dee Z dS )zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)tqdm )IntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc e Zd ZU dZdZeed< dZeed< dZ eed< dZ
eed< dZ eed< dZ eed < d
Z
ed
z ed < dZeed < dZeed
< dZeed< d
Zeeeef ed< d
Zed
z ed< d
Zed
z ed< d
Zed
z ed< dZeed< dZeed< dZeed< d
Z ed
z ed< d
Z eeeez ez ez f d
z ed< d
Z ed d
z ed< d Z d efd Z e!d efd Z"d Z#d! Z$d
S )" TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`list[dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_global_step (`int`, *optional*):
When tracking the best model, the step at which the best metric was encountered.
Used for setting `best_model_checkpoint`.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`list[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevant callbacks should implement a `state` and `from_state` function.
r epoch global_step max_stepsi
logging_steps
eval_steps
save_stepsNtrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_global_stepbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc | j g | _ | j i | _ d S t | j t rd S i }| j D ]}t |t s t dt
| |j j }||v rUt || t s || g||< ||
| | ||< || _ d S )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r!
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr! callbacknames [/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/trainer_callback.py
__post_init__zTrainerState.__post_init__t s, # !D " *&(D # # #
/
6
6 9 D "$ 3
@
@!(_>> #yimnviwiwyy )2--- &&8&>EE N4Ft4L3M*40&t,33HNN4D4DEEEE/7~~/?/?&t,,&8D # # # json_pathc t j t j | dd dz }t |dd 5 }| | ddd dS # 1 swxY w Y dS ) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r- r3 json_stringfs r0 save_to_jsonzTrainerState.save_to_json s j!3D!9!9!tTTTW[[
)S7
3
3
3 !q
GGK ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s A""A&)A&c t |d 5 }| }ddd n # 1 swxY w Y | di t j | S )z3Create an instance from the content of `json_path`.r: r; N )rA readr= loads)clsr3 rD texts r0 load_from_jsonzTrainerState.load_from_json s )g
.
.
. !6688D s&&TZ%%&&&s 377c dD ]H}t || d }|1|dk rt j ||z }t | | d| IdS )z
Calculates and stores the absolute value for logging,
eval, and save steps based on if it was a proportion
or not.
)r
evalsave_stepsNr )getattrmathceilsetattr)r- argsr step_kind num_stepss r0
compute_stepszTrainerState.compute_steps st 5 ? ?I &:&:&:;;I$ q== $ )i*? @ @I 2 2 2I>>> ? ?r2 c |j &|j | |j | _ d| _ |ddlm} || | _ || _ || _ | | _ | | _ dS )zI
Stores the initial training references needed in `self`
Nr ) hp_params)
hp_name_trialr r transformers.integrationsrZ r r r r )r- trainerr r trialrZ s r0 init_training_referencesz%TrainerState.init_training_references s ? &7>+E &oogn = =DO ; ; ; ; ; ; ) % 0 0D " 0 %,%B%B%D%D"%,%B%B%D%D"""r2 )%r)
__module__ __qualname____doc__r
float__annotations__r intr r r r r r r r r r* r$ strr r r r boolr r r r r! r1 rE classmethodrL rX r` rG r2 r0 r r " s* 9 9v E5KIs M3 JJ#'cDj''' c !"3"""J*.Kd3:&'... $K $$$#'cDj'''(,3:,,,"&4&&&"&4&&&"'4''' !Jd
!!!?CL$sC%K# - 445<CCC9= ./$6===9 9 96!c ! ! ! ! 's ' ' ' [' ? ? ?E E E E Er2 r c 4 e Zd ZdZdefdZed ZdS )r% aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc t d )NzInternal class that just calls the list of callbacks in order.c g | _ |D ]}| | || _ || _ || _ || _ d | _ d | _ t d | j D s$t
d| j z d S d S )Nc 3 @ K | ]}t |t V d S N)r# DefaultFlowCallback.0cbs r0 z+CallbackHandler.__init__.. s- PP2:b"566PPPPPPr2 zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r- r r r r r r s r0 __init__zCallbackHandler.__init__ s " "B b ! ! ! !
0 " ( $ # PP PPPPP NN$ $%
r2 c 0 t |t r
| n|}t |t r|n|j }|d | j D v r)t d| ddz | j z | j | d S )Nc g | ] }|j
S rG )r( )r cs r0
z0CallbackHandler.add_callback.. s <<< <<>N))"--- III "
r2 c t |t r9| j D ]/}t || r | j | d S 0d S | j | d S r r r s r0 remove_callbackz CallbackHandler.remove_callback s h % % ,n
b(++ N))"---FF
N ! !( + + + + +r2 c J d d | j D S )Nr8 c 3 . K | ]}|j j V d S r )r( r) r s r0 r z0CallbackHandler.callback_list.. s' HH2 .HHHHHHr2 )joinr rn s r0 r z CallbackHandler.callback_list s% yyHH HHHHHHr2 rU r, r c $ | j d|||fi |S )Nr
call_eventr s r0 r zCallbackHandler.on_init_end " t}dE7MMfMMMr2 c 2 d|_ | j d|||fi |S )NFr )rx r r s r0 r z CallbackHandler.on_train_begin s+ ',$ t /ugPPPPPr2 c $ | j d|||fi |S )Nr r r s r0 r z CallbackHandler.on_train_end " t~tUGNNvNNNr2 c 2 d|_ | j d|||fi |S )NFr )ry r r s r0 r z CallbackHandler.on_epoch_begin s+ $)! t /ugPPPPPr2 c $ | j d|||fi |S )Nr r r s r0 r z CallbackHandler.on_epoch_end r r2 c N d|_ d|_ d|_ | j d|||fi |S )NFr )r| r{ rz r r s r0 r z CallbackHandler.on_step_begin s: ""' # teWOOOOOr2 c $ | j d|||fi |S )Nr r r s r0 r z%CallbackHandler.on_pre_optimizer_step s$ t 6eWWWPVWWWr2 c $ | j d|||fi |S )Nr r r s r0 r z!CallbackHandler.on_optimizer_step s# t 2D%SSFSSSr2 c $ | j d|||fi |S )Nr r r s r0 r z CallbackHandler.on_substep_end s# t /ugPPPPPr2 c $ | j d|||fi |S )Nr r r s r0 r zCallbackHandler.on_step_end r r2 c 6 d|_ | j d|||fd|i|S )NFr r )r{ r r s r0 r zCallbackHandler.on_evaluate
s0 "' t}dE7^^G^W]^^^r2 c ( | j d|||fd|i|S )Nr r r r s r0 r zCallbackHandler.on_predict s( t|T5']]7]V\]]]r2 c 2 d|_ | j d|||fi |S )NFr )rz r r s r0 r zCallbackHandler.on_save s* # ty$wII&IIIr2 c 6 d|_ | j d|||fd|i|S )NFr logs)r| r )r- rU r, r r r s r0 r zCallbackHandler.on_log s/ " txugSSDSFSSSr2 c $ | j d|||fi |S )Nr r r s r0 r z"CallbackHandler.on_prediction_step s# t 3T5'TTVTTTr2 c $ | j d|||fi |S )Nr r r s r0 r z CallbackHandler.on_push_begin s" teWOOOOOr2 c | j D ]D} t || |||f| j | j | j | j | j | j d|}||}E|S )N)r r r r r r )r rQ r r r r r r )r- eventrU r, r r r. results r0 r zCallbackHandler.call_event s ! !H-WXu-- j!%!6 . ! .!%!6 $ 4 F ! r2 N) r) ra rb rc r r r r propertyr r r rw r r r r r r r r r r r r r r r r r rG r2 r0 r r s HH & " " "
, , , I I XIN 1 N, NQ_ N N N NQ#4 Q\ QTb Q Q Q QO!2 O< OR` O O O OQ#4 Q\ QTb Q Q Q QO!2 O< OR` O O O OP"3 PL PSa P P P P X*; XL X[i X X X XT&7 T TWe T T T TQ#4 Q\ QTb Q Q Q QN 1 N, NQ_ N N N N_ 1 _, _Q_ _ _ _ _^ 0 ^ ^P^ ^ ^ ^ ^J - Jl J^ J J J JT, T\ TN T T T TU'8 U UXf U U U UP"3 PL PSa P P P P r2 r c : e Zd ZdZdededefdZdededefdZdS )r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rU r, r c l |j dk r|j rd|_ |j t j k r|j |j z dk rd|_ |j t j k r*|j |j z dk r|j |j k rd|_
|j t j k r%|j
dk r|j |j
z dk rd|_ |j |j k rbd|_ |j t j k r*|j |j z dk r|j |j k rd|_
|j t j k rd|_ |S )Nr Tr )r logging_first_stepr| logging_strategyr STEPSr
eval_strategyr
eval_delayr{
save_strategyr r rz r rx r s r0 r z DefaultFlowCallback.on_step_end8 sL ! !d&= !!%G $4$: : :u?PSXSf?fjk?k?k!%G
"2"8 8 8!E$44995#444&*G #
,"4 4 4 1$$!E$4499"&G / /+/G ( "&6&<<<%(88A==Ou'888*.'!\%777&*#r2 c |j t j k rd|_ |j t j k r|j |j k rd|_ |j t j k rd|_
|S )NT) r r EPOCHr| r r r
r{ r r rz r s r0 r z DefaultFlowCallback.on_epoch_end` sf $4$: : :!%G !1!7 7 7DOu{ z/. Consider increasing `max_str_len` if needed.].4gr )
r r rq r# rg r r rd poprB )
r- rU r, r r r shallow_logsrs rt _s
r0 r zProgressCallback.on_log s & 74+<+H L
(
(1 a%% (#a&&43C*C*CHA H H4K[ H H H !OO 5)) ()*jjLOO&'LOO t44A # #C $5$5 6 6 6 6 6! 7 7+H+Hr2 c X |j r"| j d | _ d S d S r )r r r r s r0 r z ProgressCallback.on_train_end s: & % # # % % % $D % %r2 )r r )
r) ra rb rc rf r r r r r r r r rG r2 r0 r r p s
' 'C ' ' ' '
2 2 2
* * * *' ' ' ' ' ' 7 7 7 7&% % % % %r2 r c e Zd ZdZddZdS )PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc | dd }|j r1| d | D }t | d S d S )Nr c L i | ]!\ }}|t |t r|d n|"S )r )r# rd )r rs rt s r0
z*PrinterCallback.on_log.. s5 ```41a*Q*>*> EqJJJA```r2 )r
r rq print)r- rU r, r r r r s r0 r zPrinterCallback.on_log s_ HH\4 ( ( & ``SWS]S]S_S_``` $KKKKK r2 r )r) ra rb rc r rG r2 r0 r r s2 r2 r c H e Zd ZdZd
dededz fdZd Zd Zd
Z d e
fd Z dS )EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
r early_stopping_patienceearly_stopping_thresholdNc 0 || _ || _ d| _ d S )Nr r r early_stopping_patience_counter)r- r r s r0 r z EarlyStoppingCallback.__init__ s '>$(@%/0,,,r2 c |j r t j n t j }|j 1 |||j r)t ||j z
| j k r d| _ d S | xj dz
c_ d S )Nr r )greater_is_betternpgreaterlessr absr r )r- rU r, r metric_valueoperators r0 check_metric_valuez(EarlyStoppingCallback.check_metric_value s} !%!7D2::RW $ H\5#4 5 5 %L5#44558UUU34D 0 0 0 0 0A 5 0 0 0 0r2 c |j st d |j
J d |j t
j k s
J d d S )NzUsing EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.zBEarlyStoppingCallback requires metric_for_best_model to be definedzAEarlyStoppingCallback requires IntervalStrategy of steps or epoch)load_best_model_at_endr r metric_for_best_modelr r NOr s r0 r z$EarlyStoppingCallback.on_train_begin sp * NN^
)55 P 655 !%5%8888 O 98888r2 c |j }| d sd| }| | }| t d| d d S | |||| | j | j k r d|_ d S d S )Neval_z@early stopping required metric_for_best_model, but did not find z so early stopping is disabledT) r*
startswithgetr r r' r r rx )r- rU r, r r r metric_to_checkr% s r0 r z!EarlyStoppingCallback.on_evaluate s 4 ))'22 8 7o 7 7O {{?33 NN Sb
F eWlCCC /43O O O+/G ( ( ( P Or2 rk c 4 | j | j dd| j idS )N)r r r r r rn s r0 r, zEarlyStoppingCallback.state s8 ,0+G,0,I
243W
r2 )r r ) r) ra rb rc rf rd r r' r r r$ r, rG r2 r0 r r s 1 1 1SX[_S_ 1 1 1 1 6 6 6
0 0 0"
t
r2 r ) rc r? r= rR r numpyr! tqdm.autor
trainer_utilsr r r
training_argsr utilsr
get_loggerr) r r r% rw r r r r r r rG r2 r0 r8 s ! ! ! ! ! ! E E E E E E E E E E , , , , , ,
H % % WE WE WE WE WE WE WE WEt) ) ) ) ) ) ) )X :
:
:
:
:
_ :
:
:
zC C C C C C C C LC C C C Co C C CL: : : : :/ : : :zG% G% G% G% G% G% G% G%T
o
I
I
I
I
I
O_ I
I
I
I
I
r2