tjDP&dZddlmZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZddgZGd deZd d ed ed e d e d ed ze_deedeedeedeedeedeedededededededededededdf"d Zdeedeedeedeedeedeedededededededededededdf"d!Ze e" d&deedeedeedeedeedeed$edzdededededededededededdf$d%ZdS)'z)Implementation for the RMSprop algorithm.)castN)Tensor)_capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _maximize_doc _params_doc _to_scalar_use_grad_for_differentiable _view_as_real OptimizerParamsTRMSproprmspropceZdZ ddedeezd ed ed ed ed edededzdededdffd ZfdZdZ e ddZ xZ S)r{Gz?Gz?:0yE>rFNparamslralphaeps weight_decaymomentumcentered capturableforeachmaximizedifferentiablereturnc t|tr'|dkrtdd|kstd|d|kstd|d|kstd|d|kstd|d|kstd||||||||| | | d } t || dS) NrzTensor lr must be 1-elementgzInvalid learning rate: zInvalid epsilon value: zInvalid momentum value: zInvalid weight_decay value: zInvalid alpha value: ) rrrrr rr!r"r#r$) isinstancernumel ValueErrorsuper__init__)selfrrrrrrr r!r"r#r$defaults __class__s Q/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/optim/rmsprop.pyr+zRMSprop.__init__s& b& ! ! 000E+,$(-(8)>)))E*%   u\2 3 3 3   uV} - - -Z 1$$$++E2C,DEEEZ  4  z!2333r0c|d}|5tj5|}dddn #1swxYwY|jD]}g}g}g}g}g}g} |||||||| } t |||||| |d|d|d|d|d|d|d|d |d |d | |S) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrr r"r#r$r!) rrrrrr r"r#r$r!rY)'_accelerator_graph_capture_health_checkr= enable_gradr8rZr) r,closurelossrArSrTrUrWrVrXrYs r/r2z RMSprop.stepsv 44666  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&$ $ E-/ "$E(*K&(I13 (*K** $K  $;Gn%L">2z*z*i(z*$%56 .'#     ( s AA A) rrrrrFFNFFN) __name__ __module__ __qualname__rr?rboolr+r7rZrr2 __classcell__)r.s@r/rrs+" #$'+'+'+ FN'+ '+  '+  '+'+'+'+'+'+'+ '+'+'+'+'+'+R*111f"444"!44444r0aj Implements RMSprop algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \alpha \text{ (alpha)}, \: \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)}, \: centered, \: \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \: \textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t \hspace{8mm} \\ &\hspace{5mm} \tilde{v_t} \leftarrow v_t \\ &\hspace{5mm}if \: centered \\ &\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\ &\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\ &\hspace{5mm}if \: \mu > 0 \\ &\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} + g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\ &\hspace{5mm} else \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `lecture notes `_ by G. Hinton. and centered version `Generating Sequences With Recurrent Neural Networks `_. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma` is the scheduled learning rate and :math:`v` is the weighted moving average of the squared gradient. z Args: a0 lr (float, Tensor, optional): learning rate (default: 1e-2) alpha (float, optional): smoothing constant (default: 0.99) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) momentum (float, optional): momentum factor (default: 0) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance z z rrTrUrWrVrXrrrrrr r#r$r!rYr%c ,tjst|}t |D]U\}}||}tjsK|rIt}|jj |jj kr|jj |vstd|d||}| s|n| }||}|dz }| dkr| || }tj |}|rz(_multi_tensor_rmsprop..rs]  4 HMT[- - > !==      r0T)strictrgrhg?cpu)r5rirrjrl) r<rsr=rprqr allziprr"_group_tensors_by_device_and_dtypevaluesrlistrrMr _foreach_negis_cpu _foreach_add_r@ _foreach_add _foreach_mul__foreach_addcmul__foreach_lerp__foreach_addcmul_foreach_sqrt_ _foreach_sqrt_foreach_addcdiv_r' _foreach_mul _foreach_div_)"rrTrUrWrVrXrrrrrr r#r$r!rYgrouped_tensorsgrouped_params_grouped_grads_grouped_square_avgs_grouped_grad_avgs_grouped_momentum_buffer_list_grouped_state_steps__grouped_params grouped_gradsgrouped_square_avgsgrouped_state_stepsstate_and_gradsgrouped_momentum_buffer_listgrouped_grad_avgsr momentum_lrrs" @r/_multi_tensor_rmsproprVs& 6{{aDBCCC > & & ( ( Z 'H'J'J$    v{4@@@       !{\x{{{  BBB  Y0DkRO  " " $ $YWYW     ) d6lO<<T&\>:: "4<1EFF"4<1EFF  <,.ABO!||/3L"?00, &&'CDDD :$(f7I$J$J!&&'8999 . ;? ; ; ; ;  >!.}==M ~**,, 81DQ1G1N 8  #U\#e%D%D%DC        3Q 7 7 7 1   #M>VVVVV % 2!>!!!  /777   QY      * $T&\3E F F   !2M1u9 M M M(#%68IQSC   % % %  S ) ) ) )%&9::C  S ) ) ) a<<+/V ;,, (   .s?55() 1el##555555r0zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizers) rrrrrr r#r!r$rY) r=rprqrrOrrmrnrr)rrTrUrWrVrXr"r#r$r!rYrrrrrr rfuncs r/rrs0: > & & ( ( 55-855522  ^   1 Ne   7U59))++USTTT&uy--//&$%D   !%!r0)NFFFF)__doc__typingrr=r optimizerrrrr r r r r rrrrrr__all__rrr?rdrrrrFr0r/rs00 $ i gggggigggV+X     Y<BJ1 LJ1 <J1fJ1F| J1 v, J1 f J1 J1 J1 J1J1J1J1J1J1 !J1"#J1$ %J1J1J1J1ZCW LCW <CWfCWF| CW v, CW f CW CW CW CWCWCWCWCWCW !CW"#CW$ %CWCWCWCWL 1GHHH  AA LA <AfAF| A v, A f AD[AAAAA A !A" #A$%A&'A()A* +AAAIHAAAr0