tjo7UddlZddlZddlmZmZmZddlmZddlm Z m Z ddl Z ddl m Z ddl mZddlmZddlmZgZeeed<ejeZGd d ejZd eed efd ZdS)N)Callable CollectionMapping)deepcopy)Anyoverload)optim) ShardedTensor)FullyShardedDataParallel__all__c2eZdZdZ ddeeejezfde j de eee fdzde jdzdee dfd eee fd dfd Zdd Zd eee ffd ZedddZedegefd efdZddegefdzd edzfdZed eeje ffdZdeee fd dfdZdeee fd dfdZddZdeee fd eee ffdZdeee fd eee ffdZdS)_NamedOptimizera ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualified name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to be updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns an FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. Nnamed_parametersoptimizer_class param_groupsmoduleargs.kwargsreturnctjd||_|t ||_||jn|}||g|Ri||_||_ |,t|j |_ ntjddd|jD}g} |D]?} | dD]4} | |vrt!d| d| || 5@| |_ |jj|_dS) Nz'torch.distributed.optim._NamedOptimizerzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.) stacklevelci|]\}}|| Sr.0keyparams e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/distributed/optim/named_optimizer.py z,_NamedOptimizer.__init__..]sWWW:3E3WWWparamszExpect param name z% found in param group but is missing.)torch_C_log_api_usage_oncer_param_groups_checkdictrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrrrparams_for_optimizer param_to_keyr,grouprs r__init__z_NamedOptimizer.__init__?s $$%NOOO;G   """ $%5 6 6.:.BD ! ( ( * * *  */          &*4+@+E+E+G+G&H&HD # # MN     XW9N9T9T9V9VWWWL!# % C C"8_CCEL00(]]]]'--l5.ABBBB C '9D # O8r!c|j|jD]}t|tstdd|vrtd|d}t|tjr|g}t |}|D]@}t|tjs$tdt j|zA||d<dSdS)Nparam group must be a dictr"z#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is ) r isinstancer'AssertionErrorr#Tensorr* TypeErrortypename)r2 param_groupr"rs rr&z#_NamedOptimizer._param_groups_checkjs   (#0 / / !+t44G()EFFF;..()NOOO$X.fel33&$XFf#E%eU\::'8:?.:O:OP )/ H%%! ) ( / /r!cj}|d}fd|dD}g}|D]n}fd|dD}dt|i}|D]\}} |dkrt | ||<||o||dS)z Return the ``state_dict`` of the optimizer. Instead of using number to index parameters, we will use module fully qualified name (FQN) as the key. rc2i|]\}}j||Srr,)rst_key state_valr2s rr z._NamedOptimizer.state_dict..s7   !   #F +Y   r!statec*g|]}j|SrrA)rrr2s r z._NamedOptimizer.state_dict..s!VVVU$1%8VVVr!r")rDr)r) state_dictr/sortedrr1_post_state_dict) r2rGr ret_state ret_groupsr5 param_keys ret_groupkvs ` rrGz_NamedOptimizer.state_dict}s_//11 !.1     %/%8%>%>%@%@    ! ) )EVVVVeHoVVVJ!6*#5#56I  / /1==#+A;;IaL   i ( ( ( ($$y*%U%UVVVr!closurecdSNrr2rPs rstepz_NamedOptimizer.steps25#r!cdSrRrrSs rrTz_NamedOptimizer.steps;>3r!c8|j|S)z Perform a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. rP)r)rTrSs rrTz_NamedOptimizer.steps##G#444r!c|jjSrR)r)rD)r2s rrDz_NamedOptimizer.states $$r!rGc  |j}||}|d}|d}t|dkrt dt |jD]B\}}||vr t||t||kr>t dt||d|dt||||D]\}}|||vrt d|d|d|||} t|trt| tstt| } t| } | | krt d | d | d |d |t| | D]6\} } | j | j 75t|t jrJt| t jst|| t%| |||<D|d }|d }i}|D])}t'|d}||t)|<*i}|D]A}g}|dD]"}||j|#||t)|<B|D]\}}||vr ||}t|t|kr3t dt|d|d t|d|D]:}||vrt d|d|d|dkrt%||||<;|j|dS)a Define the default behavior to load a state_dict for ``_NamedOptimizer``. Sample Code ``` my_model = MyModule() optimizer = _NamedOptimizer(my_model.named_parameters(), Adagrad) ... optim_state_dict = optimizer.state_dict() ... ... optimizer.load_state_dict(optim_state_dict) ... ``` Args: state_dict (dict[str, Any]) : A ``state_dict`` to load into the optimizer. Note that this state dict update is performed in place. .. note:: PyTorch is using lazy init to initialize the optim states. So it is possible that there is no optim state when user call ``load_state_dict`` and for ``_NamedOptimizer`` we make it stricter that users can only call ``load_state_dict`` after the state is initialized. By doing this, we can validate the optim ``state_dict`` to be loaded. rDrzJExpects the optim to be initialized before load but found not initialized.zExpects equal length as z for parameter z but found: zExpects state z but not found.z"Expects equal number of shards as z but found z for /rr"z"Expects equal param_group size as z for group .zExpects group key z to be in group z in `state_dict` but is missing.N)r)rG_pre_load_state_dictlenr0 enumerater,r/r9r r: local_shardsziptensordetachcopy_r#r;rr*_gen_param_group_keyr1load_state_dict)r2rGnew_state_dictrD new_stateidx param_key state_keyrC src_state_val num_shardsnum_new_shardsshard src_shardsrc_param_groupsnew_param_groups src_group_mapr5rL new_group_map new_group group_key src_grouprNs rrez_NamedOptimizer.load_state_dicts63355..z:: 7#"7+ y>>Q  \ ((?@@" H" HNC%%5#$$IcN(;(;;; Bs9S>/B/BBBS\BBjmnst}n~jjBB)2#(<(<(>(> H H$ 9E)$444$]]]9]]]!&i 0 ; i77H%m]CC-,,!$Y%;%;%=%=!>!>J%()C)C)E)E%F%FN!^33(EEE\fEEmvEEzCEE-0!..00-2L2L2N2N--FF(y ++--33I4DEEEEF 5<88H%mU\BB-,,$$&&,,];;;;080G0GIcN9--3 H8&n5).9 % D DEeHo..J>CM.z:: ; ; ) H HIJ&x0 F F !!$"9)"DEEEE>GM.z:: ; ;$1$7$7$9$9 : : Iy --%i0I9~~Y// {Y{{T]{{jmnwjxjx{{{ : :I%%$kQkk kkk==#+IaL#9#9IaL  : ''77777r!r>ct|tstd|d}t|tjr|g|d<nt ||d<d|jD}|dD]5}||vrtd|j ||6|j ||j j |_ dS)z Add a param group to the :class:`_NamedOptimizer` s `param_groups`. Warning: This API is still in development and subject to change. r8r"ci|]\}}|| Srrrs rr z3_NamedOptimizer.add_param_group..#sSSSzsEsSSSr!z%some parameters are not in the moduleN)r9r'r:r#r;r*rr/r0r,r1r)add_param_groupr)r2r>r"r4rs rryz_NamedOptimizer.add_param_groups +t,, ? !=>> >X& fel + + 1%+HK ! !$(LLK !SST5J5P5P5R5RSSS  * @ @EL(( !HIII  # * *<+> ? ? ? ? '' 444 O8r!c|jD]A}|jr8tj|}tj||_B|ddS)z Run a dummy optimizer step, which allows us to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. NrW) rr( requires_gradr# zeros_likeautogradVariablegradrT)r2rts r init_statez_NamedOptimizer.init_state-so *1133 8 8E" 8$U++"^44Q77  $ r!c~t|jtr"tj|j|j|dS|S)NT)is_named_optimizer)r9rFSDPoptim_state_dict_to_loadr)r2rGs rr\z$_NamedOptimizer._pre_load_state_dict:sG dk4 ( ( 0 T_jT r!czt|jtr tj|j|j||SrR)r9rroptim_state_dictr)rs rrIz _NamedOptimizer._post_state_dictCs8 dk4 ( ( L  !$+t K K Kr!)NN)rNrR)rPNrN)__name__ __module__ __qualname____doc__rstrr#r;r r OptimizerrrnnModuletupler'r6r&rGrrTrfloatpropertyrDreryrr\rIrr!rrrs))^>B#' )9)9!#u|m'C"CD)9)9!c!23d: )9  D )9 S#X )9sCx.)9 )9)9)9)9V////&WDcNWWWW45555X5 >HRY/>E>>>X>55HRY/$65%$,5555%wu|S01%%%X%h8$sCx.h8Th8h8h8h8T9738+<999992     tCH~$sCx.4S>d38nr!rrLrcFdt|S)zFConcatenate all param keys as a unique identifier for one param group.rZ)joinrH)rLs rrdrdKs 88F:&& ' ''r!)loggingr-collections.abcrrrcopyrtypingrrr#torch.nnrr 'torch.distributed._shard.sharded_tensorr torch.distributed.fsdpr rr r*r__annotations__ getLoggerrloggerrrrdrr!rrs?9999999999 AAAAAACCCCCCc  8 $ $uuuuueouuup (T#Y(3((((((r!