tjUdZddlmZddlmZddlmZddlmZddl Z ddl m Z m Z m Z mZmZddlmZiZeed fed <gd ZGd d eZd ededdfdZdedededede jdede fdZdedededede jdede fdZedeededededee jzdede f dZde fdZede dedfd Z dS)!z This is an experimental new API for PyTorch Distributed. This is actively in development and subject to change or deletion entirely. This is intended as a proving ground for more flexible and object oriented distributed APIs. ) Generator)contextmanager) timedelta)ProtocolN)_current_process_group_set_process_group ProcessGroupReduceOpStore) rendezvousProcessGroupFactory _BACKENDS)r r r register_backend new_groupcurrent_process_group process_groupc@eZdZdZdededededejde de fd Z d S) r z%Protocol for process group factories.storerank world_sizetimeoutdevicekwargsreturnc dS)N)selfrrrrrrs V/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/distributed/_dist2.py__call__zProcessGroupFactory.__call__'s sN) __name__ __module__ __qualname____doc__r intrtorchrobjectr rrr rr r $st//      r namefuncrcR|tvrtd|d|t|<dS)z Register a new process group backend. Args: name: The name of the backend. func: The function to create the process group. Backend z already registeredN)r ValueError)r(r)s rrr2s6 y=D===>>>IdOOOr rrrrrrc lddlm}t|dkrtd|||||}|t |||}|t jj| |t jj|| tj dt jj|tj r8| tj dt jj||S)Nr)ProcessGroupGlooz'Gloo backend received unexpected kwargscpucuda)torch.distributedr.lenAssertionError_set_sequence_number_for_groupr _set_default_backend BackendTypeGLOO_register_backendr&rr0 is_available) rrrrrrr. backend_classpgs r _gloo_factoryr<@s&322222 6{{aFGGG$$UD*gFFM00222 eT: . .BL49:::!9!> NNN U\5:M z     L ,":"?    Ir c ddlm}|}||_|D]8\}} t ||st d|t||| 9|||||} | | |t|||} | tj j | |tj j | | S)Nr)ProcessGroupNCCLzUnknown option )r1r>Options_timeoutitemshasattrKeyErrorsetattrr4eager_connect_single_devicer r5r6NCCLr8) rrrrrrr>optskvr:r;s r _nccl_factoryrJ_s322222  # # % %DDM 1tQ 20Q0011 1a$$UD*dCCM00222--f555 eT: . .BL49:::!9!> NNN Ir glooncclbackendc |tvrtd|dtj|}t t t d\}}}||t||||||fi|S)aF Create a new process group with the given backend and options. This group is independent and will not be globally registered and thus not usable via the standard torch.distributed.* APIs. Args: backend: The backend to use for the process group. timeout: The timeout for collective operations. device: The device to use for the process group. **kwargs: All remaining arguments are passed to the backend constructor. See the backend specific documentation for details. Returns: A new process group. r+z not registeredzenv://)rr,r&rnextiterr set_timeout)rMrrrrrrs rrrs*i>E4 g W eT:w Q Q& Q QQr ctS)zn Get the current process group. Thread local method. Returns: The current process group. )rrr rrrs " # ##r r;)NNNc#Kt}t| dVt|dS#t|wxYw)zs Context manager for process groups. Thread local method. Args: pg: The process group to use. N)rr)r;prev_pgs rrrsX$%%Gr$ 7#####7####s 6A)!r$collections.abcr contextlibrdatetimertypingrr&torch._C._distributed_c10drrr r r torch.distributed.rendezvousr rdictstr__annotations____all__r rr%rr'r<rJrrrrr rr_s &%%%%%%%%%%% 433333/1 4** +000        (    3 &9 d        L   >    L   8''''''R R R %, R R  RRRR@$|$$$$ $l $y1A'B $ $ $ $ $ $r