tj$dZddlZddlZddlmZmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZd d lmZerddlmZGddeZdedefdZdedefdZGddeZGddeZ dS)a Distributed computing variable tracking classes for PyTorch Dynamo. This module implements variable tracking for distributed computing components: - Process Groups (for collective communication) - Device Meshes (for distributed tensor sharding) - Placement Types (for specifying distribution strategies) - Distributed Tensors and their operations - Backward hooks for distributed module operations These classes are responsible for tracking distributed operations during graph compilation while maintaining proper guards and handling distributed-specific behaviors. They ensure correct handling of distributed components like process groups, device meshes, and placement strategies while preserving proper semantics for distributed tensor operations in the compiled code. The implementation provides special handling for distributed package availability checks and proper tracking of distributed state and operations across processes. N)Any TYPE_CHECKING) BackwardState)compiled_autograd) trace_wrapped) unimplemented)%call_module_hooks_from_backward_state) GuardBuilder install_guard) AttrSource)VariableTracker)InstructionTranslatorBaseceZdZdZdededdffd ZdefdZede fdZ d ede e e ffd Z d Zd ede fd ZxZS)DistributedVariablea{ The base distributed variable that encapsulates common methods for the distributed objects (i.e. ProcessGroup, DeviceMesh, etc.). Concrete distributed objects could inherit this class and add object specific logic. i.e. It provides the check on the distributed package existence and hold the tracking value for the corresponding distributed object. valuekwargsreturnNc tjdi|tst ddddg||_dS)Nz+torch.distributed package is not available!zPThe PyTorch package doesn't include torch.distributed when building from source.zESet USE_DISTRIBUTED=1 to enable it when building PyTorch from source.gb_typecontext explanationhints)super__init__r is_availabler r)selfrr __class__s a/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/_dynamo/variables/distributed.pyrzDistributedVariable.__init__4si""6""""//11  En[      c*t|jSNtyperr!s r# python_typezDistributedVariable.python_typeADJr$c>tjSr&)torch distributedr rr$r#r z DistributedVariable.is_availableDs --///r$txc.t|jdfS)NF)hashr)r!r/s r# hash_implzDistributedVariable.hash_implIsDJ&&r$c*ddlm}|||||S)Nr)object_richcompare)object_protocolr4)r!r/otheropr4s r#richcompare_implz$DistributedVariable.richcompare_implLs+777777!!$E2666r$r6ct|to)||kSr&) isinstanceras_python_constant)r!r6s r#is_python_equalz#DistributedVariable.is_python_equalQs: uo . . H''))U-E-E-G-GG r$)__name__ __module__ __qualname____doc__rrr(r* staticmethodboolr tupleintr2r8objectr< __classcell__r"s@r#rr)s c S T       T    0$000\0'C'E#t)$4''''777  V         r$rrrc~tsdSddlm}t j|o||juS)NFr)DTensor)rr torch.distributed.tensorrIinspect isfunction from_local)rrIs r# is_from_localrNXsK  + + - -u000000  e $ $ D'2D)DDr$ctsdSddlm}m}m}m}m}|||||g}tj |o||vS)NFr)_get_group_size_by_name_get_group_tag_rank_not_in_group$_resolve_group_name_by_ranks_and_tagget_process_group_ranks) rr "torch.distributed.distributed_c10drPrQrRrSrTrKrL)rrPrQrRrSrTconstant_processgroup_functionss r#is_constant_pg_functionsrW`s  + + - -u , '#  e $ $ Q2Q)QQr$c\eZdZdZededefdZdefdZ ddde de ffd Z xZ S) WorldMetaClassVariablez Tracks torch.distributed.GroupMember and torch.distributed.group, which are instances of the metaclass _WorldMeta. rrc\|sdSddlm}t||uS)NFr) _WorldMeta)r rUr[r()clsrr[s r#is_group_member_typez+WorldMetaClassVariable.is_group_member_type}s>!! 5AAAAAAE{{j((r$c*t|jSr&r'r)s r#r*z"WorldMetaClassVariable.python_typer+r$r/rnamec@|dkrx|jstdt|jd}t|t jtj||j j |S|dkrx|jstdt|jd}t|t jtj||j j |St ||S)NWORLDzr?r@ classmethodrErBr]r(r*strrrjrFrGs@r#rYrYws ))D)))[) T    ---58- ----------r$rYc eZdZdZedddedededdf dZd ejj dededed e dd f fd Z dejj fd Z de fdZdddedeedeeefdef fd ZdddededefdZxZS)BackwardHookVariablezU Handles torch.utils.hooks.BackwardHook for module-level backward hooks. r/rmodule user_hooksuser_pre_hooksrc tjstddddgdtdtjjjf fd }|j |d\}|j |\ }|j |\ }|j d ||fi}tjjdd d |j j d <t||||S) Nz7Module-level backwards hooks require compiled autograd.rzREnable compiled autograd by setting torch._dynamo.config.compiled_autograd = True.rbw_staterc tjjdt jt t|ft jt t|fS)a  Rather than installing the user hooks in the graph (which don't survive AotAutograd), we install hooks that will call trace_wrapped in the backward pass that CompiledAutograd can turn into actual hook calls. N)fnrs hooks_name module_name)r-utilshooks BackwardHook functoolspartialrr )rsrwuser_hooks_nameuser_pre_hooks_names r#_in_graph_bw_hooksz7BackwardHookVariable.create.._in_graph_bw_hookssy;$11%%@!)#2$/ %%@!)#6$/  r$mod call_functionr example_value)rcompiled_autograd_enabledr rr-rxryrzoutputadd_backward_state_hook create_proxynodemetarn) r/rorprqrbw_state_proxy_proxyrwr}r~s @@@r#createzBackwardHookVariable.creates1!:  Qh      #  [  +        >')i&G&GPU&V&V# ^!#!B!B>!R!RQY>>zJJ &&         ,1;+<+I+I$PRTV+W+W (#E6:~NNNr$roptionsNc rtjdi|||_||_||_||_dS)Nr)rrrrorprq)r!rrorprqrr"s r#rzBackwardHookVariable.__init__sE ##7###  $,r$c|jSr&)rr)s r#as_proxyzBackwardHookVariable.as_proxys zr$c.tjjjSr&)r-rxryrzr)s r#r*z BackwardHookVariable.python_types{ --r$r_argsrcx|dvr|j||g|Ri|St||||S)N)setup_input_hooksetup_output_hook) _setup_hookr call_method)r!r/r_rrr"s r#rz BackwardHookVariable.call_methodsT < < <#4#B>t>>>v>> >ww""2tT6:::r$hook_method_namec ddlm}|||jd|||fiS)Nr) wrap_fx_proxyr)builderrrrr)r!r/rrrs r#rz BackwardHookVariable._setup_hooksc +*****}  I " " $--//2      r$)r=r>r?r@rArrr-fxProxyrrrr(r*rllistdictrrrFrGs@r#rnrns 9O '9O9O$9O( 9O 9O9O9O\9Ov -x~ -  -$ - ( -  -  - - - - - -%(..T.... ; ' ; ;?# ; S/)* ;  ; ; ; ; ; ; '             r$rn)!r@r{rKtypingrrr-%torch.fx.experimental._backward_staterrr_trace_wrapped_higher_order_oprexcr external_utilsr guardsr r rer rbrtorch._dynamo.symbolic_convertrrrErBrNrWrYrnrr$r#rs(%%%%%%%% ?????? ::::::BBBBBB00000000!!!!!!IHHHHHH, , , , , /, , , ^EEDEEEERFRtRRRR.%-%-%-%-%-0%-%-%-Pq q q q q ?q q q q q r$