tja+ddlZddlZddlmZmZmZmZmZmZddl m Z d dddZ d dZ d dZ dd Zdddd Zdddd ZdS)N)_flatten_dense_tensors_get_device_index_handle_complex_reorder_tensors_as _take_tensors_unflatten_dense_tensors)nccl)outct|}|du|duz std|d||,d|D}tj||Stj||S)aBroadcasts a tensor to specified GPU devices. Args: tensor (Tensor): tensor to broadcast. Can be on CPU or GPU. devices (Iterable[torch.device, str or int], optional): an iterable of GPU devices, among which to broadcast. out (Sequence[Tensor], optional, keyword-only): the GPU tensors to store output results. .. note:: Exactly one of :attr:`devices` and :attr:`out` must be specified. Returns: - If :attr:`devices` is specified, a tuple containing copies of :attr:`tensor`, placed on :attr:`devices`. - If :attr:`out` is specified, a tuple containing :attr:`out` tensors, each containing a copy of :attr:`tensor`. NzFExactly one of 'devices' and 'out' must be specified, but got devices=z and out=c,g|]}t|Sr.0ds T/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/nn/parallel/comm.py zbroadcast..+!999A$Q''999)r RuntimeErrortorch_C _broadcast_broadcast_out)tensordevicesr s r broadcastrs*V $ $F _ -  lU\ l lgj l l   99999x""67333x&&vs333rctd|D}d|D}tj|||S)a.Broadcast a sequence of tensors to the specified GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations. Args: tensors (sequence): tensors to broadcast. Must be on the same device, either CPU or GPU. devices (Iterable[torch.device, str or int]): an iterable of GPU devices, among which to broadcast. buffer_size (int): maximum size of the buffer used for coalescing Returns: A tuple containing copies of :attr:`tensor`, placed on :attr:`devices`. c,g|]}t|Sr rrs rrz'broadcast_coalesced..As!555 ##555rc,g|]}t|Sr rrts rrz'broadcast_coalesced..B 333aq!!333r)rr_broadcast_coalesced)tensorsr buffer_sizes rbroadcast_coalescedr)2sD65W555G337333G 8 ( ('; G GGrc $ t|d}|d}d t|D]\}}|jjdkrt d|d||kr| ||krhdd |D}dd |D}td |d |d |É tdt|dkr|dStj |r2tj| }tj|| ntj| jj|} fdt|D} | | d|dz}| ddD],} || |d-|S)aSum tensors from multiple GPUs. All inputs should have matching shapes, dtype, and layout. The output tensor will be of the same shape, dtype, and layout. Args: inputs (Iterable[Tensor]): an iterable of tensors to add. destination (int, optional): a device on which the output will be placed (default: current device). Returns: A tensor containing an elementwise sum of all inputs, placed on the :attr:`destination` device. T)optionalrNcpuz7reduce_add expects all inputs to be on GPUs, but input z is on CPUxc34K|]}t|VdSNstrrr-s r zreduce_add..`s(66a3q66666666rc34K|]}t|VdSr/r0r2s rr3zreduce_add..as(;;1A;;;;;;rzinput z has invalid size: got z, but expected zLreduce_add expects destination to be on the same GPU with one of the tensors)outputrootc&g|] \}}|k |Sr r )rir$ root_indexs rrzreduce_add..rs"FFFAa:oo1ooor)device non_blocking)rsize enumerater;typeAssertionError get_devicejoin ValueErrorrlenr is_availabler empty_likereducetoadd_) inputs destination input_sizer9inpgotexpectedresultdestination_devicenonroototherr:s @r reduce_addrTFsP$K$???K!!JJF##  3 :?e # # W!WWW  >>  { * *J 88:: # #((66388::66666Cxx;; ;;;;;HQQQ3QQxQQ  $  Z    6{{aay   P!&"455 F6 ;;;;;"\&*<*C*H+VVFFFF6!2!2FFF #gajmm%D'4' '  QRR[ P PE KK(:NN O O O O Mrcd|D}g}g}t|ddiD]}td|DrAt||}||||d\t||dD]5\}} || jr| n| 6||ddƈfd|D} t| ddiD]Q} d | D} t| |} t | | dD]} || jRtt||S) a\Sum tensors from multiple GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations. Args: inputs (Iterable[Iterable[Tensor]]): iterable of iterables that contain tensors from a single device. destination (int, optional): a device on which the output will be placed (default: current device). buffer_size (int): maximum size of the buffer used for coalescing Returns: A tuple of tensors containing an elementwise sum of each group of inputs, placed on the ``destination`` device. cg|]}gSr r )r_s rrz(reduce_add_coalesced..s 4 4 4 4 4 4rstrictTc3$K|] }|jV dSr/) is_sparser#s rr3z'reduce_add_coalesced..s$33qq{333333rr)rXc0g|]}t|Sr )r)rr'r(s rrz(reduce_add_coalesced..s# M M MGM'; / / M M Mrc,g|]}t|Sr )r)rchunks rrz(reduce_add_coalesced..s.   .3 "5 ) )   r) zipallrTappendrZto_denserdatatupler)rJrKr( dense_tensorsr6 ref_ordertensor_at_gpusrPcollr$itrschunks flat_tensors flat_results ` rreduce_add_coalescedrm|s&!5 4V 4 4 4M FIv3d3333 33N333 3 3 3 <AJJLLLQ????   ]1-b1 2 2 2 2 M M M M} M M MDt)D)) " "  7=   !{;; )+vayAA " "A MM!& ! ! ! !  " $VY77 8 88rc Jt|}|.rrzI'devices' must not be specified when 'out' is specified, but got devices=zQ'chunk_sizes' must not be specified when 'out' is specified, but got chunk_sizes=)rrdrr_scatterr _scatter_out)rr chunk_sizesdimstreamsr s rscatterrusDV $ $F {99999UX&&vw S'RRSSS  e\cee   "qdoqq UX**63WEEFFFrc2d|D}|U|dkrtjdtdt|dd}tj|||S|td |tj|||S) aGathers tensors from multiple GPU devices. Args: tensors (Iterable[Tensor]): an iterable of tensors to gather. Tensor sizes in all dimensions other than :attr:`dim` have to match. dim (int, optional): a dimension along which the tensors will be concatenated. Default: ``0``. destination (torch.device, str, or int, optional): the output device. Can be CPU or CUDA. Default: the current CUDA device. out (Tensor, optional, keyword-only): the tensor to store gather result. Its sizes must match those of :attr:`tensors`, except for :attr:`dim`, where the size must equal ``sum(tensor.size(dim) for tensor in tensors)``. Can be on CPU or CUDA. .. note:: :attr:`destination` must not be specified when :attr:`out` is specified. Returns: - If :attr:`destination` is specified, a tensor located on :attr:`destination` device, that is a result of concatenating :attr:`tensors` along :attr:`dim`. - If :attr:`out` is specified, the :attr:`out` tensor, now containing results of concatenating :attr:`tensors` along :attr:`dim`. c,g|]}t|Sr r"r#s rrzgather..r%rNr[zjUsing -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".) stacklevelT) allow_cpur+zQ'destination' must not be specified when 'out' is specified, but got destination=) warningswarn FutureWarningrrr_gatherr _gather_out)r'rsrKr s rgatherrs4437333G { "   M@      ( tdSSS xk:::  "qdoqq x##GS#666rr/)r)Nr)NNrN)rN)r{r torch._utilsrrrrrr torch.cudar rr)rTrmrurr rrrs0 4444444DHHHH(3333l,9,9,9,9^0GPT0G0G0G0G0Gf*7D*7*7*7*7*7*7*7r