tj pddlmZddlZddlmZddlmZddlm Z ej e Z ddd Z dd ZdS)) annotationsN)logging)is_dist_initializedFtensor torch.Tensor with_gradboolreturnctr|r%tjj}n[t j}fdt|D}t j|t j}||<tj |dSd}t |S)ap Gathers a tensor from each distributed rank into a list. Always retains gradients for the local rank's tensor, and optionally retains gradients for the gathered tensors if `with_grad` is True. Args: tensor (torch.Tensor): The tensor to gather from each rank. with_grad (bool, optional): If True, the local rank's tensor retains its gradients. Defaults to False. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. c8g|]}tjS)torch zeros_like).0_rs d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/sentence_transformers/util/distributed.py zall_gather.. s$TTTQ 0 8 8TTTr)dimzTrying to gather while torch.distributed is not available or has not been initialized, returning the original (local) tensor. This is expected if you are only using one GPU; consider not using gathering to remove this warning.) rr distributednn all_gatherdistget_world_sizerangeget_rankcatlogger warning_once)rr gathered_tensors world_size local_rankwarnings` rrr s 2  2$03>>vFF  ,..JTTTT% BSBSTTT  O,f 5 5 5J+1 Z (y)q1111 S     Mrc$t|dS)a Gathers a tensor from each distributed rank into a list, retaining gradients for the local rank's tensor. Args: tensor (torch.Tensor): The tensor to gather from each rank. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. T)r )r)rs rall_gather_with_gradr&4s f - - --r)F)rrr r r r)rrr r) __future__rrtorch.distributedrrtransformers.utilsr environmentr get_logger__name__rrr&rrrr-s"""""" &&&&&&,,,,,,  H % %$$$$$N . . . . . .r