tj3ddlZddlmZddlmZddlZddlmZddl m Z ddl m Z dgZ edd ZGd de eZdS) N)Iterator)TypeVar)Dataset)SamplerDistributedSampler_T_coT) covariantceZdZdZ ddededzdedzd ed ed ed dfd Zd ee fdZ d efdZ ded dfdZ dS)ra' Sampler that restricts data loading to a subset of the dataset. It is especially useful in conjunction with :class:`torch.nn.parallel.DistributedDataParallel`. In such a case, each process can pass a :class:`~torch.utils.data.DistributedSampler` instance as a :class:`~torch.utils.data.DataLoader` sampler, and load a subset of the original dataset that is exclusive to it. .. note:: Dataset is assumed to be of constant size and that any instance of it always returns the same elements in the same order. Args: dataset: Dataset used for sampling. num_replicas (int, optional): Number of processes participating in distributed training. By default, :attr:`world_size` is retrieved from the current distributed group. rank (int, optional): Rank of the current process within :attr:`num_replicas`. By default, :attr:`rank` is retrieved from the current distributed group. shuffle (bool, optional): If ``True`` (default), sampler will shuffle the indices. seed (int, optional): random seed used to shuffle the sampler if :attr:`shuffle=True`. This number should be identical across all processes in the distributed group. Default: ``0``. drop_last (bool, optional): if ``True``, then the sampler will drop the tail of the data to make it evenly divisible across the number of replicas. If ``False``, the sampler will add extra indices to make the data evenly divisible across the replicas. Default: ``False``. .. warning:: In distributed mode, calling the :meth:`set_epoch` method at the beginning of each epoch **before** creating the :class:`DataLoader` iterator is necessary to make shuffling work properly across multiple epochs. Otherwise, the same ordering will be always used. Example:: >>> # xdoctest: +SKIP >>> sampler = DistributedSampler(dataset) if is_distributed else None >>> loader = DataLoader(dataset, shuffle=(sampler is None), ... sampler=sampler) >>> for epoch in range(start_epoch, n_epochs): ... if is_distributed: ... sampler.set_epoch(epoch) ... train(loader) NTrFdataset num_replicasrankshuffleseed drop_lastreturnc|5tjstdtj}|5tjstdtj}||ks|dkrt d|d|dz d||_||_||_d|_ ||_ |j r\t|j|jzdkr Pc$,//$2CCqHH $yT\""T%66$:KK  D   $yT\):):T=N)NOOD *T->>  c|jrptj}||j|jztjt|j| }n.ttt|j}|j sk|j t|z }|t|kr||d|z }nB||tj|t|z zd|z }n|d|j }t||j kr(t!dt|d|j d||j|j |j}t||jkr(t!dt|d|jdt)|S)N) generatorzNumber of indices (z) does not match total_size ()zNumber of subsampled indices (z) does not match num_samples ()rtorch Generator manual_seedrrrandpermrr tolistlistrangerr rrAssertionErrorr r riter)r!gindices padding_sizes r"__iter__zDistributedSampler.__iter__ks < 5!!A MM$)dj0 1 1 1nS%6%6!DDDKKMMGG5T\!2!23344G~ 1?S\\9Ls7||++7=L=11Gdi s7||0K&L&LL!\M //0G w<<4? * * cc'llccQUQ`ccc  $)do8IIJ w<<4+ + + pWpp]a]mppp  G}}r$c|jS)N)r)r!s r"__len__zDistributedSampler.__len__s r$rc||_dS)a1 Set the epoch for this sampler. When :attr:`shuffle=True`, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering. Args: epoch (int): Epoch number. N)r)r!rs r" set_epochzDistributedSampler.set_epochs r$)NNTrF) __name__ __module__ __qualname____doc__rintboolr#rrr4r6r8r$r"rrs..f$('''Dj'Dj '  '  '' ''''R"(5/""""H      s t      r$)rcollections.abcrtypingrr(torch.distributed distributedrtorch.utils.data.datasetrtorch.utils.data.samplerr__all__rrr?r$r"rGs $$$$$$ ,,,,,,,,,,,,  4(((LLLLLLLLLLr$