tj[bddlZddlZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z ddlmZmZmZgdZejeZ d ZGd d ZGd d eZeejdejZdZGddZGddZde dede e fdZ!dej"dededeej"fdZ#dZ$ d#de%edfde&e'efdzdede%edfdzd e&e'efdzde%e e%e e&ff d!Z(de efd"Z)dS)$N)Sequence)Anycast)DTensor map_aggregate) BlockMask) tree_flattentree_maptree_unflatten)TensorChunkSpecsplit_args_kwargs_into_chunks merge_chunksFceZdZdZdZdS)_CustomReducera$ Custom reducer class that can be used to specify a custom operation that reduces losses of multiple microbatches into one value. Example: >>> # xdoctest: +SKIP >>> sum_reducer = _CustomReducer( >>> torch.tensor(0.0), >>> lambda a, b: a + b >>> ) c"||_||_dSN) init_value reduce_fn)selfrrs e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/distributed/pipelining/microbatch.py__init__z_CustomReducer.__init__,s$"N)__name__ __module__ __qualname____doc__rrrrrs-  #####rrceZdZdS) _LossReducerNrrrrrrr r 1Drr gceZdZUdZdZeed<dZdZe de edffdZ e de e effd Zd S) r z2 Class used to specify chunking of inputs c||_dSr split_dim)rr&s rrzTensorChunkSpec.__init__As "rr&cJ|jjd|jjd|jdS)N.()) __class__rrr&rs r__repr__zTensorChunkSpec.__repr__Fs/~( V V4>+B V VT^ V V V rcd|jdS)NzTensorChunkSpec(r*r%r,s r__str__zTensorChunkSpec.__str__Ks3$.3333r chunk_dims.c(t|d}|S)a A helper for creating a tuple of `TensorChunkSpec` from a tuple of chunk dimensions (int's). Example: >>> # xdoctest: +SKIP >>> # There are three positional arguments to the model, and >>> # we are chunking them along dimension 0, 0 and 1, respectively >>> args_chunk_spec = TensorChunkSpec.from_tuple((0, 0, 1)) c t|Srr dims rz,TensorChunkSpec.from_tuple..],,rr)r0args_chunk_specs r from_tuplezTensorChunkSpec.from_tupleNs$(  , ,  rc(t|d}|S)a\ A helper for creating a dictionary of `TensorChunkSpec` from a dictionary of chunk dimensions (int's). Example: >>> # xdoctest: +SKIP >>> # Chunk dimension 0 for the "id" argument, 1 for the "mask" argument >>> kwargs_chunk_spec = TensorChunkSpec.from_dict({"id": 0, "mask": 1}) c t|Srr3r4s rr6z+TensorChunkSpec.from_dict..or7rr)r0kwargs_chunk_specs r from_dictzTensorChunkSpec.from_dictas%*  , ,  ! rN)rrrrrint__annotations__r-r/ staticmethodtupler9dictstrr=rrrr r <s###NNN   444#s(O\$!cN!!!\!!!rr ceZdZdS) _ReplicateNr!rrrrErEur"rrE block_mask num_chunksreturnc jddkrg|zSjd|kstdd}tjj||}tjj||}jtjj||ndg|z}jtjj||ndg|z}g}d}t|D]~} fd} | tj || || || || j | |j ||| dz }|S)a Given a block mask, split the block mask along the batch dimension (dim0). Args: block_mask: Block mask to split num_chunks: Number of chunks to split the block mask into Returns: chunk_block_masks: List of chunked block masks rz;Block mask has fewer batch size than the number of chunks. Ncfd}|S)Ncdtj|}||z|||Sr)torch full_likemask_mod)bhq_idxkv_idxb_offsetrFidxs rbatch_offset_mask_modzI_split_block_mask..create_mask_mod..batch_offset_mask_mods2 ?1c22!**1x<E6JJJrr)rUrVrFs` rcreate_mask_modz*_split_block_mask..create_mask_mods0 K K K K K K) (r) kv_num_blocks kv_indicesfull_kv_num_blocksfull_kv_indices BLOCK_SIZErO seq_lengths)rXsizeAssertionErrorrM tensor_splitrYrZr[rangeappendr from_kv_blocksr\r]) rFrG batch_dimkv_num_blocks_chunkskv_indices_chunksfull_kv_num_blocks_chunksfull_kv_indices_chunkschunk_block_masks batch_offset chunk_idxrWs ` r_split_block_maskrlys$$Q''1,,|j((  # ( ( + +z 9 9 I   I - *i*:+@*iXX  ( 4 :8*iPPPVj   % 1 :5z9MMMVj  L:&&@@  ) ) ) ) )    $29=,Y7#._expand_chunkss . .E&t,,G%**T^444E#(;;-',">F%*3%6%6F4> "#GFO OOG $ $ $ 5::dn-- -CCXrc3dK|]+}tt|V,dSr)rrto_local).0cs r z _split_tensor..s8 A Aad7A'')) A A A A A Arc Dg|]}tj|dS)Frp)r from_local)rt global_shape global_stridemesh placementss r z!_split_tensor..sO      "$        r)r^r&r_ isinstancerr device_meshrrMr`rqdivmodrr enumeratelistrbrSize requires_gradis_leaf retain_grad_debug_mask_minibatchesTensorrA)rmrnrG _is_dtensor local_tensor local_chunksglobal_split_sizequotient remainderchunk_tensors_listi local_chunk chunk_shape chunk_tensorsr{rlocal_expandedrrrrs ` @@@@r _split_tensorrs ;;t~ & &* 4 4 S6;;t~66 S S S   VW--KO & !(( ), DNSS "L8$%6 CC)  13' 55  NA{v|,,K*21y==aaa*PK '  % %"*[11(#     1C *6:t~NN   "  E        " l %*\  u|S !      <& !|   ' OO   A A= A A A           $    NN6:M:::;;;rc |sdt|DSt|t|ksStdt|dt||tdt |d\}t |d\}}g}t ||d D]\}}|tust|tr| |:t|tj rbt|tstd t|| ||jt|t rt|tstd t||jd kstd |jd dkr| |N| |jd }t%d|d|dt'g||R} dt| D} t ||d D]\}}g} |tust|tr|g| z} nht|tj rt)||| } n.5s...q...rzargs_dict.keys() = z args_chunk_spec.keys() = Nz.args_chunk_spec should have been set by callerc,t|tSrrr xs rr6z%_shard_dict_of_args..@sZ9%=%=rrc,t|tSrrrs rr6z%_shard_dict_of_args..Cs:a+C+CrTstrictzExpected TensorChunkSpec, got rz#BlockMask only supports split_dim=0rJzUnsupported chunk spec: z and value: z combination.cg|]}gSrrrs rrz'_shard_dict_of_args..as$J$J$JAR$J$J$Jrc0g|]}t|Sr)r )r_flat_split_result tree_specs rrz'_shard_dict_of_args..ts4     )955   r)ralenr_rkeysr ziprErrbrMrr typer^r&r rX ValueErrorminrrl) args_dictr8rGvalues chunk_specsr split_sizesvrnresult_num_chunksflat_split_resultsv_splitsr_v_splitrs @r_shard_dict_of_argsr"s$ /..E*--.... y>>S11 1 1 G$y~~'7'7"8"8 G G(,_-A-A-C-C(D(D G G   MNNN$==FI"!C!CNK Kv{48884 :  D*!=!=    z * * * * 5< ( ( dO44 T$%Rd4jj%R%RSSS   qvvdn55 6 6 6 6 9 % % dO44 T$%Rd4jj%R%RSSS>Q&&$%JKKK##A&&!++"":....""1?#7#7#:#:;;;;M4MMQMMM 5[5*555$J$J7H1I1I$J$J$Jv{4888004"$ :  D*!=!= s..HH 5< ( ( $Q.?@@HH 9 % % (,=>>HHM4MMQMMM -0 - - -  0 0 (   % %h / / / / 0     "4   rargs.kwargsrur8r<c|i}d}|t||d}|t||d}ttt|tt||}t |}t|||}t ||krTt |}ttt|tt||}t |t |kr/t dt |dt |d|D} | |fS) a Given a sequence of args and kwargs, split them into a number of chunks according to their respective chunking specs. Args: args: Tuple of args kwargs: Dict of kwargs chunks: Number of chunks to split the args and kwargs into args_chunk_spec: chunking specs for args, in same shape as args kwargs_chunk_spec: chunking specs for kwargs, in same shape as kwargs Returns: args_split: List of sharded args kwargs_split: List of sharded kwargs Nct|tjtzrt t St Sr)rrMrr r DEFAULT_CHUNK_DIMrErs r default_specz3split_args_kwargs_into_chunks..default_specs4 a 1 2 2 "#455 5<< rc,t|tSrrrs rr6z/split_args_kwargs_into_chunks..s*Q 2J2Jrrc,t|tSrrrs rr6z/split_args_kwargs_into_chunks..sJq)4L4Lrz;args and kwargs are split into different number of chunks: z, c zg|]7tfdttD8S)c3(K|] }|V dSrr)rr chunk_argss rrz;split_args_kwargs_into_chunks...s' < <jm < < < < < .sT   < < < r_j }tt!D]$}!|!| }%||!d }tdt!D]5}!| |ks!td|d!| 6||tE||S)z Given a list of chunks, merge them into a single value according to the chunk spec. Args: chunks: list of chunks chunk_spec: Chunking spec for the chunks Returns: value: Merged value NrzChunk z did not match chunk spec c,g|]}|Srr)rrkarg_idxchunks_flatteneds rrz merge_chunks..3s3!+G4rrJzExpected shape z, got devicemeta)sectionsr5z6Expected len(partial_values) == len(meta_chunks), got z != Trc8g|]}t|tSr)rrrrs rrz merge_chunks..^s"KKKZ733KKKrzRmerge_chunks: expected all values to be DTensors or none to be DTensors, got a mixz*merge_chunks: placement mismatch at chunk z : expected c6g|]}|Sr)rrs rrz merge_chunks..os 999aQZZ\\999rr4c3<K|]}|jjVdSr)rqr&)rrargs rrzmerge_chunks..ss=///0AGCM*//////rFrpz Expected )#r r rrrrbrrrarrqr_rMr`emptyr&rr^rxryanyallrrcatrsumrrrrrrrrr )"ru chunk_specspec_flattened flatten_spec chunk0_flatr{chunk_flattenedrargs_flattenedpartial_values overall_shapeval meta_chunks values_to_catchunk_start_idx partial_value meta_chunk chunk_end_idx slice_indicessliced dtensor_flagsrrrr local_cat cat_shape cat_stride reduced_valrkvaluerrrs" @@@rrrsZ'3J'?'?$ %1$;$;! \)*;<<=K@P@PP11)%00   3~#6#6 6 6SeSSzSSTT T0000 !#N!.11c)c)  c? + +b )!&s+;'<'v>> 00  !# "#>**c+.>.>>>(|QTUcQdQd||jmnyjzjz||25"K222 4 4-M:%4joocm6T6T$TM%*4t%<%<$= @R$RM38-3X3XM#-0*=9F!((000&3OO 4!/ LK]KKKM=!!$ S=))(9 +1-8 %mABB&7;;DAq|z11,III(2II:;,II2 %Q'3!I99=999  !q!1!788 +.////4A///,, #-("Jy11 *1-4466 %%&!"')"'     %%ei 3=&Q&Q&QRRRR ^ , , ).K"3'7#8#899   !mm!1)!rrlrrrrArBrCrrrrrr s $$$$$$ ,,,,,,''''''777777FFFFFFFFFF     8 $ $  ########$     >   l<5<,,hl;; 5!5!5!5!5!5!5!5!r        >>> )_>>>>Be< Le< e<e<el e<e<e<e