tjZdZddlmZddlZddlmZmZddlmZddl m Z ddl m Z ddl Z ddlZddl mZdd lmZmZGd d Zd/dZd0dZd1dZ d2d3dZd4dZd5d Zd6d,ZGd-d.ZdS)7adShared GradCache machinery (https://huggingface.co/papers/2101.06983). A "cached" loss trades compute for memory in three steps: (1) a quick embedding step without gradients/computation graphs, to get all the embeddings; (2) calculate the loss, backward up to the embeddings, and cache the gradients wrt. the embeddings; (3) a 2nd embedding step with gradients/computation graphs, connecting the cached gradients into the backward chain. Only one mini-batch of activations is alive at a time in (1) and (3), which is what bounds the memory. The second forward pass reproduces the first exactly by replaying its RNG state (:class:`RandContext`), so dropout draws the same masks and the cached gradients belong to the embeddings they are applied to. :class:`CachedLossMixin` implements all of this. A loss only has to provide ``calculate_loss``. ) annotationsN)IterableIterator) nullcontext)partial)Any)Tensor)get_device_statesset_device_statesc*eZdZdZddZddZddZdS) RandContextzSnapshot the CPU/CUDA/MPS RNG at init and restore it on enter, so the cached second forward replays the first's randomness (e.g. dropout). Ref: https://github.com/luyug/GradCache.returnNonectj|_td|Drtjnd|_t d|D}t|\|_|_ dS)Nc3jK|].}t|tjo|jjdkV/dSmpsN isinstancetorchr devicetype.0ts i/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/sentence_transformers/base/losses/gradcache.py z'RandContext.__init__..)s<[[a:a..I18=E3I[[[[[[c3nK|]0}t|tjr|jjdk,|V1dSrrrs rrz'RandContext.__init__..,sEssa:a;V;Vs[\[c[hlq[q[q[q[q[q[qssr) r get_rng_state fwd_cpu_stateanyr fwd_mps_statetupler fwd_gpu_devicesfwd_gpu_states)selftensorsnon_mps_tensorss r__init__zRandContext.__init__#s"022 [[SZ[[[[[ EI # # % % %   ss7sssss4E4W1d111rctj|jd|_|jtj|j|jGtj |_ tj |jt|j|j dS)NT)devicesenabled)rrandomfork_rngr%_fork __enter__ set_rng_stater!r#rr _mps_state_outsider r&)r's rr1zRandContext.__enter__/s\**43GQU*VV   D.///   )',i&=&=&?&?D # I # #D$6 7 7 7$.0CDDDDDrc|j$tj|j|j|||d|_dSN)r#rrr2r3r0__exit__)r'exc_typeexc_valexc_tbs rr6zRandContext.__exit__:sI   ) I # #D$; < < < Hgv666 rN)rr)__name__ __module__ __qualname____doc__r*r1r6rrr r sd__ X X X X E E E Err sentence_featuredict[str, Any]rintc d|vrt|ddz SdD];}||vr5t||tjr||jdcS.OsU ESXS_A`A`ejeorseses Aesesesesr)lenrrr rInextvalues)r?keys r_get_batch_sizerOAs***#O45599/22 " " "z2B32G'V'V "#C(.q1 1 1 1 $4$;$;$=$=  rbeginendc: d|vrt|}t||}i}dD]y\}}}||}||} || 6||} | z| d} |dkrdn)t | |dz } t | |dz } | | f||<n|jd|kr||} } n| | kr|d}|d}| dkrdn)t || dz }t || dz }nd\}}||f||<{d}d}|d}t|tj r|j d kr|jd}||| d}| rSt | }|dz|kr|dz}i}|D]\}}t|tj s|||<%||vr||\}}|||||<B|.|j d kr#|jd|kr|||d|f||<r|||||<|S|d}t!|dz }t||}t ||}t ||}t |d }|||dz||z }i}|D]\}}|d vr|||<|d vrL|dd|dd z } t | ||<_|d kr|d||f|z ||<xt|tj r,|j dkr!|jd |kr|d||f||<|||<|S)a^Create a mini-batch from sentence features, handling padded, flattened, and VLM inputs. With padded inputs, this slices along the batch dimension and drops trailing padding columns shared by the whole mini-batch, so short-sequence mini-batches aren't embedded at the full batch width (else ``mini_batch_num_tokens`` would not bound memory). Leading padding is kept to preserve positions. With flattened inputs (from ``DataCollatorWithFlattening``), this extracts the token ranges for sequences ``begin:end`` and rebuilds the metadata (``cu_seq_lens_q``, ``seq_idx``, etc.). VLMs like Qwen2-VL flatten per-sample visual tokens into a single tensor (e.g. ``pixel_values`` shape ``(total_visual_tokens, hidden_dim)``) with a grid tensor (e.g. ``image_grid_thw`` shape ``(num_items, 3)``) whose per-row product gives the token count per item. ``num_images_per_sample`` / ``num_videos_per_sample`` (precomputed by ``Transformer.preprocess``) map grid rows to samples; when unavailable we fall back to assuming one grid row per sample when ``grid.shape[0] == batch_size``. rC))image_grid_thw pixel_valuesnum_images_per_sample)video_grid_thwpixel_values_videosnum_videos_per_sampleNrdimrD)rrrF)rC cu_seq_lens_k) max_length_q max_length_kseq_idx.)rOmingetcumsumrAitemrIprodrrr rHr"nonzeromaxitemsrK)!r?rPrQ batch_size custom_rangesgrid_key pixel_key count_keygridrTnum_per_sample cumsum_items grid_begingrid_endtokens_per_item token_cumsum token_begin token_endtoken_axis_end seq_widthrFactive_columns last_activeresultrNrJr_beginr_endrCnum_seqs total_tokensnew_cu_seq_lens mb_seq_lenss! r_create_minibatchrTs ...$%566 #z""46 /  @ @ *Hi$''11D+// ::L||3-11)<?? nel 3 3 58Kq8P8P&,Q/I+E#I6::q:AAN!!## 5!."8"8":":">">"@"@"E"E"G"GHH ?Y..%01_N!#*0022 / /JCeU\22 /#s  %%!.s!3#GEM2s + aEKPQNV_D_D_#E#I$>?s #E#I.s  $_5M=!!A%H c8  CmE*//1122KM#&++--..I}R(--//00L#EC!GO4}U7KKOF&,,..  U 4 4 4)F3KK 4 4 4)!""-0DDKkoo//446677F3KK I  [%: :;eCF3KK u| , , qU[QS_XdEdEd [%: :;F3KKF3KK Mrlossrboolc$t|ddS)aWhether ``loss`` defers its backward pass to a hook on the loss tensor it returns. Such a loss re-embeds each mini-batch during the *backward* pass, by which time a decorator that patched ``SentenceTransformer.forward`` for the duration of the forward pass has been removed again. ``MatryoshkaLoss`` and ``AdaptiveLayerLoss`` both work by patching that forward, so they have to treat these losses specially: MatryoshkaLoss decorates ``calculate_loss`` instead, and AdaptiveLayerLoss warns that the combination is unsupported. Losses report this by setting ``uses_gradient_cache``. :class:`CachedLossMixin` sets it to True, and a loss that can turn the caching off at construction time (``MegaBatchMarginLoss``) overrides it per instance. uses_gradient_cacheF)getattr)rs rrrs 4. 6 66rmini_batch_sizemini_batch_num_tokens int | Nonelist[tuple[int, int]]c0t||fdtdDSd|vr#|ddd}nVd|vrC|ddd}nt dg}d}|kr]|dkr ||dz nd}t j|||z}t||dz}| ||f|}|k]|S) a,Compute the ``(begin, end)`` sequence ranges that split a batch into mini-batches. If ``mini_batch_num_tokens`` is None, every range spans ``mini_batch_size`` sequences. Otherwise, each range greedily packs as many sequences as possible while keeping the total number of non-padding tokens at or below ``mini_batch_num_tokens``. A single sequence whose length exceeds the budget forms its own mini-batch. Per-sequence token counts are read from ``cu_seq_lens_q`` for flattened inputs, or from the attention mask for padded inputs. Nc:g|]}|t|zfSr>)ra)rrPrirs r z%_minibatch_ranges..s-uuueEO3Z@@AuuurrrCrDrFrYzmini_batch_num_tokens requires per-sequence token counts, but the tokenized inputs contain neither 'cu_seq_lens_q' (flattened inputs) nor 'attention_mask' (padded inputs). Use mini_batch_size instead.) rOrangetolistsumrc ValueErrorbisect bisect_rightrgappend) r?rrcumulative_num_tokensrangesrPprevious_num_tokensrQris ` @r_minibatch_rangesrsg!!122J$uuuuuuUVXbdsOtOtuuuu*** 0 A!"" E L L N N - - - 01A B F F1 F M M T TYZ T [ [ b b d d +   %'F E *  BG!))3EAI>>QR!"79LOd9dee#uqy!! ucl### *   Mrrc8||dkrtddSdS)Nrz9mini_batch_num_tokens must be a positive integer or None.)r)rs r_validate_mini_batch_num_tokensrs0(-Ba-G-GTUUU)(-G-Grmodelcddlm}mt|d|r)d|djDn|dg}t fd|DS)a-Whether the model embeds its inputs with a StaticEmbedding, directly or behind a Router. StaticEmbedding features are an EmbeddingBag (``input_ids``, ``offsets``) with no batch dimension, so they cannot be sliced into mini-batches, meaning losses that mini-batch must reject such models. r)RouterStaticEmbeddingcg|] }|d S)rr>)rroutes rrz.has_static_embedding_input..s===eq===rc38K|]}t|VdSr5)r)rmodulers rrz-has_static_embedding_input..s-OOvz&/22OOOOOOr)2sentence_transformers.sentence_transformer.modulesrrr sub_modulesrMr")rr input_modulesrs @rhas_static_embedding_inputrs [ZZZZZZZBLERSHV\A]A]m==uQx3::<<====dijkdlcm OOOOOOO O OOr grad_outputr sentence_featuresIterable[dict[str, Tensor]]loss_objcachelist[list[Tensor]] random_stateslist[list[RandContext]]r"list[list[tuple[int, int]] | None]c tj5t||||D]\}}}} t||dd|| |D]\^} } } | jstj| | |z} |  ddddS#1swxYwYdS)a%A backward hook to backpropagate the cached gradients mini-batch by mini-batch. ``loss_obj`` only needs an ``embed_minibatch_iter(sentence_feature, with_grad, copy_random_state, random_states, ranges)`` iterator whose items *start* with the embeddings tensor. Extra elements are ignored (e.g. ``CachedGISTEmbedLoss`` also yields the guide model's embeddings). :class:`CachedLossMixin` provides the standard implementation, and the cross-encoder ``CachedMultipleNegativesRankingLoss`` satisfies the contract with its own adapter. ``cache``, ``random_states`` and ``ranges`` belong to one specific forward pass and are passed in rather than read off ``loss_obj``, so that a second forward pass before the first backward pass cannot make this hook back-propagate the wrong batch's gradients. Replaying the forward pass's mini-batch ``ranges`` also matters because modules may modify the features in place between the two passes (e.g. ``Pooling`` with ``include_prompt=False`` zeroes prompt tokens in the attention mask), which would change recomputed token-budget boundaries. Every mini-batch is scaled by ``grad_output``, which is whatever the outer backward pass hands us, so the fp16 gradient scaler and the gradient accumulation division reach all of them. TF)r? with_gradcopy_random_staterrN) r enable_gradzipembed_minibatch_iter requires_graddotflattenfloatbackward)rrrrrrr?grad random_state column_rangesreps_mb_grad_mb surrogates r_backward_hookr sk4    %%CF umVD D  % % ? dL-+.--%5"&+".( . + + % %& 1w,"Igoo&7&7&=&=&?&?ARARAXAXAZAZ[[^ii ""$$$$# % %%%%%%%%%%%%%%%%%%%sCC))C-0C-ceZdZUdZded<ded<dZded<d Zd ed <d Zd Zd ed < d*d dd+dZ d*d,d Z d-d.d&Z d*d/d)Z dS)0CachedLossMixina.The GradCache forward pass, shared by the losses that cache the gradients wrt. their embeddings. Subclasses must be an ``nn.Module`` holding a ``model``, must set ``mini_batch_size``, and must implement :meth:`calculate_loss`. They then call :meth:`forward_cached` from their ``forward``. rrrArNrrFrshow_progress_barTr with_backwardrepsrlabels Tensor | Nonerrr ct)alCompute the loss over the whole batch, from the per-mini-batch embeddings. When ``with_backward`` is set, back-propagate the loss (chunk by chunk, if the implementation chunks it) and return the detached total, so that no part of the loss graph outlives its own backward pass. Losses that don't need the labels simply ignore them. )NotImplementedError)r'rrrs rcalculate_losszCachedLossMixin.calculate_lossOs "!rr?dict[str, Tensor]rPrQrrrRandContext | None!tuple[Tensor, RandContext | None]cf|rtn tj}|tn|}t|||} |5|5|rt | nd}|| d} dddn #1swxYwYdddn #1swxYwY| |fS)zEmbed a mini-batch of inputs.Nsentence_embedding)rrno_gradrr rMr) r'r?rPrQrrr grad_contextrandom_state_contextsentence_feature_minibatchrs rembed_minibatchzCachedLossMixin.embed_minibatchZs`'0B{{U] 0<0D{}}},%67GPS%T%T" ! T T T TTeo{,F,M,M,O,OPPko zz"<==>RS T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T\!!s5 B$;B  B$ B B$B B$$B(+B(rlist[RandContext] | Nonerlist[tuple[int, int]] | None+Iterator[tuple[Tensor, RandContext | None]]c #K|t||j|j}tt j|d|j D]/\}\}}|||||||dn||V0dS)zUDo a forward pass on every mini-batch of the input features and yield the embeddings.NzEmbed mini-batches)descdisable)r?rPrQrrr)rrr enumeratetqdmrr) r'r?rrrrirPrQs rrz$CachedLossMixin.embed_minibatch_iterms >&'79MtOijjF( I) 22       OA|s&&!1#"3%2%:TT a@P '      rrrc t|}tj}fd|D}g}g}t||D]\}}g} g} |d||D]S\} } | | | | T|| || |s||S||d} | } d|D}dt|D}|r3tdj j d d |d | tt |||| | S) zDRun the three-step GradCache forward pass. See the module docstring.cFg|]}t|jjSr>)rrr)rr?r's rrz2CachedLossMixin.forward_cached..s<     .0DdF` a a   rF)r?rrrTrc&g|]}d|DS)cg|] }|j Sr>)r)rreps rrz=CachedLossMixin.forward_cached...s...s#(...rr>)rrep_mbss rrz2CachedLossMixin.forward_cached..s'CCC7..g...CCCrcdg|]-\}}td|Dt|.S)c3K|]}|duV dSr5r>)rgs rrz..s*SpSpbcTUY]T]SpSpSpSpSpSpr)r"str)rindexgrad_mbss rrz2CachedLossMixin.forward_cached..s@qqqPSSpSpgoSpSpSpPpPpq#e**qqqrzThe loss computation of z did not use input column(s) z, z: their embeddings received no gradient. Every input column is embedded (twice, with gradient caching), so remove the unused column(s) from the dataset instead.)rrrrr)listris_grad_enabledrrrdetachrequires_grad_rrr __class__r:join register_hookrr)r'rr grad_enabledrrrr?rreps_mbsrandom_state_mbsrrrrunused_columnss` rforward_cachedzCachedLossMixin.forward_cacheds8 !233,..     $5    /23Df/M/M 3 3 + mH! )-)B)B!1#/$ *C** 6 6% 0 0 ? ? A ABBB '' 5555 KK ! ! !  !1 2 2 2 2 5&&tV44 4""4t"DD{{}}++--CCdCCCqqIe+B##99^,,###   "3+      rr5)rrrrrrrr )r?rrPrArQrArrrrrrrr)NN) r?rrrrrrrrrrr)rrrrrr ) r:r;r<r=__annotations__rrrequires_media_countsrrrrrr>rrrr<s JJJ(,,,,,#####!!%$$$$AE "_d " " " " " "$,0"""""037/3 6@@@@@@@rr)r?r@rrA)r?r@rPrArQrArr@)rrrrr5)r?r@rrArrrr)rrrr)rrrr)rr rrrrrrrrrrrr)r= __future__rrcollections.abcrr contextlibr functoolsrtypingrrrr torch.utils.checkpointr r r rOrrrrrrrr>rrrs #""""" ........"""""" GGGGGGGGD&ffffR 7 7 7 7&)-&&&&&RVVVV P P P P/%/%/%/%dLLLLLLLLLLr