tjL ddlZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZmZerdd lmZerdd lmZdd lmZddlmZGd dZGddeZ Gdde Z!Gdde j"Z#Gdde j"Z$GddZ%GddZ&Gdde!Z'GddeZ(Gdd e Z)Gd!d"e Z*Gd#d$e Z+d%e,e-efd&e.d'e/d(e,e-effd)Z0d%e,e-efd&e.d'e/d(e,e-effd*Z1d%e,e-efd&e.d(e,e-effd+Z2dS),N)Iterable) TYPE_CHECKINGAnyOptionalcast)prune_linear_layer) ModelOutputis_sklearn_available)GenerationConfig)LogitsProcessorListMinLengthLogitsProcessorSuppressTokensLogitsProcessor) roc_curve)PreTrainedModel)PreTrainedTokenizerBaseceZdZUdZdZeed<dejde ejej ffdZ dejdej de fd Z d S) CandidateGeneratorz`Abstract base class for all candidate generators that can be applied during assisted generation.Frequires_model_outputs input_idsreturnc 0t|jd)ag Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and, optionally, a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. zT is an abstract class. Only classes inheriting this class can call `get_candidates`.NotImplementedError __class__)selfrkwargss i/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/generation/candidate_generator.pyget_candidatesz!CandidateGenerator.get_candidates,s$"~ s s s   scores num_matchesc0t|jd) Updates the candidate generation strategy based on the outcomes. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) scores (`torch.FloatTensor` of shape `(batch_size, candidate_length, config.vocab_size)`): Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search num_matches (`int`): The number of matches between the candidate sequences and the model predictions. z_ is an abstract class. Only classes inheriting this class can call `update_candidate_strategy`.rrrr"r#s rupdate_candidate_strategyz,CandidateGenerator.update_candidate_strategy=s*"~ + + +   r!N)__name__ __module__ __qualname____doc__rbool__annotations__torch LongTensortuple FloatTensorr intr'r!rrr'sjj#(D((( (8 uUM]_d_pMpGq    " 53C UM^ mp      r!rcpeZdZdZ ddejddddded ejdzd ed f d Z dejd e ejej ffdZ dejdej de fdZdejd e e e ffdZ ddejde de d efdZdejde de d efdZded e ejej dzffdZdS)AssistedCandidateGeneratorar `CandidateGenerator` class to be used for assisted generation and speculative decoding. This class generates candidates through the use of a smaller model. Read the following blog post for more information: https://huggingface.co/blog/assisted-generation Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrassistant_modelrgeneration_configr model_kwargs inputs_tensorlogits_processorrc|j}||}|||}||_tj|j|_|j}|jjdi|ddi|jj |_ |jj |_ |j |j_ i} | D]a\} } | dvrXt| tjr'| |ntj| | | <bd| vr|s| d=|jjrC|||jj| \}} } ||| | |j} nd|vr |d| d<| |_|jjrd|_ngd| vr\d|_|jdtj|jd d f|jtj |jd <nd|_||n t;|_tj||_d|j_d|j_ |j |j_ d|j_!|jj"|_#d|j_"d|j_$|jj%|_&d|j_%d |jD|_d|j_'tQr2|jj r(tS|tTurg|_+g|_,dSdSdSdS)N defaults_onlyT)encoder_outputspast_key_valueslogits_to_keepr=decoder_input_idsrdecoder_attention_maskrr devicedtypeattention_maskc<g|]}t|t|Sr3) isinstancer).0 processors r z7AssistedCandidateGenerator.__init__..s8! ! ! #JyZrDsDs! ! ! ! r! dynamic_fullr3)-rCtor6copydeepcopyr7assistant_generation_config_get_default_generation_paramsupdatenum_assistant_tokensassistant_confidence_threshold eos_token_iditemsrGr.Tensordetach_supports_logits_to_keepconfigis_encoder_decoder_prepare_model_inputs bos_token_id._prepare_encoder_decoder_kwargs_for_generationassistant_kwargs input_ids_keygetonesshapelongrr:return_dict_in_generate output_scores is_assistant min_lengthmain_model_min_lengthmin_new_tokens max_lengthmain_model_max_lengthcache_implementationr typer5probsmatches) rrr6r7r8r9r:rCglobal_defaultsr^keyvaluemodel_input_names r__init__z#AssistedCandidateGenerator.__init__gs!'LL((  $),,V44M /,0=9Z+[+[(:YY[[/(/VV/VVQUVVVV$($D$Y!.2.N.m+9J8V(5&,,..  JC@@@1;E5<1P1PjELLNN%%f---VZVcdiVjVj!% / / /8`8`8b8b / !12  ! 4 R@O@e@et?LN^AA =M+-= /]]/1A4Cc    , . .2>?P2Q . / 0  ! 4 -!4D   "2 2 2!,D 6:6K6O6O( IOA.29;KSXS]^^^77D !"2 3 3 "-D 5E4P 0 0ViVkVk!%/@!A!A9=6/3,@D@c=.2+&*%;%F",0)04-%)%;%F",0)! ! '+'<! ! !  7E3 ! " " 0O T 888DJDLLL     98r!rc ||jj}||\}}|dkr|dfS||||||}||\}}||fS)Z Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. rN)rLr6rC_calculate_new_tokens_update_past_and_masks_prepare_generation_args_generate_candidates)rrrrimax_new_tokensgeneration_args candidate_idscandidate_logitss rr z)AssistedCandidateGenerator.get_candidatessLL!5!<== )-)C)CI)N)N& Q  d? " ##I...77 >Sabb*.*C*CO*T*T' '...r!r"r#cv|jjdvrJ|t|ddz kr|xjdz c_nt d|jdz |_t rI|jjr>t|tur)|j dg|zt|j t|j kr|j dt|j t|j z }|dkr |j | d=t|j dkrwddh |j r]t|j |j \}}}d|z }|d|zz} tj| } || } | |j_dSdSdSdSdSdS)r%> heuristicheuristic_transientrr rN)rOnum_assistant_tokens_schedulelenrRmaxr rSrmr5roextendrnappendissubsetrnpargmin) rrr"r# excess_lengthfprtpr thresholdsfnrcostsoptimal_threshold_indexbest_thresholds rr'z4AssistedCandidateGenerator.update_candidate_strategys  + IN   c&)nnq000))Q.))),/43Lq3P,Q,Q) ! " " a0O aT 888 L  k 1 2 2 24:T\!2!222 ##A&&&  OOc$,.?.??Mq  J ~/DJ!##A (E(E#'0tz'J'J$S*#ga#g +-)E*:*:'!+,C!DR`0OOO9 a a a a98$###r!c|jd}tt|j|j|z dz }t t||j|z d}||fS)zCCalculate the minimum and maximum number of new tokens to generate.r r)rbminr2rRrkrrh)rr new_cur_lenr{ris rrwz0AssistedCandidateGenerator._calculate_new_tokenssdob) S!:;;T=WZe=ehi=ijjS1Kk1YZZ\]^^~--r!rr remove_from_pkvnum_added_tokensc|jdddu}|r|jddz |z }|jd||z t |j|jd|jjj|_t|j|jd|jjj|_t|j|jd|_d|j _ |S)zLUpdate past key values and attention masks for subsequent generation rounds.r>Nrr ) r^r`rbcrop_prepare_attention_maskr6rYrZ_prepare_position_ids_prepare_token_type_idsr7rl)rrrrhas_past_key_valuesnew_cache_sizes rrxz1AssistedCandidateGenerator._update_past_and_masks$s#3778I4PPX\\  ?&_R014FN  !"3 4 9 9.K[:[ \ \ \$;%yr':DrdimrNr r3)r6generater^r>r rOrSrmr5r.catr"softmax sequencesrrangernrtoliststack) rr|assistant_output scores_tensorscores_softmaxidspr~r}s rrzz/AssistedCandidateGenerator._generate_candidatesDs 84/8dd?ddNcdd3C3S/0 " " *0O *T 888!I&6&=1EEEM"]=bAAAN",R#6F6M2N2N1N1P1P-PQCuSXX34A J  ahhjj ) ) ) ;'7'>AFFF(2 ...r!NN)rr )r(r)r*r+r.r/dictrVrrtr0r1r r2r'rwr,rxryrzr3r!rr5r5Ps8.2<@aa#a+a. a  a |d* a##89aaaaF/(8/uUM]_d_pMpGq////28a53C8aUM^8amp8a8a8a8at.u/?.E#s(O....^_##)#TT ,..00  ELLNN++ %**,,'' Il++ %i0(3!222r!c||jk}tj|stj|}|t }|sdSt |\}}|d|z}|d|z}|j d|z }|dd||zdf} |dd|||zf} || | fS)a Input: prompt: 2D array of shape (batch_size, prompt_length), represents the original prompt tokens prompt_plus_new_tokens: 2D array of shape (batch_size, prompt_length), represents the suffix of the original prompt, with additional new tokens. Output: discrepancy_length: int, represents the number of tokens that need to be replaced from prompt new_tokens_only: 2D array of shape (batch_size, new_token_length), represents the new tokens that are not in prompt discrepancy_only: 2D array of shape (batch_size, discrepancy_length), represents the new tokens that are in prompt but not in prompt_plus_new_tokens NNNrr N) Tr. is_tensortensorrLr2anyitemrrrb) promptprompt_plus_new_tokens compare_matcompare_mat_intlongest_locationlongest_diag_lengthnew_token_start_indexdiscrepancy_with_olddiscrepancy_lengthnew_tokens_onlydiscrepancy_onlys r_get_tokens_diagz>AssistedCandidateGeneratorDifferentTokenizers._get_tokens_diags 6 88 {++ 4,{33K%..--""$$))++ $##0]0u0u 1 1 --!1 36I I/25HH$l1o0DDJJLL04IL^4^4`4`1`a1 AA$'r3)r2rRrLr6rC_prepare_assistant_input_idsrrrrhrbrxryr^popr_process_assistant_outputsrrr>) rrrr{rassistant_input_idsrir|rnew_target_idss rr z > >d? "t##r!c|j|jd}d}|j?|j|jkr.|j|jz }|j|dd|dffi|}|jd}|jdd| df}|||\}} } |j} | |dkrx| jddkrg|| jdkr | | dd| df<nG|| jdkr6|| jdz } | ddd| f} | | dd| jd df<|}| jddkrtj | | gd} n9tj | |gd} n |j|fi|} |jd|_| |fS)zIConverts target input IDs to assistant input IDs, handling discrepancies.rrrNr rr) rrrrrrrbrr.r) rrconvert_kwargsrstart_index_in_target_windownew_assistant_idsprompt_use_length prompt_userrrrdiscrepancy_length_diffs rrzJAssistedCandidateGeneratorDifferentTokenizers._prepare_assistant_input_ids#s;!% 5%)%=    " .43KdNd3d3d+/+CdF\+\ ( K K!!!9:::;!!?M!! !2 7 : 05F4F4G4G1GHJDHDYDY-EE A 1A#'"9 *%)).>.DQ.G!.K.K)-=-CA-FFFGW+AAA0B/B/C/C,CDD+.>.DQ.GGG2DGWG]^_G`2`/.A!!!E^G^F^E^B^._+O_+AAA0@0Fq0I/I/K/K,KL&8O"(+a//*/)5H/4Z`b*c*c*c'',i1DFW0X^`&a&a&a##"M$"Mi"j"j[i"j"j '0q'9D $"O33r!assistant_sequencescttj|j}|jd}||jz }||dd|df|j|j}|jd}|dd| df}| ||\} } } |} | *| jddkrtj | | gd} ntj | |gd} |j | ddd|j f} | S)z7Processes assistant outputs to obtain target input IDs.r Nrrrr) rr.r/rrbrrrrrrrk) rrrrnum_prev_assistantstart_assistant_look_indexnew_target_ids_from_windowtarget_prompt_use_lengthtarget_prompt_use_target_new_tokens_onlyrs rrzHAssistedCandidateGeneratorDifferentTokenizers._process_assistant_outputsQsE"%"2D4KLL/5a8%7$:S%S"%)%P%P #=#>#> > ?!5"&"7&Q& & " $>#CA#F %aaa*B)B)C)C&CD'+'<'<=NPj'k'k$ !1" ! -%+A.22!&N'^?Z^C^$ t' ( (1 , ,+ 0K38<;BBDDEE*74440 $3#H#H#J#J !78H$Jj!k!k$55nEEE,;,P,P,R,R)'9:SUYUu'v'v$-445IJJJ,@))))<243O3O3Q3QSWSoppq**&&&' - ,r!cd|jr&t|jdkrd|j_dSdSdS)as Disables the mapping of input ids despite the assistant pruning for the language model head being enabled. This method is required for the first forward pass of `_MapInputEmbedding` where input ids are already in the assistant vocabulary space. By disabling the mapping, it ensures that the input ids are processed correctly without remapping. rFN)r.rr9r?r"rs runmap_input_idsz+AssistantToTargetTranslator.unmap_input_idssE  ' 2C0H,I,IA,M,M,1D % ) ) ) 2 2,M,Mr!cR |j}|j}d}||dd}t|dkr|j|dd ||dd}t|dkrM|j|dd kr! fd|D}t |}tj |dzf|j t}i}|D]&\} } | | } | | || <| || <'| |j|fS) N F)rrrcti|]4\}}|r|dn||5Sr ) startswithreplace)rHtokrassistant_space_signtarget_space_signs r zRAssistantToTargetTranslator._get_assistant_to_target_input_ids.. s_''' %C #~~.BCC%CKK(<>OQRSSS!$ '''r!r rD)r2 get_vocabr3rconvert_ids_to_tokensrUrrr.fullr,r2r`rLr4)r target_vocabassistant_vocab space_strtarget_space_idsassistant_space_idsmax_assistant_indexassistant_to_target_input_idsr7rJ assistant_id target_idrKrLs @@rr5z>AssistantToTargetTranslator._get_assistant_to_target_input_idss-7799 3==?? 11)PU1VVWbc  1 $ $ $ 6 L LM] ^ ^_` abc d "&";";IZ_";"`"`al"m &''!++'+'@'V'VWj'k'klm'nop'q$$(<<<''''' )8(=(=(?(? '''O"/"8"8":":;;(- 4G!4K3MtOemp(q(q(q%8:%!0!6!6!8!8 H H C$((--I$>G-l;;G-i8,//0LMMOlllr!rcRtj|j|jkdS)z` Get the input ids that are in the assistant vocab but not in the target vocab. r)r.wherer6r,rBs rr8z3AssistantToTargetTranslator._get_suppress_input_idss%{4>$BXXYYZ[\\r!assistant_candidate_idsc t|d|jdz }|dkr|S|d| df}|jr |j|}|j|}t j||dfdS)aS Return the target candidate ids that correspond to the assistant candidate ids. Note that we have already the target ids for the prompt and we only need to find the target ids for the new tokens. Moreover, assistant ids of the original prompt does not necessarily appear in _assistant_to_target_input_ids. rr Nr)rrbr.rr6r.rr$)rrtarget_input_idsr]num_new_tokenslast_candidate_idstransformed_slices rget_target_idsz*AssistantToTargetTranslator.get_target_ids$s4Q788;N;TUV;WW Q  # #"9^O> ? ?%5c;iT=`=fgh=i;i6i%j "BXY\^tYtBuM#>> ?r!NF)r(r)r*r+floatr+r-r,r2rr,rtrCr5rr8r.r/rcr1rlr3r!rr)r)sC(!5<<-L%'''s8<(- ''3'7' ' ""34 ' "& ''''R 2 2 2!m!m!mF]c]]]] XNSN^X  XXXX*%2CHYr!r)ceZdZdZejZe ddddddede d d e d e f d Z ed Z dS)AssistantVocabTranslatorCachez Cache for `AssistantToTargetTranslator` instances. The instances are computed at pre-processing time, and this cache allows us to avoid recomputing them. NFrrrr-r6rr.rc|j|}|tj}||j|<||}|t |||||}|||<|Sr)_cacher`weakrefWeakKeyDictionaryr))clsrrr-r6r.assistant_dictmappings rget_translatorz,AssistantVocabTranslatorCache.get_translatorWs(899  !$688N+9CJ' ( $$%899 ?1 #!' G3:N. /r!cd|jD}|D] }|j|= |jD]}d|D}|D]}||=dS)z Clean up dead references in the cache. This removes entries where either the target_tokenizer or assistant_tokenizer has been garbage collected. cg|]}||Srr3rHrqs rrJz9AssistantVocabTranslatorCache.cleanup..zs>>>S#+S+++r!cg|]}||Srr3r{s rrJz9AssistantVocabTranslatorCache.cleanup..sFFF#++++r!N)rrr)ru dead_keysrqrvs rcleanupz%AssistantVocabTranslatorCache.cleanuprs?>CJ>>>   C 3"j//11 ( (NFFFFFI  ( ("3'' ( ( (r!rm)r(r)r*r+rsrtrr classmethodr2rr,r)rxr~r3r!rrprpOs 'W & ( (F 8<(- 37  ""34  "&  %[4(([(((r!rpceZdZdZ ddejddddddd d d ed ed ejdzde dffd Z dejde ejej ffdZ ddejdedeffd ZdejdejfdZxZS)%UniversalSpeculativeDecodingGeneratorz `CandidateGenerator` class to be used for Universal Speculative Decoding (USD): speculative decoding with different tokenizers for the assistant and main models. This class generates candidates through the use of a smaller model. Nrr6rrrrr7r r8atm_translatorr9r:rc ||_t|||||||| d|_d|_dS)Nr)_atm_translatorrrt_target_seq_len_with_candidates_prev_assistant_ids) rrr6rrr7r8rr9r:rs rrtz.UniversalSpeculativeDecodingGenerator.__init__sY .          56,<@   r!rc V||jj}||\}}||\}}|dkr|dfS|||||||}d|d_d|d_|j j |j j |d<| |\|_ } |j |||j } | jd|_|j | } | | fS)zk Simplified version of get_candidates that uses the translator cache for token conversion. rNrTr7r:r)rLr6rCrrwrxryrerdrr:rzrrcrbrrl) rrrr_rrrir{r|assistant_candidate_logitstarget_candidate_idstarget_candidate_logitss rr z4UniversalSpeculativeDecodingGenerator.get_candidatessQ%<<(<(CDD040Q0QRb0c0c--)-)C)CDT)U)U& Q  d? " ##$7JZ#[[[778K^]kll>B+,:GK+,D   1 =262F2XO. /?C?X?XYh?i?i< "< $3BB !143K  0D/I"/M,"&"6"H"HIc"d"d#%<<->*>?!  /! 3 3 $ 4 R V VWefgWhWmWmWoWo p p   $"3::DW[;O!% 8 8E$!9!!!2d29::  !& /@.A-B4K_Kf g g g   # +"3  #CaG.X !##+/+CAAAGYIYHYGYDY+Z("')T-EGX,Y_a"b"b"b 1445:4FF ,,..."C(9!(<$=$===r!rrG)r(r)r*r+r.r/rr)rVrrtr0r1r r2r,rxrrrs@rrrss.2<@AA#A+A4 A 7 A . AA4A|d*A##89AAAAAA8=(8=uUM]_d_pMpGq====@ f f%:J f^a fjn f f f f f f&>U=M&>RWRb&>&>&>&>&>&>&>&>r!rceZdZdZ ddejdzdeded ed ed d edzf d Zdej de ej ej ffdZ dej dej defdZ dS)PromptLookupCandidateGeneratora) `CandidateGenerator` class to be used for prompt lookup generation. This class generates candidates by looking up likely continuations in the provided prompt (input_ids) itself. Read the following blog post for more information: https://github.com/apoorvumang/prompt-lookup-decoding Args: eos_token_id (`torch.Tensor`, *optional*): The token id of the end of sequence token. num_output_tokens (`int`, *optional*, defaults to 10): The number of tokens to be output as candidate tokens. max_matching_ngram_size (`int`, *optional*, defaults to 2): The maximum ngram size to be considered for matching in the prompt max_length (`int`, *optional*, defaults to 20): The number of total maximum tokens that can be generated. For decoder-only models that includes the prompt length. Defaults to 20, which is the max length used as default in generation config. logits_processor (`LogitsProcessorList`, *optional*): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. In prompt lookup assisted generation, they are not used to manipulate probabilities, but rather to find forbidden tokens (p = -inf) and block them from being valid candidates. vocab_size (`int`, *optional*): The size of the vocabulary. Required if `logits_processor` is provided. N rrTnum_output_tokensmax_matching_ngram_sizerjr:r vocab_sizec||_||_||_||_||_||_|jdks |jdkrt ddS)Nrz4Invalid max_matching_ngram_size or num_output_tokens)rrrjrTr:r ValueError)rrTrrrjr:rs rrtz'PromptLookupCandidateGenerator.__init__se"3'>$$( 0$  '1 , ,0F!0K0KSTT T1L0Kr!rrc 8|j\}}|j|dzkr|dfSd}d}tt|j|dz ddD]}|d|d}|d| df} || kd} | d d} | D]} | |z} | |jz}t|||j}| |krq|d| |f}|j |}tj ||j f|j tj }t|D]\}}| ||}|d|f}|t!d  tj|jjfvr |d|}n7tj||d|dzdfd}|jddkr+d }|jjtj||j}tj|}|dkr$|d}|d|}n|rn|r|t3|dkr|dfS|d}tj||fd}|dfS) a Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(num_candidates, candidate_length)`: The candidate sequences to be tried. r NFrr) dimensionsizesteprrT)as_tuplerBr*)rbrjrrrunfoldallrrr:r.rarrCfloat32 enumeraternfinforDrr$rTisinnumelrr)rrrbsz input_length chosen_ids match_found ngram_sizewindows ngram_tensorro match_indicesrrend_idxsequence_with_candidatefake_input_logits candidate_idxnew_candidate_tokenfake_output_logitsfake_candidate_logitsmaskmatch_indices_eosfirst_eos_indexcandidate_input_idss rr z-PromptLookupCandidateGenerator.get_candidates(s>&O\ ?lQ. . .d? "  D$@,QRBR S SUVXZ[[@ @ J&&!&LLG%Q _5L,.333::G$OOTO::1=M %. . *, #d&<<g|T_EEw&&!*1i.?+?!@J,82;/,1J $/29;KSXS`---)CLJBWBW " ">M+>151F1FG^`q1r1r.4FqJ]G]4^14%,, TiToHpHpHt8uuu-7-G %:?)%. ;N]Q=N;N0O0Y0YZ[0\0\$]cd;";";" 7 7&+A.!33$"&K (4$z*d6GHH,1M$,?,?),2244q88.?.B.G.G.I.IO)34D_4D)EJES'T   #j0C OOq4H4Hd? " ))!,, #iJ(?QGGG"D((r!r"r#cdS)r%Nr3r&s rr'z8PromptLookupCandidateGenerator.update_candidate_strategys  r!)NrrrNN)r(r)r*r+r.rVr2rrtr/r0r1r r'r3r!rrrs4-1!#'(<@!%UUlT)UU"% U  U ##89 U$JUUUU&])(8])uUM]_d_pMpGq])])])])~53CUM^mpr!rceZdZdZ ddejddddded ejdzd ed f fd Z dejd e ejej fffd Z xZ S)EarlyExitCandidateGeneratora `CandidateGenerator` class to be used for assisted generation and speculative decoding. This class generates candidates through the use of **the model itself**, exiting early. Can only be used with models that support early exit, e.g., `facebook/layerskip-llama3.2-1B`. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The original model. This model must support early exit (i.e. is trained to compute logits in earlier layers). generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrr6rr7r r8r9r:rct|||||||jj|_d|j_dS)N)rr6r7r8r9r:)rrtr7assistant_early_exit)rrr6r7r8r9r:rs rrtz$EarlyExitCandidateGenerator.__init__sY +/%'-    %)$:$O!6:333r!rc t|j|jj}|jj}|j|j_t |\}}||j_||fSr)getattrr6base_model_prefixrYnum_hidden_layersrrr )rrr base_modeloriginal_num_hidden_layersr}r~rs rr z*EarlyExitCandidateGenerator.get_candidatessiT143G3YZZ %/%6%H".2.G +*/''*@*@*K*K' '.H +...r!r)r(r)r*r+r.r/rrVrrtr0r1r rrs@rrrs:.2<@;;#;+;. ;  ; |d* ;##89;;;;;;,/(8/uUM]_d_pMpGq//////////r!rc<eZdZUdZdZeed<dddZee e fed< dde j dd d e jd d d ede jdzdeddeeeze jzdzffd Zde j d ee e fdedededee j e jdzff dZxZS)*SinglePositionMultiTokenCandidateGeneratora Candidate generator for predicting multiple draft tokens from a single token position using the MTP method. The Multi-Token Prediction (MTP) method defines an assisted candidate generator with an assistant model that autoregressively drafts multiple candidate tokens from a constant `position_ids` value, the last "seen" token, with the goals of reducing assistant model compute, increasing drafted token acceptance rates, and increasing target model throughput. This capability is enabled by three concrete requirements for assistant models: * **Use KV cache sharing techniques for the entire assistant model** (see Gemma 4) to reduce attention computation in the assistant model. This method of cache sharing passes the `key_states` and `value_states` from the main model in a dictionary, allowing the assistant model to skip pre-fill entirely, and reducing attention computations (e.g., `k_proj`, `k_norm`, `v_proj`, and `v_norm` in the Gemma 4 architecture) as applicable to the main and assistant models' architectures. This architectural feature effectively locks the assistant into a constant `position_ids` value. * **Concatenate the embedding and hidden states for the seen last token** from the target model as the input to the assistant. This adaptation allows the assistant to retain rich context about the preceding (possibly drafted) tokens while working from a constant `position_ids` value. For the first token drafted after pre-fill, the last seen token will be the last token from the prompt. For subsequent drafting steps, the last seen token will be the last token generated by the assistant (within a drafting round) or the last token accepted by the target model (between drafting rounds). * **Use cross-attention** to allow Q from the assistant to attend to KV from the main model. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. target_model_input_embeddings (`torch.nn.Embedding`): The input embedding table from main model, used to get the embeddings from the last seen token. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. model_kwargs (`dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. logits_processor (`LogitsProcessorList`, *optional*): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. eos_token_id (`int` or `list[int]` or `torch.Tensor`, *optional*): The token id of the end of sequence token. If not `None`, only the provided values will be used. If None, values will be inferred from the available `generation_config` and `assistant_generation_config`. Tr)output_hidden_statesreturn_shared_kv_statesmodel_kwargs_overridesNrr6rtarget_model_input_embeddingsr7r r8r9r:rrTc >d|jjvr+d|jjvrtd|jjdt||||||||_|@t }t|jj tr | |jj n>t|jj tr| |jj t|jj tr | |jj n>t|jj tr| |jj |r-tjt#|tjnd|_ dSt|tjs?t|tr|g}tj|tj|_ dS||_ dS)NGemma4AssistantGemma4UnifiedAssistantzfExpected assistant_model to be a Gemma4AssistantForCausalLM or Gemma4UnifiedAssistantForCausalLM. Got z This candidate generator requires that the assistant model is able to work from a shared_kv_states dictionary. Currently, only the Gemma4AssistantForCausalLM and Gemma4UnifiedAssistantForCausalLM support this.rN)rr(rrrtrrrGr7rTrrQr2rrOr.rrrcrV) rrr6rr7r8r9r:rTrs rrtz3SinglePositionMultiTokenCandidateGenerator.__init__s _%>%G G G(0I0RRRBzIzSz\BBB  O5F Vceuvvv-J*   #L$0=xHH F##D$:$GHHHHD2?EE F  !7!DEEE$:GRR P##D$D$QRRRRD<I3OO P  !A!NOOOVb l T,-?-?uz R R R RhlD   L%,77 4,,, . ,~ % \ L L LD    , 1 1 3 3D   r! model_outputsis_first_iterationn_last_matchesrc $|r|dfStt|j|j|jdz dz }|dkr|dfS| t |drt |dst d|jd}|j} |jdfd| D} |dd||dzf}|ddddf} tj |jddz ggtj |j j } tj|jdtj|j } g} g}t#|D]}|| }tj||gd }tj5| ||d | | d }dddn #1swxYwY|jd } |j}| r| d}tj||jj| } | tj|dtj|j|jn| |j|| |j ntj!| tj"| #d|j $| j} | %rntj|tj|d gd }tj| d }||fS)z2Generate draft token candidates using the drafter.Nr rrshared_kv_statesz_`model_outputs` cannot be None, and they need to contain `hidden_states` and `shared_kv_states`rc i|]:\}}||ddddddddf|ddddddddff;S)rNr r3)rHkvcurrent_lengths rrMzMSinglePositionMultiTokenCandidateGenerator.get_candidates..Xsw   TXTUWXA!QQQ?N?AAA-.!QQQ?N?AAA5M0NO   r!r1rrEF) inputs_embedsrErr use_cache)&rr2rRrkrbhasattrrrrrUr.rrcr6rCzerosr,rrrno_gradr`logitsrlast_hidden_staterr$r\r7 pad_token_idr zeros_likerT logical_orrsqueezerLr)rrr8rrrrr{rr last_token_idrsequence_stoppeddrafted_logitsdrafted_tokensrlast_token_embeddingroutputsstoppedr}r~rs @rr z9SinglePositionMultiTokenCandidateGenerator.get_candidates3s  #d? "S!:;;T=WZcZijkZl=lop=pqq Q  d? "  !=/:: "=*<== "q +8*Eb*IIVIg#+    \l\r\r\t\t    .aaa.STBT1T.TU!!!!RSS&) |ioa&81&<%=$>ejY]YmYtuuu  ;yq'9T]Tdeee~&&# # A#'#E#Em#T#T !I';=N&OUWXXXM  .."/#/#3#34D#E#E!-%5# /               $N11b199M ' 9  ##%% 6*44Q77 % GT5K5XZg h h %%K 1 1" 5 5u7G7W7WY`Yghh%%gn555  ! !- 0 0 0 ,#(#3$J}44Q779J9M9MmNb9c9cdd$$ $''))E 9eiA.N.N.N"OUVWWW  9^;;;...s.GG G r)r(r)r*r+rr,r-rrstrrr.r/r&r'rVrr2rrtr r0r1r rrs@rrrs++Z$(D'''!%#'..DcN.2<@>B+4+4#+4++4(*| +4 . +4  +4|d*+4##89+4DIo 4t;+4+4+4+4+4+4Z[/#[/38n[/# [/ ! [/  [/ u!2T!99 :[/[/[/[/[/[/[/[/r!rceZdZUdZeed<ddiZeee fed< ddddd d eee fd e d fd Z de j d eee fdedededee j e jdzff dZdZdS)MTPCandidateGeneratorTrrrN main_modelrr7r r8r:rc ddlm m m m m}ddlm}t|j dd|_ |j td| }td|jdD|_||d|ji |_||j  } fd |jD|_ fd |jD|_||_|j|j|_||_d |_dS)Nr) DynamicLayerDynamicSlidingWindowLayer$LinearAttentionAndFullAttentionLayer-LinearAttentionAndSlidingWindowAttentionLayerMtpCache)MtpModelnum_mtp_layerszgCould not find `num_mtp_layers` in the model config. This model probably has no associated mtp weights.c3$K|] }|jV dSrrf)rHxs r z1MTPCandidateGenerator.__init__..s$PP18PPPPPPr!r) device_map)rYcJg|]}t|ur n| Sr3)rm)rHlayerrrs rrJz2MTPCandidateGenerator.__init__..sA   V[d5kk-FFFLLNNNE   r!cXg|]&}t|ur|jn|'S))number_of_states)rmr)rHrrrs rrJz2MTPCandidateGenerator.__init__..sV   E{{KKK 1 0%BX Y Y Y Y   r!T) cache_utilsrrrrrmodeling_layersrrrYget_text_configrr get_decodernextlayers parametersrCfrom_pretrained mtp_modelget_mtp_config mtp_cacheactivate_past_recording do_sampler:is_main_model_prefill) rrr7r8r:rrrr rrrrs @@@@rrtzMTPCandidateGenerator.__init__s               /.....%j&7&G&G&I&IK[]abb   &   ++-- PPZ->r-B-M-M-O-OPPPPP !11*"dkIZ1[[HJ$5$D$D$F$FGGG      _h_o        #)     # ..000+4 0%)"""r!rrrrrc |r|dfS|t|dstd|jd}|js|dzn|jddz }|ddd|f|j}|jdkr1|jr||_n"tj |j|gd|_|jdkr||jdz kr|js|j |j |dd| |jz df} |ddd| |jz df} |ddd| |jz df} |jdd| |jz dddf}n9|dd| df} |ddd| df} |ddd| df} |j d | || | |j |j|j|d|\} } }d |_tj || gd} | | fS) NrzG`model_outputs` cannot be None, and they need to contain `hiden_states`rr rrrE)rlast_hidden_statesrErr r r:full_input_idsFr3)rrrr rbrLrCrfull_seq_last_hidden_statesr.rr rrr r:)rrr8rrrrrnum_last_main_model_tokens mtp_input_idsmtp_position_idsmtp_attention_maskr}r~rs rr z$MTPCandidateGenerator.get_candidatess  #d? "   (O(O fgg g+8+Fr+J@D?Y%u^a%7%7_h_nop_qtu_u"03N4N3N0NORRSWS^__   " ") 3E0038957IJPQ4440   " "~8Ka8O'O'OX\Xr'O N  !4 4 5 5 5&aaa*D)DtGZ)Z)\)\&\]M+N;AAA@Z?Z]a]p?p?r?r!?D^C^aeatCtCvCv@v!w !%!A..1DDFFI"   &aaa*D)D)E)E&EFM+N;AAA@Z?Z?[?[<[\ !-.>!?D^C^C_C_@_!` -;T^ . #1-)nn!2$ . .  . . * '&+" 9m"<"EEE ...r!cdSrr3)rargsrs rr'z/MTPCandidateGenerator.update_candidate_strategysr!r)r(r)r*rr,r-rrrrrrtr.r/r r2r0r1r r'r3r!rrrs!#'D'''.Dd-KDcNKKK=A /*/*%/*./*38n /* ##89 /*/*/*/*bS/#S/38nS/# S/ ! S/  S/ u!2T!99 :S/S/S/S/jr!rr8 new_lengthrZrc|rdnd}||vr|S||}||jdz }|dkr|ddd|f||<nA|dkr;tj|||jd|fgd||<d|vrl|d}|dkr|ddd|f|d<n|dkrE|ddddddddfd|dd}tj||gd|d<nkd |vrg|d }|dkr|ddd|f|d <nG|dkrA|ddddddfd|d}tj||gd|d <|S) zNExpands or crops the model's mask for decoding purposes, to the defined lengthrArEr rNrrcross_attention_maskimage_attention_mask)rbr.rnew_onesrepeat)r8rrZmask_keyrmask_length_diff cross_masknew_masks rrr!s,>S''CSH|##  !D!DJqM1!!%aaa):*:):&:!; X A  !&D$--AP`@a2b2b+cik!l!l!l X--!"89 a  3=aaaARBRAR>R3SL/ 0 0  ! !!!!!RSS!!!QQQ,/66q:JAqQQH389j(=SYZ3[3[3[L/ 0 < / /!"89 a  3=aaaARBRAR>R3SL/ 0 0  ! !!!!!RSS!!!),33A7GKKH389j(=SYZ3[3[3[L/ 0 r!c|rdnd}|||S||}||jdz }|dkr|ddd|f||<n{|dkrudg|dz zdgz}tj||j|jj||dddfzdz}tj||gd }|||<|S) zVExpands or crops the model's position ids for decoding purposes, to the defined lengthdecoder_position_idsrNrrr r1.r) r`rbrr.arangerDrCviewr)r8rrZ position_key positionsposition_length_diff required_dimnext_position_idss rrrEs.@S))^L %%-\*I% (;;a%.qqq2G3G2G/G%H \""  ! !simmoo12bT9 cEL-Y_YM] ^ ^ ^ ceq rRSS! "   "Iy2C&D"MMM%6 \" r!cL|d|S|d}|dddfd}||jdz }|dkr|ddd|f|d<n=|dkr7|d|}t j|d|gd|d<|S)zXExpands or crops the model's token_type_ids for decoding purposes, to the defined lengthtoken_type_idsNrr rr)r`r$rbrr.r)r8rr,final_token_typetype_length_difftoken_type_copiess rrr_s())1""23N%aaae,66r::!N$8$;;!)7;Lr<sF $$$$$$555555555555 ......55555555111111hhhhhhhhhh*))))))6000000AAAAAA555555& & & & & & & & RD/D/D/D/D/!3D/D/D/NYYYYY4NYYYxRY.55555555D````````F3(3(3(3(3(3(3(3(lt>t>t>t>t>,Yt>t>t>nYYYYY%7YYYx5/5/5/5/5/"<5/5/5/p~/~/~/~/~/1K~/~/~/BMMMMM6MMM`!$sCx.!c!_c!hlmprumuhv!!!!HS#XC]afjknpsksft4$sCx.cdSVX[S[nr!