(Y KKD^DTTUVW  & .!" E#N# $!&&(GMM9  '5#/gs Cc<\rSrSr\SSj5r\SSj5rSrg)RwkvLinearAttentionONc UR5upxn U[R:a [SUS[RS35eXy-[ U S5-S:wa[SUSU S[ U S5S 35eUR UlURRS :wdNURRS :wd4URRS :wdURRS :wa [S 5e[R"UR5R55*nUR [R:Xa0UR5nUR5nUR5nUR5nUR5nUR5n[R"U[RS 9n U(dUbUcT[R "UU S [R"UR[RS9nUSS2SS2S4==S-ss'OA[R$"UV s/sHoR'S5PM sn SS9R5nUR [R(:Xa[R*n O[R,n U "XX4X5 OHUR [R(:Xa[R.O[R0n U "XX4U 5 UR3XX4U 5 Ub3[R4"US SS9V s/sHoR7S5PM nn U R9UR 5U4$s sn fs sn f)NzCannot process a batch with z+ tokens at the same time, use a maximum of z with this model. rzThe product of batch size (z) and hidden size (z") needs to be a round multiple of rcudazUCalling the CUDA kernel for wkv attention requires all tensors to be on CUDA devices. memory_formatr)dtypedevicer7籡*G)dim)sizer"r# ValueErrorminr8 input_dtyper9typetorchexpfloat contiguousfloat16 empty_likecontiguous_formatzerosfloat32cat unsqueezebfloat16forward_with_state_bf16forward_with_state forward_bf16forwardsave_for_backwardchunksqueezeto) ctx time_decay time_firstkeyvaluestate return_state batch_sizeseq_len hidden_sizeoutputs forward_funcs r.rQRwkvLinearAttention.forwardPs+.88:( [ %44 4.wi7b#2233DF   #c+r&: :a ?-j\9L[MZ";34A7  ))    " "f ,  %%/zz&(||  F*tu uii 0 0 2 = = ?@@ 99 %#))+J))+CKKME**, nn  "!!#U5L5LM 5,} --::"'"9"9 aAg$& 5"A5a;;q>5"AqITTVyyENN*/GG /BB V K>$A%..u}} '' %BYBYZ   E4K4KL""58O8OP %-- '~~'H:E:W(66]m]v]v          !      OOK ( OOK ( HH[ ! JJ{ #     NFN)__name__ __module__ __qualname____firstlineno__ staticmethodrQrg__static_attributes__rqrpr.r1r1Os)<1<1|% % rpr1cUR5upgn[R"U5nUc[R"USS2S4[RS9n [R"USS2S4[RS9n [R"USS2S4[RS9S- n OUupn [R"U5*n[ U5GH n USS2U 4R 5n USS2U 4n[R"XU-5n[R"X- 5n[R"X-U- 5nUU -UU--nUU -U-nUU- RUR5USS2U 4'[R"X-U 5n[R"X-U- 5n[R"U U- 5nUU -UU--n UU -U-n Un GM U(dUbXU /nX4$)Nr)r8r;) r=rB zeros_likerJrCrangerDmaximumrUr8)rWrXrYrZr[r\_ seq_lengthr` num_state den_state max_state current_index current_key current_valuemax_for_outpute1e2 numerator denominator max_for_states r.rwkv_linear_attention_cpursxxzA1   c "F }$$SAYemmD $$SAYemmD $$SAYemmDtK */' i ))J''Jz* !]*+113 a./ y 2JK YYy1 2 YY{/.@ ANR-%77 9nr) $- $;#?#? #Mq-  i& YY{]2 3NR-%77 NR' ! %+(u(y1 =rpc [SXX#455nURS5S:Hn[bU(dU(a [XX#XES9$[R XX#XE5$)Nc3P# UHoRRS:gv M g7f)r5N)r9rA).0ts r. (rwkv_linear_attention..sX3Wa((--6)3Ws$&rr[r\)anyr=r"rr1apply)rWrXrYrZr[r\no_cuda one_tokens r.rwkv_linear_attentionrsZXJC3WXXG q I7i(SXtt"((Uaarpc@^\rSrSrSU4SjjrSSjrSSjrSrU=r$) RwkvSelfAttentionc>[TU]5 Xl[SL=(a [RUR :Hn[ 5(a,[5(aU(d[UR 5 X l URnURb UROUnXPl[R "["R$"U55Ul[R "["R$"U55Ul[R "["R$"SSU55Ul[R "["R$"SSU55Ul[R "["R$"SSU55Ul[R0"S5Ul[R4"XESS9Ul[R4"XESS9Ul[R4"XESS9Ul[R4"XTSS9Ulg![a [RS5 GNf=f)Nz9Could not load the custom CUDA kernel for RWKV attention.rrrrFbias)super__init__configr"r#r(rrr/ Exceptionr$r%layer_idr_attention_hidden_sizer ParameterrBemptyrWrX time_mix_keytime_mix_valuetime_mix_receptance ZeroPad2d time_shiftLinearrYrZ receptancer`)selfrr kernel_loadedr_r __class__s r.rRwkvSelfAttention.__init__s  (4q9I9X9X\b\q\q9q   $;$=$=m Y$V%:%:;! (( ,2,H,H,TF ( (Ze &;",,u{{3H'IJ,,u{{3H'IJLLQ;)GH ll5;;q![+IJ#%<< Aq+0N#O ,,}599[eLYY{N ))KUSii 5O ) Y WX Ys(H%%IIcpURS5S:XaUbUSSS2SS2UR4nO4URU5nUb USSS2SS2UR4USS2S4'XR-USUR- --nXR-USUR- --nXR -USUR - --nUR U5nURU5n[R"URU55nUb USS2S4USSS2SS2UR4'XdXR4$Nrrr) r=rrrrrrYrZrBsigmoidr)rhiddenr[shiftedrYrZrs r.extract_key_value#RwkvSelfAttention.extract_key_values2 ;;q>Q 5#4Ahq!T]]23Goof-G %aAt}})< =1 (((7a$:K:K6K+LL,,,w!d>Q>Q:Q/RR666AH`H`D`9aa hhsm 5!]]4??:#>?  ,21b5ME!HQ4==( ),,rpc ^TRXS9upEpbUb[U4SjUSS55OSn[TRTRUUUUS9upUbTUSUSSS2SS2TR 4'USUSSS2SS2TR 4'USUSSS2SS2TR 4'TR XH-5U4$) Nr[c3L># UHoSS2SS2TR4v M g7frsr)rrars r.r,RwkvSelfAttention.forward..$sFIqaDMM12Is!$r:rrrr)rtuplerrWrXrr`) rrr[ use_cacherrYrZ layer_staters ` r.rQRwkvSelfAttention.forward"s(,(>(>v(>(S% JOJ[eFE!"IFFae 1 OO OO  "    ",7NE!HQ4==( ),7NE!HQ4==( ),7NE!HQ4==( ){{:,-u44rp) rrrYrr`rrWrXrrrrrZrrsrr) rtrurvrwrrrQry __classcell__rs@r.rrsP<-&55rprc6^\rSrSrSU4SjjrSSjrSrU=r$)RwkvFeedForwardi6c8>[TU]5 XlX lURnUR b UR OSUR-n[ R"S5Ul[ R"[R"SSU55Ul [ R"[R"SSU55Ul [ R"X4SS9Ul[ R"X3SS9Ul[ R"XCSS9Ulg)NrrrFr)rrrrr_intermediate_sizerrrrrBrrrrrYrrZ)rrrr_rrs r.rRwkvFeedForward.__init__7s   (( (.(@(@(LF $ $RSV\VhVhRh ,,}5LLQ;)GH#%<< Aq+0N#O 99[%H))K5IYY0EJ rpcURS5S:XaUbUSSS2SS2UR4nO4URU5nUb USSS2SS2UR4USS2S4'XR-USUR- --nXR-USUR- --n[ R "[ R"URU555nURU5n[ R"URU55nUb USS2S4USSS2SS2UR4'XV-U4$r) r=rrrrrBsquarerelurYrZrr)rrr[rrYrrZs r.rQRwkvFeedForward.forwardHs# ;;q>Q 5#4Ahq!T]]23Goof-G %aAt}})< =1 (((7a$:K:K6K+LL666AH`H`D`9aa ll5::dhhsm45 3]]4??:#>?  ,21b5ME!HQ4==( )!5((rp)rrYrrrrrrZrrsrtrurvrwrrQryrrs@r.rr6sK"))rprc2^\rSrSrU4SjrSSjrSrU=r$) RwkvBlocki\c>[TU]5 XlX lUS:Xa.[R "UR URS9Ul[R "UR URS9Ul [R "UR URS9Ul [X5Ul [X5Ulg)Nr)eps)rrrrr LayerNormr_layer_norm_epsilonpre_lnln1ln2r attentionr feed_forward)rrrrs r.rRwkvBlock.__init__]s   q=,,v'9'9v?X?XYDK<< 2 28Q8QR<< 2 28Q8QR*6<+F=rpcURS:XaURU5nURURU5X#S9upRX-nUR UR U5US9upbX-nX4nU(aXu4- nU$US- nU$)Nr)r[rrrs)rrrrrr)rrr[routput_attentionsrroutputss r.rQRwkvBlock.forwardks ==A [[(F>>$((6*:%>] #"//0@/N &/  | #G w Grp)rrrrrrr)NFFrrs@r.rr\s >rprc\\rSrSr%\\S'SrS/rSS/rSr Sr S\ R4S jr S rg ) RwkvPreTrainedModeli~rrrrWrXTmodulec  [U[5(GaUURnURRnURR nUR nX#S- - nSX#- - n[R"[U5Vs/sHoU- PM snURRURRS9n U SSSS24n [U5V s/sHn SSXS- - SSU-----PM n n [R"XRRURRS9n [R"[U5Vs/sH oS-S -S- PM snURRURRS9S -n XRl[R "UR["R$"S 5-U -5URl[R&"X5URl[R&"X5S U--UR(l[R&"U S U-5UR*lg[U[,5(aURnURRnURR nSX#- - n[R"[U5Vs/sHoU- PM snURRURRS9n U SSSS24n [R&"X5URl[R&"X5UR*lg[U[.R05(aUR2RR4n SnSnUR6b$UR6RR95 U S U S:a["R:"U S U S- 5nU S URR<:XaU SURR :XaS nX-n[.R>RAUR2US 9 g[U[.RB5(arUR2RR4n S["R:"[EU S U S55-n[.R>RAUR2US 9 g[U[.RF5(aJUR2RRIS5 UR6RR95 ggs snfs sn fs snfs snf)zInitialize the weights.rg?r8r9Ngffffff?g?rg?g333333?r)gaing-C6?)% isinstancerrrnum_hidden_layersr_rrBtensorr|rr8r9rWrXdata ones_likemathlogpowrrrrrweightshaperzero_sqrt vocab_sizeinit orthogonal_ Embeddingmaxrfill_)rrrrr_r ratio_0_to_1ratio_1_to_almost0i time_weighth decay_speedzigzagrrscales r. _init_weights!RwkvPreTrainedModel._init_weightssu f/ 0 0H & ? ?  --33K$*$@$@ !#1'<=L!$(D!E ,,*/ *<=* L*>s U1U6U; VrqN)rtrurvrwr__annotations__base_model_prefix_no_split_modules_keep_in_fp32_modulessupports_gradient_checkpointing _is_statefulrModulerryrqrpr.rr~s> $ )<8&*#LI%BIII%rprz+ Class for the RWKV model outputs. ) custom_introc\rSrSr%SrSr\\R\ S'Sr \\ \R\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S'S rg) RwkvOutputiz state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Nlast_hidden_stater[. hidden_states attentionsrq)rtrurvrw__doc__rrrB FloatTensorrr[listrrrryrqrpr.rrsw 6:x 1 129/3E8D**+ ,3=AM8E%"3"3S"89:A:>Ju00#567>rprzK Base class for causal language model (or autoregressive) outputs. c\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr \\\RS4\ S'Sr\\\RS4\ S 'S rg) RwkvCausalLMOutputiaP loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Nlosslogitsr[.rrrq)rtrurvrwrrrrBrrrr[rrrrryrqrpr.rrs)-D(5$$ %,*.FHU&& './3E8D**+ ,3=AM8E%"3"3S"89:A:>Ju00#567>rprc^\rSrSrU4SjrSrSr\SS\\ RS\\ RS\\ RS\\ \ RS \\ S \\ S \\ S \\ S \\\44Sjj5rSrSrSrU=r$) RwkvModelic >[TU]U5 [R"URUR 5Ul[R"[UR5Vs/sH n[XS9PM sn5Ul [R"UR 5Ul SUlSUlUR!5 gs snf)NrF)rrrrrr_ embeddings ModuleListr|rrblocksrln_outlayers_are_rescaledgradient_checkpointing post_init)rridxrs r.rRwkvModel.__init__s  ,,v'8'8&:L:LMmmPUV\VnVnPo$pPoYv%DPo$pq ll6#5#56 #( &+# %qs(C cUR$rsrrs r.get_input_embeddingsRwkvModel.get_input_embeddingss rpcXlgrsr'rnew_embeddingss r.set_input_embeddingsRwkvModel.set_input_embeddingss(rp input_idsattention_mask inputs_embedsr[rroutput_hidden_states return_dictreturnc UbUOURRnUbUOURRnUbUO(UR(dURROSnUbUOURR nUb[ RS5 URUR:XaUR5 UbUb [S5eUcUc [S5eUcURU5nU(aUcURS5URRURR4n [S5V s/sHCn [ R""XS::a UR$O[ R&UR(S .6PME nn US ==S -ss'UR*(a/UR(aU(a[ RS 5 SnUn U(aS OSn U(aS OSn [-UR.5HzupU"XXVS9upnUR(a?URR0S:a%US-URR0-S:XaU S- n U(aX4-n U(dMtU U4-n M| UR3U 5n U(aX4-n U(d[5SXX455$[7U UU U S9$s sn f)a( input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. NFz<`attention_mask` was passed, but it is unused in this model.zDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsrrrrgꌠ9Y>)FzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...rq)r[rrr:c3,# UH ocMUv M g7frsrq)rxs r.r$RwkvModel.forward..ust$bq$bs )rr[rr)rrr3trainingruse_return_dictr$ warning_oncer!_rescale_layersr>rr=r_rr|rBrIr8rJr9r" enumerater rescale_everyr rr)rr0r1r2r[rrr3r4rrrall_self_attentionsall_hidden_statesr$blockrs r.rQRwkvModel.forwards>2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IZ^ZgZgT[[=R=Rmr %0%cd d  =#8TU U   OOI6M "''*DKK,C,CT[[EbEbcE q "A a-"5"5U]][h[o[o"  !H H  & &4==##p" % $5b4"6BD#DKK0JC/4i0 ,M* ((KK--11W 9 99Q> - 1 #$58H$H!  &9ZM&I#!1$ M2  14D D t];L$btt t++*   [s A Kc URUR(+:XagURRS:Ga[R "5 [ UR5GHupUR(aURRRRS[XRR-5-5 URRRRS[XRR-5-5 M[URRRS5(aURRRR R#S[XRR-5-5 URRRR R#S[XRR-5-5 GM[URRRS5(aOUR%URRU5 UR%URRU5 GM!URRRR#S[XRR-5-5 URRRR#S[XRR-5-5 GM SSS5 UR(+Ulg!,(df  N%=f)Nrr:SCB quant_state)r!r;rr@rBno_gradr?rrr`rmul_intrrZhasattrrFdiv_ _bnb_4bit_dequantize_and_rescale)rblock_idrCs r.r>RwkvModel._rescale_layers~s?  # #DMM(9 :  ;; $ $q ('0'=OH}}..55::1HP[P[PiPiDi@j;jk**0077<>qCT_T_TmTmHmDn?no!..44;;@@c(VaVaVoVoJoFpApq(>!"(,}}#4 #!s KM Mc[5(d [S5eSSKnURR UR R UR R5nURS[X RR-5-5 URRURS5SS9RUR5n[!USU5 g) z Perform the dequantization and rescaling of the weights of a given layer. After that operation the layer will be quantized again. z/Please install bitsandbytes to use this method.rNr:cpuF) requires_gradr)r ImportError bitsandbytes functionaldequantize_4bitrrrGrLrJrr@r Params4bitrUr9setattr)r target_layerrNbnbdequant_weights quant_weights r.rM*RwkvModel._bnb_4bit_dequantize_and_rescales )**OP P"..889L9L9Q9QS_SfSfSrSrsQ#h++2K2K&K"LLMvv((););E)BRW(X[[\k\r\rs  h 5rp)rrr"r!r )NNNNNNNN)rtrurvrwrr)r.r rrB LongTensorrrboolrrrrQr>rMryrrs@r.rrs )15595937$(,0/3&*g E,,-g !!1!12g   1 12 g U../0 g D> g $D>g 'tng d^g  uj !g g R5066rprz The RWKV Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). c@^\rSrSrS/rU4SjrSrSrSSjr\ SS\ \ RS\ \ RS \ \ RS \ \\ RS \ \ RS \ \S \ \S\ \S\ \S\\\44Sjj5rSrU=r$)RwkvForCausalLMiz head.weightc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 g)NFr) rrrrrrr_rheadr#)rrrs r.rRwkvForCausalLM.__init__sH  f% IIf00&2C2C%P  rpcUR$rsrcr(s r.get_output_embeddings%RwkvForCausalLM.get_output_embeddingss yyrpcXlgrsrfr,s r.set_output_embeddings%RwkvForCausalLM.set_output_embeddingss" rpc jUbUSS2S4RS5nUbUcSU0nOSU0nX&S'XFS'U$)Nrr2r0r[r)rL)rr0r[r2rkwargs model_inputss r.prepare_inputs_for_generation-RwkvForCausalLM.prepare_inputs_for_generationsY  !!R%(2226I  $+];L'3L %W$-[!rpr0r1r2r[labelsrrr3r4r5c xU bU OURRn URUUUUUUU S9n U Sn URU 5n SnUb*UR"U U4SURR 0U D6nU (dU 4U SS-nUbU4U-$U$[ UU U RU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. N)r2r[rrr3r4rrr)rrr[rr) rr<rrc loss_functionrrr[rr)rr0r1r2r[rqrrr3r4rm rwkv_outputsrrrr`s r.rQRwkvForCausalLM.forwardsJ&1%F $D>F 'tnF d^F  u(( )F F rpra)rarrrr)/rr dataclassesrpathlibrtypingrrrBtorch.utils.checkpointr generationr modeling_layersr modeling_utilsr utilsr r rrrrconfiguration_rwkvr get_loggerrtr$r"r/autogradFunctionr1rrr rrrrrrrra__all__rqrpr.rs{  !" )9-+   H %5@g %..11g T)XbC5 C5L#)bii#)L*DQ%/Q%Q%h  ? ?  ?  ??  ?$j6#j6j6Z i )?i  i X Brp