tj0}dZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZmZmZmZmZmZmZd d lmZejeZdad ZGddejjZ d(dZ!d(dZ"Gddej#Z$Gddej#Z%Gdde Z&eGddeZ'edeGddeZ(edeGd d!eZ)eGd"d#e'Z*ed$Gd%d&e'e Z+gd'Z,dS))zPyTorch RWKV model.N) dataclass)nn)initialization)GenerationMixin)GradientCheckpointingLayer)PreTrainedModel) ModelOutputauto_docstringis_bitsandbytes_availableis_kernels_availableis_ninja_availableis_torch_cuda_availablelogging) RwkvConfigc~tstdddlm}|dda|t_dS)NzFkernels is not installed, please install it with `pip install kernels`r) get_kernelzkernels-community/rwkvr)version)r ImportErrorintegrations.hub_kernelsrrwkv_cuda_kernelmax_seq_length)context_lengthrs d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/rwkv/modeling_rwkv.pyload_wkv_cuda_kernelr-sW  ! !dbccc666666!z":AFFF&4###c>eZdZeddZeddZdS)RwkvLinearAttentionNFc B|\}}} |tjkr td|dtjd|| zt | dzdkr'td|d| dt | dd |j|_|jjd ks0|jjd ks |jjd ks|jjd krtd tj |  }|jtj kr<| }| }| }| }| }| }tj|tj } |s||Ktj|| d tj|jtj}|dddddfxxdzcc<n2tjd|Dd }|jtjkr tj} n tj} | ||||| |n<|jtjkr tjn tj} | ||||| |||||| |!dtj|d dD}| |j|fS)NzCannot process a batch with z+ tokens at the same time, use a maximum of z with this model. rzThe product of batch size (z) and hidden size (z") needs to be a round multiple of .cudazUCalling the CUDA kernel for wkv attention requires all tensors to be on CUDA devices. memory_formatr)dtypedevicer%籡*Gc8g|]}|dSr() unsqueeze.0ss r z/RwkvLinearAttention.forward..gs""A"A"Aa1;;q>>"A"A"Ar)dimc8g|]}|dSr+)squeezer-s rr0z/RwkvLinearAttention.forward..ts"HHHaQYYq\\HHHr)sizerr ValueErrorminr& input_dtyper'typetorchexpfloat contiguousfloat16 empty_likecontiguous_formatzerosfloat32catbfloat16forward_with_state_bf16forward_with_state forward_bf16forwardsave_for_backwardchunkto) ctx time_decay time_firstkeyvaluestate return_state batch_sizeseq_len hidden_sizeoutput forward_funcs rrGzRwkvLinearAttention.forward9sG+.88::( G[ %4 4 4FwFF#2FFF   #c+r&:&: :a ? ?7j77[77";33777  )   "f , , %//z&((| F**tuu ui 0 0 2 2 = = ? ?@@@ 9 % %#))++J))++CKKMME**,, nn  ""!#U5LMMM  E5,} -:"'"9 aaaAg$& "A"A5"A"A"AqIIITTVVyEN**/G /B LZeVU K K K Kr?rCrAr=r;r backward_bf16backwardr<rJ)rKg_outputg_stater7rLrMrNrOrU g_time_decay g_time_firstg_keyg_value backward_funcs rrZzRwkvLinearAttention.backwardxseo 585F2 JUF' 1$/5>$A$A%..u}   ' %BYZZZ  E4KLLL"58OPPP %- ' '~~''H:E:W:W(66]m]v          ! !      OOK ( ( OOK ( ( HH[ ! ! JJ{ # #     rNFN)__name__ __module__ __qualname__ staticmethodrGrZrrrr8sS<1<1<1\<1|% % % \% % % rrFc|\}}}tj|}|tj|dddftj} tj|dddftj} tj|dddftjdz } n|\} } } tj| }t |D] } |dd| f} |dd| f}tj| | |z}tj| |z }tj| |z|z }|| z||zz}|| z|z}||z |j |dd| f<tj| |z| }tj| |z|z }tj| |z }|| z||zz} || z|z} |}  |s|| | | g}||fS)Nr)r&r)) r4r9 zeros_likerAr:ranger;maximumrJr&)rLrMrNrOrPrQ_ seq_lengthrU num_state den_state max_state current_index current_key current_valuemax_for_outpute1e2 numerator denominator max_for_states rrwkv_linear_attention_cpur{s$xxzzAz1  c " "F }$SAYemDDD $SAYemDDD $SAYemDDDtK */' 9i )J'''Jz**"" !!!]*+1133 aaa./ y+ 2JKK Yy>1 2 2 Y{Z/.@ A ANR-%77 9nr) $- $;#?#? #M#Mqqq-  i*& > Y{]2 3 3NR-%77 NR' ! 2u(Iy1 5=rctd||||fD}|ddk}t|s|rt||||||St||||||S)Nc36K|]}|jjdkVdS)r#N)r'r8)r.ts r z(rwkv_linear_attention..s+XXa!(-6)XXXXXXrrrPrQ)anyr4rr{rapply)rLrMrNrOrPrQno_cuda one_tokens rrwkv_linear_attentionrsXXJ CQV3WXXXXXG q I7i(ZeSXgstttt"((ZeUT`aaarc0eZdZdfd ZddZd dZxZS) RwkvSelfAttentionrcdt||_tduotj|jk}t rPtrB|s@ t|jn*#t$rt dYnwxYw||_ |j }|j|jn|}||_tjt#j||_tjt#j||_tjt#jdd||_tjt#jdd||_tjt#jdd||_tjd|_tj||d|_tj||d|_tj||d|_tj||d|_dS)Nz9Could not load the custom CUDA kernel for RWKV attention.rrrrFbias)super__init__configrrrrrr Exceptionloggerinfolayer_idrTattention_hidden_sizer Parameterr9emptyrLrM time_mix_keytime_mix_valuetime_mix_receptance ZeroPad2d time_shiftLinearrNrO receptancerU)selfrr kernel_loadedrTr __class__s rrzRwkvSelfAttention.__init__s  (4q9I9X\b\q9q    Y$;$=$= Ym Y Y$V%:;;;; Y Y Y WXXXXX Y  ( ,2,H,TF ( (Ze &;",u{3H'I'IJJ,u{3H'I'IJJLQ;)G)GHH l5;q![+I+IJJ#%< Aq+0N0N#O#O ,}559[*?eLLLY{,ANNN )K1FUSSSi 5{OOO s&A;;$B"!B"Ncb|ddkr||ddddd|jf}n8||}|!|ddddd|jf|dddf<||jz|d|jz zz}||jz|d|jz zz}||jz|d|jz zz}||}||}tj | |}|!|dddf|ddddd|jf<||||fSNrrr) r4rrrrrrNrOr9sigmoidr)rhiddenrPshiftedrNrOrs rextract_key_valuez#RwkvSelfAttention.extract_key_valuesW ;;q>>Q  5#4Ahqqq!!!T]23GGoof--G %aAAAt})< =1 t((7a$:K6K+LL,,w!d>Q:Q/RRd66AH`D`9aa hhsmm 5!!]4??:#>#>??  ,2111b5ME!HQQQ4=( )3u,,rFc||\}}}}|#tfd|ddDnd}tjj||||\}}|W|d|dddddjf<|d|dddddjf<|d|dddddjf<||z|fS) NrPc3@K|]}|ddddjfVdSrcr)r.r/rs rrz,RwkvSelfAttention.forward.. s9FFqAaaaDM12FFFFFFrr(rrrr)rtuplerrLrMrrU) rrrP use_cacherrNrO layer_staterwkvs ` rrGzRwkvSelfAttention.forward s(,(>(>vU(>(S(S% CJOJ[eFFFFE!""IFFFFFFae 1 O O  "    k  ",7NE!HQQQ4=( ),7NE!HQQQ4=( ),7NE!HQQQ4=( ){{:,--u44rrrcrb)rdrerfrrrG __classcell__rs@rrrskPPPPPP<----&55555555rrc(eZdZdfd ZddZxZS)RwkvFeedForwardrc0t||_||_|j}|j|jn d|jz}t jd|_t j tj dd||_ t j tj dd||_ t j||d|_t j||d|_t j||d|_dS)NrrrFr)rrrrrTintermediate_sizerrrrr9rrrrrNrrO)rrrrTrrs rrzRwkvFeedForward.__init__ s    ( (.(@(LF $ $RSV\VhRh ,}55LQ;)G)GHH#%< Aq+0N0N#O#O 9[*;%HHH)K5IIIY0+EJJJ rNc||ddkr||ddddd|jf}n8||}|!|ddddd|jf|dddf<||jz|d|jz zz}||jz|d|jz zz}t jt j||}| |}t j | |}|!|dddf|ddddd|jf<||z|fSr) r4rrrrr9squarerelurNrOrr)rrrPrrNrrOs rrGzRwkvFeedForward.forward1sK ;;q>>Q  5#4Ahqqq!!!T]23GGoof--G %aAAAt})< =1 t((7a$:K6K+LLd66AH`D`9aa l5:dhhsmm4455 3]4??:#>#>??  ,2111b5ME!HQQQ4=( )E!5((rrrcrdrerfrrGrrs@rrrsWKKKKKK"))))))))rrc&eZdZfdZddZxZS) RwkvBlockct||_||_|dkr%t j|j|j|_t j|j|j|_ t j|j|j|_ t|||_ t|||_dS)Nr)eps)rrrrr LayerNormrTlayer_norm_epsilonpre_lnln1ln2r attentionr feed_forward)rrrrs rrzRwkvBlock.__init__Fs    q==,v'9v?XYYYDK< 28QRRR< 28QRRR*68<<+FH==rNFc4|jdkr||}|||||\}}||z}||||\}}||z}||f}|r||fz }n|dz }|S)Nr)rPrrrc)rrrrrr)rrrProutput_attentionsrroutputss rrGzRwkvBlock.forwardTs =A  [[((F>>$((6*:*:%S\>]] 5)#"//0@0@/NN e,&5/    | #GG w Gr)NFFrrs@rrrEsL > > > > >rrc~eZdZUeed<dZdgZddgZdZdZ e j de j ffd ZxZS) RwkvPreTrainedModelrrrrLrMTmodulec  t|t|tr|j}|jj}|jj |j ||dz z d||z z }tj fdt D|j j |j j}|ddddf} fdt D}tj ||jj |jj}tj dt D|jj |jjdz}t#j|j|t#j|jtj|jt)jd z|zt#j|j tj||t#j|jtj||d zzt#j|jtj|d|zdSt|t2r|j}|jj}|jj d||z z }tj fd t D|j j |j j}|ddddf}t#j|j tj||t#j|jtj||dSt|t4jr|jj}d} d} |jt#j|j|d |dkr#t)j |d |dz } |d |jj!kr|d|jjkrd} | | z} t#j"|j| dSt|t4j#rZ|jj}d t)j tI|d |dz} t#j"|j| dSdS)zInitialize the weights.rg?cg|]}|z Srhrhr.irTs rr0z5RwkvPreTrainedModel._init_weights..~===Q[===rr&r'Nc>g|]}dd|dz z ddzzzzzS)rgffffff?g?rh)r.hr ratio_0_to_1s rr0z5RwkvPreTrainedModel._init_weights..sMQ!4q89sS.s$KKKa!eq[1_KKKrg?g333333?cg|]}|z Srhrhrs rr0z5RwkvPreTrainedModel._init_weights..rrr)gaing-C6?)%r _init_weights isinstancerrrnum_hidden_layersrTrr9tensorrkrr&r'rLrMinitcopy_ ones_likemathlogpowrrrrrweightshaperzeros_sqrt vocab_size orthogonal_ Embeddingmax)rrrrratio_1_to_almost0 time_weight decay_speedzigzagrrscalerrTrrs @@@rrz!RwkvPreTrainedModel._init_weightspsa f%%% f/ 0 0D 7H & ?  -3K$*$@ !#'81'<=L!$3D(D!E ,====% *<*<===)/*1K &dD!!!m4K455K ,{&:K:QZ`ZkZrsssK KKe4I.J.JKKK +1!,3    Jv(+ 6 6 6 Jv(%/&:KdhWZmm:[^d:d*e*e f f f Jv*EIkCU,V,V W W W Jv,ei EW.X.X[^am[m.m n n n Jv159[#PbJb3c3c d d d d d  0 0 7H & ?  -3K!$3D(D!E ,====% *<*<===)/*1K &dD!!!m4K Jv*EIkCU,V,V W W W Jv159[J\3]3] ^ ^ ^ ^ ^  * * 7M'EDE{& FK(((Qx%(""yqE!H!455Qx4;111eAh$+BY6Y6Y EMD  V] 6 6 6 6 6 6  - - 7M'E$)Ca%($;$;<<M5*C/047>>>7;Je',-4;;;;;rrzK Base class for causal language model (or autoregressive) outputs. ceZdZUdZdZejdzed<dZejdzed<dZ e ejdzed<dZ e ejdfdzed<dZ e ejdfdzed<dS) RwkvCausalLMOutputap loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. NlosslogitsrP.rr)rdrerfrrr9rrrrPrrrrrhrrrrs&*D% d ")))'+FE  $+++,0E4! "T )000:>M5*C/047>>>7;Je',-4;;;;;rrceZdZfdZdZdZe ddejdzdejdzdej dzde ej dzd e dzd e dzd e dzd e dzd e e zfdZdZdZxZS) RwkvModelcttjjj|_tjfdtj D|_ tj j|_ d|_ d|_|dS)Nc2g|]}t|S)r)r)r.idxrs rr0z&RwkvModel.__init__..s&$p$p$pYv%D%D%D$p$p$prF)rrrrrrT embeddings ModuleListrkrblocksrln_outlayers_are_rescaledgradient_checkpointing post_initrrrs `rrzRwkvModel.__init__s    ,v'8&:LMMm$p$p$p$pPUV\VnPoPo$p$p$pqq l6#566 #( &+# rc|jSrcr rs rget_input_embeddingszRwkvModel.get_input_embeddingss rc||_dSrcrrnew_embeddingss rset_input_embeddingszRwkvModel.set_input_embeddingss (rN input_idsattention_mask inputs_embedsrPrroutput_hidden_states return_dictreturnc  :||n |jj}||n |jj}||n|js |jjnd}||n |jj}|t d|j|jkr| |td|td| ||rZ|X d|jj |jjffdtdD}|d xxd zcc<|jr%|jr|rt d d}} |rd nd} |rd nd} t#|jD]Z\} }|| ||| \} }}|jr+|jjdkr| dz|jjzdkr| dz } |r| | fz} |r| |fz} [|| } |r| | fz} |st+d| || | fDSt-| || | S)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. NFz<`attention_mask` was passed, but it is unused in this model.zDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsrclg|]0}tj|dkrjn tjjd1S)rr)r9r@r&rAr')r.rrrs rr0z%RwkvModel.forward..3sY a-"5"5U][h[orrgꌠ9Y>)FzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...rh)rPrrrr(c3K|]}||V dSrcrh)r.xs rrz$RwkvModel.forward..^s(ttqfgfsfsfsfsfsttr)rrPrr)rrrtrainingrrr warning_oncer_rescale_layersr5r r4rTrrkr enumerater  rescale_everyrrr)rrrrrPrrrrkwargsrall_self_attentionsall_hidden_statesr blockrrs ` @rrGzRwkvModel.forwards@2C1N--TXT_Tq$8$D $+Jj "+!6IIZ^Zg=rT[=R=Rmr %0%cdd d  =#8TUU U   OOI66M  "''**DK,CT[EbcEq E !HHH HHH  & "4= " "##p" % $5?bb4"6@BBD#DK00 J JJC/4uUiSd000 ,M5* ( 2K-111W 99Q>> - 1 # I$58H$H!  J&9ZM&I# M22  E 1]4D D  utt]E;LNa$bttttt t++*     rc ,|j|j krdS|jjdkrbt j5t |jD] \}}|jr|jj j dt||jjzz|j jj dt||jjzzt|jj j dr|jj j jdt||jjzz|j jj jdt||jjzz=t|jj j drB||jj |||j j||jj j dt||jjzz|j jj dt||jjzz" dddn #1swxYwY|j |_dS)Nrr(SCB quant_state)rr&rr*r9rr)r rrUrmul_intrrOhasattrr0div_ _bnb_4bit_dequantize_and_rescale)rblock_idr.s rr(zRwkvModel._rescale_layersgs  #DM(9 : : F ; $q ( ( r r'0'='=rrOHe} r.5::1HPTP[PiDi@j@j;jkkk*07<>qCTXT_TmHmDnDn?nooo!.4;@@c(VZVaVoJoFpFpApqqqqr r r r r r r r r r r r r r r r"(,}#4   sH7I<<JJctstdddl}|j|jj|jj}|dt||j j zz|j |dd|j}t!|d|dS) z Perform the dequantization and rescaling of the weights of a given layer. After that operation the layer will be quantized again. z/Please install bitsandbytes to use this method.rNr(cpuF) requires_gradr)r r bitsandbytes functionaldequantize_4bitrdatar1r5r3rr*r Params4bitrJr'setattr)r target_layerr7bnbdequant_weights quant_weights rr6z*RwkvModel._bnb_4bit_dequantize_and_rescales )** QOPP P"""".889L9QS_SfSrssQ#h$+2K&K"L"LLMMMv((););E)B)BRW(XX[[\k\rss  h 55555r)NNNNNNNN)rdrerfrrrr r9 LongTensorrrboolrrrGr(r6rrs@rrrsT     ))).2262604!%)-,0#'h h #d*h (4/h (4/ h E%&- h $; h  $;h #Tkh D[h   h h h ^h T55506666666rrz The RWKV Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ceZdZddiZfdZdZdZe ddej dzd ej dzd ej dzd e ej dzd ej dzd e dzde dzde dzde dzde ejzdeezfdZxZS)RwkvForCausalLMz head.weightzrwkv.embeddings.weightct|t||_t j|j|jd|_| dS)NFr) rrrrrrrTrheadrrs rrzRwkvForCausalLM.__init__s`    f%% If0&2C%PPP  rc|jSrcrJrs rget_output_embeddingsz%RwkvForCausalLM.get_output_embeddingss yrc||_dSrcrLrs rset_output_embeddingsz%RwkvForCausalLM.set_output_embeddingss " rNrrrrrPlabelsrrrrlogits_to_keepr c | | n |jj} |||||||| } | d} t| trt | dn| }|| dd|ddf}d}||jd|||jjd| }| s|f| ddz}||f|zn|St||| j | j | j S)aJ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. N)rrPrrrrr)rrPrr)rrrPrrrh) rrrrr3slicerJ loss_functionrrrPrr)rrrrrPrPrrrrrQr+ rwkv_outputsr slice_indicesrrrUs rrGzRwkvForCausalLM.forwards2L&1%SV8W8Wk~ot444]k =M111)<=>>  %4%pVFt{OeppioppD FYabb!11F)-)9TGf$$v E!$&4#.     r) NNNNNNNNNr)rdrerf_tied_weights_keysrrMrOr r9rErrrFr3TensorrrrGrrs@rrHrHss()AB###.2262604*.!%)-,0#'-.D D #d*D (4/D (4/ D E%&- D  4' D $;D  $;D #TkD D[D el*D  # #D D D ^D D D D D rrH)rHrrrb)-rr dataclassesrr9rrr generationrmodeling_layersrmodeling_utilsr utilsr r r r rrrconfiguration_rwkvr get_loggerrdrrrautogradFunctionrr{rrrrrrrrrrH__all__rhrrrds !!!!!! &&&&&&))))))999999------+*****  H % %555g g g g g %.1g g g T))))XbbbbC5C5C5C5C5 C5C5C5L#)#)#)#)#)bi#)#)#)L*DP7P7P7P7P7/P7P7P7f   < < < < < < <   <  <<<<<<<   <$k6k6k6k6k6#k6k6k6\ V V V V V )?V V  V r B A Ar