tjrddlmZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z dd lmZdd lmZmZdd lmZmZdd lmZmZdd lmZmZddlmZddlmZm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0m1Z1ddl2m3Z3e!j4e5Z6Gdde+Z7Gdde*Z8Gddej9Z:Gdd e.Z;Gd!d"e0Z<Gd#d$ej9Z=Gd%d&eZ>Gd'd(eZ?e>e?d)Z@Gd*d+eZAe Gd,d-eAZBGd.d/e1ZCGd0d1eeAZDgd2ZEdS)3)CallableN)nn)initialization)ACT2FN)Cache DynamicCache)lazy_load_kernel)force_accelerate_hooks)create_causal_maskcreate_recurrent_attention_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)merge_with_config_defaults)resolve_internal_import)OutputRecordercapture_outputs)LlamaAttention LlamaRMSNormeager_attention_forward) MistralMLP)MixtralExpertsMixtralForCausalLM) JambaConfigceZdZdS) JambaRMSNormN__name__ __module__ __qualname__e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/jamba/modular_jamba.pyr&r&/Dr,r&ceZdZdedeffd Z d dejdejdzdedzde e d e ejejdzff d Z xZ S) JambaAttentionconfig layer_idxct||tj|j|j|jzd|_tj|j|j|jzd|_ tj|j|j|jzd|_ tj|j|jz|jd|_ dSNFbias) super__init__rLinear hidden_sizenum_attention_headshead_dimq_projnum_key_value_headsk_projv_projo_proj)selfr1r2 __class__s r-r8zJambaAttention.__init__4s +++i 2F4NQUQ^4^ejkkk i 2F4NQUQ^4^ejkkk i 2F4NQUQ^4^ejkkk i :T] JFL^ejkkk r,N hidden_statesattention_maskpast_key_valueskwargsreturnc |jdd}g|d|jR}|||dd}|||dd}|||dd} |||| |j\}} tj |j j t} | |||| |f|jsdn|j|jd|\} } | jg|dR} || } | | fS)Nr#r)dropoutscaling)shaper<r=view transposer?r@updater2r get_interfacer1_attn_implementationrtrainingattention_dropoutrMreshape contiguousrA) rBrDrErFrG input_shape hidden_shape query_states key_states value_statesattention_interface attn_output attn_weightss r-forwardzJambaAttention.forward;s$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r,NN)r(r)r*r$intr8torchTensorrrrtupler` __classcell__rCs@r-r0r03sl{lsllllll/3(, ")")|") t+") ") +, ") u|U\D00 1 ")")")")")")")")r,r0ceZdZdZdeffd Z d dejdedzdej dzfdZ d dedzdej dzfd Z e d  d dedzdej dzfd Z xZS) JambaMambaMixeru Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) r1c t||_||_|j|_|j|_|j|_|j |jz|_ |j |_ |j |_|j|_t#j|j |j |j|j|j |jdz |_|j|_t,|j|_t#j|j|j dz|j|_t#j|j |j |jdzzd|_t#j|j |j d|_t9jd|jdzdddf}||j d}t#j t9j!||_"t#j t9j#|j |_$t#j|j |j|j|_%tM|j |j'|_(tM|j|j'|_)tM|j|j'|_*tWd }tY|d da-tY|d da.tWd }t_|d a0tY|dda1tY|dda2tgt`tbt\tZtdfa4thstj6d|j7||_8dS)Nr#) in_channels out_channelsr6 kernel_sizegroupspaddingrr5FTrJepsz causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmz8ops.triton.selective_state_update.selective_state_update) chained_pathselective_scan_fnmamba_inner_fnaThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d.)9r7r8r1r2r: mamba_d_statessm_state_size mamba_d_convconv_kernel_size mamba_expandintermediate_size mamba_dt_ranktime_step_rankmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1d hidden_act activationractr9in_projx_projdt_projrcarangeexpandrW ParameterlogA_logonesDout_projr& rms_norm_eps dt_layernorm b_layernorm c_layernormr getattrrrrsrselective_state_updaterurvallis_fast_path_availablelogger warning_once layer_types layer_type)rBr1r2A causal_conv1d mamba_ssmrCs r-r8zJambaMambaMixer.__init__hs'  "!-$2 & 3!'!4v7I!I$2#3. i./#-))A-     !+&+,y!143IA3MTXTabbb i 68KdNadeNe8elqrrr y!4d6LSWXXX  LD/!3 4 4T111W = HHT+R 0 0 ; ; = =\%)A,,// ej)?@@AA $"8$:JQUQ^___ ()<&BUVVV'(;ATUUU'(;ATUUU)99 &}6LdSS"=2DdKK%[11 !8 $^" " " $I/BDII ,> t  %)N,D,DQ,G,GGM{)..t{/A/F/Fq/I/I4;K]KbKbcdKeKeff ! x0%%b))#DN3?B   M*33B77MM' m// @UXeXklnXo@oqr?stt ..{DNKKK,]L$+JZgkgvwwwM  %)N,D,DQ,G,GGM]%<%>q!DD ]__ 4 4%3DL  " 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4Ytz'')) * * *3A3M--///SW ! O1#DN3DQGf%"6*!!!Q$!!!Q$V     imm L'8" Aq!! Aq!! #"& ' ' ' #L)$)A33It~NNN!% l.D.DQ.J.J K K$$s$3KKKL::L>L>c  |j\}}}|j}||dd}|dd\} } || |dz} |J||jr0|j|jj d } n)tj ||j |jf| j|} |`||jr|dkr|| |jd|j df} tj| |jjdddddfzd} |jr| |jjz } || |d} nt2j| |j| jdz df} || |jd|j df} ||| dd|f} n2||| dd|f} || |dz} || dd} tj| |j|j|jgd\}}}||}| |}|!|}|"|}t2j#|dd}tj$|j%& }tj$|ddddddf|dddddddfz}|dddddddf|dddddddf&z}|| dddddddf&z}g}tO|D]}|dddd|ddf| z|dddd|ddfz} tj(| ||dd|ddfd}|)|dddddftj*|d}|| |j+ddddfzz}||| z}||,| |j|-|dd}|S)Nr#rrr)devicedtype.rJ).rNrrrPrrrr2rrclonerczerosr|rxrrrzsumrrrr6rtorrrrrr~rrrrsoftplusrrrrangematmulappendstackrrr)rB input_statesrrErrrrrrDrr conv_staterrrrrr discrete_A discrete_BdeltaB_uri scan_outputrs r- slow_forwardzJambaMambaMixer.slow_forwards!-!3 GQ"<< 55??1EE.44QA4>> t  %)N,D,DQ,G,GGM  # (G(G(W(W #$+DN;LQOUUWWII T3T5HI$+5I  #..t~>> T7a<<);;M4>ZZ[^aeav`v`w`w[wx % *t{7I!!!QPQPQPQ'7R*RXZ [ [ [ %6!T[%55M $ 7 7 : :5 A A K KB O O ]..!*]-@-DDaH *;;JWWX[^b^s]s]t]tXtu $])C)CC'M)R S S  HHT[[%?%?XgX %NOOM  %)N,D,DQ,G,GGM]%<% J J J!% k.C.CAq.I.I J J$$r,rc|jjrEtrd|jjjjvr&tdd|j_|jjr| |||S| |||S)NcudazFast Mamba kernels are not available. Make sure that they are installed and that the mamba module is on a CUDA device. Turning off the fast path `config.use_mamba_kernels=False` and falling back to the slow path.F) r1use_mamba_kernelsrrrrtyperrrr)rBrDrrEs r-r`zJambaMambaMixer.forward`s ; ( 2& 2*0 8J8Q8V*V*V   V    -2DK ) ; ( Z,,]L.YY Y   nMMMr,ra)r(r)r*__doc__r$r8rcrdr LongTensorrrr r`rfrgs@r-riri`sKC8{C8C8C8C8C8C8P&*26 c%c%|c%dlc%(4/ c%c%c%c%LJ%J%ut|J%\a\los\sJ%J%J%J%ZH%%&*26 NNdlN(4/ NNN&%NNNNNr,riceZdZdS)JambaMLPNr'r+r,r-rrvr.r,rceZdZdS) JambaExpertsNr'r+r,r-rrzr.r,rcReZdZdZdeffd ZdZdejdejfdZ xZ S)JambaSparseMoeBlocka This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accommodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. r1ct|j|_|j|_|j|_|j|_tj |j|jd|_ t||_ dSr4)r7r8r: hidden_dimr|ffn_dim num_expertsnum_experts_per_toktop_krr9routerrexpertsrBr1rCs r-r8zJambaSparseMoeBlock.__init__sr  ,/ !-/ i1ANNN #F++ r,ctjj|dtj}tj||jd\}}|||jfS)NrJ)rrr) rcrrsoftmaxrtopkrrr)rBrD router_logitsrouting_weights top_k_weights top_k_indexs r-route_tokens_to_expertsz+JambaSparseMoeBlock.route_tokens_to_expertss_(-55mSXS^5__%*ZQS%T%T%T" {M,,]-@AAAAr,rDrHc|j\}}}|d|}||}|||\}}||||}||||}|S)NrJ)rNrOrrrrV)rBrDrsequence_lengthrrrrs r-r`zJambaSparseMoeBlock.forwards2?2E/ OZ%**2z::  M22 %)%A%A-Q^%_%_" ] ]KOO %--j/:VV r,) r(r)r*rr$r8rrcrdr`rfrgs@r-rr~s  ,{,,,,,,BBB U\elr,rceZdZdedeffd Z ddejdejdzdejdzd e dzd e dzd e e d ej fd ZxZS)JambaAttentionDecoderLayerr1r2cft|jr |j|nd}t|||_|dkrt nt }|||_t|j |j |_ t|j |j |_ dS)Nr#rp) r7r8layers_num_expertsr0 self_attnrr feed_forwardr&r:rinput_layernormpre_ff_layernormrBr1r2rffn_layer_classrCs r-r8z#JambaAttentionDecoderLayer.__init__s >D>W^f/ ::]^ ' ::1FDWXXX ,V-?VEX Y Y Yr,NFrDrE position_idsrF use_cacherGrHc |}||}|jd|||||d|\}}||z}|}||}||}||z}|S)N)rDrEr rFr r+)rrrr) rBrDrEr rFr rGresidualrs r-r`z"JambaAttentionDecoderLayer.forwards!,,];; )4> ')%+       q!=0  --m<< ))-88  =0 r,)NNNF)r(r)r*r$rbr8rcrdrrboolrr FloatTensorr`rfrgs@r-rrsZ{ZsZZZZZZ/304(,!& | t+&-    $; +,  r,rc eZdZdedeffd Z d dejdejdzdejdzde dzd e e d ej f d Z xZS) JambaMambaDecoderLayerr1r2cht|jr |j|nd}t|||_|dkrt nt }|||_t|j |j |_ t|j |j |_ dS)Nr#)r1r2rp) r7r8rrimambarrrr&r:rrrrs r-r8zJambaMambaDecoderLayer.__init__s >D>W^f/ ::]^ $FiHHH 1FDWXXX ,V-?VEX Y Y Yr,NrDrEr rFrGrHc |}||}||||}||z}|}||}||}||z}|S)N)rDrrE)rrrr)rBrDrEr rFrGr s r-r`zJambaMambaDecoderLayer.forwards!,,];;  '()#  !=0  --m<< ))-88  =0 r,)NNN)r(r)r*r$rbr8rcrdrrrrrr`rfrgs@r-rrsZ{ZsZZZZZZ/304(, | t+&-    +,   r,r) attentionrceZdZUeed<dZdZddgZdgZdZ dZ dZ dZ e egeeejdd Zejfd ZxZS) JambaPreTrainedModelr1modelTrrrFr) layer_name)rD attentionsrcpt|t|trt jd|jdzdddf}||jd }tj |j t j |tj|jdSt|t rNtj|jd|jjtj|jd|jjdSdS)Nr#rJrK)meanstd)r7 _init_weights isinstancerircrrxrr|rWinitcopy_rrones_rrnormal_ gate_up_projr1initializer_range down_proj)rBmodulerrCs r-rz"JambaPreTrainedModel._init_weightss  f%%% fo . . X Q 5 9::47CA1266AACCA Jv|UYq\\ 2 2 2 Jvx  - - X L,3DKFDWXXX&+# r,N input_idsrEr rF inputs_embedsr rGrHc |du|duz rtd|||}|r|t|j}|V||nd}t j|jd|j|z}| d}t|x} ts&|j||||d} td i| td i| d} |} t|jD])\} } | | f| |jj| |||d|} *|| } t%| | S) Nz:You must specify exactly one of input_ids or inputs_embeds)r1rr#)r)r1rDrErFr )full_attentionlinear_attention)rEr rFr )last_hidden_staterFr+) ValueErrorr9r r1get_seq_lengthrcrrNrrrdictr r enumeraterrr>r)rBrCrEr rFrDr rGpast_seen_tokenscausal_mask_mapping mask_kwargsrDr decoder_layers r-r`zJambaModel.forwards  -t"; < [YZZ Z   --i88M  ?0*$+>>>O  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L?-FF +!."0#2 , K#5"C"C{"C"C$C$R$Rk$R$R## & )$+ 6 6   A})M24;3J13MN) /#   MM,,];; %++    r,)NNNNNN)r(r)r*r$r8rrrrcrrdrrrrrrr`rfrgs@r-r3r3 s {$ .2.204(,26!%6 6 #d*6  t+6 &- 6  6 (4/ 6 $;6 +,6  6 6 6 ^_ 6 6 6 6 6 r,r3ceZdZdeffd Z ddejdzdejdzdejdzdedzd ej dzd ejdzd e dzd e dzd e ejzde e deffd ZxZS)JambaForCausalLMr1cbt||j|_dS)N)r7r8rrs r-r8zJambaForCausalLM.__init__[s,    !-r,NrrCrEr rFrDlabelsr output_router_logitslogits_to_keeprGrHc Ftj|||||||| fi| S)aj labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, JambaForCausalLM >>> model = JambaForCausalLM.from_pretrained("ai21labs/Jamba-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)r7r`) rBrCrEr rFrDrTr rUrVrGrCs r-r`zJambaForCausalLM.forward_sGFuww              r,) NNNNNNNNr)r(r)r*r$r8rcrrdrrrrbrrrr`rfrgs@r-rRrRZs0.{...... .2.204(,26*.!%,0-.- - #d*-  t+- &- -  - (4/ -  4'- $;- #Tk- el*- +,-  #- - - - - - - - - - r,rRceZdZdS)JambaForSequenceClassificationNr'r+r,r-rYrYr.r,rY)rRrYr3r)Fcollections.abcrrcrrr  activationsr cache_utilsrr integrationsr integrations.accelerater masking_utilsr r modeling_layersrrmodeling_outputsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrutils.import_utilsrutils.output_capturingrrllama.modeling_llamarrrmistral.modeling_mistralr mixtral.modeling_mixtralr!r"configuration_jambar$ get_loggerr(rr&r0Modulerirrrrrr;rr3rRrY__all__r+r,r-rps)&%$$$$$ &&&&&&!!!!!!........,,,,,,======PPPPPPPP[[[[[[[[QQQQQQQQFFFFFFFF&&&&&&@@@@@@@@@@777777999999EEEEEEEEXXXXXXXXXX111111IIIIIIII,,,,,,  H % %     <   *)*)*)*)*)^*)*)*)ZSNSNSNSNSNbiSNSNSNl     z        >   """""")"""J#####!;###L7B)CMcddXXXXX?XXX:L L L L L %L L L ^2 2 2 2 2 )2 2 2 j     %EG[    g f fr,