tj2FddlZddlmcmZddlmZddlmZddlm Z m Z ddl m Z ddl mZmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZd dlmZmZd dlm Z m!Z!d dl"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)GddeZ*GddeZ+Gddej,Z-Gdde$Z.Gddej,Z/Gdde Z0Gdd e#Z1eGd!d"e!eZ2eGd#d$e&Z3Gd%d&e%Z4gd'Z5dS)(N)nn)initialization)Cache DynamicCache)create_causal_mask)MoeCausalLMOutputWithPastMoeModelOutputWithPast)PreTrainedModel)Unpack)TransformersKwargsauto_docstring)can_return_tuplemerge_with_config_defaults)capture_outputs)GraniteRMSNormGraniteRotaryEmbedding)LlamaAttentionLlamaPreTrainedModel)MixtralDecoderLayerMixtralExpertsMixtralForCausalLM MixtralModelload_balancing_loss_func)GraniteMoeConfigceZdZdS)GraniteMoeRMSNormN__name__ __module__ __qualname__o/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/granitemoe/modular_granitemoe.pyrr)Dr%rceZdZdS)GraniteMoeRotaryEmbeddingNr r$r%r&r)r)-r'r%r)creZdZdZdeffd ZdejdeejejejffdZ xZ S)GraniteMoeTopKRouteraTop-k gating that returns the routing decisions without grouping tokens by expert. Returns ``(top_k_index, top_k_weights, router_logits)``; the grouping/scattering used to live here (via ``expert_size.tolist()``, which broke fullgraph compile) and now happens inside the experts forward via ``use_experts_implementation`` so the default ``grouped_mm`` / ``batched_mm`` paths can compile cleanly. configct|j|_|j|_t jtj |j|j |_ dSN) super__init__num_local_experts num_expertsnum_experts_per_toktop_kr Parametertorchempty hidden_sizeweightselfr, __class__s r&r0zGraniteMoeTopKRouter.__init__:sU !3/ l5;t/?AS#T#TUU r% hidden_statesreturnctj||j}||jd\}}t j|d|}|||fS)N)dim) Flinearr9floattopkr4r6softmaxtype_as)r;r= router_logits top_k_logits top_k_index top_k_weightss r&forwardzGraniteMoeTopKRouter.forward@so <<BBDD $1$6$6tzr$6$J$J! k l;;;CCMRR M=88r%) r!r"r#__doc__rr0r6TensortuplerL __classcell__r<s@r&r+r+1sV/VVVVVV 9U\9eEL%,X]Xdc|\}}}|d|}||\}}}||||} | |||jS)Nr@)sizereshaperXrYviewrW) r;rZbszlengthemb_sizer=rJrK_ layer_outputs r&rLzGraniteMoeMoE.forwardTsv + 0 0 2 2VX#++B99 (, M(B(B% ]A||M; NN   fdo>>>r%) r!r"r#rMrr0r6rNrLrPrQs@r&rUrUKsqSS1/111111 ?5<?EL????????r%rUc(eZdZdedeffd ZxZS)GraniteMoeAttentionr, layer_idxcft||||j|_dSr.)r/r0attention_multiplierscalingr;r,rfr<s r&r0zGraniteMoeAttention.__init__]s- vy1112 r%)r!r"r#rintr0rPrQs@r&rere\sL3/3C3333333333r%rec eZdZdedeffd Z d dejdejdzdedzde ejejfdzd ejf d Z xZ S) GraniteMoeDecoderLayerr,rfcdt||t|||_t ||_t |j|j|_ t |j|j|_ |` t ||_|j |_ dS)N)r,rfeps) r/r0re self_attnrUblock_sparse_moerr8 rms_norm_epsinput_layernormpost_attention_layernormmlpresidual_multiplierrjs r&r0zGraniteMoeDecoderLayer.__init__cs +++,FiPPP -f 5 501CI\]]](9&:LRXRe(f(f(f% H -f 5 5#)#=   r%Nr=attention_maskpast_key_valuesposition_embeddingsr>c |}||}|jd||||d|\}}|||jzz}|}||}||}|||jzz}|S)N)r=rxryrzr$)rtrqrwrurr)r;r=rxryrzkwargsresidualrbs r&rLzGraniteMoeDecoderLayer.forwardms!,,];; )4> ')+ 3       q!=43K#KK  55mDD --m<<  =43K#KK r%)NNN) r!r"r#rrkr0r6rNrrOrLrPrQs@r&rmrmbs>/>C>>>>>>/3(,HL | t+  #5<#=>E  r%rmcjeZdZUeed<dZdZdgZdgZdZ dZ dZ e j dZdS)GraniteMoePreTrainedModelr,modelTrmrycptj||t|trNt j|jd|jjt j|j d|jjdSt|tr(t j|j d|jjdSdS)Ng)meanstd) r _init_weights isinstancerSinitnormal_ gate_up_projr,initializer_range down_projr+r9)r;modules r&rz'GraniteMoePreTrainedModel._init_weightss%dF333 f/ 0 0 U L,3DKz,GraniteMoeModel.__init__..s$ h h h9 #FI 6 6 h h hr%ro) r/r0r ModuleListrangenum_hidden_layerslayersrr8rsnormembedding_multiplierr:s `r&r0zGraniteMoeModel.__init__s~    m h h h hfNfHgHg h h h   &f&8f>QRRR $*$?!!!r%N input_idsrx position_idsry inputs_embeds use_cacher|r>c Z|du|duz rtd|r|t|j}|||}|V||nd}t j|jd|j|z}| d}t|j||||} ||j z}|} | | |} |j d|jjD]} | | f| | |||d|} || } t!| |S) Nz:You must specify exactly one of input_ids or inputs_embeds)r,rr)device)r,rrxryr)rzrxrryr)last_hidden_statery) ValueErrorrr, embed_tokensget_seq_lengthr6arangeshaper unsqueezerr rotary_embrrrr ) r;rrxrryrrr|past_seen_tokens causal_maskr=rz decoder_layers r&rLzGraniteMoeModel.forwards  -t"; < [YZZ Z  ?0*$+>>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    &(AA % #oom\JJ![)H4;+H)HI  M)M$7*) /# MM -00 %++    r%)NNNNNN)r!r"r#rr0rrrr6 LongTensorrNr FloatTensorboolr r r rLrPrQs@r&rrs@/@@@@@@ .2.204(,26!%5 5 #d*5  t+5 &- 5  5 (4/ 5 $;5 +,5  5 5 5 ^_ 5 5 5 5 5 r%rceZdZdeffd Zee ddejdzdej dzdejdzde dzd ej dzd ejdzd e dzd e ej zd eezfdZxZS)GraniteMoeForCausalLMr,ct|t||_|j|_dSr.)r/r0rrlogits_scalingr:s r&r0zGraniteMoeForCausalLM.__init__s;    $V,, $3r%Nrrrxrryrlabelsoutput_router_logitslogits_to_keepr>c 2||n |jj}|jd|||||d| } | j} t |t rt | dn|} || dd| ddf} | |jjz } d}||j | |fd|jj i| }d}|rHt| j |j |j|}|%||j||jzz }t%||| | j| j| j| j S)al labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, GraniteMoeForCausalLM >>> model = GraniteMoeForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rrxrryr vocab_size)lossaux_losslogitsryr= attentionsrHr$)r,rrrrrkslicelm_headr loss_functionrrrHr2r3router_aux_loss_coeftorr ryr=r)r;rrxrryrrrrr|outputsr= slice_indicesrrrs r&rLzGraniteMoeForCausalLM.forwardsJ%9$D $+Jj $* )%+'       1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA$+44  %4% ;1 D  M/% ( H !1HKK 4L4LLL(#3!/)!/    r%)NNNNNNNr)r!r"r#rr0rrr6rrNrrrrkrOr rLrPrQs@r&rrs54/444444 .2.204(,26*.,0-.Q Q #d*Q  t+Q &- Q  Q (4/ Q  4'Q #TkQ el*Q  * *Q Q Q ^Q Q Q Q Q r%r)rrr)6r6torch.nn.functionalr functionalrBrr cache_utilsrr masking_utilsrmodeling_outputsr r modeling_utilsr processing_utilsr utilsr r utils.genericrrutils.output_capturingrgranite.modeling_graniterrllama.modeling_llamarrmixtral.modeling_mixtralrrrrrconfiguration_granitemoerrr)Moduler+rSrUrermrrr__all__r$r%r&rst  &&&&&&........//////QQQQQQQQ------&&&&&&77777777IIIIIIII555555MMMMMMMMGGGGGGGG766666             6   9999929999,        ?????BI???"33333.333 !!!!!0!!!HUUUUU 4oUUU(A A A A A lA A A HY Y Y Y Y .Y Y Y x T S Sr%