tj*dZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z dd l mZdd lmZdd lmZmZdd lmZdd lmZmZmZddlmZddlmZddlmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ej-e.Z/Gdde!Z0Gdde"Z1GddeZ2GddeZ3Gdde&Z4Gd d!e*Z5Gd"d#ej6Z7Gd$d%e Z8eGd&d'eZ9eGd(d)e(Z:Gd*d+e'e Z;gd,ZcLt||dSN)super__init__)self hidden_sizeeps __class__s e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/olmoe/modular_olmoe.pyr'zOlmoeRMSNorm.__init__.s# c*****)r#)__name__ __module__ __qualname__r' __classcell__r+s@r,r"r"-s=++++++++++r-r"ceZdZdS)OlmoeRotaryEmbeddingNr.r/r0r-r,r4r42Dr-r4ceZdZdS)OlmoeMLPNr5r6r-r,r9r96r7r-r9ceZdZd dededzffd Z d dejdeejejfdejdzde dzd e e d eejejdzeejdzff d Z xZ S) OlmoeAttentionNconfig layer_idxct||t|j|j|_t|j|jz|jz|j|_dS)Nr*) r&r'r"r) rms_norm_epsq_normnum_attention_headsnum_key_value_headsk_normr(r<r=r+s r,r'zOlmoeAttention.__init__;sm +++"6#56;NOOO "  6#= =A[ [agat    r- hidden_statesposition_embeddingsattention_maskpast_key_valueskwargsreturnc |jdd}g|d|jR}|||}|||} ||} |jj| |jj |jj| |jj |jj| |jj |jj|j | dd}| j | dd} | j | dd} |\} } t|| | | \}} || | | |j\} } tj|jjt$} | ||| | |f|jsdn|j|jt-|jddd|\}}|jg|dR}||}||fS)N)minmaxrrsliding_window)dropoutscalingrQ)shapehead_dimrAq_projrDk_projv_projr<clip_qkvclamp_view transposerupdater=r get_interface_attn_implementationrtrainingattention_dropoutrSgetattrreshape contiguouso_proj)r(rFrGrHrIrJ input_shape hidden_shape query_states key_states value_statescossinattention_interface attn_output attn_weightss r,forwardzOlmoeAttention.forwardBsf$)#2#. 88b8$-88 {{4;;}#=#=>> [[]!;!;<< {{=11 ;  +   T[%9$9t{?S  T T T   4;#7"7T[=Q  R R R   T[%9$9t{?S  T T T(|(,7AA!QGG $Z_l3==aCC (|(,7AA!QGG &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL"4;0@$GG % %  % % ! \*k);;;;;;FFHH kk+.. L((r-r%)r.r/r0r intr'torchTensortuplerrrrpr1r2s@r,r;r;:s  { sTz      )- /)/)|/)#5<#=>/) t+ /)  /) +, /) u|U\D0% 2E2LL M/)/)/)/)/)/)/)/)r-r;ceZdZdS) OlmoeExpertsNr5r6r-r,rvrvtr7r-rvceZdZdS)OlmoeTopKRouterNr5r6r-r,rxrxxr7r-rxcBeZdZfdZdejdejfdZxZS)OlmoeSparseMoeBlockctt||_t ||_dSr%)r&r'rxgatervexpertsr(r<r+s r,r'zOlmoeSparseMoeBlock.__init__}s; #F++ #F++ r-rFrKc|j\}}}|d|}||\}}}|||||||}|S)NrM)rTr[r|r}rc) r(rF batch_sizesequence_length hidden_dim_ top_k_weights top_k_indexfinal_hidden_statess r,rpzOlmoeSparseMoeBlock.forwardss2?2E/ OZ%**2z:: (, -(@(@%=+"ll=+}UU]]   #"r-)r.r/r0r'rrrsrpr1r2s@r,rzrz|s^,,,,, #U\#el########r-rzc(eZdZdedeffd ZxZS)OlmoeDecoderLayerr<r=c8t|||j|_t|||_t ||_t|j|j|_ t|j|j|_ dS)N)r<r=r?) r&r'r)r; self_attnrzmlpr"r@input_layernormpost_attention_layernormrEs r,r'zOlmoeDecoderLayer.__init__s +++!-'vKKK&v..+F,>FDWXXX(4V5GVM`(a(a(a%%%r-)r.r/r0r rqr'r1r2s@r,rrsWb{bsbbbbbbbbbbr-rceZdZUeed<dZdZdgZdgZdZ dZ e e de edZdZejd Zd S) OlmoePreTrainedModelr<modelTrrIr)index) router_logitsrF attentionscptj||t|trNt j|jd|jjt j|j d|jjdSt|tr(t j|j d|jjdSdS)NrP)meanstd) r _init_weights isinstancervinitnormal_ gate_up_projr<initializer_range down_projrxweight)r(modules r,rz"OlmoePreTrainedModel._init_weightss%dF333 fl + + U L,3DKz'OlmoeModel.__init__..s$ c c ci vy 1 1 c c cr-r?r<)r&r'r Embedding vocab_sizer) padding_idx embed_tokens ModuleListrangenum_hidden_layerslayersr"r@normr4 rotary_embr~s `r,r'zOlmoeModel.__init__s    L):F>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    & #oom\JJ![)H4;+H)HI  M)M$7*) /# MM -00 %++    r-)NNNNNN)r.r/r0r r'rr LongTensorrsr FloatTensorboolrrr rpr1r2s@r,rrs>{>>>>>>.2.204(,26!%5 5 #d*5  t+5 &- 5  5 (4/ 5 $;5 +,5  5 5 5 5 5 5 5 5 r-rc0eZdZddiZfdZfdZxZS)OlmoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightct|t||_|j|_dSr%)r&r'rr num_expertsr~s r,r'zOlmoeForCausalLM.__init__s;    '' !-r-c 6tjdi|S)u labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OlmoeForCausalLM >>> model = OlmoeForCausalLM.from_pretrained("allenai/OLMoE-1B-7B-0924") >>> tokenizer = AutoTokenizer.from_pretrained("allenai/OLMoE-1B-7B-0924") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'Hey, are you conscious? Can you talk to me?\nI’m not sure if you’re conscious of this, but I’m' ``` r6)r&rp)r( super_kwargsr+s r,rpzOlmoeForCausalLM.forwards!0uww.....r-)r.r/r0_tied_weights_keysr'rpr1r2s@r,rrs]*,GH..... /////////r-r)rrr)=__doc__collections.abcrrrrrr cache_utilsrr generationr masking_utilsr modeling_outputsr modeling_utilsr r processing_utilsrutilsrrrutils.output_capturingrgemma.modeling_gemmarllama.modeling_llamarrrrrrmixtral.modeling_mixtralrrrqwen2_moe.modeling_qwen2_moerconfiguration_olmoer get_loggerr.loggerr"r4r9r;rvrxModulerzrrrr__all__r6r-r,rs$$$$$$ &&&&&&........))))))//////666666FFFFFFFF&&&&&&@@@@@@@@@@444444++++++XWWWWWWWWW======,,,,,,  H % %+++++<+++      /        x   7)7)7)7)7)^7)7)7)t     >        (    # # # # #") # # # bbbbb)bbbUUUUU?UUU4? ? ? ? ? ? ? ? D / / / / /)? / / /F E D Dr-