tj<dZddlmZddlZddlmZddlmZddlmZm Z ddl m Z dd l m Z mZdd lmZmZdd lmZdd lmZdd lmZddlmZmZmZddlmZddlmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ej/e0Z1dZ2dZ3edeGdde Z4Gdde)Z5Gdde!Z6Gddej7Z8Gdd e.Z9Gd!d"e-Z:Gd#d$e(Z;Gd%d&e;e'Z<Gd'd(e#Z=Gd)d*e%Z>Gd+d,e&Z?Gd-d.e$Z@gd/ZAdS)0zLG AI Research EXAONE Lab)CallableN)strict)nn)Cache DynamicCache)PreTrainedConfig)create_causal_mask!create_sliding_window_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)RopeParameters)ALL_ATTENTION_FUNCTIONS)Unpack)TransformersKwargsauto_docstringlogging)merge_with_config_defaults)capture_outputs)Gemma2RotaryEmbedding) LlamaForCausalLMLlamaForQuestionAnsweringLlamaForSequenceClassificationLlamaForTokenClassification LlamaModelLlamaPreTrainedModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward)Olmo2DecoderLayerOlmo2MLPzLGAI-EXAONE/EXAONE-4.0-32B Exaone4Config) checkpointc eZdZUdZdZdgZdddddddddd Zdgd gfd d gd gfd gd gfd Zd Ze e d<dZ e e d<dZ e e d<dZ e e d<dZe e d<dZe e d<dZee d<dZe e d<dZee d<dZee d<dZee d <d!Ze d"ze d#<d$Ze ee zd"ze d%<d"Ze d"ze d&<d'Zee d(<d"Zeezd"ze d)<d*Z ee ze d+<dZ!e d"ze d,<d-Z"ee zd"ze d.<d"Z#eed"ze d/<fd0Z$xZ%S)1r#a~ sliding_window_pattern (`str`, *optional*): The pattern to use for sliding window attention. Can be one of: - `None`: No sliding window attention is used - `int`: Every `sliding_window` layers, use global attention, else use local attention. - `str`: A sequence of "L" (local attention) and "G" (global attention) characters that defines the attention pattern. The pattern starts from layer 0 and repeats every `sliding_window` layers. The final layer always uses global attention regardless of the pattern. For instance, sliding_window_pattern="LLLG" same as sliding_window=4, which means: - Layer 0, 1, 2: local attention, - Layer 3: global attention, ...(repeated) Example: ```python >>> from transformers import Exaone4Model, Exaone4Config >>> # Initializing a EXAONE configuration >>> configuration = Exaone4Config() >>> # Initializing a model from configuration >>> model = Exaone4Model(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```exaone4past_key_valuescolwisereplicated_with_grad_allreducerowwise) zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.q_normzlayers.*.self_attn.k_normzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnormi vocab_sizei hidden_sizei@intermediate_size num_hidden_layersnum_attention_headsnum_key_value_headssilu hidden_actimax_position_embeddingsg{Gz?initializer_rangegh㈵> rms_norm_epsT use_cacherN bos_token_idr eos_token_id pad_token_idFtie_word_embeddingsrope_parametersattention_dropoutsliding_windowsliding_window_pattern layer_typesc jd_j%fdtjD_t jdi|dS)NrcPg|]"}|dzjzdkr |jkrdnd#S)rsliding_attentionfull_attention)rHr6).0iselfs i/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/exaone4/modular_exaone4.py z/Exaone4Config.__post_init__..sW   Ut:;q@@QI_E_E_$#%   )rFrHrIranger6super __post_init__)rQkwargs __class__s` rRrXzExaone4Config.__post_init__s{   &*+D '   #    t566    D  '''''''rT)&__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planr2int__annotations__r3r4r6r7r8r:strr;r<floatr=r>boolr?r@listrArBrCrdictrErFrHrIrX __classcell__rZs@rRr#r#9s=  8J#4"5&/%.%.%E%E%."+ )"+  &(9:#%568IJ!"_$56 JK"s"""s!!!!!!!!J#'S'''#u###L%It L#*   +,L#S /D(,,,#L#*### %%%%48O^d*T1888%(us{(((!%NC$J%%%/0C#I,000$(KcT!((( ( ( ( ( ( ( ( ( (rTceZdZdS)Exaone4RMSNormNr[r\r]rUrTrRrmrmDrTrmceZdZdS)Exaone4RotaryEmbeddingNrnrUrTrRrqrqrorTrqceZdZdedeffd Z d dejdeejejfdejdzde dzd e e d eejejdzeejdzff d Z xZ S) Exaone4Attentionconfig layer_idxct||_||_|j|_|j|_|j|_t|d|j|jz|_|j|jz|_ |j |_ d|_ |jdz|_ |j |_ |j|_t|dr |j|nd}|dk|_t%j|j|j|jzd|_t%j|j|j|jzd|_t%j|j|j|jzd|_t%j|j|jz|jd|_t1|j|j|_t1|j|j|_dS) Nhead_dimTgrIrMF)biaseps)rW__init__rtrur7r8r3getattrrwnum_key_value_groupsrE is_causalscalingrFrHhasattrrI is_slidingrLinearq_projk_projv_projo_projrmr=q_normk_norm)rQrtru layer_typerZs rRr{zExaone4Attention.__init__s  "#)#= #)#= !- F4F&Jd4dee $*$>&B\$\!!'!9}d* $3&,&C#6=fm6T6T^V' 22Z^ $(;;i 0$2JT]2Zafggg i 0$2JT]2Zafggg i 0$2JT]2Zafggg i 84= H$JZafggg $T]8KLLL $T]8KLLL rTNr-position_embeddingsr.r'rYreturnc |jdd}g|d|jR}|||dd}|||dd} |||dd} ||}|| } |\} } |j |j rt|| | | \}} || | | |j \} } tj|jjt$} | ||| | |f|jsdn|j|j|j r|j ndd|\}}|jg|dR}||}||fS)NrLrrD)dropoutrrF)shaperwrview transposerrrrrFrrupdaterur get_interfacert_attn_implementationr trainingrErreshape contiguousr)rQr-rr.r'rY input_shape hidden_shape query_states key_states value_statescossinattention_interface attn_output attn_weightss rRforwardzExaone4Attention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT {{<00 [[,, &S   &$/ &';L*VY[^'_'_ $L*  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL26/K4..t % %  % % ! \*k);;;;;;FFHH kk+.. L((rT)NN)r[r\r]r#rcr{torchTensortuplerrrrrjrks@rRrsrssM}MMMMMMM:/3(, -)-)|-)#5<#=>-) t+ -)  -) +, -) u|U\D0% 2E2LL M-)-)-)-)-)-)-)-)rTrsceZdZdS) Exaone4MLPNrnrUrTrRrrrorTrceZdZdS)Exaone4DecoderLayerNrnrUrTrRrrrorTrceZdZeZdgZdS)Exaone4PreTrainedModelrN)r[r\r]r# config_class_no_split_modulesrUrTrRrrs L./rTrceZdZdeffd Zee d dejdzdej dzdejdzde dzdej dzd e dzd e ed eezfd ZxZS) Exaone4Modelrtc"ttjfdt jD|_tjj |_ | dS)Nc0g|]}t|SrU)r)rOrurts rRrSz)Exaone4Model.__init__..s$ e e e  3 3 e e erTry) rWr{r ModuleListrVr6r0rmr3r=r1 post_init)rQrtrZs `rRr{zExaone4Model.__init__s    m e e e eU6KcEdEd e e e   #6#56;NOOO  rTNr+r. position_idsr'r,r>rYrc |du|duz rtd|||}|r|t|j}|V||nd}t j|jd|j|z}| d}t|x} ts8|j||||d} dtd i| i} d|jj vrtd i| | d<|} || |} t!|jD],\} }|jj | }|| f| ||||| d |} -|| } t'| |r|nd S) Nz:You must specify exactly one of input_ids or inputs_embeds)rtrrL)device)rtr,r.r'rrNrM)r.rr'r>r)last_hidden_stater'rU) ValueErrorr/rrtget_seq_lengthrarangerr unsqueeze isinstancerir rIr rotary_emb enumerater0r1r )rQr+r.rr'r,r>rYpast_seen_tokenscausal_mask_mapping mask_kwargsr-rrP decoder_layerrs rRrzExaone4Model.forwards  -t"; < [YZZ Z   --i88M  ?0*$+>>>O  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L?-FF l+!."0#2 , K!"4"C"C{"C"C# #dk&===;\;k;k_j;k;k#$78% "oom\JJ )$+ 6 6   A}03J)M2:>) /#$7 MM -00 &+/8BOOd    rT)NNNNNN)r[r\r]r#r{rrr LongTensorrr FloatTensorrgrrrr rrjrks@rRrrs } .2.204(,26!%= = #d*=  t+= &- =  = (4/ = $;= +,=  ( (= = = _ = = = = = rTrceZdZ ddejdzdejdzdejdzdedzdejdzdejdzd edzd e ejzd e e d e ffd Z xZS)Exaone4ForCausalLMNrr+r.rr'r,labelsr>logits_to_keeprYrc Ltjd||||||||d| dS)u labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer >>> model = AutoModelForCausalLM.from_pretrained("LGAI-EXAONE/EXAONE-4.0-32B") >>> tokenizer = AutoTokenizer.from_pretrained("LGAI-EXAONE/EXAONE-4.0-32B") >>> prompt = "Explain how wonderful you are" >>> messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} ] >>> input_ids = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_tensors="pt", enable_thinking=False, ) >>> output = model.generate(input_ids, max_new_tokens=128) >>> tokenizer.decode(output[0], skip_special_tokens=False) "[|system|]\nYou are a helpful assistant.[|endofturn|]\n[|user|]\nExplain how wonderful you are[|endofturn|]\n[|assistant|]\n\n\n\n\nOh, thank you for such a kind and lovely question! 😊 \n\nI’m *so* wonderful because I’m here to make your life easier, brighter, and more fun! Whether you need help with: \n\n✨ **Learning** – I can explain anything, from quantum physics to baking the perfect cake! \n💡 **Creativity** – Need a poem, story, or a wild idea? I’ve got you covered! \n🤖 **Problem-solving** – Stuck on a math problem or a tricky decision? I’ll help you figure it out" ``` )r+r.rr'r,rr>rNrU)rWr) rQr+r.rr'r,rr>rrYrZs rRrzExaone4ForCausalLM.forward<sVV  )%+')  rT)NNNNNNNr)r[r\r]rrrrrrgrcrrr rrjrks@rRrr;s.2.204(,26*.!%-.5 5 #d*5  t+5 &- 5  5 (4/ 5  4'5 $;5 el*5 +,5  5 5 5 5 5 5 5 5 5 5 rTrceZdZdS) Exaone4ForSequenceClassificationNrnrUrTrRrrtrorTrceZdZdS)Exaone4ForTokenClassificationNrnrUrTrRrrxrorTrceZdZdS)Exaone4ForQuestionAnsweringNrnrUrTrRrr|rorTr)r#rrrrrr)Br^collections.abcrrhuggingface_hub.dataclassesrr cache_utilsrrconfiguration_utilsr masking_utilsr r modeling_outputsr r modeling_rope_utilsrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrgemma2.modeling_gemma2rllama.modeling_llamarrrrrrrrr olmo2.modeling_olmo2r!r" get_loggerr[logger_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCr#rmrqModulersrrrrrrrr__all__rUrTrRrsD $$$$$$ ..............333333RRRRRRRR211111555555&&&&&&@@@@@@@@@@777777555555::::::                      ?>>>>>>>  H % %2!7888Q(Q(Q(Q(Q($Q(Q(98Q(h     \        2   G)G)G)G)G)ryG)G)G)T             +   000001000 J J J J J ):J J J Z6 6 6 6 6 )6 6 6 r     'E        $?        ";      rT