tjjddlZddlmZddlmZddlmZddlmZddl m Z ddl m Z m Z d d lmZmZmZmZd d lmZd d lmZd dlmZe jeZdZGddeZGddeZGddeZGddeZ GddeZ!GddeZ"gdZ#dS)N)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack)TransformersKwargslogging) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-0414ceZdZdS)Glm4MLPN__name__ __module__ __qualname__c/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/glm4/modular_glm4.pyrr#Drrc eZdZdedeffd Z ddejdejdzdejdzd e dzd e dzd e ejejfdzd e e d e eje ejejfdzffdZxZS)Glm4DecoderLayerconfig layer_idxct|j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ dS)N)r r!)eps) super__init__ hidden_size Glm4Attention self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfr r! __class__s rr%zGlm4DecoderLayer.__init__(s !-&f JJJ6??*6+=6CVWWW(3F4FFL_(`(`(`%(3F4FFL_(`(`(`%"-f.@fFY"Z"Z"ZrNF hidden_statesattention_mask position_idspast_key_values use_cacheposition_embeddingskwargsreturnc "|}||}|jd||||||d|\}} ||}||z}|}||}||}||}||z}|S)N)r1r2r3r4r5r6r)r+r(r-r,r)r.) r/r1r2r3r4r5r6r7residual_s rforwardzGlm4DecoderLayer.forward3s!,,];; )4> ')%+ 3      q55mDD  =0  55mDD // // >>  =0 r)NNNFN)rrrrintr%torchTensor LongTensorrbooltuplerr FloatTensorr< __classcell__r0s@rrr's  [z [c [ [ [ [ [ [/304(,!&HL| t+&-    $; #5<#=>E-. u %(95;L(L"MPT"TT UrrceZdZdS)r'Nrrrrr'r'Urrr'c:eZdZdeedeezffd ZxZS)Glm4ForCausalLM super_kwargsr8c 6tjdi|S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```r)r$r<)r/rIr0s rr<zGlm4ForCausalLM.forwardZs!4uww.....r) rrrrr rBrr<rDrEs@rrHrHYs[/12/ ' '//////////rrHceZdZdS)Glm4ForSequenceClassificationNrrrrrLrLwrrrLceZdZdS)Glm4ForTokenClassificationNrrrrrNrN{rrrN)Glm4PreTrainedModel Glm4ModelrHrLrN)$r> cache_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrprocessing_utilsrutilsr r glm.modeling_glmr r rrphi3.modeling_phi3rconfiguration_glm4r modeling_glm4r get_loggerrlogger_CHECKPOINT_FOR_DOCrrr'rHrLrN__all__rrrr_s   BBBBBB999999666666&&&&&&00000000tttttttttttt((((((******&&&&&&  H % %+     g   +++++1+++\     L   /////n///<     $@        !:      r