[ TU ] 5 UR U l [ XS9U l [ U5 U l [ UR UR S9U l [ UR UR S9U l
[ UR UR S9U l [ UR UR S9U l g )N)r& r' )eps)
super__init__ hidden_size
Glm4Attention self_attnr mlpr rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfr& r' __class__s r! r+ Glm4DecoderLayer.__init__* s
!--&fJ6? *6+=+=6CVCV W (3F4F4FFL_L_(`%(3F4F4FFL_L_(`%"-f.@.@fFYFY"Z r
hidden_statesattention_mask position_idspast_key_value use_cachecache_positionposition_embeddingskwargsreturnc Un U R U5 nU R " SUUUUUUUS.UD6u pU R U5 nX- nUn U R U5 nU R U5 nU R U5 nX- nU$ )N)r8 r9 r: r; r< r= r> r )r1 r. r3 r2 r/ r4 ) r5 r8 r9 r: r; r< r= r> r? residual_s r! forwardGlm4DecoderLayer.forward5 s ! ,,];
>>
')%) ) 3
55mD
0
55mD
/
//
>
0
r )r, r1 r/ r2 r4 r3 r. )NNNFNN)r r r r r intr+ torchTensorr
LongTensorr booltupler
r FloatTensorrD r
__classcell__r6 s @r! r$ r$ ) s [z [c [ 2637*.$)59KO! ||! ! .! u / /0 !
! ! D>
! !!1!12! &eELL%,,,F&G H! -.!
u (51B1BEDUDU1U+V"WW X! ! r r$ c \ rS rSrSrg)r- Y r Nr r r r! r- r- Y r" r r- c D ^ \ rS rSrS\\ S\\\4 4U 4S jjr Sr
U =r $ )Glm4ForCausalLM] super_kwargsr@ c $ > [ TU ] " S0 UD6$ )a
labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
(masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
Example:
```python
>>> from transformers import AutoTokenizer, Glm4ForCausalLM
>>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414")
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414")
>>> prompt = "Hey, are you conscious? Can you talk to me?"
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> # Generate
>>> generate_ids = model.generate(inputs.input_ids, max_length=30)
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
"Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
```r )r* rD )r5 rT r6 s r! rD Glm4ForCausalLM.forward^ s 4 w. ..r r ) r r r r r
r r rK r rD r rM rN s @r! rR rR ] s0 / 12/
u,, -/ /r rR c \ rS rSrSrg) Glm4ForSequenceClassification{ r Nr r r r! rX rX { r" r rX c \ rS rSrSrg)Glm4ForTokenClassification r Nr r r r! r[ r[ r" r r[ )Glm4PreTrainedModel Glm4ModelrR rX r[ )'typingr r rG cache_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr processing_utilsr
utilsr r glm.modeling_glmr r r r phi3.modeling_phi3r configuration_glm4r
modeling_glm4r
get_loggerr logger_CHECKPOINT_FOR_DOCr r$ r- rR rX r[ __all__r r r! rn s # B 9 6 & 0 t t ( * &
H %+ g - 1 - ` L /n /< $@ !: r