tjdddlmZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZddlmZd d lmZejeZd ZGd d eZGddeZdZddZeGdde ZGdde ZGdde ZGdde ZgdZdS))OptionalN)logging)no_inherit_decorator)LlamaAttentionLlamaForCausalLMLlamaForSequenceClassificationLlamaForTokenClassificationLlamaRotaryEmbedding)Phi3MLP) GlmConfigzTHUDM/glm-4-9bceZdZdS)GlmMLPN__name__ __module__ __qualname__a/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/glm/modular_glm.pyrr&Drrc feZdZe d dedzdeddedzdedeffdZ dS) GlmRotaryEmbeddingNconfigdevicez torch.deviceseq_lenreturnz torch.TensorcV|jd}|jdd}t|ddp|j|jz}t ||z}d}d|t jd|dt j |t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrr)dtype)rr$) rope_parametersgetgetattr hidden_sizenum_attention_headsinttorcharangeint64tofloat) rrrbaser"r#dimattention_factorinv_freqs rcompute_default_rope_parametersz2GlmRotaryEmbedding.compute_default_rope_parameters+s&%l3 & 6 : :;RTW X X6:t44h8JfNh8h(2233 U\!S!5;???BB&X]XcBddgjj k )))r)NNN) rrr staticmethodrrr*tupler/r4rrrrr*s|#'+/"**D *(*t* ~u$ % ***\***rrc|ddddf}|ddddf}tj| |fddS) z*Rotates half the hidden dims of the input..rNrrr1)r+stackflatten)xx1x2s r rotate_halfr@LsQ 319B 319B ;Ryb ) ) ) 1 1" 5 55rcT||}||}|dd|jddzfdd}|dd|jddzfdd}|jd}|dd|f|d|df}}|dd|f|d|df} }||zt||zz} ||zt||zz} t j| |gd} t j| | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr8rr9) unsqueezeshaperepeat_interleaver@r+cat) qkcossin unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds rapply_rotary_pos_embrRSs\$ -- & &C -- & &C c'SYr]a''' ( : :1" : E EC c'SYr]a''' ( : :1" : E EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{{511C78Gs{{511C78Gi&)r222Gi&)r222G G rc0eZdZddededzffd ZxZS) GlmAttentionNr layer_idxct||tj|j|jz|jd|_dS)NF)bias)super__init__nnLinearr)r#r(o_proj)selfrrU __class__s rrYzGlmAttention.__init__}sG +++i :T] JFL^ejkkk r)N)rrrrr*rY __classcell__)r^s@rrTrT{sallylS4ZllllllllllrrTceZdZdS)GlmForCausalLMNrrrrrararrraceZdZdS)GlmForSequenceClassificationNrrrrrcrcrrrcceZdZdS)GlmForTokenClassificationNrrrrrererrre)GlmPreTrainedModelGlmModelrarcre)r) typingrr+torch.nnrZutilsr utils.genericrllama.modeling_llamarr r r r phi3.modeling_phi3r configuration_glmr get_loggerrlogger_CHECKPOINT_FOR_DOCrrr@rRrTrarcre__all__rrrrss- 111111)(((((((((((  H % %&     W   *****-***D666%%%%Plllll>lll      %        #A        ;      r