tjqW|dZddlmZddlmZddlZddlmcmZ ddlm Z m Z mZddl m ZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddl m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3e,j4e5Z6dZ7Gddej8Z9Gddej:Z;dZdZ=Gddej:Z>dej d e?d!ej fd"Z@ d?d$ej:d%ej d&ej d'ej d(ej dzd)eAd*eAd+e'e)fd,ZBGd-d.ej:ZCGd/d0eZDe*Gd1d2e%ZEe*Gd3d4eEZFGd5d6eEeZGGd7d8eeEZHGd9d:eeEZIGd;dTNnormalized_shapeepselementwise_affinebiascTt||||||dSr$)super__init__)selfr4r5r6r7devicedtype __class__s r.r:zNemotronLayerNorm1P.__init__?s0 )30BD&RWXXXXXr0inputreturnc|jjdkr |jjnd}t|||j|jdz|j|j}t|jjd5tj |cdddS#1swxYwYdS)Nmpscpu?Fr+enabled) r<typer/r4weightr7r5rF layer_norm)r;r?r+r,s r.forwardzNemotronLayerNorm1P.forwardJs+0<+<+E+Eel''5 (  5t{S7H$)UYU]   (95 I I I ' '<& ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' 's%BBB)r3TTNN) __name__ __module__ __qualname__intlistrfloatboolr:rrK __classcell__r>s@r.r2r2>s#' Y YS /D0 Y Y! Y  Y Y Y Y Y Y'V'''''''''r0r2c eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)NemotronRotaryEmbeddinginv_freqNconfigct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrWF) persistentoriginal_inv_freq) r9r:max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrXrope_parametersrZcompute_default_rope_parametersrattention_scalingregister_bufferclone)r;rXr< rope_init_fnrWr>s r.r:z NemotronRotaryEmbedding.__init__Ws "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr0r<z torch.deviceseq_lenr@z torch.TensorcV|jd}|jdd}t|ddp|j|jz}t ||z}d}d|t jd|dt j |t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorrDhead_dimNrr=)r<r=) ragetgetattr hidden_sizenum_attention_headsrOr%arangeint64torQ) rXr<rgbaserjrkdimattention_factorrWs r.rbz7NemotronRotaryEmbedding.compute_default_rope_parametersgs(%l3 & 6 : :;RTW X X6:t44h8JfNh8h(2233 U\!S!5;???BB&X]XcBddgjj k )))r0cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) Nrr!rBrCFrErlrvrm)rWrQexpandshapertr< isinstancerGstrr transposer%catcosrcsinr=) r;x position_idsinv_freq_expandedposition_ids_expandedr+freqsembrrs r.rKzNemotronRotaryEmbedding.forwards!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*r$)NNN)rLrMrNr%r__annotations__r"r: staticmethodrrOtuplerQrbno_gradrrKrSrTs@r.rVrVTslVV~VVVVVV )-+/"**%*(*t* ~u$ % ***\*>U]__ < <_ < < < <