tj-~ddlmZddlmZddlmZddlZddlmZddlm Z ddl m Z m Z dd l mZmZdd lmZdd lmZdd lmZmZdd lmZmZddlmZmZddlmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-Gddej.Z/Gddej.Z0dZ1eddCdZ2dej3de4d ej3fd!Z5 dDd#ej.d$ej3d%ej3d&ej3d'ej3dzd(e6d)e6d*ee!fd+Z7ee2Gd,d-ej.Z8Gd.d/ej.Z9Gd0d1ej.Z:Gd2d3eZ;e"Gd4d5eZ<e"d67eGd8d9eZ=e"d:7Gd;deGd=d>e Z?e"d?7Gd@dAe$>??  DKK $>$>?? %//155 ||M***r;) __name__ __module__ __qualname__r!r)torchTensorrA __classcell__r9s@r:r#r#,sk 0      $+el+u|++++++++r;r#c eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)LasrEncoderRotaryEmbeddinginv_freqNr$ct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrKF) persistentoriginal_inv_freq) r(r)max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr$rope_parametersrMcompute_default_rope_parametersrattention_scalingregister_bufferclone)r8r$device rope_init_fnrKr9s r:r)z#LasrEncoderRotaryEmbedding.__init__Ks "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr;rYz torch.deviceseq_lenr=z torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rrdtype)rYr`) rTgetattrr,num_attention_headsrEarangeint64tofloat)r$rYr[basedimattention_factorrKs r:rUz:LasrEncoderRotaryEmbedding.compute_default_rope_parameters[s&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r;cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) NrrmpscpuF) device_typeenabledrrhr_)rKrfexpandshapererY isinstancetypestrrr?rEcatcosrVsinr`) r8x position_idsinv_freq_expandedposition_ids_expandedrnfreqsembrwrxs r:rAz"LasrEncoderRotaryEmbedding.forwardys!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*N)NNN)rBrCrDrErF__annotations__r!r) staticmethodrinttuplerfrUno_gradrrArGrHs@r:rJrJHslVV0VVVVVV +/+/"**!D(*(*t* ~u$ % ***\*:U]__ < <_ < < < <