.longrope_frequency_update+s))L)A- 4;; B C C/3{{/[/[ ,/3{{/R/R ,  5411(,(9(9KKTU1U):)%"A  T-?-?E R&*%;%;%>%>v%FD "  T-C-CPU Vc[R"U5S-nX0R:a8URURX#S9uo@lUR SUSS9 X0lX0R:ahURUR:aMURRU5UlUR SURSS9 URUlggg)z dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rrrFrN) rrmax_seq_len_cachedrrattention_scalingroriginal_max_seq_lenrr)rrrrrs r dynamic_frequency_update5dynamic_rope_update..dynamic_frequency_update>s ))L)A- ,, ,/3/@/@f/@/f ,H,  X% H&- # .. .43J3JTMfMf3f&*%;%;%>%>v%FD "  T-C-CPU V&*&?&?D # 4g .r#c>SUR;aT"XURS9 O!URS:XaT"XURS9 T"XU5$)Ndynamic)rlongrope) rope_typer)rxrr(r! rope_forwards r wrapper$dynamic_rope_update..wrapperQsD  & $T I ^^z ) %d JD\22r#r)r/r0r(r!s` @@r dynamic_rope_updater2s/W&@& <33 Nr#rrz torch.devicerreturnz torch.Tensorc jURn[US5(a UROSn[USS5=(d URUR -n[ XT-5nSnSU[R"SUS[RS9RU[RS9U- -- nX4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). partial_rotary_factor?head_dimNrdtyperr:) rope_thetarr5getattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrrbaser5r7dimattention_factorrs r _compute_default_rope_parametersrG\s$   Dr?r@rrJ isinstancerTensormaximumtensorr:rrrArBrrC) rrrrDr5r7rErrIrFrs r _compute_dynamic_ntk_parametersrRs@&   D>6A:NTWab[bTcd dDdu||AsAU[[ILLTZbgbmbmLnqttuvH  %%r#cj^URn[US5(a UROSn[USURUR -5n[ XT-5nURSnURRS5nURRS5n URRS5n SUR;aURSn URU - nO URn SS jn Uc1U (a"U (a[U "Xy5U "Xz5- 5nOU "U5nURRS 5=(d S n URRS 5=(d S nSmU4SjnSnU[R"SUS5RU[RS9U- -nSU- nSUU-- nURRSS5nU"XXcU U5unnS U"UUUS-5RU[RS9- nUS U- -UU--nUU4$)a] Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r5r6r7rIrFmscalemscale_all_dimr rcNUS::agSU-[R"U5-S-$)Nrr6g?)mathlog)scalerTs r get_mscale,_compute_yarn_parameters..get_mscales( A:V|dhhuo-33r# beta_fast beta_slowcU[R"X0S-[R-- 5-S[R"U5-- $)zPInverse dimension formula to find the dimension based on the number of rotationsr8)rWrXpi) num_rotationsrErDrs r find_correction_dim5_compute_yarn_parameters..find_correction_dims@dhh6!:Kdgg:UVWW\]`d`h`him`n\noor#c>T"XX45nT"XX45nU(a,[R"U5n[R"U5n[US5[ XrS- 54$)z.Find dimension range bounds based on rotationsrr)rWfloorceilrmin) low_rothigh_rotrErDrtruncatelowhighrbs r find_correction_range7_compute_yarn_parameters..find_correction_rangesR!'N"8$P  3#99T?D3{CAg...r#cX:XaUS- n[R"U[RS9U- X- - n[R"USS5nU$)NgMbP?r9rr)rrAfloat32clamp)rgrrE linear_func ramp_funcs r linear_ramp_factor4_compute_yarn_parameters..linear_ramp_factorsH : 5LC||Cu}}=C R KK Q2 r#rr8r;rjT)r)r<rr5r=r>r?r@rJgetrrCrrAr)rrrrDr5r7rErIrFrTrUr rZr\r^rmrt pos_freqsinv_freq_extrapolationinv_freq_interpolationrjrkrlinv_freq_extrapolation_factorrrbs @r _compute_yarn_parametersr{sI$   DN *V-@-@@+1+>+>?a+b(//2RR+1+I+I(4  n$Z%?*VBd%de )&1 ##'' 4:I##'' 49Ip/aa03363UX[[\I 9_ FY$67""&&z48H%iCGgiqrIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r#cURn[US5(a UROSn[USURUR -5n[ XT-5nURSnURSnURRS5n URRS5n [US5(a&URn URUR- n O URn U cMU S::aSn OD[R"S [R"U 5[R"U 5- -5n U(a*X+:a%[R"U[R US 9n O$[R"U[R US 9n [R""S US [R$US 9R'5U- n SXU --- nX4$) aJ Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r5r6r7 long_factor short_factorrIrFr rrMrr8)r<rr5r=r>r?r@rJrvr rrWsqrtrXrrQrprArBrC)rrrrDr5r7rEr}r~rIrFr ext_factorsinv_freq_shapers r _compute_longrope_parametersr"s$   D))*LMO&8':$''kH$G[[!;X=NPXYN$.@EUEghM]*n= 1, got rJrvrrrrNrCrr)rrrJr-rrrIs r (_validate_linear_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M ))+,M=Z ( #F ~Z66&3,QRXQYZ[;Gr#c.URnURSURSS55nSS1nS1n[UR55n[ X6XEUS9 USnUb[ U[ 5(aUS:a[RSU35 gg)Nr-rrIr rr6rr)rrrJr-rrrrIs r )_validate_dynamic_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M78M ))+,M=]hi ( #F ~Z66&3,QRXQYZ[;Gr#c*URnURSURSS55nSS1n1Skn[UR55n[ X6XEUS9 USnUb[ U[ 5(aUS:a[RSU35 URS5nUb3[ U[ 5(aUS :a[RS U35 URS 5n U b-[ U [ 5(d[RS U 35 URS 5n U b-[ U [ 5(d[RSU 35 U =(d SU =(d S:a[RSU SU S35 gg)Nr-rrI>rTrjr\r^rUrFr rr6rrFrL`rope_scaling`'s attention_factor field must be a float greater than 0, got r\z6`rope_scaling`'s beta_fast field must be a float, got r^z6`rope_scaling`'s beta_slow field must be a float, got r]rzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r) rrrJr-rrrrIrFr\r^s r _validate_yarn_parametersrs&&L  l.>.>vt.LMI (+MM ))+,M=]hi ( #F ~Z66&3,QRXQYZ[#''(:;#Z8H%-P-PTdghThZ[kZl m   -IZ 5%A%AOPY{[\  -IZ 5%A%AOPY{[\RIN+]^g]hi66?[@X Z ,r#c*URnURSURSS55n1Skn1Skn[UR55n[ X6XEUS9 [ US5(a UR OSn[USURUR-5n[X-5n URS 5n [U [5(d/[S U 55(a[RS U 35 [!U 5U S -:wa'[RS U S -S[!U 535 URS5n [U [5(d/[SU 55(a[RSU 35 [!U 5U S -:wa'[RSU S -S[!U 535 [ US5(a[R#S5 gURS5n U c[RS5 O3[U [$5(aU S:a[RSU 35 URS5n U b5[U [$5(aU S:a[RSU 35 ggg)Nr-r>r-r}r~>rIrFr rr5r6r7r~c3L# UHn[U[[45v M g7fNrNr@rC.0r.s r 0_validate_longrope_parameters..s!1dWcRS*Qe 2M2MWc"$zC`rope_scaling`'s short_factor field must be a list of numbers, got r8z5`rope_scaling`'s short_factor field must have length z, got r}c3L# UHn[U[[45v M g7frrrs r rrs!0bVaQRAU|1L1LVarzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.rIz1Missing required keys in `rope_scaling`: 'factor'rrFgr)rJrvrrrrr5r=r>r?r@rNlistallrrlen warning_oncerC)rrrJr-rrrr5r7rEr~r}rIrFs r _validate_longrope_parametersrsZ&&L  l.>.>vt.LMI@MVM ))+,M=]hi NNN OFE**fsl NNUV\U]^ _'++,>?  '.66:JS:Pbcsbtu;Q (r#c URnURSURSS55n1Skn[UR55n[ X5XAS9 USnUb[ U[ 5(aUS:a[RSU35 USnUS nUb[ U[ 5(d[RS U35 Ub[ U[ 5(d[RS U35 X::a[RS US U35 USn U b[ U [5(d[RSU 35 XR:a&[RSU SUR35 gg)Nr-r>rIr-rrr rrIr6rrrz<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rJrvrrrrNrCrrr@r) rrrJr-rrrIrrr s r _validate_llama3_parametersr&s&&L  l.>.>vt.LMIvM ))+,M=Z ( #F ~Z66&3,QRXQYZ["#45O#$67j%&H&HUVeUfghz2BE'J'JVWgVhij* q 5o5F H (44V'W$'/zBbdg7h7h ^/0 2 (+I+II u/00MfNlNlMm o Jr#c[USS5nUcgURSURSS55n[RU5nUbU"XS9 g[R SUS35 g) zG Validate the RoPE config arguments, given a `PretrainedConfig` object rJNr-rrrzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')r=rvROPE_VALIDATION_FUNCTIONSrr)rrrJr- validation_fns r rope_config_validationrUsu6>48L  l.>.>vy.QRI-11)rs 1.   H %;~*.'+!& % && ^ $&c]& >5 ! &>*.'+!& % && ^ $&c]& >5 ! &>*.'+!*& % &*& ^ $*&c]*& >5 ! *&\PTZ& Z&&4Z&?G}Z& >5 !Z&|PT3& 3&&43&?G}3& >5 !3&nPT&, &,&4&,?G}&, >5 !&,Z05. $,( $(!% llllC= l # l:[.>[XVY][ \5E \T\]`Ta \ \6F \U]^aUb \$ &6$ Xc]$ N/*:/RU/d! (8! xPS}! L168 %-)  #3 (3- r#