tjT:ddlmZddlmZddlZddlmZddlmZddlm Z ddl m Z m Z dd l mZdd lmZmZmZdd lmZdd lmZmZmZmZdd lmZmZddlmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.Gddej/Z0edGddej/Z1Gddej/Z2dZ3eddAdZ4d ej5d!e6d"ej5fd#Z7 dBd%ej/d&ej5d'ej5d(ej5d)ej5dzd*e8d+e8d,e$e&fd-Z9ee4Gd.d/ej/Z:Gd0d1eZ;eGd2d3e"Zed67Gd:d;ee<Z?ed67Gd<d=ee<Z@ed67Gd>d?ee<ZAgd@ZBdS)C)Callable)OptionalN)nn)auto_docstring)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GenericForQuestionAnswering GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargscan_return_tuple)maybe_autocastmerge_with_config_defaults)capture_outputs) ArceeConfigc$eZdZfdZdZxZS)ArceeMLPc`t||_|j|_|j|_t j|j|j|j|_t j|j|j|j|_ t|j |_ dS)Nbias) super__init__config hidden_sizeintermediate_sizerLinearmlp_biasup_proj down_projr hidden_actact_fnselfr) __class__s f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/arcee/modeling_arcee.pyr(zArceeMLP.__init__3s  !-!'!9y!143IPVP_``` 4#94;KRXRabbbV./ cx||||SN)r/r1r.)r3xs r5forwardzArceeMLP.forward<s*~~dkk$,,q//::;;;r6)__name__ __module__ __qualname__r(r: __classcell__r4s@r5r#r#2sG00000<<<<<<epsreturnNcttjt j||_||_dS)z; ArceeRMSNorm is equivalent to T5LayerNorm N)r'r(r Parametertorchonesweightvariance_epsilon)r3r*rDr4s r5r(zArceeRMSNorm.__init__BsD l5:k#:#:;; #r6 hidden_statesc|j}|tj}|ddd}|tj||jzz}|j||zS)NT)keepdim) dtypetorHfloat32powmeanrsqrtrKrJ)r3rL input_dtypevariances r5r:zArceeRMSNorm.forwardJs|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r6cHt|jjd|jS)Nz, eps=)tuplerJshaperK)r3s r5 extra_reprzArceeRMSNorm.extra_reprQs& )**II$2GIIIr6)rC) r;r<r=floatr(rHTensorr:r\r>r?s@r5rBrB@s$$$$$$$$$$;U\;el;;;;JJJJJJJr6rBc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)ArceeRotaryEmbeddinginv_freqNr)ct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultraF) persistentoriginal_inv_freq) r'r(max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr)rope_parametersrccompute_default_rope_parametersrattention_scalingregister_bufferclone)r3r)device rope_init_fnrar4s r5r(zArceeRotaryEmbedding.__init__Xs "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr6roz torch.deviceseq_lenrEz torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rrNrQ)rorQ) rjgetattrr*num_attention_headsrHarangeint64rRr])r)rorqbasedimattention_factorras r5rkz4ArceeRotaryEmbedding.compute_default_rope_parametershs&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r6cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) NrrOr mpscpuF) device_typeenabledrNr{ru)rar]expandr[rRro isinstancetypestrr transposerHcatcosrlsinrQ) r3r9 position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r5r:zArceeRotaryEmbedding.forwards!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*r8NNN)r;r<r=rHr^__annotations__r!r( staticmethodrintrZr]rkno_gradrr:r>r?s@r5r`r`UslVV{VVVVVV %)+/"**d"*(*t* ~u$ % ***\*:U]__ < <_ < < < <