tj' nddlZddlmZddlmZddlZddlmZddlm Z ddl m Z ddl mZmZmZmZddlmZmZmZdd lmZmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7m8Z8ddl9m:Z:m;Z;ddlm?Z?m@Z@mAZAmBZBGddejCZDGddejCZEGddejCZFdZGeddSd ZHd!ejId"eJd#ejIfd$ZK dTd&ejCd'ejId(ejId)ejId*ejIdzd+eLeJzd,eLdzd-eLdzd#eMejIejIffd.ZNeeHGd/d0ejCZOeeHGd1d2ejCZPGd3d4e ZQGd5d6e ZRGd7d8ejCZSGd9d:ejCZTGd;dejVZWe3Gd?d@e.ZXdUdBeJd#efdCZYGdDdEeXZZGdFdGeXZ[GdHdIeXZ\e3GdJdKeXZ]GdLdMeXeZ^e3GdNdOeXZ_e3GdPdQeXZ`gdRZadS)VN)Callable)Optional)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache StaticCache)GenerationConfigGenerationMixinGenerationMode)use_kernel_func_from_hubuse_kernelized_func)create_bidirectional_maskcreate_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPoolingSeq2SeqLMOutputSeq2SeqModelOutputSequenceClassifierOutputTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) AutoModel)T5Gemma2ConfigT5Gemma2DecoderConfigT5Gemma2EncoderConfigT5Gemma2TextConfigc<eZdZddedeffd ZdZdZdZxZ S) T5Gemma2RMSNormư>dimepsct||_tjt j||_dSN)super__init__r5nn Parametertorchzerosweight)selfr4r5 __class__s l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/t5gemma2/modeling_t5gemma2.pyr9zT5Gemma2RMSNorm.__init__8s? l5;s#3#344 c|tj|ddd|jzzS)Nr*T)keepdim)r<rsqrtpowmeanr5)r?xs rA_normzT5Gemma2RMSNorm._norm=s85;quuQxx}}R}>>IJJJJrBc||}|d|jzz}||S)N?)rJfloatr>type_as)r?rIoutputs rAforwardzT5Gemma2RMSNorm.forward@sLAGGII&&3!2!2!4!445~~a   rBcHt|jjd|jS)Nz, eps=)tupler>shaper5r?s rA extra_reprzT5Gemma2RMSNorm.extra_reprGs% )**<<$(<<*3MN 4@LYc4d4d4d 1M1  J!9!9!9=UZ [ [ [  J!B!B!BMDWDWDYDYfk l l l DZ;;;=S T T T T U UrBrz torch.deviceseq_lenrwreturnz torch.Tensorc|j|d}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a| Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. layer_type (`str`, *optional*): The current layer type if the model has different RoPE parameters per type. Should not be used unless `config.layer_types is not None` Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNrLrr*dtyperr) rgetattrrbnum_attention_headsr<arangeint64torM)r^rrrwbaser4attention_factorrrs rArz7T5Gemma2RotaryEmbedding.compute_default_rope_parametersvs2%j1,?fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))rBc|t||d}t||d}|ddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d 5||z dd } tj | | fd } | |z} | |z} dddn #1swxYwY| |j | |j fS) Nrxr|rrDr,mpscpuF) device_typeenabledr*r4r)rrMexpandrSrr isinstancetypestrr& transposer<catcossinr) r?rI position_idsrwrrattention_scalinginv_freq_expandedposition_ids_expandedrfreqsembrrs rArPzT5Gemma2RotaryEmbedding.forwards4J!9!9!9::#DZ*K*K*KLL$T111d]399;;BB > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 0 0&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))//C''))//C  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 vvAGv$$cff17f&;&;;;s-BE==FFr7)NNNN)rVrWrXr<Tensor__annotations__r0r9 staticmethodrrYrrRrMrno_gradrrPrZr[s@rArqrq^slUU1UUUUUU*,0+/"!% !*!*"T)!*(!*t!*$J !* ~u$ % !*!*!*\!*FU]__<<<_<<<<