tjddlZddlmZddlmZddlZddlmZddlmcm Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!dd l"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8e-deGdde#Z9e-deGdde+Z:edGddej;Z<Gdd ej;Z=d!ej>d"e?d#ej>fd$Z@ ddd&ej;d'ej>d(ej>d)ej>d*ej>dzd+eAd,eAd-ej>dzfd.ZBGd/d0ej;ZCGd1d2ej;ZDeGd3d4ej;ZEGd5d6ej;ZFGd7d8ej;ZGGd9d:ej;ZHd;ZIed< ded!ej>d=ej>d>ejJd?ejJdzd@eKdzf dAZLedB ded!ej>d>ejJd?ejJdzd@eKdzfdCZMeeLeMgGdDdEej;ZNGdFdGe!ZOe-GdHdIe'ZPe-GdJdKePZQe-GdLdMePeZRGdNdOej;ZSGdPdQePZTGdRdSej;ZUdTe?d#eVe?fdUZW dfdWe?dXe?dYe?dZe?d#ejXf d[ZYGd\d]ePZZe-d^Gd_d`ePZ[e-d^GdadbePeZ\gdcZ]dS)gN)Callable) dataclass)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_experts_implementationuse_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)force_accelerate_hooks)create_causal_maskcreate_recurrent_attention_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)merge_with_config_defaults)capture_outputs)InklingAudioConfig InklingConfigInklingTextConfigInklingVisionConfigzL Base class for Inkling outputs, with hidden states and attentions.  custom_introc2eZdZUdZdZejdzed<dS)InklingModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__r)torch FloatTensor__annotations__j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/inkling/modeling_inkling.pyr(r(4s7 59*T188888r2r(zS Base class for Inkling causal language model (or autoregressive) outputs. ceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZejdzed<dS) InklingCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr))r*r+r,r-r6r.r/r0r7r8rr9tupler:r)r1r2r3r5r5Ds  &*D% d ")))'+FE  $+++$(OUT\(((59M5*+d299926Je'(4/66648*T188888r2r5RMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) InklingRMSNormư>epsreturnNcttjt j||_||_dS)z= InklingRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr.onesweightvariance_epsilon)self hidden_sizer@ __class__s r3rDzInklingRMSNorm.__init__dsD l5:k#:#:;; #r2r9c|j}|tj}|ddd}|tj||jzz}|j||zS)NT)keepdim) dtypetor.float32powmeanrsqrtrIrH)rJr9 input_dtypevariances r3forwardzInklingRMSNorm.forwardls|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r2cHt|jjd|jS)Nz, eps=)r;rHshaperI)rJs r3 extra_reprzInklingRMSNorm.extra_reprss& )**II$2GIIIr2)r?) r*r+r,floatrDr.TensorrYr\ __classcell__rLs@r3r>r>bs$$$$$$$$$$;U\;el;;;;JJJJJJJr2r>cleZdZdZdedeffd Zdejdejdejdejfd ZxZ S) InklingRelativeLogitsa|hidden states conditioned relative position bias. `proj` is a trained bank of bias-vs-distance profiles; each token's `relative_states` mixes them into one bias value per backward distance (`sglang RelLogitsProj` + the FA4 `score_mod`, materialized densely). The bias is zero outside `0 <= distance < rel_extent`; causality and padding stay in the attention mask. d_rel rel_extentct||_tjt j|||_dSN)rCrDrdrErFr.emptyproj)rJrcrdrLs r3rDzInklingRelativeLogits.__init__~sA $LUJ!?!?@@ r2relative_statesquery_positions key_positionsrAcx||jzdd}|dddf|dddfz ddddddf}|d|jdz jg|jddddR}|d|}||dk||jkzdS)Nr rNrrO)rh transposeclamprdexpandr[gather masked_fill)rJrirjrk rel_logitsdistance gather_index position_biass r3rYzInklingRelativeLogits.forwards& 1<