tjXLddlmZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z dd l mZmZmZdd lmZdd lmZdd lmZmZdd lmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*dZ+edd8dZ,dej-de.dej-fdZ/ d9dej0dej-dej-d ej-d!ej-dzd"e1d#e1d$ee!fd%Z2ee,Gd&d'ej0Z3ed(Gd)d*ej0Z4Gd+d,ej0Z5Gd-d.eZ6e"Gd/d0eZ7Gd1d2ej0Z8e"Gd3d4e7Z9e"Gd5d6e7e Z:gd7Z;dS):)Callable)OptionalN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)capture_outputs) GraniteConfigc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/granite/modeling_granite.py rotate_halfr+,s] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''rotary_pos_embc||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer+)qkcossin unsqueeze_dimq_embedk_embeds r*apply_rotary_pos_embr73sc& -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r, hidden_statesn_repreturnc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r$expandreshape)r8r9batchnum_key_value_headsslenhead_dims r* repeat_kvrBMs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr,modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nr!rr )r#dtype)ptrainingr)rBnum_key_value_groupsr%matmul transposer functionalsoftmaxfloat32torMrJrO contiguous) rDrErFrGrHrIrJrK key_states value_states attn_weights attn_outputs r*eager_attention_forwardr\Ys3 ;<  " F4F&Jd4dee $*$>&B\$\!2 !'!9i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    r,r8position_embeddingsrHpast_key_valuesrKr:c "|jdd}g|d|jR}|||dd}|||dd} |||dd} |\} } t|| | | \}} ||| | |j \} } tj |j j t} | ||| | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)Nr rr!rC)rJrI)r$rArnviewrRrorpr7updater`r get_interfacer__attn_implementationr\rOrjrIr=rWrq)rsr8rurHrvrK input_shape hidden_shape query_statesrXrYr2r3attention_interfacer[rZs r*forwardzGraniteAttention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r,NNNN)__name__ __module__ __qualname____doc__rintrer%Tensortuplerrrr __classcell__rts@r*r^r^rsGG  } t      4IM.2(, &)&)|&)#5<#=>E&) t+ &)  &) +, &) u|U\) *&)&)&)&)&)&)&)&)r,r^RMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) GraniteRMSNormư>epsr:Ncttjt j||_||_dS)z= GraniteRMSNorm is equivalent to T5LayerNorm N)rdrer Parameterr%onesweightvariance_epsilon)rsrgrrts r*rezGraniteRMSNorm.__init__sD l5:k#:#:;; #r,r8c|j}|tj}|ddd}|tj||jzz}|j||zS)Nr!r T)keepdim) rMrVr%rUpowmeanrsqrtrr)rsr8 input_dtypevariances r*rzGraniteRMSNorm.forwards|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r,cHt|jjd|jS)Nz, eps=)rrr$r)rss r* extra_reprzGraniteRMSNorm.extra_reprs& )**II$2GIIIr,)r) rrrfloatrer%rrrrrs@r*rrs$$$$$$$$$$;U\;el;;;;JJJJJJJr,rc$eZdZfdZdZxZS) GraniteMLPct||_|j|_|j|_t j|j|j|j|_t j|j|j|j|_ t j|j|j|j|_ t|j |_ dS)Nrb)rdrer_rgintermediate_sizerrlmlp_bias gate_projup_proj down_projr hidden_actact_fnrsr_rts r*rezGraniteMLP.__init__s  !-!'!94#3T5KRXRabbby!143IPVP_``` 4#94;KRXRabbbV./ r,c||||||z}|Sr)rrrr)rsr'rs r*rzGraniteMLP.forwardsANN4;;t~~a/@/@#A#ADLLQROO#STT r,)rrrrerrrs@r*rrsG00000r,rceZdZdedeffd Z ddejdejdzdejdzd e dzd e dzd e ejejfdzd e e d ejfdZxZS)GraniteDecoderLayerr_r`cLt|j|_t|||_t ||_t|j|j|_ t|j|j|_ |j |_ dS)N)r_r`r) rdrergr^ self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormresidual_multiplierrrs r*rezGraniteDecoderLayer.__init__s !-)9MMMf%%-f.@fFYZZZ(6v7IvOb(c(c(c%#)#=   r,NFr8rH position_idsrv use_cacherurKr:c |}||}|jd||||||d|\}} |||jzz}|}||}||}|||jzz}|S)af Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r8rHrrvrru)rrrrr) rsr8rHrrvrrurKresidual_s r*rzGraniteDecoderLayer.forwards<!,,];; )4> ')%+ 3      q!=43K#KK  55mDD //  =43K#KK r,)NNNFN)rrrrrrer%r LongTensorrboolrrrrrrs@r*rrs>}>>>>>>>/304(,!&HL22|2 t+2&- 2  2 $; 2#5<#=>E2+,2 22222222r,rcLeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)GranitePreTrainedModelr_modelTrrv)r8 attentionsN)rrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr^_can_record_outputsrr,r*rrsl &*#./#4"5N!"&,&r,rc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)GraniteRotaryEmbeddinginv_freqNr_ct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrF) persistentoriginal_inv_freq) rdremax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr_rope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)rsr_device rope_init_fnrrts r*rezGraniteRotaryEmbedding.__init__1s "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr,rz torch.deviceseq_lenr:z torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetarANg?rr!rM)rrM) rrfrgrhr%arangeint64rVr)r_rrbaser#attention_factorrs r*rz6GraniteRotaryEmbedding.compute_default_rope_parametersAs&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r,cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) Nrr rmpscpuF) device_typeenabledr!r"r)rrr<r$rVr isinstancetypestrrrRr%r&r2rr3rM) rsr'rinv_freq_expandedposition_ids_expandedrfreqsembr2r3s r*rzGraniteRotaryEmbedding.forward_s!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*rr)rrrr%rrrre staticmethodrrrrrno_gradrrrrs@r*rr.slVV}VVVVVV '++/"**$*(*t* ~u$ % ***\*:U]__ < <_ < < < < z)GraniteModel.__init__..xs$ e e e  3 3 e e er,rr_F)rdre pad_token_id padding_idx vocab_sizer Embeddingrg embed_tokens ModuleListrangenum_hidden_layerslayersrrnormr rotary_embgradient_checkpointingembedding_multiplier post_initrs `r*rezGraniteModel.__init__qs    !. +L):F>>O  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    & "oom,oWW![)H4;+H)HI  M)M*) /#$7 MM -00 &++    r,)NNNNNN)rrrrrerrrr%rrr FloatTensorrrrrrrrs@r*rros }" .2.204(,26!%5 5 #d*5  t+5 &- 5  5 (4/ 5 $;5 +,5  !5 5 5 ^_ 5 5 5 5 5 r,rceZdZddiZddiZddgdgfiZfdZee dd e j dzd e j dzd e j dzd e dzde j dzde j dzdedzdee j zdeedefdZxZS)GraniteForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr8logitsct|t||_|j|_t j|j|jd|_| dS)NFrb) rdrerrrrrlrgrr rs r*rezGraniteForCausalLM.__init__sj    !&))  +y!3V5FUSSS  r,Nrr rHrrvr labelsrlogits_to_keeprKr:c n|jd||||||d| } | j} t|trt | dn|} || dd| ddf} | |jjz } d}||jd| ||jj d| }t|| | j | j | j S)a Example: ```python >>> from transformers import AutoTokenizer, GraniteForCausalLM >>> model = GraniteForCausalLM.from_pretrained("meta-granite/Granite-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-granite/Granite-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)r rHrrvr rN)rrr)lossrrvr8rr)rr rrslicerr_logits_scaling loss_functionrrrvr8r)rsr rHrrvr rrrrKoutputsr8 slice_indicesrrs r*rzGraniteForCausalLM.forwards>,64:, )%+' , , , ,  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA$+44  %4%pVFt{OeppioppD%#3!/)     r,)NNNNNNNr)rrr_tied_weights_keys_tp_plan_pp_planrerrr%rrrrrrrrrrrrs@r*rrsK*,GH23H_-z:;H.2.204(,26*.!%-.6 6 #d*6  t+6 &- 6  6 (4/ 6  4'6 $;6 el*6 +,6  6 6 6 ^6 6 6 6 6 r,r)rrr)r)rC)<collections.abcrtypingrr%r activationsr cache_utilsrr generationr integrationsr r r masking_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_graniterr+r7rrrBModulerr\r^rrrrrrr__all__rr,r*r8sP,%$$$$$ !!!!!!........))))))ffffffffff//////999999OOOOOOOOKKKKKKKKFFFFFFFF&&&&&&IIIIIIIIIIGGGGGGGG555555000000(((*++,+2 UU\ U# U%, U U U U&%% I% <% % < % L4' %  %%'(%%%%2)**@)@)@)@)@)ry@)@)+*@)FY''JJJJJRYJJ('J(         =====4===@_$><><><><><><>