tj`XddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZdd lmZddlmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(Gddej)Z*Gddej)Z+dej,de-dej,fdZ. d4dej)dej,dej,dej,d ej,dzd!e/d"e/d#eefd$Z0d%Z1d5d&Z2Gd'd(ej)Z3Gd)d*ej)Z4Gd+d,eZ5e Gd-d.eZ6e Gd/d0e6Z7e Gd1d2e6e Z8gd3Z9dS)6)Callable)OptionalN)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)capture_outputs) Cohere2Configc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)Cohere2RotaryEmbeddinginv_freqNconfigct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrF) persistentoriginal_inv_freq) super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersr!compute_default_rope_parametersrattention_scalingregister_bufferclone)selfrdevice rope_init_fnr __class__s j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/cohere2/modeling_cohere2.pyr&zCohere2RotaryEmbedding.__init__-s "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr0z torch.deviceseq_lenreturnz torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rdtype)r0r<) r*getattr hidden_sizenum_attention_headstorcharangeint64tofloat)rr0r5basedimattention_factorrs r3r+z6Cohere2RotaryEmbedding.compute_default_rope_parameters=s&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r4c|jddddf|jddd}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj |dd }| |j z}||j z} dddn #1swxYwY||j | |j fS) NrrmpscpuF) device_typeenabledr:rFr;)rrDexpandshape isinstancer0typestrr transposer@repeat_interleavecosr,sinrCr<) r/x position_idsinv_freq_expandedposition_ids_expandedrLfreqsembrVrWs r3forwardzCohere2RotaryEmbedding.forward[s!M$4-8>>@@GG HZ[\H]_acdee ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)%;;;C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s4B E  EEN)NNN)__name__ __module__ __qualname__r@Tensor__annotations__rr& staticmethodrinttuplerDr+no_gradrr^ __classcell__r2s@r3rr*slVV}VVVVVV '++/"**$*(*t* ~u$ % ***\*:U]__ < <_ < < < < Fcttjt j||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)r%r&nn Parameterr@onesweightvariance_epsilon)r/r>epsbiasr2s r3r&zCohere2LayerNorm.__init__lsB l5:k#:#:;; #r4c|j}|tj}|dd}||z ddd}||z tj||jzz}|jtj|z}||S)NrIT)keepdimr:) r<rCr@float32meanpowrsqrtrsrr)r/ hidden_states input_dtyperyvariances r3r^zCohere2LayerNorm.forwardrs#) %((77 !!"d!33!D(--a0055b$5GG&-XH]=]1^1^^  u}55 E  ,,,r4)NrmFr`rarbr&r^rirjs@r3rlrlksL$$$$$$ -------r4rlr|n_repr6c|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rPrOreshape)r|rbatchnum_key_value_headsslenr9s r3 repeat_kvr|s 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr4modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nr:rrI)rFr<)ptrainingr)rnum_key_value_groupsr@matmulrTro functionalsoftmaxrxrCr<rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r3eager_attention_forwardrs3 ;<>C/0G3w;q>>C/0G ::E: " "GJJUJ$;$; ;;r4ceZdZdZd dededzffd Z d dejde ejejfdejdzd e dzd e e d e ejejdze ejdzff d Z xZS)Cohere2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxct||_||_t |d|j|jz|_|j|jz|_ |jdz|_ |j |_ d|_ |j |dkr|jnd|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|jz|j|j|_dS)Nr9gTsliding_attentionru)r%r&rrr=r>r?r9rrrattention_dropout is_causal layer_typessliding_windowroLinearattention_biasq_projk_projv_projo_projr/rrr2s r3r&zCohere2Attention.__init__sf  " F4F&Jd4dee $*$>&B\$\!}d* !'!97=7I)7TXk7k7kf33qui   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    r4r|position_embeddingsrpast_key_valuesrr6c <|jdd}g|d|jR}|||dd}|||dd} |||dd} |\} } |jt|| | | \}} || | | |j \} } tj |j jt} | ||| | |f|jsdn|j|j|jd|\}}|jg|dR}||}||fS)NrIrr:r)rrr)rPr9rviewrTrrrrupdaterr get_interfacer_attn_implementationrrrrrrr)r/r|rrrr input_shape hidden_shape query_statesrrrVrWattention_interfacerrs r3r^zCohere2Attention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S   *';L*VY[^'_'_ $L*  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL. % %  % % ! \*k);;;;;;FFHH kk+.. L((r4r_)r`rarb__doc__rrfr&r@rcrgrrrr^rirjs@r3rrsGG  } t      :)- ()()|()#5<#=>() t+ ()  () +, () u|U\D0% 2E2LL M()()()()()()()()r4rc$eZdZfdZdZxZS) Cohere2MLPct||_|j|_|j|_t j|j|jd|_t j|j|jd|_t j|j|jd|_ t|j |_ dSNFr) r%r&rr>intermediate_sizeror gate_projup_proj down_projr hidden_actact_fnr/rr2s r3r&zCohere2MLP.__init__ s  !-!'!94#3T5KRWXXXy!143IPUVVV 4#94;KRWXXXV./ r4c||||||z}|Sr_)rrrr)r/rXrs r3r^zCohere2MLP.forwardsANN4;;t~~a/@/@#A#ADLLQROO#STT r4rrjs@r3rr sG00000r4rceZdZdedeffd Z ddejdeejejfdzdejdzd e dzd e dzd e e d eej eej ej fdzffd ZxZS)Cohere2DecoderLayerrrct|j|_t|||_t ||_t|j|j|_ dS)N)rrr>rt) r%r&r>r self_attnrmlprllayer_norm_epsinput_layernormrs r3r&zCohere2DecoderLayer.__init__si !-)9MMMf%%/V=OV\Vklllr4NFr|rrr use_cacherr6c |}||}|jd|||||d|\}} ||} ||z| z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r|rrrr)rrr) r/r|rrrrrresidualhidden_states_attention_hidden_states_mlps r3r^zCohere2DecoderLayer.forward#s4!,,];; %3T^& ' 3)+ & &  & & "!HH]33 #::=NN r4)NNNF)r`rarbrrfr&r@rcrgrboolrr FloatTensorr^rirjs@r3rrs m}mmmmmmmIM.2(,!& ''|'#5<#=>E' t+ '  ' $; '+,' u %(95;L(L"MPT"TT U''''''''r4rcLeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)Cohere2PreTrainedModelrmodelTrr)r| attentionsN)r`rarbrrdbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsrr4r3rrMsl &*#./#4"5N!"&,&r4rceZdZdeffd Zeee d dej dzdej dzdej dzde dzdej dzd e dzd eed efd ZxZS) Cohere2Modelrctj|_j|_t jjj|j|_t j fdtj D|_ tjj|_t!|_d|_|dS)Nc0g|]}t|Sr)r).0rrs r3 z)Cohere2Model.__init__..hs$ e e e  3 3 e e er4rF)r%r& pad_token_id padding_idx vocab_sizero Embeddingr> embed_tokens ModuleListrangenum_hidden_layerslayersrlrnormr rotary_embgradient_checkpointing post_initrs `r3r&zCohere2Model.__init__bs    !. +L):F>>O  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L?-FF +!."0#2 , K#5"C"C{"C"C%F%U%U%U%U##  & "oom\JJ )$+ 6 6   A})M24;3J13MN$7 /#) MM -00 &++    r4)NNNNNN)r`rarbrr&rrrr@ LongTensorrcrrrrrr r^rirjs@r3rr`s  }       .2.204(,26!%7 7 #d*7  t+7 &- 7  7 (4/ 7 $;7 +,7  !7 7 7 ^_ 7 7 7 7 7 r4rceZdZddiZddiZddgdgfiZfdZee dd e j dzd e j dzd e j dzd e dzde j dzde j dzdedzdee j zdeedefdZxZS)Cohere2ForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr|logitsc.t|t||_|j|_t j|j|jd|_|j |_ |j |_ | dSr) r%r&rrrrorr>r logit_scaletie_word_embeddingsrrs r3r&zCohere2ForCausalLM.__init__s    !&))  +y!3V5FUSSS !-#)#=  r4NrrrrYrr labelsrlogits_to_keeprr6c d|jd||||||d| } | j} t|trt | dn|} || dd| ddf} | |jz} d}||jd| ||jj d| }t|| | j | j | j S)a Example: ```python >> from transformers import AutoTokenizer, Cohere2ForCausalLM >> model = Cohere2ForCausalLM.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rrrYrr rN)rrr)lossrrr|rr)rr rQrfslicerr loss_functionrrrrr|r)r/rrrYrr rrr routputsr| slice_indicesrr"s r3r^zCohere2ForCausalLM.forwards>,64:, )%+' , , , ,  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA$**  %4%pVFt{OeppioppD%#3!/)     r4)NNNNNNNr)r`rarb_tied_weights_keys_tp_plan_pp_planr&rrr@rrcrrrrfrrrr^rirjs@r3rrsK*,GH23H_-z:;H     .2.204(,26*.!%-.6 6 #d*6  t+6 &- 6  6 (4/ 6  4'6 $;6 el*6 +,6  6 6 6 ^6 6 6 6 6 r4r)rrr)r)r):collections.abcrtypingrr@torch.nnro activationsr cache_utilsrr generationr masking_utilsr r modeling_layersr modeling_outputsr rmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_cohere2rModulerrlrcrfrrDrrrrrrrrr__all__rr4r3r<s*%$$$$$ !!!!!!........))))))RRRRRRRR999999OOOOOOOOKKKKKKKKFFFFFFFF&&&&&&IIIIIIIIIIGGGGGGGG555555000000><><><><><><>