tj*rdZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z dd lmZd d lmZd d lmZmZmZmZmZmZmZddlmZejeZGddeZGddeZ GddeZ!GddeZ"GddeZ#GddeZ$gdZ%dS)zPyTorch BitNet model.)CallableN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack)logging)GemmaMLP)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward) BitNetConfigceZdZdS) BitNetRMSNormN__name__ __module__ __qualname__g/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/bitnet/modular_bitnet.pyrr)Drrc*eZdZdeffd ZdZxZS) BitNetMLPconfigct|t|j|j|_dSN)eps)super__init__rintermediate_size rms_norm_eps ffn_sub_norm)selfr" __class__s rr'zBitNetMLP.__init__.s<    )&*BH[\\\rc |||||||z}|SN) down_projr*act_fn gate_projup_proj)r+xr/s rforwardzBitNetMLP.forward2sUNN4#4#4T[[PQARAR5S5SVZVbVbcdVeVe5e#f#fgg r)rrrrr'r4 __classcell__r,s@rr!r!-sZ]|]]]]]]rr!ceZdZdedeffd Z d dejdeejejfdejdzde dzd e e d eejejdzff d Z xZ S) BitNetAttentionr" layer_idxct||t|j|j|_dSr$)r&r'r hidden_sizer) attn_sub_norm)r+r"r9r,s rr'zBitNetAttention.__init__8s> +++*6+=6CVWWWrN hidden_statesposition_embeddingsattention_maskpast_key_valueskwargsreturnc L|jdd}g|d|jR}|||dd}|||dd} |||dd} |\} } t|| | | \}} ||| | |j \} } tj |j j t} | ||| | |f|jsdn|j|jd|\}}|jg|dR}||}||}||fS)Nrr g)dropoutscaling)shapehead_dimq_projview transposek_projv_projrupdater9r get_interfacer"_attn_implementationrtrainingattention_dropoutrFreshape contiguousr<o_proj)r+r=r>r?r@rA input_shape hidden_shape query_states key_states value_statescossinattention_interface attn_output attn_weightss rr4zBitNetAttention.forward<s$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH ((55 kk+.. L((rr.)rrrrintr'torchTensortuplerr rr4r5r6s@rr8r87sX|XXXXXXX)- ')')|')#5<#=>') t+ ')  ') -. ') u|U\D00 1')')')')')')')')rr8ceZdZdS)BitNetDecoderLayerNrrrrrerefrrreceZdZdS) BitNetModelNrrrrrgrgjrrrgc4eZdZddiZdZdZdeffd ZxZS)BitNetForCausalLMzlm_head.weightzmodel.embed_tokens.weightNrBc 6tjdi|S)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BitNetForCausalLM >>> model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: ' >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=100) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "User: Hey, are you conscious? Can you talk to me?Assistant: No, I'm not conscious. I'm an artificial intelligence designed to assist with information and tasks. How can I help you today?" ```r)r&r4)r+ super_kwargsr,s rr4zBitNetForCausalLM.forwardss!4uww.....r) rrr_tied_weights_keys_tp_plan_pp_planrr4r5r6s@rririns]*,GHHH/ //////////rri)rirgBitNetPreTrainedModel)&__doc__collections.abcrra cache_utilsrmodeling_flash_attention_utilsrmodeling_outputsrmodeling_utilsrprocessing_utilsr utilsr gemma.modeling_gemmar llama.modeling_llamar rrrrrrconfiguration_bitnetr get_loggerrloggerrr!r8rergri__all__rrrr~s=$$$$$$ BBBBBB666666555555&&&&&&++++++/.....  H % %     L   ,),),),),)n,),),)^     *        *   /////(///D   r