tj+dZddlmZddlmZddlmZddlmZedeGd d eZ edeGd d eZ d gZ d S)zMpt configuration)Literal)strict)PreTrainedConfig)auto_docstringzmosaicml/mpt-7b) checkpointceZdZUdZdZdZeded<dZe ed<dZ e ed <d Z e d zed <d Ze d zed <d Zeed<d Zeed<d Zeed<dZeed<dZe ed<d S)MptAttentionConfigaD attn_type (`str`, *optional*, defaults to `"multihead_attention"`): type of attention to use. Options: `"multihead_attention"`, `"multiquery_attention"`. attn_pdrop (`float`, *optional*, defaults to `0.0`): The dropout probability for the attention layers. attn_impl (`str`, *optional*, defaults to `"torch"`): The attention implementation to use. One of `"torch"`, `"flash"`, or `"triton"`. clip_qkv (`float`, *optional*): If not `None`, clip the queries, keys, and values in the attention layer to this value. softmax_scale (`float`, *optional*): If not `None`, scale the softmax in the attention layer by this value. If `None`, will default to `1/sqrt(hidden_size)`. prefix_lm (`bool`, *optional*, defaults to `False`): Whether the model should operate as a Prefix LM. This requires passing an extra `prefix_mask` argument which indicates which tokens belong to the prefix. Tokens in the prefix can attend to one another bi-directionally. Tokens outside the prefix use causal attention. qk_ln (`bool`, *optional*, defaults to `False`): Whether to apply layer normalization to the queries and keys in the attention layer. attn_uses_sequence_id (`bool`, *optional*, defaults to `False`): Whether to restrict attention to tokens that have the same token_type_ids. When the model is in `train` mode, this requires passing an extra *token_type_ids* argument which indicates which sub-sequence each token belongs to. Defaults to `False` meaning any provided *token_type_ids* will be ignored. alibi (`bool`, *optional*, defaults to `True`): Whether or not to use the alibi bias instead of positional embedding. alibi_bias_max (`int`, *optional*, defaults to 8): The maximum value of the alibi bias. attn_configmultihead_attention)r multiquery_attention attn_typer attn_pdroptorch attn_implNclip_qkv softmax_scaleF prefix_lmqk_lnattn_uses_sequence_idTalibialibi_bias_max)__name__ __module__ __qualname____doc__base_config_keyrr__annotations__rintrstrrfloatrrboolrrrrg/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mpt/configuration_mpt.pyr r s8$OH]IwDE]]]JIs!Hedl!!!"&M54<&&&ItE4"'4'''E4NCr%r ceZdZUdZdZdeiZddddZdZe e d<d Z e e d<d Z e e d<d Z e e d <dZe e d <dZe e d<dZee ze d<dZee d<dZee ze d<dZee d<dZeezdze d<dZee d<dZeezdze d<dZee d<dZee d<dZee d<d Zee d!<d"Zee d#<dZ ee d$<dZ!e dze d%<dZ"e dze d&<dZ#e e$e zdze d'<fd(Z%xZ&S)) MptConfigaZ expansion_ratio (`int`, *optional*, defaults to 4): The ratio of the up/down scale in the MLP. max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. layer_norm_epsilon (`float`, *optional*, defaults to 1e-05): The epsilon to use in the layer normalization layers. learned_pos_emb (`bool`, *optional*, defaults to `True`): Whether to use learned positional embeddings. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. init_device (`str`, *optional*, defaults to `"cpu"`): The device to use for parameter initialization. Defined for backward compatibility logit_scale (`float`, *optional*): If not None, scale the logits by this value. no_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in all linear layers. embedding_fraction (`float`, *optional*, defaults to 1.0): The fraction to scale the gradients of the embedding layer by. norm_type (`str`, *optional*, defaults to `"low_precision_layernorm"`): Type of layer norm to use. All MPT models uses the same layer norm implementation. Defined for backward compatibility. Example: ```python >>> from transformers import MptConfig, MptModel >>> # Initializing a Mpt configuration >>> configuration = MptConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = MptModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` mptr n_headsd_modeln_layers)num_attention_heads hidden_sizenum_hidden_layersiexpansion_ratio max_seq_leni vocab_sizeg resid_pdropgh㈵>layer_norm_epsilon emb_pdropTlearned_pos_embNcpu init_device logit_scaleno_biasg?embedding_fractionlow_precision_layernorm norm_typeF use_cacheg{Gz?initializer_rangetie_word_embeddings pad_token_id bos_token_id eos_token_idc |jt|_n0t|jtrtdi|j|_t jdi|dS)Nr$)r r isinstancedictsuper __post_init__)selfkwargs __class__s r&rKzMptConfig.__post_init__so   #133D   ($ / / F1EED4DEED '''''''r%)'rrrr model_typer sub_configs attribute_mapr+r rr*r,r3r4r5r6r"r7r8r9r#r rIr;r!r<r=r>r@rArBrCrDrErFlistrK __classcell__)rNs@r&r(r(Es%%NJ "45K( 'M GSGSHcOSKJ"K""" $$$$ Ius{    OT   48K**T1888K&*Kt#***GT ####.Is...It#u### $$$$#L#*####L#*###+/L#S /D(///(((((((((r%r(N) rtypingrhuggingface_hub.dataclassesrconfiguration_utilsrutilsrr r(__all__r$r%r&rYs ......333333######,---((((()((.-(V,---L(L(L(L(L( L(L(.-L(^ -r%