tj4dZddlmZddlmZddlmZddlmZeedd Gd d eZ eed d Gd deZ edeGddeZ dgZ dS)zDBRX model configuration)strict)PreTrainedConfig)RopeParameters)auto_docstringz4This config is used to instantiate attention layers.z$transformers-community/dbrx-instruct) custom_intro checkpointcTeZdZUdZdZdZeezed<dZ eezdzed<dZ eed<dS) DbrxAttentionConfigaz attn_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the attention layers. clip_qkv (`float`, *optional*): If set, clip the queries, keys, and values in the attention layer to this value. kv_n_heads (`int`, *optional*, defaults to 1): For grouped_query_attention only, allow user to specify number of kv heads. attn_config attn_pdropNclip_qkv kv_n_heads) __name__ __module__ __qualname____doc__base_config_keyrfloatint__annotations__rri/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/dbrx/configuration_dbrx.pyr r s` $O!J !!!#'HcEkD '''Jrr z6This config is used to instantiate feedforward layers.ceZdZUdZdZdZeed<dZe dzed<dZ eed<d Z eed <d Z eed <dZ edzed <dZeed<dZedzed<fdZxZS) DbrxFFNConfiga ffn_act_fn (`dict`, *optional*, defaults to `None`): A dict specifying activation function for the FFN. The dict should have a key 'name' with the value being the name of the activation function along with any additional keyword arguments. If `None`, then set to `{"name": "silu"}`. ffn_hidden_size (`int`, *optional*, defaults to 3584): The hidden size of the feedforward network. moe_num_experts (`int`, *optional*, defaults to 4): The number of experts in the mixture of experts layer. moe_top_k (`int`, *optional*, defaults to 1): The number of experts to use in the mixture of experts layer. moe_jitter_eps (`float`, *optional*, defaults to `None`): If not `None`, the jitter epsilon for the mixture of experts layer. moe_loss_weight (`float`, *optional*, defaults to 0.01): The loss weight for the mixture of experts layer. moe_normalize_expert_weights (`float`, *optional*, defaults to 1.0): The normalization factor for the expert weights. ffn_configi hidden_sizeN ffn_act_fniffn_hidden_sizemoe_num_expertsr moe_top_kmoe_jitter_epsg{Gz?moe_loss_weightg?moe_normalize_expert_weightsc |j ddi|_dD]}||vr||t|dkrtd|t jdi|dS)Nnamesilu) model_typeattn_implementationexperts_implementationtransformers_version _commit_hash torch_dtypedtyperzFound unknown kwargs=r)r!poplen ValueErrorsuper __post_init__)selfkwargsk __class__s rr7zDbrxFFNConfig.__post_init__Qs ? "%v.DO  AF{{ 1 v;;!  7f7788 8'''''''r)rrrrrr rrr!dictr"r$r%r&rr'r(r7 __classcell__r;s@rrr-s &#OK"Jt """OSOSIs#'NEDL'''!OU!!!14 %$,444(((((((((rr)r ceZdZUdZdZeedZdddddZd Z e d ze d<d Z e d ze d<d Z e d ze d<d Ze d ze d<d Ze e d<dZed ze d<dZed ze d<d Zeezd ze d<d Zeezd ze d<dZee d<dZee d<dZed ze d<d Zeezd ze d<d Ze d ze d<d Ze d ze d<d Ze ee zd ze d<dZ ee d<fdZ!d Z"xZ#S)! DbrxConfiga max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. ffn_config (`dict`, *optional*): A dictionary used to configure the model's FFN module. Example: ```python >>> from transformers import DbrxConfig, DbrxModel >>> # Initializing a Dbrx configuration >>> configuration = DbrxConfig(n_layers=2, d_model=256, n_heads=8, vocab_size=128) >>> # Initializing a model (with random weights) from the configuration >>> model = DbrxModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` dbrx)r rn_headsd_modeln_layers max_seq_len)num_attention_headsr num_hidden_layersmax_position_embeddingsiNi} vocab_sizer resid_pdrop emb_pdropr rT use_cacheg{Gz?initializer_rangeFoutput_router_logitsrope_parameters pad_token_id bos_token_id eos_token_idtie_word_embeddingsc |jt|_n0t|jtrtdi|j|_|jt |_n0t|jtrt di|j|_|jj|_tj di|dS)Nr) r r isinstancer<rrrnum_key_value_headsr6r7)r8r9r;s rr7zDbrxConfig.__post_init__s   #244D   ($ / / G2FFT5EFFD  ? "+ooDOO  . . ?+>>do>>DO#'#3#> '''''''rc2|jrtddS)zOPart of `@strict`-powered validation. Validates the architecture of the config.z5tie_word_embeddings is not supported for DBRX models.N)rUr5)r8s rvalidate_architecturez DbrxConfig.validate_architectures)  # VTUU U V Vr)$rrrrr,r r sub_configs attribute_maprCrrrBrDrErKrLrrMr r<rrNboolrOrPrQrrRrSrTlistrUr7rZr=r>s@rr@r@fs.J"5]SSK( '#0 MGS4ZGS4ZHcDj"Kt"""J #K###!Iut|!!!59K$t+d2999.2J $t+222It#u###(-$+---48O^d*T1888#L#*####L#*###+/L#S /D(/// %%%% ( ( ( ( (VVVVVVVrr@N) rhuggingface_hub.dataclassesrconfiguration_utilsrmodeling_rope_utilsrutilsrr rr@__all__rrrrds{......333333111111######G5*  "I51(1(1(1(1($1(1(  1(hABBBDVDVDVDVDV!DVDVCBDVN .r