tjdZddlZddlmZddlmZddlmZedeGd d eZd gZ dS) zMAMBA configurationN)strict)PreTrainedConfig)auto_docstringzstate-spaces/mamba-2.8b) checkpointceZdZUdZdZdZeed<dZeed<dZ eed<d Z eed <d Z e ed <d Z edzed<d Zedzed<d Zeeezdzed<dZeed<dZeed<dZeed<dZeed<dZeed<dZe ed<dZeed<dZeezed <d!Ze ed"<d#Ze ed$<dZe ed%<d&Zeed'<d(Ze ed)<dZ eed*<dZ!eed+<dZ"eed,<dZ#eed-<dZ$eed.<fd/Z%e&d0Z'xZ(S)1 MambaConfiga: layer_norm_epsilon (`float`, *optional*, defaults to 1e-05): The epsilon to use in the layer normalization layers. expand (`int`, *optional*, defaults to 2): Expanding factor used to determine the intermediate size. use_bias (`bool`, *optional*, defaults to `False`): Whether or not to use bias in ["in_proj", "out_proj"] of the mixer block use_conv_bias (`bool`, *optional*, defaults to `True`): Whether or not to use bias in the convolution layer of the mixer block. residual_in_fp32 (`bool`, *optional*, defaults to `True`): Whether or not residuals should be in `float32`. If set to `False` residuals will keep the same `dtype` as the rest of the model rescale_prenorm_residual (`bool`, *optional*, defaults to `False`): Whether or not to rescale `out_proj` weights when initializing. use_mambapy (`bool`, *optional*, defaults to `False`): Determines the fallback strategy during training if the CUDA-based official implementation of Mamba is not available. If `True`, the mamba.py implementation is used. If `False`, the naive and slower implementation is used. Consider switching to the naive version if memory is limited. use_associative_scan (`bool`, *optional*, defaults to `True`): Whether to use PyTorch's `torch._higher_order_ops.associative_scan` for the parallel scan instead of the naive sequential implementation. The associative scan is only active during `torch.compile` tracing and requires torch >= 2.9.0. Both paths are tested to produce numerically identical results (see `test_associative_scan_matches_sequential`). Set to `False` to fall back to the sequential loop. Example: ```python >>> from transformers import MambaConfig, MambaModel >>> # Initializing a Mamba configuration >>> configuration = MambaConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = MambaModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mambaih vocab_sizei hidden_size state_size num_hidden_layersgh㈵>layer_norm_epsilonrN pad_token_id bos_token_id eos_token_idexpand conv_kernelFuse_biasT use_conv_biassilu hidden_actg?initializer_rangeresidual_in_fp32autotime_step_rankg?time_step_scalegMbP? time_step_min time_step_maxrandomtime_step_init_schemeg-C6?time_step_floorrescale_prenorm_residual use_cache use_mambapyuse_associative_scantie_word_embeddingsc t|j|jz|_|jdkrt j|jdz n|j|_tjdi|dS)Nrr ) intrr intermediate_sizer mathceilsuper __post_init__)selfkwargs __class__s k/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mamba/configuration_mamba.pyr3zMambaConfig.__post_init__^su!$T[43C%C!D!D040Cv0M0MDId&+ , , ,SWSf  '''''''cdg|jzS)Nlinear_attention)r)r4s r7 layer_typeszMambaConfig.layer_typeses"#d&<<rLs ......333333######4555M=M=M=M=M="M=M=65M=` /r8