tj1!dZddlmZddlmZddlmZddlmZddl m Z ed eGd d eZ ed eGd deZ dd gZ dS)zMoshi model configuration)strict)PreTrainedConfig)RopeParameters)auto_docstring) AutoConfigzkmhf/hf-moshiko) checkpointceZdZUdZdZdgZdZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ edzed<dZeed<dZeed<dZeed<dZedzed<dZeed<dZeed<dZeed<dZeezed<dZeed <d!Zeed"<dZeed#<d$Zeed%<dZedzed&<dZedzed'<dZee ezdzed(<fd)Z!d*Z"xZ#S)+MoshiDepthConfiga input_size (`int`, *optional*, defaults to 4096): Dimensionality of the input hidden states. Used to connect the main decoder to the depth decoder. audio_vocab_size (`int`, *optional*, defaults to 2048): Vocabulary size of the audio part of model. Defines the number of different tokens that can be represented by the `audio_codes` passed when calling the Moshi models. ffn_dim (`int`, *optional*, defaults to 5632): Dimensionality of the "intermediate" (often named feed-forward) layer in the depth decoder block. Must be even. Example: ```python >>> from transformers import ( ... MoshiDepthConfig, ... MoshiDepthDecoder, ... ) >>> configuration = MoshiDepthConfig() >>> # Initializing a MoshiDepthDecoder (with random weights) from the kmhf/hf-moshiko style configuration >>> model = MoshiDepthDecoder(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` moshi_depthpast_key_values} vocab_sizei hidden_size input_sizenum_hidden_layersnum_attention_headsNnum_key_value_headsiaudio_vocab_size max_position_embeddingssilu hidden_acthead_dim{Gz?initializer_rangeT use_cachesliding_windowattention_dropoutiffn_dim:0yE> rms_norm_eps num_codebooksFtie_word_embeddings pad_token_id bos_token_id eos_token_idc |j|jn|j|_|jp|j|jz|_t jdi|dS)N)rrrrsuper __post_init__)selfkwargs __class__s k/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/moshi/configuration_moshi.pyr1zMoshiDepthConfig.__post_init__Ns`(,(@(LD $ $RVRj   U)9T=U)U '''''''cR|jdzdkrtd|jddS)OPart of `@strict`-powered validation. Validates the architecture of the config.r `ffn_dim=` must be even.N)r& ValueErrorr2s r5validate_architecturez&MoshiDepthConfig.validate_architectureUs8  __classcell__r4s@r5r r s  4J#4"5JKJs!!!!&*t*** c   #$S$$$JHcDj#u###ItNC%(us{(((GSL%M3 %%%%#L#*####L#*###+/L#S /D(///(((((HHHHHHHr6r ceZdZUdZdZdgZeedZdZ e e d<dZ e e d<d Z e e d <d Ze e d <d Ze d ze d <d Ze d ze d<dZe e d<d Zeezd ze d<dZee d<d Ze d ze d<dZee d<dZee d<dZe e d<dZee ze d<dZe e d<dZee d<d Z e e d!<d"Z!ee d#<d Z"e d ze d$<d Z#e d ze d%<d Z$e e%e zd ze d&<d Z&ee'zd ze d'<d Z(ee'zd ze d(<fd)Z)d*Z*e+d+Z,e-d'e'fd,Z.xZ/S)- MoshiConfiga audio_vocab_size (`int`, *optional*): Vocabulary size of the audio part of model. Defines the number of different tokens that can be represented by the `audio_codes` passed when calling the Moshi models. ffn_dim (`int`, *optional*, defaults to 22528): Dimensionality of the "intermediate" (often named feed-forward) layer in the main decoder block. Must be even. audio_encoder_config (`PreTrainedConfig | dict`, *optional*): Configuration for the audio encoder. depth_decoder_config (`PreTrainedConfig | dict`, *optional*): Configuration for the depth decoder. Example: ```python >>> from transformers import ( ... MoshiConfig, ... MoshiForConditionalGeneration, ... ) >>> configuration = MoshiConfig() >>> # Initializing a MoshiForConditionalGeneration (with random weights) from the kmhf/hf-moshiko style configuration >>> model = MoshiForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> # Saving the model, including its configuration >>> model.save_pretrained("kmhf/hf-moshiko") >>> # loading model and config from pretrained folder >>> moshi_config = MoshiConfig.from_pretrained("kmhf/hf-moshiko") >>> model = MoshiForConditionalGeneration.from_pretrained("kmhf/hf-moshiko", config=moshi_config) ```moshir)audio_encoder_configdepth_decoder_configrrrr rrNrri rrope_parametersrrrrr Tr!r#r$r%iXr&r'r(r"r)Fr*r+r,r-rPrQc |j|jn|j|_|jp|j|jz|_t |jt r8|jdd}tj |fi|j|_n |jtj d|_|j |jj n|j |_ t |j t rJ|j |j |j|j|jdt!di|j |_ n|j t!|_ t#jdi|dS)NrCmimi)rrrr)r/)rrrr isinstancerPdictpopr for_modelr codebook_sizerQupdaterr)r r0r1)r2r3audio_encoder_model_typer4s r5r1zMoshiConfig.__post_init__s{(,(@(LD $ $RVRj   U)9T=U)U d/ 6 6 E'+'@'D'D\SY'Z'Z $(2(<=U(s(sY]Yr(s(sD % %  & .(2(z!MoshiConfig.validate_architectures propertyr_ classmethodrcrKrLs@r5rNrN[st! ! FJ#4"5+5O_``KJKs!!!!&*t***#'cDj'''#'S'''48O^d*T1888JHcDj#u###ItNC%(us{(((GSL%M3 %%%%#L#*####L#*###+/L#S /D(///;?$!11D8???;?$!11D8???(((((<77X7 .   [     r6rNN)rBhuggingface_hub.dataclassesrconfiguration_utilsrmodeling_rope_utilsrutilsrauto.configuration_autor r rN__all__r/r6r5rms* ......333333111111######000000,--->H>H>H>H>H'>H>H.->HB,---| | | | | "| | .-| ~ , -r6