tj<ddlmZmZddlmZddlmZddlmZm Z ddl m Z e j e ZedeGd d eZedeGd d eZedeGd deZedeGddeZgdZdS))AnyLiteral)strict)PreTrainedConfig)auto_docstringlogging)intervalzgoogle/gemma-4-e2b-it) checkpointcbeZdZUdZdZdZeed<dZeed<dZ eed<d Z e ed <d Z e eeeefzed <d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<d Zeed!<ed"d#$d%&Zeed'<fd(ZxZS))Gemma4AudioConfiga subsampling_conv_channels (`list[int]`, defaults to `[128, 32]`): Channel sizes for the convolutional layers in the Sub-sample Convolution Projection. residual_weight (`float`, defaults to `0.5`): Scaling applied to hidden_states prior to combining with the residual in the feedforward. attention_chunk_size (`int`, defaults to `12`): The sub-sequence size for attention processing. attention_context_left (`int`, defaults to `13`): The leftward context size for the attention chunk. attention_context_right (`int`, defaults to `0`): The rightward context size for the attention chunk. attention_logit_cap (`float`, defaults to `50.0`): Cap applied to attention weights. attention_invalid_logits_value (`float`, defaults to `1e-9`): Value to use for invalid logits in attention. use_clipped_linears (`bool`, defaults to `True`): If true, apply clipping to the Linear layers, drawing bounds from the model checkpoint. gradient_clipping (`float`, defaults to `1e10`): Clipping value used to stabilize extremely large gradient values. output_proj_dims (`int`, defaults to `1536`): Dimension of the final linear projection from `hidden_size` to the model's output. gemma4_audioi hidden_size num_hidden_layersnum_attention_headssilu hidden_act) subsampling_conv_channelsconv_kernel_sizeg?residual_weightattention_chunk_size attention_context_leftrattention_context_rightgI@attention_logit_capgeattention_invalid_logits_valueTuse_clipped_linearsư> rms_norm_epsg _Bgradient_clippingioutput_proj_dimsg?)minmax{Gz?)defaultinitializer_rangec t|jtrt|j|_t jdi|dS)N) isinstancertuplelistsuper __post_init__selfkwargs __class__s m/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/gemma4/configuration_gemma4.pyr3zGemma4AudioConfig.__post_init__NsO d4e < < R-1$2P-Q-QD *''''''') __name__ __module__ __qualname____doc__ model_typerint__annotations__rrrstrrr1r0rrfloatrrrr r!r"boolr$r%r&r r,r3 __classcell__r7s@r8r r s. JKs    J>GtCy5c?:FFFc OU   "#""""$C$$$#$S$$$!%%%%,2"E222 $$$$L%#u### c   9xxCS999$GGGuGGG(((((((((r9r c eZdZUdZdZdgZddddddddddddd Zdddd d d dd Zd gdgfddgdgfdgdgfdZdZ e e d<dZ e e d<dZ e e d<dZe e d<dZe e d<dZe e d<dZe e d<d Zee d!<d"Ze e d#<d$Zee d%<d&Zee d'<d(Zee d)<d*Ze d+ze d,<d-Ze ee zd+ze d.<d/Ze d+ze d0<d(Zee d1<d+Ze d+ze d2<d3Z!ee d4<d5Z"e ezd+ze d6<d7Z#e e d8<d+Z$eed+ze d9<d+Z%ed+ze d:<d+Z&e'd;d+ze d<<dZ(e e d=<dZ)e e d><d+Z*e d+ze d?<d7Z+e e d@<d3Z,ee dA<d*Z-e e dB<d3Z.ee dC<d3Z/ee dD<d+Z0e d+ze dE<d+Z1e d+ze dF<d+Z2e d+ze dG<fdHZ3dIZ4xZ5S)JGemma4TextConfigaS use_bidirectional_attention (`str`, *optional*): Controls bidirectional attention behavior. When set to `"vision"`, vision tokens attend bidirectionally while text tokens use causal attention. When set to `"all"`, all tokens use bidirectional attention. vocab_size_per_layer_input (`int`, defaults to 262144): Vocabulary size for the per-layer input embeddings (PLE). Used by models with per-layer residual streams where a smaller embedding is added at each decoder layer. hidden_size_per_layer_input (`int`, defaults to 256): Per-layer hidden dimension for the PLE system. The actual embedding weight has shape `[vocab_size_per_layer_input, num_hidden_layers * hidden_size_per_layer_input]` because all layers are packed into a single table. See the [Gemma4](https://huggingface.co/docs/transformers/main/en/model_doc/gemma4#per-layer-embeddings-ple) docs for a description of the full PLE pipeline. num_global_key_value_heads (`int`, *optional*): Number of key-value heads for global (full) attention layers. If `None`, defaults to `num_key_value_heads`. global_head_dim (`int`, defaults to 512): Dimension of each attention head in global (full) attention layers. attention_k_eq_v (`bool`, defaults to `False`): Whether keys and values share the same projection weights. When `True`, the key projection output is reused as the value projection. num_kv_shared_layers (`int`, defaults to 0): Number of consecutive decoder layers that share the same key-value projections. A value of 0 means no sharing (each layer has independent KV projections). enable_moe_block (`bool`, defaults to `False`): Whether to enable Mixture-of-Experts (MoE) blocks in the decoder layers. When `True`, eligible layers will use a sparse MoE feed-forward network. use_double_wide_mlp (`bool`, defaults to `False`): Whether to use a double-width MLP with fused gate and up projections. top_k_experts (`int`, *optional*): Number of experts activated per token in MoE layers. Only used when `enable_moe_block=True`. moe_intermediate_size (`int`, *optional*): Intermediate (hidden) size of each expert's feed-forward network in MoE layers. Only used when `enable_moe_block=True`. gemma4_textpast_key_valuescolwisereplicated_with_grad_allreducerowwisepacked_colwisemoe_tp_experts) zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.q_normzlayers.*.self_attn.k_normzlayers.*.self_attn.o_projlayers.*.mlp.gate_projlayers.*.mlp.up_projlayers.*.mlp.down_projlayers.*.experts.gate_up_projlayers.*.experts.down_projlayers.*.experts ep_router grouped_gemm)rOrPrQzlayers.*.routerrRrSrT input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnormi vocab_sizei ri$intermediate_sizerrrnum_key_value_headshead_dimgelu_pytorch_tanhhidden_activationmax_position_embeddingsr*r,r#r$T use_cacherN pad_token_id eos_token_id bos_token_idtie_word_embeddingsrope_parametersFattention_biasr'attention_dropoutisliding_window layer_typesfinal_logit_softcapping)allvisionuse_bidirectional_attentionvocab_size_per_layer_inputhidden_size_per_layer_inputnum_global_key_value_headsglobal_head_dimattention_k_eq_vnum_kv_shared_layersenable_moe_blockuse_double_wide_mlp num_experts top_k_expertsmoe_intermediate_sizec |jdkrd|_|jdzdz|_|j'dfdt |jD|_|jr;|jdx}dkr(t d |d d|jd<d d d ddddd}|j||_tj di|dS)NrvFrmrkcBg|]}t|dzzrdndS)rksliding_attentionfull_attention)rC).0isliding_window_patterns r8 z2Gemma4TextConfig.__post_init__..sG   (,QU6L,L'M'Mc##Sc   r9rz/Last layer must use `full_attention`, but got `z*`. Forcing last layer to `full_attention`.r+g@ rope_type rope_theta proportionalg?g.A)rpartial_rotary_factorr)rrr.) rx is_causalrsrtrangerloggerwarningrpr2r3)r5r6last_layer_typedefault_rope_paramsrr7s @r8r3zGemma4TextConfig.__post_init__s,  +u 4 4"DN#'#6!#;q"@D    #%& "    t566   D    4D4DR4H!HM] ] ] NN}/}}}   $4D R 09!Q!Q,:UYituuf f    '#6D '''''''r9c |S)Nr.)r5r6s r8convert_rope_params_to_dictz,Gemma4TextConfig.convert_rope_params_to_dicts r9)6r:r;r<r=r>keys_to_ignore_at_inferencebase_model_tp_planbase_model_ep_planbase_model_pp_planr^r?r@rr_rrrbrdrfrArhr,rBr$rirCrjrlr1rnrorpdictrqrrrsrtrurxrryrzr{r|r}r~rrrrrr3rrDrEs@r8rGrGUsr##JJ#4"5%.%.%.%E%E%."+ )"+)9&/,   #, )"+&)7&4,  &(9:#%568IJ!"_$56 JK!s!!!s        Hc0s000#*S***#u###L%It L#*   +,L#S /D(,,, L#*   $$$$#'OTD[''' ND   ,/sU{T)///NC$(KcT!(((,0UT\000CG!9D!@GGG&----'****-1d 111OS"d""" !#!!!"d""" %%%%"Kt""" $M3:$$$(,3:,,,(((((6r9rGc ReZdZUdZdZdddddddddd ZdZdZee d <d Z ee d <d Z ee d <dZ ee d<dZ ee d<dZee d<dZee d<dZee d<dZee d<dZedze d<dZedze d<dZedze d<dZee d <d Zee d!<d"Zee d#<dZee d$<dZee d%<d&Zee d'<fd(ZxZ S))Gemma4VisionConfiga pooling_kernel_size (`int`, *optional*): Spatial pooling kernel size applied after patchification. position_embedding_size (`int`, defaults to 10240): Maximum number of position embeddings for the vision encoder. Controls the size of the learned 2D position embedding table used by the patch embedder. use_clipped_linears (`bool`, defaults to `False`): Whether to use weight-clipped linear layers. When enabled, linear layer weights are clamped to a fixed range during the forward pass to improve numerical stability. standardize (`bool`, defaults to `False`): If true, applies a bias and scale to the soft tokens returned from the pooler. gemma4_visionrJrKrL) z!encoder.layers.*.self_attn.q_projz!encoder.layers.*.self_attn.k_projz!encoder.layers.*.self_attn.v_projz!encoder.layers.*.self_attn.q_normz!encoder.layers.*.self_attn.k_normz!encoder.layers.*.self_attn.o_projzencoder.layers.*.mlp.gate_projzencoder.layers.*.mlp.up_projzencoder.layers.*.mlp.down_projY@iri r_rrrrb@rdrerfr#r$rgrhFNrqr'rrrprpooling_kernel_size patch_sizei(position_embedding_sizer" standardizer*r,c \|j ddd|_tjdi|dS)Nr+rrr.)rpr2r3r4s r8r3z Gemma4VisionConfig.__post_init__s?   '1:%#P#PD '''''''r9)!r:r;r<r=r>r default_thetarr?r@r_rrrbrdrfrAr$rBrhrqrCrrrprrrrr"rr,r3rDrEs@r8rrs  !J-6-6-6-M-M-6*3(1*3  MK!s!!!s!!!!!!!!Hc0s000L%#*S***"'ND4K'''&)ut|)))#'OTD['''    J#,S,,, %%%%K#u###(((((((((r9rc`eZdZUdZdZeeedZdZ ee e e fzdze d<dZee e e fzdze d<dZee e e fzdze d<dZedze d <d Zedze d <d Zedze d <dZedze d<dZedze d<dZedze d<dZedze d<dZedze d<dZee d<fdZxZS) Gemma4Configag boi_token_id (`int`, *optional*, defaults to 255999): The begin-of-image token index to wrap the image prompt. eoi_token_id (`int`, *optional*, defaults to 258882): The end-of-image token index to wrap the image prompt. boa_token_id (`int`, *optional*, defaults to 256000): The begin-of-audio token index to wrap the audio prompt. eoa_token_index (`int`, *optional*, defaults to 258883): The end-of-audio token index to wrap the audio prompt. Example: ```python >>> from transformers import ( >>> Gemma4AudioConfig, >>> Gemma4Config, >>> Gemma4ForConditionalGeneration, >>> Gemma4TextConfig, >>> Gemma4VisionConfig, >>> ) >>> # Initializing a Gemma 4 Audio config. >>> audio_config = Gemma4AudioConfig() >>> # Initializing a Gemma 4 Text config. >>> text_config = Gemma4TextConfig() >>> # Initializing a Gemma 4 vision config. >>> vision_config = Gemma4VisionConfig() >>> # Initializing a Gemma 4 config similar to google/gemma-4-e2b-it >>> configuration = Gemma4Config(text_config, vision_config, audio_config) >>> # Initializing a model from the google/gemma-4-e2b-it configuration >>> model = Gemma4ForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```gemma4) text_config vision_config audio_configNrrri boi_token_idiB eoi_token_idi@image_token_idiDvideo_token_idi boa_token_idiCeoa_token_indexiAaudio_token_idr*r,Troc H|j.t|_tdn0t |jt rtdi|j|_|jtdt |jt rtdi|j|_|jtdt |jt rtdi|j|_tj di|dS)Nz4text_config is None. Using default Gemma4TextConfig.zHvision_config is None. Gemma4Model.vision_tower will not be initialized.zFaudio_config is None. Gemma4Model.audio_tower will not be initialized.r.) rrGrinfor/rrrrr r2r3r4s r8r3zGemma4Config.__post_init__Ws   #/11D  KKN O O O O ($ / / D/CC$2BCCD    % KKb c c c d($ / / J!3!I!Id6H!I!ID    $ KK` a a a d' . . G 1 F FD4E F FD '''''''r9)r:r;r<r=r>rGrr sub_configsrrrArr@rrrr?rrrrrrr,rBrorCr3rDrEs@r8rrs& & PJ'+)K =AK!DcN2T9@@@@DM%S#X6=DDD>BL#d38n4t;BBB&L#*&&&&L#*&&&!(NC$J(((!(NC$J(((&L#*&&&")OS4Z)))!(NC$J(((&*ut|*** $$$$(((((((((r9r)r rrGrN)typingrrhuggingface_hub.dataclassesrconfiguration_utilsrutilsrr utils.type_validatorsr get_loggerr:rr rGrr__all__r.r9r8rs ......333333,,,,,,,,------  H % %23335(5(5(5(5((5(5(435(p2333FFFFF'FF43FR23333(3(3(3(3()3(3(433(l2333N(N(N(N(N(#N(N(43N(b Z Y Yr9