tjddZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2m3Z3e,j4e5Z6e+deGddeZ7e+deGddeZ8e+deGd d!eZ9e+eGd"d#eZ:Gd$d%e j;Z<Gd&d'e j;Z=Gd(d)e j;Z>Gd*d+e j;Z?d,Z@dOd-ZAGd.d/e j;ZBd0e jCd1eDd2e jCfd3ZE dPd5e j;d6e jCd7e jCd8e jCd9e jCdzd:eFd;eFdd?e j;ZHGd@dAeZIe+GdBdCe&ZJGdDdEeJeZKe+GdFdGeJZLe+dHGdIdJeJeZMe+dKGdLdMeJeZNgdNZOdS)QzPyTorch Moshi model.N)Callable) dataclass)AnyOptional)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCache)GenerationConfigGenerationMixin)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast ModelOutputSeq2SeqLMOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)maybe_autocast) AutoModel) MoshiConfigMoshiDepthConfigzK Outputs of [`MoshiForConditionalConditionalGeneration.generate`]. ) custom_introc~eZdZUdZdZejdzed<dZej dzed<dZ ej dzed<dZ e ej dzed<dZe ej dzed<dZej dzed<dZe e ej dzed <dZe e ej dzed <dZedzed <dZej dzed <dS) (MoshiConditionalGenerationGenerateOutputa audio_sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, 1, sequence_length)`, *optional*): The generated audio waveforms. sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated text sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`Cache`, *optional*, returned when `use_cache=True`): Contains the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. audio_codes (`torch.LongTensor` of shape `(batch_size*num_return_sequences, num_codeooks, sequence_length)`, *optional*): The generated audio codes. Returned if `return_audio_codes=True`. Intermediate audio "tokens" which transforms to `audio_sequences` once passed through the audio decoder. Naudio_sequences sequencessequences_scoresscoreslogits beam_indices attentions hidden_statespast_key_values audio_codes)__name__ __module__ __qualname____doc__r&torchTensor__annotations__r' LongTensorr( FloatTensorr)tupler*r+r,r-r.r r/f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/moshi/modeling_moshi.pyr%r%,s)B,0OU\D(///)-Iu$&---15e'$.555.2FE%# $t +222.2FE%# $t +222,0L%"T)0009=JeE-./$6===<@M5u012T9@@@$(OUT\(((+/K!D(/////r;r%z% `MoshiForCausalLM` outputs. ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e dzed<dZ e ejdfdzed<dZe ejdfdzed <dS) MoshiCausalLMOutputWithPasta% loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. Nlossr*last_hidden_stater..r-r,)r0r1r2r3r?r4r8r6r*r@r.r r-r9r,r:r;r<r>r>`s  &*D% d ")))'+FE  $+++26u(4/666$(OUT\(((:>M5*C/047>>>7;Je',-4;;;;;r;r>z2 `MoshiForConditionalGeneration` outputs. ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e dzed<dZ e ejdfdzed<dZe ejdfdzed <dZejdzed <dZejdzed <dZe dzed <dZe ejdfdzed <dZe ejdfdzed<dS)(MoshiConditionalGenerationOutputWithPasta' loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `text_labels` is provided): Text language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the text language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `audio_labels` is provided): Audio language modeling loss (for next-token prediction). audio_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the audio language modeling heads. depth_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Past key-values of the depth decoder. depth_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Hidden states of the depth decoder depth_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Depth decoder's Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr?r*r@r..r-r, depth_loss audio_logitsdepth_past_key_valuesdepth_hidden_statesdepth_attentions)r0r1r2r3r?r4r8r6r*r@r.r r-r9r,rCrDrErFrGr:r;r<rBrB{sJ.&*D% d ")))'+FE  $+++26u(4/666$(OUT\(((:>M5*C/047>>>7;Je',-4;;;+/J!D(///-1L%#d*111*.54<...@Du0#56=DDD=AeE-s23d:AAAAAr;rBceZdZUdZdZejdzed<dZej dzed<dZ ej dzed<dZ ejdzed<dS)MoshiUnconditionalInputa) input_ids (`torch.Tensor `of shape `(batch_size, sequence_length), *optional*): The sequence used as a text prompt for the generation. user_audio_codes (`torch.Tensor `of shape `(batch_size, num_codebooks, sequence_length), *optional*): The audio codes used as audio user prompt for the generation. Has priority over `user_input_values` and represents the audio "tokens" of `user_input_values` once passed through the audio encoder. moshi_audio_codes (`torch.Tensor `of shape `(batch_size, num_codebooks, sequence_length), *optional*): The audio codes used as audio Moshi prompt for the generation. Has priority over `moshi_input_values` and represents the audio "tokens" of `moshi_input_values` once passed through the audio encoder. attention_mask (`torch.LongTensor`) of shape `(batch_size, sequence_length)`, *optional*): Attention mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: 1 for tokens that are **not masked**, 0 for tokens that are **masked**. N input_idsuser_audio_codesmoshi_audio_codesattention_mask) r0r1r2r3rJr4r7r6rKr5rLrMr:r;r<rIrIs~  *.Iu$&---,0elT)000-1u|d*111.2NE$t+22222r;rIc<eZdZddedeffd ZdZdZdZxZ S) MoshiRMSNormư>dimepsct||_tjt j||_dSN)super__init__rRnn Parameterr4onesweight)selfrQrR __class__s r<rVzMoshiRMSNorm.__init__s= l5:c??33 r;c|tj|ddd|jzzS)NrT)keepdim)r4rsqrtpowmeanrR)r[xs r<_normzMoshiRMSNorm._norms85;quuQxx}}R}>>IJJJJr;c||}||jz}||SrT)rdfloatrZtype_as)r[rcoutputs r<forwardzMoshiRMSNorm.forwardsEAGGII&&$+++---~~a   r;cHt|jjd|jS)Nz, eps=)r9rZshaperRr[s r< extra_reprzMoshiRMSNorm.extra_reprs% )**<<$(<<> !r;FrTrrps@r<rrsW______""""""""r;rc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)MoshiRotaryEmbeddinginv_freqNconfigct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrF) persistentoriginal_inv_freq) rUrVmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)r[rdevice rope_init_fnrr\s r<rVzMoshiRotaryEmbedding.__init__ s "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr;rz torch.deviceseq_lenreturnz torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rrdtyperr) rgetattr hidden_sizenum_attention_headsr4arangeint64torf)rrrbaserQattention_factorrs r<rz4MoshiRotaryEmbedding.compute_default_rope_parameterss&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r;cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) Nrr^r mpscpuF) device_typeenabledrrQr)rrfexpandrkrr isinstancetypestrrrzr4catcosrsinr) r[rc position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r<rizMoshiRotaryEmbedding.forward:s!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*rT)NNN)r0r1r2r4r5r6r!rV staticmethodrrnr9rfrno_gradrrirorps@r<rr slVV{VVVVVV %)+/"**d"*(*t* ~u$ % ***\*:U]__ < <_ < < < <