tj(ddlmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZmZmZmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ej1e2Z3edeGddeZ4Gdde)Z5Gdde*Z6Gdde'Z7Gdd e$Z8Gd!d"e%Z9ed#eGd$d%eZ:eGd&d'e(e:Z;Gd(d)ej<Z=ed*Gd+d,e&e Z>Gd-d.ej<Z?eGd/d0e(Z@ed1Gd2d3e:e0ZAgd4ZBdS)5) dataclassN)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging)merge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdfdzed<dZ e ejdfdzed<dZejdzed <dZejdzed <dZe dzed <dZe ejdfdzed <dZe ejdfdzed <dZejdzed<dS)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r&torch FloatTensor__annotations__r'r(rr)tupler*r+r,r-r.r/r0a/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/csm/modular_csm.pyr%r%1sK8&*D% d ")))'+FE  $+++$(OUT\(((:>M5*C/047>>>7;Je',-4;;;37)D077759%+d299926!54<666HLu'8#'=!>!ELLLEIeE$5s$:;dBIII.2M5$t+22222r:r%ceZdZdS) CsmRMSNormNr1r2r3r9r:r;r=r=bDr:r=ceZdZdS)CsmRotaryEmbeddingNr>r9r:r;rArAfr?r:rAceZdZdS)CsmMLPNr>r9r:r;rCrCjr?r:rCceZdZdS) CsmAttentionNr>r9r:r;rErEnr?r:rEceZdZdS)CsmDecoderLayerNr>r9r:r;rGrGrr?r:rGz[ The bare Csm Model outputting raw hidden-states without any specific head on top. ceZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dZ eedZejfdZxZS) CsmPreTrainedModelconfigmodel)audiotextTrGr()r)r*ct|t|trD|j}t |dz D](}t j|jd|j j )dSt|trEt j |j tj|j j|j jzdSdS)Nrg)meanstd)super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeinitnormal_weightrJinitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr5arange vocab_size)selfmodulerUi __class__s r;rRz CsmPreTrainedModel._init_weightss f%%% f. / / v"0M=1,-- Y Y V]$+:WXXXXX Y Y  : ; ; v Jv2ELAZ4[4[^b^i^t4t u u u u u v vr:)r1r2r3r r7base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendrGrE_can_record_outputsr5no_gradrR __classcell__rcs@r;rIrIvs (&*#*+#4"5N""&(" U]__vvvv_vvvvvr:rIceZdZUeed<fdZeee dde j dzde j dzde j dzde j dzde dzd e j dzd edzd eed eezfd ZxZS)CsmDepthDecoderModelrJct|tj|j|jz|j|_tj|j|j d|_ dSNF)bias) rQ__init__nn EmbeddingrUr_backbone_hidden_size embed_tokensLinear hidden_sizeinputs_embeds_projectorr`rJrcs r;rvzCsmDepthDecoderModel.__init__sd    L&*>AR*RU[Upqq')y1LfN`gl'm'm'm$$$r:N input_idsbackbone_last_hidden_stateattention_mask position_idsr( inputs_embeds use_cachekwargsreturnc |*tstdd}|du|duz rtd|r|t |j}||nd} | |jdn |jd} ||jn|j} tj | | | z| }|}tj |dz d} | |j z} | || z}|ddk}| ||dddf<n*ts|rtd ||}t!|j|||| }|}|d}||| }|jd|jjD]}||f|||||d |}||}t-||r|nd S)aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.)rJrrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)rJrrr(r)r)rrr(rposition_embeddings)last_hidden_stater()rlogger warning_once ValueErrorrrJget_seq_lengthshaperr5r^clampr_rzwarningr}r unsqueeze rotary_emblayersnum_hidden_layersnormr )r`rrrrr(rrrpast_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr)r decoder_layers r;forwardzCsmDepthDecoderModel.forwards&  #,D,F,F #   w    L  -t"; < \Z[[ [  ?0*$+>>>O?N?Z?99;;;`a6C6OM/22U^UdefUg)6)B%% HX|$46FIZ6Zcijjj  !K q(8a@@@M"T_4F --i&.@AAM+7?a+? ()5&@ aaad##/116RNNQ44]CC (;')+%    & $--a00 "oom,oWW![)H4;+H)HI  M)M*) /#$7 MM -00 &+/8BOOd    r:)NNNNNNN)r1r2r3r!r7rvrrrr5 LongTensorr6Tensorrboolr rr8r rrorps@r;rrrrsA !!!!nnnnn  .2?C.204(,26!%N N #d*N %*$5$<N  t+ N &- N  N (4/N $;N +,N  ( (N N N ^_ N N N N N r:rrc&eZdZfdZddZxZS)rTct||_tjt j|jdz |||_dS)Nr)rQrvrUrw Parameterr5emptyrY)r`r|rUr_rcs r;rvzCsmCodebooksHead.__init__sM *l5;t/AA/E{T^#_#_`` r:Nc|dz }|j|fdtjdDtjdS)Nrc g|]:}tjdd|ddf|j;SN)rw functionallinearT).0 codebook_idxcodebook_weightr)s r; z,CsmCodebooksHead.forward..sX    M qqq,/A!BOT`DaDc d d   r:rdim)rYrVrr5stack)r`r)codebook_indicesrs ` @r;rzCsmCodebooksHead.forwardsu+a/+&67     %o&;A&> ? ?    Mq999 r:rr1r2r3rvrrorps@r;rTrTsQaaaaa        r:rTa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) ceZdZdZdZdZfdZ ddejde dzde dzdejdzdej dzd e dzf fd Z ee ddejdzd ej dzdejdzd ejdzde dzdej dzdejdzde dzde ejzdeedeezfdZxZS)CsmDepthDecoderForCausalLMNct||`t|j|j|j|_t||_ dSr) rQrvlm_headrTr|rUr_codebooks_headrrrKr~s r;rvz#CsmDepthDecoderForCausalLM.__init__sQ    L.v/A6CWY_Yjkk)&11 r:Frnext_sequence_lengthr(rris_first_iterationc tj|||||fi|}|s|d|d|S)Nrr)rQprepare_inputs_for_generationpop) r`rrr(rrrr model_inputsrcs r;rz8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationsq=uww< +_nm  _e  " ;   9 : : : (((r:rrrlabelsrlogits_to_keeprrc ||nd} | |jdn |jd} ||jn|j} tj| | | z}|jd |||||||d| }|d}t | tr)| dkrtdd}nt| d}n| }| |dd|ddf||}| }d}|:|dddf }|j d |d|j j |d| }t|||j|j|jS) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrrr)rrrrr(rr.)r'rr_ shift_labels)r&r'r(r)r*r9)rrrr5r^rKrSintslicer contiguous loss_functionrJr_r r(r)r*)r`rrrrr(rrrrrrseq_lenrroutputsr) slice_indicesr'r&rs r;rz"CsmDepthDecoderForCausalLM.forward3s0@O?Z?99;;;`a,9,E-%a((9?[\K])6)B%% HX <???BRR$*  'A)%+'        nc * * +"" %a %~ot < < *M$$]111mQQQ3F%GIYZgIhii""$$  !#qrr'?5577L%4%dt{7M\hlrD&#3!/)     r:)NNNNF) NNNNNNNNr)r1r2r3_tied_weights_keys_tp_plan_pp_planrvr5rrrr6rrrrrr rr8r rrorps@r;rr sHH22222,0(,2626*/#"Dj  (4/  (4/ !4K,.2?C.204(,26*.!%-.A A #d*A %*$5$<A  t+ A &- A  A (4/A  4'A $;A el*A +,A  ' 'A A A ^A A A A A r:rc$eZdZfdZdZxZS)r[cttj|j|jz|j|_|dtj |j|jzddS)Nr]F) persistent) rQrvrwrxrU codebook_sizer|embed_audio_tokensregister_bufferr5r^r~s r;rvz#CsmBackboneModelEmbeddings.__init__zs "$,0DvG[0[^d^p"q"q  "EL1E$F$FI]$]jo      r:cl|||jz}|d}|S)Nrr)rr]sum)r`rrs r;rz"CsmBackboneModelEmbeddings.forwards9// D>r:c 6tjdi|S)a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r9)rQr)r` super_kwargsrcs r;rzCsmBackboneModel.forwards! uww.....r:) r1r2r3rvrrrrrorps@r;rrsr?????  / / / /^_  / / / / /r:rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c8eZdZddiZfdZdZdZefdZfdZ dd e j dzd e j dzd e j dzd e j dzd e j dzf dZ dd e j dedzdedzde j dzde jdzf fd Zee dd e j dzd e j dzde j dzd e j dzde j dzdedzde jdzd e j dzdedzdee j zdeed eezfdZxZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightct||j|_tj|j|jd|_tj|j|j|_ t ||_ t |j|_t!j|j|_|dSrt)rQrvr_rwr{r|rrxtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initr~s r;rvz$CsmForConditionalGeneration.__init__s     +y!3V5FUSSS !#f.DfFX!Y!Y.;;FCC7DDVE`aa$01DEE r:c|jjSrrrz)r`s r;get_input_embeddingsz0CsmForConditionalGeneration.get_input_embeddingss"//r:c||j_dSrr)r`values r;set_input_embeddingsz0CsmForConditionalGeneration.set_input_embeddingss+0(((r:c|ddrtj|i|\}}ntj|i|}dtfdt |jD}t |jjddi||D]}t|j|zd|vr||fS|S)Noutput_loading_infoFdepth_decoder_cVi|]%\}}||d|&Sr) startswith)rattrrprefix prefix_lens r; z?CsmForConditionalGeneration.from_pretrained..sJ   ev&&  u   r:_from_model_config) getrQfrom_pretrainedlenvarsgeneration_configitemsrupdatedelattr) clsargsrrK loading_infodepth_decoder_attrsrrrrcs @@r;rz+CsmForConditionalGeneration.from_pretraineds' ::+U 3 3 ="9%''"94"J6"J"J E<<+EGG+TzQCsmForConditionalGeneration._merge_input_ids_with_input_values..s(&O&Orrx{&O&O&O&O&O&Or:c tg|]4}tj|ddd|jdz f5S)r)rwrpadr)rrmax_audio_framess r;rzRCsmForConditionalGeneration._merge_input_ids_with_input_values..sBrrrZ\R]&&rAq!5EQR 5S+TUUrrrr:)rdtypeiTas_tuple)rr)$rrwrrdiffr5r^maxrexpandrrrnziprVrrencode audio_codes transposeappendrget_audio_codes_maskrJaudio_token_idrrzonesrUlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)r`rr r rr audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrb start_idxend_idx audio_batch codec_outputs codebook_idsbatched_audio_token_idsaudio_codes_maskr#audio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxsrs @r;"_merge_input_ids_with_input_valuesz>CsmForConditionalGeneration._merge_input_ids_with_input_valuess*..y99  ##%=#4#45I6#R#R 01E1JKPPRRM)-!*;q$A "O9O&P (,(8(?(? @U@UVW@X@X(Y(Y '4'@'J'J1b'Q'Q )00aAAAA B$'&O&O=N&O&O&O#O#O */+rrrr`qrrr++'$(#3#H#HIZ#[#[ ! \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$"[7N(N: .;;r) r`rrr(rrrr merged_inputsrcs r;rz9CsmForConditionalGeneration.prepare_inputs_for_generation1s=uww< !5+)'        Y^q%8%8\=M=Mo=^=^=f CC##ZZ77%+ZZ0F%G%Gzz(++ DM   "/"@MZbLcrvww   r:rrrrrc |5|jdkr*|||||} | d}| d}d}|jd|||||| d| } | d}t| trt | dn| }||dd|ddf}d}d}d}d}||dddddf}|jd|||jj d| }|ddddddfd k d }||d d|jj dz f}tj |d d}|d}||d|ddz ddf}||}|jd||| d|d| }|j}||z}t%||||| j| j| j||jnd||jnd||jnd||jnd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrr)rrrr(rrr)r'rr_rrrr.r)rTr)rrr return_dictr) r&r0r+r'r(r)r*r,r-r.r/r9)r@r>rrSrrrrrJr_allrUrwrrr*rr&r%r(r)r*r')r`rr rr rr(rrrrrrAbackbone_outputsbackbone_hidden_statesrbackbone_logitsr&r0r+depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r;rz#CsmForConditionalGeneration.forwardPsd  Y^q%8%8 CC<)=vM*/:M"8,FI.4. )%+'     "2!!48B>SV8W8Wk~ot444]k ,,'=aaaPQPQPQ>Q'RSS ! $  $QQQ1WoO.D.&4;KaekM "!!!QQQ(+t388R8@@@J&,Z&8>] @Y\]@]>]9]&^ #&(m&7&78OQW_`&7&a&a ##++T+::J*@APZ[\P]`aPacdcdcdAd*e '#)*#5 $6D$6%1+F# + %%  %% !"7!;  #55D '1",<*8'2AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%Ept    r:)NNNN) NNNNNNNNNr)r1r2r3rrvrr classmethodrr r5rr>rrrr6rrrrr rr8r%rrorps@r;rrs @Aj000111[411111*.,048&* PBPB<$&PBlT)PB$lT1 PB  t# PB   PBPBPBPBj,0(,2626 #"Dj  (4/  (4/ >.2,0.24804(,26*.!%-.Y Y #d*Y lT)Y  t+ Y $lT1 Y &- Y Y (4/Y  4'Y $;Y el*Y +,Y  " "Y Y Y ^Y Y Y Y Y r:r)rIrrrrr)C dataclassesrr5torch.nnrwrrW cache_utilsrr generationr masking_utilsr modeling_outputsr r modeling_utilsr processing_utilsr utilsrrrr utils.genericrutils.import_utilsrutils.output_capturingrautorllama.modeling_llamarrrrrrrrconfiguration_csmr r!generation_csmr" get_loggerr1rr%r=rArCrErGrIrrModulerTrr[rr__all__r9r:r;rds"!!!!! &&&&&&........))))))//////OOOOOOOO------&&&&&&KKKKKKKKKKKK777777::::::555555                    @???????......  H % %  '3'3'3'3'3 '3'3   '3V             -        X        >        '    vvvvvvv  v<Y Y Y Y Y :'9Y Y Y xry(d d d d d !1?d d d N        /////z///0 F F F F F "46HF F  F R    r: