tjmddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6m7Z7ddl8m9Z9e+j:e;Z<e)deGdde'Z=edGdd ej>Z?Gd!d"ej>Z@Gd#d$ej>ZAd%ZBed&dLd'ZCd(ejDd)eEd*ejDfd+ZF dMd-ej>d.ejDd/ejDd0ejDd1ejDdzd2eGd3eGd4e%e(fd5ZHeeCGd6d7ej>ZIGd8d9eZJe)d:e)Gd;deKZLGd?d@ej>ZMe)dAGdBdCeKeZNGdDdEej>ZOe)GdFdGeKZPe)dHGdIdJeKe9ZQgdKZRdS)N)Callable) dataclass)OptionalN)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdfdzed<dZ e ejdfdzed<dZejdzed <dZejdzed <dZe dzed <dZe ejdfdzed <dZe ejdfdzed <dZejdzed<dS)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r*torch FloatTensor__annotations__r+r,r r-tupler.r/r0r1r2r3r4b/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/csm/modeling_csm.pyr)r)3sK8&*D% d ")))'+FE  $+++$(OUT\(((:>M5*C/047>>>7;Je',-4;;;37)D077759%+d299926!54<666HLu'8#'=!>!ELLLEIeE$5s$:;dBIII.2M5$t+22222r>r)RMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) CsmRMSNormư>epsreturnNcttjt j||_||_dS)z9 CsmRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr9onesweightvariance_epsilon)self hidden_sizerD __class__s r?rHzCsmRMSNorm.__init__esD l5:k#:#:;; #r>r-c|j}|tj}|ddd}|tj||jzz}|j||zS)Nr!T)keepdim) dtypetor9float32powmeanrsqrtrMrL)rNr- input_dtypevariances r?forwardzCsmRMSNorm.forwardms|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r>cHt|jjd|jS)Nz, eps=)r<rLshaperMrNs r? extra_reprzCsmRMSNorm.extra_reprts& )**II$2GIIIr>)rC) r5r6r7floatrHr9Tensorr\r` __classcell__rPs@r?rBrBcs$$$$$$$$$$;U\;el;;;;JJJJJJJr>rBc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)CsmRotaryEmbeddinginv_freqNconfigct|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| ddS)N rope_typedefaultrgF persistentoriginal_inv_freq) rGrHmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrhrope_parametersrjcompute_default_rope_parametersrattention_scalingregister_bufferclone)rNrhdevice rope_init_fnrgrPs r?rHzCsmRotaryEmbedding.__init__{s "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUUUUr>rwz torch.deviceseq_lenrEz torch.Tensorc|jd}t|ddp|j|jz}d}d|t jd|dtj|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr!rTrwrT) rrgetattrrOnum_attention_headsr9arangeint64rUra)rhrwrybasedimattention_factorrgs r?rsz2CsmRotaryEmbedding.compute_default_rope_parameterss&%l3fj$//c63EIc3c U\!S!5;???BB&X]XcBddgjj k )))r>cN|jddddf|jddd|j}|dddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}tj ||fd }| |jz}||jz} dddn #1swxYwY||j | |j fS) NrrRr#mpscpuF) device_typeenabledr!rr})rgraexpandr^rUrw isinstancetypestrr transposer9catcosrtsinrT) rNx position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r?r\zCsmRotaryEmbedding.forwards!M$4-8>>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s BE&&E*-E*NNNN)r5r6r7r9rbr;r$rH staticmethodrintr<rarsno_gradrr\rcrds@r?rfrfxslVVyVVVVVV #'+/"**D *(*t* ~u$ % ***\*:U]__ < <_ < < < < rfc$eZdZfdZdZxZS)CsmMLPct||_|j|_|j|_t j|j|j|j|_t j|j|j|j|_ t j|j|j|j|_ t|j |_ dS)Nbias)rGrHrhrOintermediate_sizerILinearmlp_bias gate_projup_proj down_projr hidden_actact_fnrNrhrPs r?rHzCsmMLP.__init__s  !-!'!94#3T5KRXRabbby!143IPVP_``` 4#94;KRXRabbbV./ r>c||||||z}|Sr)rrrr)rNrrs r?r\zCsmMLP.forwardsANN4;;t~~a/@/@#A#ADLLQROO#STT r>r5r6r7rHr\rcrds@r?rrsG00000r>rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..NrRr!r)r^r9r)rx1x2s r? rotate_halfrs] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r>rotary_pos_embc||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrr unsqueeze_dimq_embedk_embeds r?apply_rotary_pos_embrsc& -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r>r-n_reprEc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r#N)r^rreshape)r-rbatchnum_key_value_headsslenr|s r? repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr>modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nr!rrR)rrT)ptrainingr#)rnum_key_value_groupsr9matmulrrI functionalsoftmaxrVrUrTrr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r?eager_attention_forwardrs3 ;<ceZdZdZdedeffd Z d dejde ejejfdzdejdzd e dzd e e d e ejejff d Z xZS) CsmAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrh layer_idxct||_||_t |d|j|jz|_|j|jz|_ |jdz|_ |j |_ d|_ tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|jz|j|j|_dS)Nr|gTr)rGrHrhrrrOrr|rrrattention_dropout is_causalrIrattention_biasq_projk_projv_projo_projrNrhrrPs r?rHzCsmAttention.__init__sB  " F4F&Jd4dee $*$>&B\$\!}d* !'!9i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    r>Nr-position_embeddingsrr,rrEc "|jdd}g|d|jR}|||dd}|||dd} |||dd} |\} } t|| | | \}} ||| | |j \} } tj |j j t} | ||| | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)NrRr#r!r)rr)r^r|rviewrrrrupdaterr get_interfacerh_attn_implementationrrrrrrr)rNr-rrr,r input_shape hidden_shape query_statesrrrrattention_interfacerrs r?r\zCsmAttention.forward*s$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r>r)r5r6r7r8r$rrHr9rbr<r rrr\rcrds@r?rrsGG y S      4IM.2(, &)&)|&)#5<#=>E&) t+ &)  &) +, &) u|U\) *&)&)&)&)&)&)&)&)r>rceZdZdedeffd Z ddejdejdzdejdzd e dzd e dzd e ejejfdzd e e d ejfdZxZS)CsmDecoderLayerrhrc4t|j|_t|||_t ||_t|j|j|_ t|j|j|_ dS)N)rhrrD) rGrHrOr self_attnrmlprB rms_norm_epsinput_layernormpost_attention_layernormrs r?rHzCsmDecoderLayer.__init__Ts !-%VyIII&>>)&*<&BUVVV(263E6K^(_(_(_%%%r>NFr-rrr, use_cacherrrEc |}||}|jd||||||d|\}} ||z}|}||}||}||z}|S)N)r-rrr,rrr=)rrrr) rNr-rrr,rrrresidual_s r?r\zCsmDecoderLayer.forward^s!,,];; )4> ')%+ 3      q!=0 !55mDD //  =0 r>)NNNFN)r5r6r7r$rrHr9rb LongTensorr boolr<rrr\rcrds@r?rrSs`y`S``````/304(,!&HL| t+&-    $; #5<#=>E+, r>rz[ The bare Csm Model outputting raw hidden-states without any specific head on top. ceZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dZ eedZejfdZxZS) CsmPreTrainedModelrhmodel)audiotextTrr,)r-r.ct|t|trD|j}t |dz D](}t j|jd|j j )dSt|trEt j |j tj|j j|j jzdSdS)Nr#r)rXstd)rG _init_weightsrCsmCodebooksHead num_codebooksrangeinitnormal_rLrhinitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr9r vocab_size)rNrrirPs r?rz CsmPreTrainedModel._init_weightss f%%% f. / / v"0M=1,-- Y Y V]$+:WXXXXX Y Y  : ; ; v Jv2ELAZ4[4[^b^i^t4t u u u u u v vr>)r5r6r7r$r;base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr9rrrcrds@r?rr~s (&*#*+#4"5N""&(" U]__vvvv_vvvvvr>rceZdZUeed<fdZeee dde j dzde j dzde j dzde j dzde dzd e j dzd edzd eed eezfd ZxZS)CsmDepthDecoderModelrhc.tj|_j|_t jjjzj|_ t j fdtj D|_ tjj|_t%|_d|_t jjjd|_|dS)Nc0g|]}t|Sr=r.0rrhs r? z1CsmDepthDecoderModel.__init__..# a a aI_VY / / a a ar>rrhFr)rGrH pad_token_id padding_idxrrI Embeddingrbackbone_hidden_size embed_tokens ModuleListrnum_hidden_layerslayersrBrOrnormrf rotary_embgradient_checkpointingrinputs_embeds_projector post_initrs `r?rHzCsmDepthDecoderModel.__init__s    !. +L&*>AR*RU[Upqqm a a a avG_A`A` a a a   v1v7JKKK ,F;;;&+#')y1LfN`gl'm'm'm$ r>N input_idsbackbone_last_hidden_staterrr, inputs_embedsrrrEc |*tstdd}|du|duz rtd|r|t |j}||nd} | |jdn |jd} ||jn|j} tj | | | z| }|}tj |dz d} | |j z} | || z}|ddk}| ||dddf<n*ts|rtd ||}t!|j|||| }|}|d}||| }|jd|jjD]}||f|||||d |}||}t-||r|nd S)aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.r$rr#rw)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.rhr4rr,rr)rrr,rrlast_hidden_stater,)rlogger warning_once ValueErrorr rhget_seq_lengthr^rwr9rclamprr)warningr0rrr.r,r+r-r)rNr2r3rrr,r4rrpast_seen_tokensinputs_seq_lengthrw codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr-r decoder_layers r?r\zCsmDepthDecoderModel.forwards&  #,D,F,F #   w    L  -t"; < \Z[[ [  ?0*$+>>>O?N?Z?99;;;`a6C6OM/22U^UdefUg)6)B%% HX|$46FIZ6Zcijjj  !K q(8a@@@M"T_4F --i&.@AAM+7?a+? ()5&@ aaad##/116RNNQ44]CC (;')+%    & $--a00 "oom,oWW![)H4;+H)HI  M)M*) /#$7 MM -00 &+/8BOOd    r>)NNNNNNN)r5r6r7r%r;rHrr rr9rr:rbr rrrr<rr\rcrds@r?rrs< !!!!  .2?C.204(,26!%N N #d*N %*$5$<N  t+ N &- N  N (4/N $;N +,N  ( (N N N ^_ N N N N N r>rc&eZdZfdZddZxZS)rct||_tjt j|jdz |||_dS)Nr#)rGrHrrIrJr9emptyrL)rNrOrrrPs r?rHzCsmCodebooksHead.__init__ sM *l5;t/AA/E{T^#_#_`` r>Nc|dz }|j|fdtjdDtjdS)Nr#c g|]:}tjdd|ddf|j;Sr)rIrlinearT)r! codebook_idxcodebook_weightr-s r?r"z,CsmCodebooksHead.forward..sX    M qqq,/A!BOT`DaDc d d   r>rr)rLrr^r9stack)rNr-codebook_indicesrQs ` @r?r\zCsmCodebooksHead.forwardsu+a/+&67     %o&;A&> ? ?    Mq999 r>rrrds@r?rr sQaaaaa        r>ra$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) ceZdZdZdZdZfdZee dde j dzde j dzde j dzde j dzde dzd e j dzd e j dzd edzd ee j zd eedeezfdZ dde j dedzde dzde j dzd e j dzdedzf fd ZxZS)CsmDepthDecoderForCausalLMNct|t||_|j|_t |j|j|j|_| dSr) rGrHrrrrrOrcodebooks_headr1rs r?rHz#CsmDepthDecoderForCausalLM.__init__*sj    )&11  +.v/A6CWY_Yjkk r>rr2r3rrr,r4labelsrlogits_to_keeprrEc ||nd} | |jdn |jd} ||jn|j} tj| | | z}|jd |||||||d| }|d}t | tr)| dkrtdd}nt| d}n| }| |dd|ddf||}| }d}|:|dddf }|j d |d|j j |d| }t|||j|j|jS) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrr#r6)r2r3rrr,r4r.)r+rXr shift_labels)r*r+r,r-r.r=)r?r^rwr9rrrrslicerWr loss_functionrhrrr,r-r.)rNr2r3rrr,r4rXrrYrrBryrwrSoutputsr- slice_indicesr+r*r[s r?r\z"CsmDepthDecoderForCausalLM.forward3s0@O?Z?99;;;`a,9,E-%a((9?[\K])6)B%% HX <???BRR$*  'A)%+'        nc * * +"" %a %~ot < < *M$$]111mQQQ3F%GIYZgIhii""$$  !#qrr'?5577L%4%dt{7M\hlrD&#3!/)     r>Fnext_sequence_lengthis_first_iterationc tj|||||fi|}|s|d|d|S)Nr3r)rGprepare_inputs_for_generationpop) rNr2r`r,rr4rar model_inputsrPs r?rcz8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationxsq=uww< +_nm  _e  " ;   9 : : : (((r>) NNNNNNNNr)NNNNF)r5r6r7_tied_weights_keys_tp_plan_pp_planrHrrr9rr:rbr rrrrr<rr\rcrcrds@r?rUrUsHH.2?C.204(,26*.!%-.A A #d*A %*$5$<A  t+ A &- A  A (4/A  4'A $;A el*A +,A  ' 'A A A ^A L,0(,2626*/#"Dj  (4/  (4/ !4Kr>rUc$eZdZfdZdZxZS)r cttj|j|jz|j|_|dtj |j|jzddS)NrFrl) rGrHrIr'r codebook_sizerOembed_audio_tokensrur9rrs r?rHz#CsmBackboneModelEmbeddings.__init__s "$,0DvG[0[^d^p"q"q  "EL1E$F$FI]$]jo      r>cl|||jz}|d}|S)Nr!r)rlrsum)rNr2r4s r?r\z"CsmBackboneModelEmbeddings.forwards9// Drrds@r?r r sG     r>r ceZdZfdZeee d dejdzdej dzdejdzde dzdej dzde dzd e ed efd ZxZS) CsmBackboneModelctj|_j|_t |_tjfdtj D|_ tj j|_t!|_d|_|dS)Nc0g|]}t|Sr=rr s r?r"z-CsmBackboneModel.__init__..r#r>rr$F)rGrHr%r&rr r)rIr*rr+r,rBrOrr-rfr.r/r1rs `r?rHzCsmBackboneModel.__init__s    !. +6v>>m a a a avG_A`A` a a a   v1v7JKKK ,F;;;&+# r>Nr2rrr,r4rrrEc H|du|duz rtd|||}|r|t|j}|V||nd}t j|jd|j|z}| d}t|j||||} |} | | |} |j d|jj D]} | | f| | |||d |} || } t| | S) a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nz:You must specify exactly one of input_ids or inputs_embedsr$rr#r6r8r9)rrrr,rr:)r>r)r rhr?r9rr^rwrrr.r,r+r-r) rNr2rrr,r4rrrBrGr-rrHs r?r\zCsmBackboneModel.forwards2  -t"; < [YZZ Z  *.*;*;I*F*FM  ?0*$+>>>O  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    & "oom,oWW![)H4;+H)HI  M)M*$7) /# MM -00 &++    r>)NNNNNN)r5r6r7rHrr rr9rrbr r:rrrrr\rcrds@r?rprps      .2.204(,26!%> > #d*>  t+> &- >  > (4/ > $;> +,>  !> > > ^_ > > > > > r>rpz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c8eZdZddiZfdZdZdZefdZfdZ dd e j dzd e j dzd e j dzd e j dzd e j dzf dZ dd e j dedzdedzde j dzde jdzf fd Zee dd e j dzd e j dzde j dzd e j dzde j dzdedzde jdzd e j dzdedzdee j zdeed eezfdZxZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightct||j|_tj|j|jd|_tj|j|j|_ t ||_ t |j|_t!j|j|_|dS)NFr)rGrHrrIrrOlm_headr'text_vocab_sizeembed_text_tokensrp _from_configbackbone_modelrUdepth_decoder_config depth_decoderr" from_config codec_config codec_modelr1rs r?rHz$CsmForConditionalGeneration.__init__s     +y!3V5FUSSS !#f.DfFX!Y!Y.;;FCC7DDVE`aa$01DEE r>c|jjSrr{r)r_s r?get_input_embeddingsz0CsmForConditionalGeneration.get_input_embeddingss"//r>c||j_dSrr)rNrs r?set_input_embeddingsz0CsmForConditionalGeneration.set_input_embeddings s+0(((r>c|ddrtj|i|\}}ntj|i|}dtfdt |jD}t |jjddi||D]}t|j|zd|vr||fS|S)Noutput_loading_infoFdepth_decoder_cVi|]%\}}||d|&Sr) startswith)r!attrrprefix prefix_lens r? z?CsmForConditionalGeneration.from_pretrained..sJ   ev&&  u   r>_from_model_config) getrGfrom_pretrainedlenvarsgeneration_configitemsr}rdelattr) clsargsrr loading_infodepth_decoder_attrsrrrrPs @@r?rz+CsmForConditionalGeneration.from_pretrained s' ::+U 3 3 ="9%''"94"J6"J"J E<<+EGG+Tc d}|jj}|dd|D]\}}t |j||z|t j|i|dS)Nrtransformers_version)r}r to_diff_dictrdrsetattrrGsave_pretrained)rNrrrrrrrPs r?rz+CsmForConditionalGeneration.save_pretrained's!"0BOOQQ 6===.4466 B BKD% D*FTM5 A A A A000000r>Nr2 input_valuesinput_values_cutoffsrXrEc||}|Ltj|d}||dk}||dk}t j||j t|d}|| dk}t j 5g}t||D]\} } | | dk} t| jddz D]} | | } | | dz} | d| | f}|j| d}|jdd}||dtd|Dt jfd |D}|j|}dddn #1swxYwY|jj}||k}|j|}||||<t jdd|jjf|jt j |jjz}|j|d}||jj k}|!|"d||<|v| d!dd|jj}||||<|||<|d k#d }d||d|dddf<|}||dS)a Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retrieve codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nr#rrr6rRr#.c30K|]}|jdVdS)rN)r^)r!els r? zQCsmForConditionalGeneration._merge_input_ids_with_input_values..as(&O&Orrx{&O&O&O&O&O&Or>c tg|]4}tj|ddd|jdz f5S)r)rIrpadr^)r!rmax_audio_framess r?r"zRCsmForConditionalGeneration._merge_input_ids_with_input_values..csBrrrZ\R]&&rAq!5EQR 5S+TUUrrrr>r~iTas_tuple)r4rX)$ryrIrrdiffr9rmaxrwrrrrziprr^rencode audio_codesrappendrRget_audio_codes_maskrhaudio_token_idr{r)rKrlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnnonzero)rNr2rrrXr4 audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsr start_idxend_idx audio_batch codec_outputs codebook_idsbatched_audio_token_idsaudio_codes_maskraudio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxsrs @r?"_merge_input_ids_with_input_valuesz>CsmForConditionalGeneration._merge_input_ids_with_input_values1s*..y99  ##%=#4#45I6#R#R 01E1JKPPRRM)-!*;q$A "O9O&P (,(8(?(? @U@UVW@X@X(Y(Y '4'@'J'J1b'Q'Q )00aAAAA B$'&O&O=N&O&O&O#O#O */+rrrr`qrrr++'$(#3#H#HIZ#[#[ ! \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$"[7N(N: .;;rrrrYrc |5|jdkr*|||||} | d}| d}d}|jd|||||| d| } | d}t| trt | dn| }||dd|ddf}d}d}d}d}||dddddf}|jd|||jj d| }|ddddddfd k d }||d d|jj dz f}tj |d d}|d}||d|ddz ddf}||}|jd||| d|d| }|j}||z}t%||||| j| j| j||jnd||jnd||jnd||jnd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nr!r4rX)r2rrr,r4rr)r+rXrr#rrRr.r)rTr)r2r3r return_dictrX) r*r4r/r+r,r-r.r0r1r2r3r=)rrr{rrr\rwr]rhrallrrIrrrr}r*r)r,r-r.r+)rNr2rrrrr,r4rXrrYrrbackbone_outputsbackbone_hidden_statesr_backbone_logitsr*r4r/depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r?r\z#CsmForConditionalGeneration.forwardsd  Y^q%8%8 CC<)=vM*/:M"8,FI.4. )%+'     "2!!48B>SV8W8Wk~ot444]k ,,'=aaaPQPQPQ>Q'RSS ! $  $QQQ1WoO.D.&4;KaekM "!!!QQQ(+t388R8@@@J&,Z&8>] @Y\]@]>]9]&^ #&(m&7&78OQW_`&7&a&a ##++T+::J*@APZ[\P]`aPacdcdcdAd*e '#)*#5 $6D$6%1+F# + %%  %% !"7!;  #55D '1",<*8'2AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%Ept    r>)NNNN) NNNNNNNNNr)r5r6r7rfrHrr classmethodrrr9rbrrrr r:rcrrrrrr<r)r\rcrds@r?rurus @Aj000111[411111*.,048&* PBPB<$&PBlT)PB$lT1 PB  t# PB   PBPBPBPBj,0(,2626 #"Dj  (4/  (4/ >.2,0.24804(,26*.!%-.Y Y #d*Y lT)Y  t+ Y $lT1 Y &- Y Y (4/Y  4'Y $;Y el*Y +,Y  " "Y Y Y ^Y Y Y Y Y r>ru)rrprrUru)r#)r)Scollections.abcr dataclassesrtypingrr9torch.nnrIrr  activationsr cache_utilsr r generationr integrationsr r r masking_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.import_utilsrutils.output_capturingr autor"configuration_csmr$r%generation_csmr& get_loggerr5r<r)ModulerBrfrrrrbrrrarrrrrrrUr rpru__all__r=r>r?rsi*%$$$$$!!!!!! &&&&&&!!!!!!........))))))ffffffffff//////999999OOOOOOOOKKKKKKKKFFFFFFFF&&&&&&______________GGGGGGGG::::::555555????????......  H % %  '3'3'3'3'3 '3'3   '3TY''JJJJJJJ('J(><><><><><><><>