"S0S1\Rh5r?"S2S3\5r@\&"S4S9\&"S5S6\ 555rA\&"S7S8\A55rB"S9S:\Rh5rC\&"S;S9"S<S=\A\55rD"S>S?\Rh5rE\&"S@SA\A55rF\&"SBS9"SCSD\A\/55rG/SEQrHg)H) dataclass)CallableOptionalUnionN)check_model_inputs)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introc\rSrSr%SrSr\\R\ S'Sr \R\ S'Sr \\ \ \R\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'Sr\\R\ S 'Sr\R\ S 'Sr\\ \ \R\ S 'Sr\\ \RS4\ S 'Sr\\ \RS4\ S'Sr\\R\ S'Srg)CsmOutputWithPast1a loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____firstlineno____doc__r&rtorch FloatTensor__annotations__r'r(tupler)r*r+r,r-r.r/r0__static_attributes__r1\/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/csm/modeling_csm.pyr$r$1s'<)-D(5$$ %, $FE  $AEOXeE%*;*;$<=>E=AM8E%"3"3S"89:A:>Ju00#567>6:!2!23:.2%++2OS!8E%8I8I2J,K#LSKO%0A0A30F*G!HOHLhuU->->-C'DEL15M8E--.5r<r$RMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$) CsmRMSNormcc>[TU]5 [R"[R "U55UlX lg)z) CsmRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr7onesweightvariance_epsilon)self hidden_sizeeps __class__s r=rDCsmRMSNorm.__init__es/ ll5::k#:; #r<cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NrT)keepdim) dtypetor7float32powmeanrsqrtrIrH)rJr) input_dtypevariances r=forwardCsmRMSNorm.forwardmsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r<c^[URR5SUR3$)Nz, eps=)r:rHshaperIrJs r= extra_reprCsmRMSNorm.extra_reprts* ))*+6$2G2G1HIIr<)rIrH)gư>) r2r3r4r5rDrZr_r; __classcell__rMs@r=r@r@cs$;JJr<r@cl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)CsmRotaryEmbeddingxconfigc>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultinv_freqF persistent)rCrDhasattr isinstancerhdictgetrimax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrfr rope_init_fnattention_scalingregister_bufferrloriginal_inv_freq)rJrfdevicerlrMs r=rDCsmRotaryEmbedding.__init__ys  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rwrfrtryrurvriN) r2r3r4r5rrDr7no_gradrrZr;rarbs@r=rdrdxs6/y//" ]]_ < [TU]5 XlURUlURUl[ R "URURURS9Ul[ R "URURURS9Ul [ R "URURURS9Ul [URUl g)Nbias)rCrDrfrKintermediate_sizerELinearmlp_bias gate_projup_proj down_projr hidden_actact_fnrJrfrMs r=rDCsmMLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r<cURURURU55URU5-5nU$r)rrrr)rJrrs r=rZCsmMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r<)rrfrrrKrrr2r3r4r5rDrZr;rarbs@r=rrs0r<rcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..NrPrr)r]r7r)rx1x2s r= rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r<cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrrr unsqueeze_dimq_embedk_embeds r=apply_rotary_pos_embrsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r<r)n_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r]rreshape)r)rbatchnum_key_value_headsslenhead_dims r= repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr<modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)NrrrP)rrR)ptrainingr)rnum_key_value_groupsr7matmulrr]rE functionalsoftmaxrTrSrRrr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r=eager_attention_forwardrs3 ; ;[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-URS9Ul[R"UR URUR-URS9Ul[R"UR URUR-URS9Ul[R"UR UR-UR URS9Ulg)NrgTr)rCrDrfrgetattrrKnum_attention_headsrrrrattention_dropout is_causalrErattention_biasq_projk_projv_projo_projrJrfrrMs r=rDCsmAttention.__init__sI  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r<r)position_embeddingsrpast_key_valuecache_positionrrc 4URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO URUR S.UD6unnUR""/UQSP76R%5nUR'U5nUU4$)NrPrr)rrreager)rr)r]rrviewrrrrupdaterrrf_attn_implementationrrrrrrr)rJr)rrrrr input_shape hidden_shape query_statesrrrr cache_kwargsattention_interfacerrs r=rZCsmAttention.forward s$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<'))!. )) ! )) !!1!12 ))+,)) u||U\\) *))))r<rc8^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \RS \ \ \R\R4S \\S \ \R4SjjrSrU=r$)CsmDecoderLayeri8rfrc >[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul g)N)rfrrL) rCrDrKr self_attnrmlpr@ rms_norm_epsinput_layernormpost_attention_layernormrs r=rDCsmDecoderLayer.__init__9si !--%VI&>)&*<*<&BUBUV(263E3E6K^K^(_%r<r)rrr use_cacherrrrc Un URU5nUR"SUUUUUUUS.UD6upX-nUn URU5nURU5nX-nU$)N)r)rrrrrrr1)rrrr) rJr)rrrrrrrresidual_s r=rZCsmDecoderLayer.forwardCs!,,]; >>  ')%)) 3     !0 !55mD /  0 r<)rKrrrr)NNNFNN)r2r3r4r5rrrDr7rrrr boolr:rrrZr;rarbs@r=rr8s`y`S`2637*.$)59KO||!.u//0  !  D> !!1!12&eELL%,,,F&GH+, u|| r<rz[ The bare Csm Model outputting raw hidden-states without any specific head on top. cb^\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr Sr \\S.rU4SjrS rU=r$) CsmPreTrainedModelierfmodelTrr()r)r*c>[TU]U5 [U[5(a]URn[ US- 5H>nUR RURSURRS9 M@ gg)Nrr)rVstd) rC _init_weightsrpCsmCodebooksHead num_codebooksrangerHdatanormal_rfinitializer_range)rJrr irMs r=r  CsmPreTrainedModel._init_weights}sn f% f. / /"00M=1,- ""1%--3DKK[TU]U5 URUlURUl[ R "URUR-UR5Ul [ R"[UR5Vs/sHn[X5PM sn5Ul[UR UR"S9Ul['US9UlSUl[ R,"URUR SS9UlUR15 gs snf)NrrfFr)rCrD pad_token_id padding_idx vocab_sizerE Embeddingr backbone_hidden_size embed_tokens ModuleListr num_hidden_layersrlayersr@rKrnormrd rotary_embgradient_checkpointingrinputs_embeds_projector post_initrs r=rDCsmDepthDecoderModel.__init__s  !.. ++LL&*>*>ARAR*RU[UpUpqmmAFvG_G_A` aA`I_V /A` a  v11v7J7JK ,F;&+#')yy1L1LfN`N`gl'm$  bsD/ input_idsbackbone_last_hidden_staterrr( inputs_embedsrrrrc (Ub:[RR5(d[R S5 SnUSLUSL- (a [ S5eU(a Uc [ 5nUciUbUR5OSn UbURSOURSn Ub URO URn [R"XU -U S9nUc[R"US- SS9n XR-nURX-5nUSS:HnUb X&SS2S4'O?[RR5(dU(a[RS5 URU5n[!UR"UUUUUS 9nUnUR%S5nUR'UU5nUR(SUR"R*HnU"U4UUUUUUS .U D6nM UR-U5n[/UU(aUS 9$SS 9$) a* backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.rrrz)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.rf input_embedsrrr(r)rrrrrrlast_hidden_stater()r7compiler is_compilinglogger warning_once ValueErrorr get_seq_lengthr]rzarangeclampr"r%warningr,rrfrr*r(r'r)r)rJr/r0rrr(r1rrrpast_seen_tokensinputs_seq_lengthrz codebook_idxsoffsetinput_ids_are_first_codebookrr)r decoder_layers r=rZCsmDepthDecoderModel.forwards7&  #ENN,G,G,I,I   M  L  -t"; <Z[ [ 0*nO  !CRC^==?de :G:S 3 3A 6YbYhYhijYk -:-F]))IL\L\F"\\*:O`<`iopN  !KK(:BM"__4F --i.@AM+9!+<+A ()5&@ad#~~22449UNNQ44]C (;;&))+%  & &//2 "oom\J![[)H4;;+H+HIM) *).#-$7  MJ -0 &+/8O  >B  r<)r%r+r,r(r)r!r*r")NNNNNNNN)r2r3r4r5r r9rDrrr7rrr8rr rrrrr:rrZr;rarbs@r=rrs !! '+BF1537+/59$(59R ##R %-U->->$?R !. R u//0 R "% R   1 12R D>R !!1!12R +,R  u-- .R R r<rc2^\rSrSrU4SjrSSjrSrU=r$)r ic>[TU]5 X l[R"[ R "URS- X55UlgNr)rCrDr rErFr7emptyrH)rJrKr r"rMs r=rDCsmCodebooksHead.__init__s: *ll5;;t/A/AA/E{#_` r<c Uc3URSnUR[R"U5nOUS- nURUn[ URS5Vs/sH9n[ R RUSS2USS24XFR5PM; nn[R"USS9nU$s snf)Nrrr) r]rHr7r?r rErlinearTstack)rJr)r seq_lengthcodebook_weightrD codebook_idxs r=rZCsmCodebooksHead.forwards  !&,,Q/J"kk%,,z*BCO*Q.M"kk-8O!&o&;&;A&> ? ?  MM q,/A!BODaDcDc d ?   Mq9   s%?B=)r rHrrrbs@r=r r sa r<r a$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) cL^\rSrSrSrSrSrU4SjrSrSr \ \ SS\ RS\\ RS\\ R S \\ RS \\\\\ R4S \\ RS \\ RS \\S\\ RS\\\ R 4S\\S\\\44Sjj55rSS\ RS \\S\\ RS \\ RS\\ R4 U4SjjjrSrU=r$)CsmDepthDecoderForCausalLMiNc>[TU]U5 [U5UlURUl[ UR URUR5UlUR5 gr) rCrDrrr"r rKr codebooks_headr-rs r=rD#CsmDepthDecoderForCausalLM.__init__sY  )&1  ++.v/A/A6CWCWY_YjYjk r<cXlgrr)rJdecoders r= set_decoder&CsmDepthDecoderForCausalLM.set_decoders r<cUR$rr\r^s r= get_decoder&CsmDepthDecoderForCausalLM.get_decoders zzr<r/r0rrr(r1labelsrrlogits_to_keeprrc UR"SUUUUUUUU S.U D6n U Sn [U [5(a!U S:Xa [SS5nO[U *S5nOU nUR U SS2USS24U bXOS5nUR 5nSnUbBUSSS24R 5nUR "SUSURRUS.U D6n[UUU RU RU RS9$) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )r/r0rrr(r1rrrrN.)r'rcr" shift_labels)r&r'r(r)r*r1) rrprslicerYr loss_functionrfr"rr(r)r*)rJr/r0rrr(r1rcrrrdroutputsr) slice_indicesr'r&rfs r=rZ"CsmDepthDecoderForCausalLM.forward"s82**  'A)%+')      nc * *" %a %~ot < *M$$ !]A- .Q_Qk0Mqu ""$  !#qr'?557L%%dt{{7M7M\hlrD&#33!//))   r<c >[T U]"XX4U40UD6nUSSS:HnU(dURS5 URS5 U$)Nrrr0r)rCprepare_inputs_for_generationpop) rJr/r(rr1rr model_inputsis_first_generation_steprMs r=rm8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationfscw< ~ Y_ $00@#A!#D#I '   9 : (r<)rYrr") NNNNNNNNNrNNNN)r2r3r4r5_tied_weights_keys_tp_plan_pp_planrDr^rarrr7rrr8rrr listrrrrr:rrZrmr;rarbs@r=rWrWsHH'+BF1537KO59-1$(5934@ ##@ %-U->->$?@ !. @ u//0 @ "%tE4E4E/F(F"GH @   1 12@ ))*@ D>@ !!1!12@ c5<</0@ +,@  u,, -@ @ J,0595959 ##"%!!1!12    1 12  !!1!12 r<rWc.^\rSrSrU4SjrSrSrU=r$)CsmBackboneModelEmbeddingsi}c>[TU]5 [R"URUR -UR 5UlURS[R"UR5UR -SS9 g)Naudio_tokens_offsetsFrm) rCrDrEr#r r"rKembed_audio_tokensrxr7r?rs r=rD#CsmBackboneModelEmbeddings.__init__~sn "$,,0D0DvGXGX0X[a[m[m"n  "ELL1E1E$FIZIZ$Zgl  r<c^URXR-5nURSS9nU$)Nrr)r{rzsum)rJr/r6s r=rZ"CsmBackboneModelEmbeddings.forwards4..y;T;T/TU #''A'. r<)r{rrbs@r=rxrx}s r<rxc^\rSrSrU4Sjr\\SS\\RS\\RS\\RS\\ S\\RS\\RS \\ S \\S \4S jj55rS rU=r$)CsmBackboneModelic >[TU]U5 URUlURUl[ U5Ul[R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)NrrF)rCrDr r!r"rxr%rEr&r r'rr(r@rKrr)rdr*r+r-rs r=rDCsmBackboneModel.__init__s  !.. ++6v>mmAFvG_G_A` aA`I_V /A` a  v11v7J7JK ,F;&+#  bs*Cr/rrr(r1rrrrc 8USLUSL- (a [S5eUcURU5nU(a Uc [5nUcDUbUR5OSn [R "XUR S-URS9nUcURS5n[URUUUUUS9n Un URX5n URSURRHn U "U 4U UUUU S.UD6n M URU 5n [U US9$) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nz:You must specify exactly one of input_ids or inputs_embedsrrr3r5)rrrrrr7)r=r%r r>r7r?r]rzrrrfr*r(r'r)r)rJr/rrr(r1rrrrBrr)rrGs r=rZCsmBackboneModel.forwardsK2  -t"; <YZ Z  *.*;*;I*FM 0*nO  !CRC^==?de +0<< ]5H5H5K"KTaThTh,N  )33A6L(;;&))+%  & "oomJ![[)H4;;+H+HIM)*).-$7 MJ -0 &++  r<)r%r+r(r)r!r*r")NNNNNNN)r2r3r4r5rDrrrr7rrr r8rrrrrZr;rarbs@r=rrs 151537+/5959$(D E,,-D !.D u//0 D "% D   1 12 D !!1!12D D>D +,D  !D D r<rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c<^\rSrSrSS/rU4SjrSrSrSr\ U4Sj5r U4S jr SS \ \ RS \ \ RS \ \ RS \ \ RS\ \ R4 SjjrSS \ R S\ \S\ \ R S\ \ R$S\ \ R 4 U4Sjjjr\\SS \ R S \ \ RS\ \ RS \ \ RS\ \ R S\ \\\\ R$4S\ \ R$S \ \ R S\ \S\ \ R S\\\ R4S\\S\\\44Sjj55rSrU=r $)CsmForConditionalGenerationiz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightc>[TU]U5 URUl[R"UR URSS9Ul[R"URUR 5Ul [RU5Ul [RUR5Ul[ R""UR$5UlUR)5 g)NFr)rCrDr"rErrKlm_headr#text_vocab_sizeembed_text_tokensr _from_configbackbone_modelrWdepth_decoder_config depth_decoderr from_config codec_config codec_modelr-rs r=rD$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r<c.URR$rrr%r^s r=get_input_embeddings0CsmForConditionalGeneration.get_input_embeddingss""///r<c$XRlgrr)rJrs r=set_input_embeddings0CsmForConditionalGeneration.set_input_embeddingss+0(r<cURR(aOURURRR UR RR5 ggr)rftie_codebooks_embeddings_tie_or_clone_weightsrr%r{rrr^s r= _tie_weights(CsmForConditionalGeneration._tie_weightssL ;; / /  & &##00CC""((55  0r<c>URSS5(a[T U]"U0UD6up4O[T U]"U0UD6nSn[U5n[ UR 5R 5VVs0sH"upxURU5(dMXvSU_M$ n nn[ URR 5RSS0U E5 U Hn[UR XW-5 M SU;aUW4$U$s snnf)Noutput_loading_infoFdepth_decoder__from_model_config) rrrCfrom_pretrainedlenvarsgeneration_configitems startswithrrdelattr) clsargsrr loading_infoprefix prefix_lenattrrdepth_decoder_attrsrMs r=r+CsmForConditionalGeneration.from_pretraineds  ::+U 3 3"''"94"J6"J E<G+TSnURRR5nURSS5 UR 5HupV[ URX5-U5 M [ TU]"U0UD6 g)Nrtransformers_version)rr to_diff_dictrnrsetattrrCsave_pretrained)rJrrrrrrrMs r=r+CsmForConditionalGeneration.save_pretrained"sq!"00BBOOQ 6=.446KD D**FM5 A7 00r<r/ input_valuesinput_values_cutoffsrcrcURU5nUGbM[RRUS5nX3S:R 5nXfS:n[ R "UR5URS9R[U5S5nXvRS5:n[ R"5 /n[X#5HupXS:n [U RSS- 5Hpn Xn XS-n U SX24nUR R#URS55nUR$R'SS5nUR)US5 Mr M [SU55n[ R*"UVs/sH7n[RRUSSSUURS- 45PM9 sn5nUR R-U5nSSS5 UR.R0nUU:HnUR2R5W5nUWUU'[ R6"SSUR.R84UR[ R:S 9UR.R<-nUR2R5U5R?S5nXR.R@:HnURCURE5S5UU'UbgURS5RCSSUR.R85nUUUU'UUU'US :HRGS S 9nS UUSUSSS24'UnXTS.$s snf!,(df  GN=f)a8 Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retreive codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nrrrr3rPr.c3<# UHoRSv M g7f)rN)r]).0els r= QCsmForConditionalGeneration._merge_input_ids_with_input_values..\s&O=Nrxx{=Ns)rzrRiTas_tuple)r1rc)$rrErpaddiffr7r?maxrzrrrrzipr r]rencode audio_codesrappendrQget_audio_codes_maskrfaudio_token_idrr%rGr longcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatr~nonzero)rJr/rrrcr1 audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsr start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesrbatched_audio_token_idsaudio_codes_maskraudio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss r="_merge_input_ids_with_input_values>CsmForConditionalGeneration._merge_input_ids_with_input_values,su*..y9  ##%==#4#45I6#R 01JKPPRM)!*;$A "<U"C&8i>O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA LGn$'&O=N&O#O */++`qr`qZ\R]]&&rAq!5EQR 5S+TU`qr+'$(#3#3#H#HIZ#[ !!$"[[77N(N: ..;;[T U]"S UUUUUS.UD6nUb|URS:XalURS5cZUR UURS5URS5URS5S9nUR USUSSS.5 U$) N)r/r(rr1rrr1rrrc)r/rrrc)r1rcr/r1)rCrmndimrrrr) rJr/r(rr1rrro merged_inputsrMs r=rm9CsmForConditionalGeneration.prepare_inputs_for_generation~sw< +)')      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw r<rrrdrc  Ub.URS:XaURXXH5n U SnU SnSnUR"SUUUUUU U S.U D6nUSn[U [5(a [ U *S5OU nUR USS2USS245nSnSnSnSnUbUSS2SS2S4nUR"SUUURRS.U D6nUSS2SS2SS24S :HRS S 9)nUUS SURRS- 24n[RRUS SS9nURSS9nUUSUSS- SS24nUUnUR "SUUU SUS.U D6nUR"nUU-n[%UUUUUR&UR(UR*Ub UR,OSUb UR&OSUb UR(OSUbUR*S9 $SS9 $)a_ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be infered from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrr1rc)r/rrr(r1rrr)r'rcr"rrrPr.r)rTr)r/r0r return_dictrc) r&r0r+r'r(r)r*r,r-r.r/r1)rrrrprrgrrhrfr"allr rErrrrr&r$r(r)r*r')rJr/rrrrr(r1rcrrrdrrbackbone_outputsbackbone_hidden_statesrjbackbone_logitsr&r0r+depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r=rZ#CsmForConditionalGeneration.forwardsf  Y^^q%8 CC)=M*/:M"8,FI..  )%+')    "2!!48B>SV8W8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO ..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $($6$6%1+F# + %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r<)rrrrrr"rr) NNNNNNNNNNr)!r2r3r4r5rsrDrrr classmethodrrrr7rrrr r8rmrrrrvrrrrr:r$rZr;rarbs@r=rrs @1 0141-1/37;)- PBELL)PBu||,PB'u||4 PB & PB %,,  PBj,0595959 ##"%!!1!12    1 12  !!1!12 >'+/3157;37KO59-1$(5934[ ##[ u||,[ !. [ 'u||4 [ u//0 [ "%tE4E4E/F(F"GH[   1 12[ ))*[ D>[ !!1!12[ c5<</0[ +,[  u'' ([ [ r<r)rrrrWrrK)r)I dataclassesrtypingrrrr7torch.nnrEtransformers.utils.genericr activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrrautorconfiguration_csmrr generation_csmr! get_loggerr2r;r$Moduler@rdrrrrrrrrrrrrr rWrxrr__all__r1r<r=rs,",, 9!.)7/9OKF&__?.   H %  )6 )6  )6XY'JJ(J(<