\655rI\&"S?S<9"S@SA\6\055rJ/SBQrKg)E)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCacheEncoderDecoderCache)use_kernel_forward_from_hub)create_causal_mask)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availableis_torchdynamo_compilinglogging) DiaConfigDiaDecoderConfigDiaEncoderConfig)DiaGenerationMixin)make_flex_block_causal_maskcD\rSrSr%\\S'SrSrSrSr Sr Sr Sr SS/r Srg ) DiaPreTrainedModel>configmodelT input_idsDiaEncoderLayerDiaDecoderLayerN)__name__ __module__ __qualname____firstlineno__r"__annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphmain_input_name_no_split_modules__static_attributes__r/\/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/dia/modeling_dia.pyr(r(>s< &*#N!!O*,=>r>r(cn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) DiaMultiChannelEmbeddingKaIn order to efficiently compute the audio embedding from the 9 different channels, we vectorize the embedding process by using a single embedding layer and an offset. Example: - num_embeds = 4 - vocab_size = 8 - num_channels = 3 We would have offsets = [0, 8, 16] If audio_codes = [0, 1, 2, 3], [1, 3, 4, 7], [5, 6, 7, 8], then tokens = audio_codes + offsets = [0, 1, 2, 3, 9, 11, 12, 15, 21, 22, 23, 24] This allows us to use a single embedding layer for all channels. r*cv>[TU]5 [R"URUR -UR 5UlUR UlUR Ul[R"UR [RS9UR-nURSUSS9 g)NdtypeoffsetsF persistent) super__init__r Embedding vocab_size num_channels hidden_sizeembedtorcharangelongregister_buffer)selfr*rF __class__s r?rJ!DiaMultiChannelEmbedding.__init__Ys \\&"3"3f6I6I"I6K]K]^ !--"//,,v22%**EHYHYY YEBr> audio_codesreturncXRRUR5-RS5nUR U5R UR SUR SSUR5nURSS9$)Nr!rdim) rFtodevicesqueezerOviewshaperNsum)rTrWtokensembedss r?forward DiaMultiChannelEmbedding.forwardass  0B0B CCLLQOF#((a+:K:KA:NPRTXTdTdezzaz  r>)rOrNrM) r0r1r2r3__doc__r#rJrPTensorrfr= __classcell__rUs@r?rArAKs7 C/C!5<<!ELL!!r>rAcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)DiaMLPgc >[TU]5 Xl[R"UR SUR -SS9Ul[R"UR UR SS9Ul[URUl g)Nr[Fbias) rIrJr*rLinearrNintermediate_size gate_up_proj down_projr hidden_act activation_fnrTr*rUs r?rJDiaMLP.__init__hsn  IIf&8&8!f>V>V:V]bc6#;#;V=O=OV[\#F$5$56r> hidden_statesrXcURU5nURSSS9up2X RU5-nURU5$)Nr[rZr\)rtchunkrwru)rTrz up_statesgates r?rfDiaMLP.forwardpsH%%m4 #//!/4 2 24 88 ~~i((r>)rwr*rurt) r0r1r2r3rJrP FloatTensorrfr=rjrks@r?rmrmgs,7)U%6%6)5;L;L))r>rmRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$) DiaRMSNormyc>[TU]5 [R"[R "U55UlX lg)z) DiaRMSNorm is equivalent to T5LayerNorm N)rIrJr ParameterrPonesweightvariance_epsilon)rTrNepsrUs r?rJDiaRMSNorm.__init__{s/ ll5::k#:; #r>cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nr[rZT)keepdim) rEr^rPfloat32powmeanrsqrtrr)rTrz input_dtypevariances r?rfDiaRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r>c^[URR5SUR3$)Nz, eps=)tuplerrbrrTs r? extra_reprDiaRMSNorm.extra_reprs* ))*+6$2G2G1HIIr>)rr)gư>) r0r1r2r3rJrfrr=rjrks@r?rrys$;JJr>rcl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)DiaRotaryEmbeddingr*c>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultinv_freqFrG)rIrJhasattr isinstancerdictgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr*r rope_init_fnattention_scalingrSroriginal_inv_freq)rTr*r_rrUs r?rJDiaRotaryEmbedding.__init__s  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+cbURSSS2S4R5RURSSS5R UR 5nUSS2SSS24R5n[ UR R[5(a0UR RS:waUR ROSn[R"USS9 UR5UR5-RSS5n[R"Xf4SS 9nUR5UR-nUR5UR-n SSS5 WR UR S 9W R UR S 94$!,(df  N@=f) NrrZr!mpscpuF) device_typeenabledr[r\rD)rfloatexpandrbr^r_rrstrrPautocast transposecatcosrsinrE) rTx position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r?rfDiaRotaryEmbedding.forwardsR!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rr*rrrrrN) r0r1r2r3r"rJrPno_gradrrfr=rjrks@r?rrs6/y//" ]]_ < rcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..NrZr[r\)rbrPr)rx1x2s r? rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r>cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrrr unsqueeze_dimq_embedk_embeds r?apply_rotary_pos_embrsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r>rzn_reprXcURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rbrreshape)rzrbatchnum_key_value_headsslenhead_dims r? repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr>modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nr[rrZ)r]rE)ptrainingr!)rnum_key_value_groupsrPmatmulrrbr functionalsoftmaxrr^rErr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r?eager_attention_forwardrs3 ; ;c0^\rSrSrSrSS\\\4S\S\ 4U4Sjjjr SS\ RS\ \ R\ R4S \\ RS \\S \\ R S \\S \ \ R\ R44SjjrSrU=r$)DiaSelfAttention=Multi-headed attention from 'Attention Is All You Need' paperr* layer_idx is_causalc>[TU]5 XlX lURUlURR UlURR=(d UR UlUR UR-Ul[USURUR -5Ul SUl SUl X0l [R"URUR UR-SS9Ul[R"URURUR-SS9Ul[R"URURUR-SS9Ul[R"UR UR-URSS9Ulg)Nrr!Frp)rIrJr*rrNnum_attention_heads num_headsrrgetattrrrattention_dropoutrrrrq_projk_projv_projo_proj)rTr*rrrUs r?rJDiaSelfAttention.__init__s@  "!--88#';;#B#B#Tdnn $(NNd6N6N$N! F4F4F$..4XY  !$"ii 0 0$..4==2PW\] ii 0 0$2J2JT]]2Zafg ii 0 0$2J2JT]]2Zafg ii >@P@PW\] r>rzposition_embeddingsrpast_key_valuecache_positionrrXc 4URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO URUR S.UD6unnUR""/UQSP76R%5nUR'U5nUU4$)NrZr!r[)rrreagerr)rr)rbrrrarrrrupdaterrr*_attn_implementationrrrrrrr)rTrzrrrrr input_shape hidden_shape query_statesrrrr cache_kwargsattention_interfacerrs r?rfDiaSelfAttention.forward s$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<')rr*rrNrrrrrrrrrr)FNN)r0r1r2r3rhrr$r#intboolrJrPrirrr LongTensorrrrfr=rjrks@r?rrsG^u%57G%GH^UX^ei^^.+/59 ))||))#5<<#=>))!. )) ! )) !!1!12 ))+,)) u||U\\) *))))r>rc^\rSrSrSrS\S\4U4SjjrSS\RS\RS\ \RS \ \ S \ \ S \\R\ \R44 S jjrS rU=r$)DiaCrossAttentioni9rr*rcR>[TU]5 XlX lURUlUR UlURR UlURRUl URUR-Ul URUl SUl SUlSUl[ R""URURUR-SS9Ul[ R""UR URUR-SS9Ul[ R""UR URUR-SS9Ul[ R""URUR-URSS9Ulg)Nr!rFrp)rIrJr*rrNcross_hidden_sizecross_num_attention_headsrcross_num_key_value_headsrrcross_head_dimrrrrrrrrrrrrTr*rrUs r?rJDiaCrossAttention.__init__<s;  "!--!'!9!9>>#';;#H#H $(NNd6N6N$N!--  !$ii 0 0$..4==2PW\] ii 6 68P8PSWS`S`8`glm ii 6 68P8PSWS`S`8`glm ii >@P@PW\] r>rzcross_attention_statesrpast_key_valuesrrXc nURSSn/UQSPURP7n/URSSQSPURP7nURU5RU5R SS5n Ub%UR R UR5OSn UbbU (a[URRURRn URRURRn OURU5RU5R SS5n URU5RU5R SS5n UbBURRU U UR5upSUR UR'[n UR R"S:wa[$UR R"n U "UU U U U4SUR&0UD6upUR)/UQSP75R+5nUR-U5nX4$)NrZr!r[FTrr)rbrrrar is_updatedrrcross_attention_cachelayerskeysvaluesrrrrr*rrrrrr)rTrzrrrrrr cross_shaper rrrr rrs r?rfDiaCrossAttention.forwardOs$))#2. 88b8$--8 M.44Sb9M2Mt}}M {{=166|DNNqRST GVGb_//33DNNChm  &:(>>EEdnnUZZJ*@@GGW^^L%;<AA+NXXYZ\]^J;;'=>CCKPZZ[\^_`L*+:+P+P+W+W NN,( >B**4>>:(? ;; + +w 6"9$++:Z:Z"[ $7      % LL % % ! "))*)rr*rrrNrrrrrrrrrrr )r0r1r2r3rhr#rrJrPrirr rrrrfr=rjrks@r?rr9sG^/^C^.269= 1)||1)!& 1)!. 1) ""56 1) -. 1) u||Xell33 41)1)r>rc^\rSrSrS\S\4U4SjjrS S\RS\ \ \R\R4S\ \RS\ \ S \ \R\ \R44 S jjr S rU=r$) r-ir*rc>[TU]5 [URURS9Ul[ XSS9Ul[URURS9Ul[U5Ul g)NrFr) rIrJrrNnorm_eps pre_sa_normrself_attention post_sa_normrmmlprs r?rJDiaEncoderLayer.__init__sZ %f&8&8fooN.vER&v'9'9vO&>r>rzrrrrXc UnURU5nUR"U4UUS.UD6upxXW-nUnURU5nURU5n XY-nX4$)Nrr)r)r*r+r,) rTrzrrrresidual normed_statesself_attn_outputself_attn_weightsmlp_outs r?rfDiaEncoderLayer.forwards!((7 .2.A.A / 3)/  / + !3  ))-8 ((=) * //r>)r,r+r)r*r )r0r1r2r3r$rrJrPrirrrrrfr=rjrks@r?r-r-s"/"C"LP15 0||0&eELL%,,,F&GH0!. 0 -. 0 u||Xell33 4 00r>r-c^\rSrSrS\4U4Sjjr\\SS\RS\ \RS\ \ S\ \ S \ \ S \\\44 S jj55rS\\RS4S \R4S jrSrU=r$) DiaEncoderir*c >[TU]U5 Xl[R"UR UR 5Ul[R"[UR5Vs/sHn[X5PM sn5Ul [UR URS9Ul[!U5Ulgs snfNr&)rIrJr*rrKrLrN embedding ModuleListrangenum_hidden_layersr-rrr(normrrotary_embeddingsrs r?rJDiaEncoder.__init__s   f&7&79K9KLmmAFvG_G_A` aA`I_V /A` a  v11vG !3F!; bs.CNr,routput_attentionsoutput_hidden_statesrrXc URU5n[R"URSURS9SSS24nUR Xg5nUR UU5nU(aSOSn U(aSOSn URH1n U(aX4-n U "U4UUS.UD6n U SnU(dM)XS4-n M3 URU5nU(aX4- n [XiU S9$)NrZr_r/r/rr!last_hidden_staterz attentions) r:rPrQrbr_r?_update_full_maskrr>r) rTr,rrArBrrzrrencoder_statesall_attentions encoder_layer layer_outputss r?rfDiaEncoder.forwards y1 ||IOOB$7 @P@PQRVXYRYZ "44]Q//    40d![[M#!/2B!B)$7- M *!,M  !/3C2E!E) -0  . .N+Vd  r> inputs_embedscrUbURRS:XaSU;aUnU$SnU$URRS:Xa[XR5nU$URRS:Xa+[ U[ R 5(a [USS9nU$[XR5nU$)Nflash_attention_2rsdpaflex_attentionFr' r*rrrErrPrir&r)rTrrNs r?rHDiaEncoder._update_full_masks  %{{//3FF343FMQ11V;"E^UhUh!i115EEnell;;%@[`%aN ")r*r:rr>r?)NFF)r0r1r2r3r$rJrrrPrirrrrrrrrfrHr=rjrks@r?r7r7s </ <26,1/4 . <<. !.. $D> . 'tn . -. .  % &. . bellD01||r>r7c|^\rSrSrS\S\4U4SjjrSS\RS\ \ \R\R4S\ \RS\ \RS \ \RS \ \ S \ \RS \ \R\ \R\ \R44S jjr SrU=r$)r.ir*rct>[TU]5 URUl[ XSS9Ul[ X5Ul[URURS9Ul [URURS9Ul [URURS9Ul [U5Ulg)NTr'r&)rIrJrN embed_dimrr*rcross_attentionrr(r) pre_ca_norm pre_mlp_normrmr,rs r?rJDiaDecoderLayer.__init__s ++.vDQ0C%f&8&8fooN%f&8&8fooN&v'9'9vO&>r>rzrrencoder_hidden_statesencoder_attention_maskrrrXc dUn [U [5(a U Rn Un URU5n UR"U UUU 4SU0UD6upX-nUn UR U5n UR "U U4UUS.UD6upX-nUn URU5n URU 5nU U-nXU4$)Nr)rr) rr self_attention_cacher)r*rYrXrZr,)rTrzrrr\r]rrrself_attn_cacher0r1r2r3 cross_statescross_attn_weightsr4s r?rfDiaDecoderLayer.forwards* o': ; ;-BBO ((7 .2.A.A     / * /  / +!3  ((7 +/+?+?  !, 2+ ,  , ( !/  ))-8 ((=) 7* 1CCCr>)rXrWr,rYrZr)r*)NNNNNN)r0r1r2r3r#rrJrPrirrr rrfr=rjrks@r?r.r.s"/"C"LP158<9=9=59-D||-D&eELL%,,,F&GH-D!. -D ( 5 -D !) 6 -D""56-D!!1!12-D u||Xell3Xell5KK L-D-Dr>r.c^\rSrSrSrS\4U4Sjjr\\SS\ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \S \ \S \ \S\ \ RS\\\44Sjj55rS \\ RS4S \\ RS4S\ R(S\ R4SjrSrU=r$) DiaDecoderi5z-Transformer Decoder Stack using DenseGeneral.r*c >[TU]U5 URUlURUl[ U5Ul[ U5Ul[R"[UR5Vs/sHn[X5PM sn5Ul [URUR S9Ulgs snfr9)rIrJrMrLrA embeddingsrr?rr;r<r=r.rrrNr(r>rs r?rJDiaDecoder.__init__8s  "// ++26:!3F!;mmAFvG_G_A` aA`I_V /A` a  v11vG  bs:B>Nr,rrr\r]rrArBrrXc "UR5SSupUbUR5OSn U c"[R"XU -URS9n Uc U SSS24nUR U5nUR X5nUc3[5(d$X-n[R"U UURS9n[URUUU UUS9nURUUURSSU5nU(aSOSnU(aSOSnU(aUbSOSnURHEnU(aUU4- nU"UUUU4UUU S.U D6nUSnU(dM.UUS 4-nUcM<UUS4-nMG URU5nU(aUU4- n[UUUUUS 9$) z input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks)`): The original `decoder_input_ids` in 3D shape to facilitate more efficient computations. [What are input IDs?](../glossary#input-ids) NrZrrD)r* input_embedsrrrrr[r/)r]rrr!)rFrrzrGcross_attentions)sizeget_seq_lengthrPrQr_rgr?rrr r*_update_cross_attn_maskrbrr>r)rTr,rrr\r]rrArBrr batch_size seq_lengthpast_key_values_lengthrzrmask_seq_lengthall_hidden_statesall_self_attnsall_cross_attentionslayerrLs r?rfDiaDecoder.forwardCs,"+!1#2!6 ETE`!?!?!Afg  !"\\&(KT]TdTdN  )$'2L 2 "44]Q  !*B*D*D4AO"ZZ OIL\L\]N+;;&))+%  "&!=!= ! "    #  " #7BD0d&7 /-  M*!,M  !/=3C2E!E(4+?=QRCSBU+U()!, -0  -!1 1 8+++%1   r>rrNcUbUbURRS:XaSU;aUnU$SnU$URRS:Xa[UURUSS9nU$URRS:Xa/[ U[ R 5(a[UUSSS9nU$[X$RUSS9nU$) NrPrrQrZ)tgt_lenrRF) query_lengthrrS)rTr\r]rrNs r?rn"DiaDecoder._update_cross_attn_masks ! ,1G1S{{//3FFCDH^C^)?&.&%/ei&.&%-11V;*M*!'''O*&$&%115EE4ellCC-H.%0_"'.*&% *D*,?,?UW*&&%r>)rgrr>rMr?rL)NNNNNFFN)r0r1r2r3rhr#rJrrrPrirrrr rrrrrfSizernr=rjrks@r?rere5s`7 H/ H4815=A=A9=,1/459Z <<Z u//0Z !. Z ((9(9: Z !))9)9 : Z ""56Z $D>Z 'tnZ !!1!12Z  8%? @Z Z z!&$U\\4%78!&!&ellD&8 9!&ZZ !& || !&!&r>rez[ The bare Dia model outputting raw hidden-states without any specific head on top. ) custom_introcr^\rSrSrS\4U4SjjrSrSr\\ SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ \\4S \ \S \ \S\ \S\ \S\ \ RS\ \\44Sjj55rSrU=r$)DiaModelir*c>[TU]U5 Xl[UR5Ul[ UR5UlUR5 gr) rIrJr*r7encoder_configencoderredecoder_configdecoder post_initrxs r?rJDiaModel.__init__sC   !&"7"78 !&"7"78  r>cUR$r)rrs r? get_encoderDiaModel.get_encoder ||r>cUR$r)rrs r? get_decoderDiaModel.get_decoderrr>r,rdecoder_input_idsdecoder_position_idsdecoder_attention_maskencoder_outputsr use_cacherArBrrXc  UcUc [S5eU bU OURRn U bU OURRn UbUOURRnUR (a/UR (aU(a[RS5 SnU(aUc[[5[55nUcUR"SUUU U S.U D6nOK[U[5(d6[US[U5S:aUSOS[U5S:aUSOSS 9nUSRSS URR R"pn Uc7[$R&"U SU4URR(UR*S 9nUR,S:Xa"UR/XU5R1SS5nUR2"SUUUUSUUU U UU S . U D6n[5UR6UR8UR:UR<UR>USUR:UR<S 9$)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size * num_codebooks, target_sequence_length) or (batch_size, target_sequence_length, num_codebooks)`, *optional*): 1. (batch_size * num_codebooks, target_sequence_length): corresponds to the general use case where the audio input codebooks are flattened into the batch dimension. This also aligns with the flat- tened audio logits which are used to calculate the loss. 2. (batch_size, sequence_length, num_codebooks): corresponds to the internally used shape of Dia to calculate embeddings and subsequent steps more efficiently. If no `decoder_input_ids` are provided, it will create a tensor of `bos_token_id` with shape `(batch_size, 1, num_codebooks)`. Indices can be obtained using the [`DiaProcessor`]. See [`DiaProcessor.__call__`] for more details. [What are decoder input IDs?](../glossary#decoder-input-ids) decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings`. [What are position IDs?](../glossary#position-ids) NzXYou should either provide text ids or the cached text encodings. Neither has been found.zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r,rrArBrr!r[rErZ)rl fill_valuer_) r,rrr\r]rrArBrr)rFrdecoder_hidden_statesdecoder_attentionsrkencoder_last_hidden_stater\encoder_attentionsr/) ValueErrorr*rArBris_gradient_checkpointingrlogger warning_oncer r rrrlenrbrrMrPfull bos_token_idr_ndimrrrrrFrrzrGrk)rTr,rrrrrrrrArBrrbszseq_lenchannelsdecoder_outputss r?rfDiaModel.forwardsXN  !8j 2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK)r*rr) NNNNNNNNNNN)r0r1r2r3r"rJrrrrrrPrrrrr rrrfr=rjrks@r?rrsJ y15598<;?=ACG9=$(,0/359k E,,-k !!1!12k $E$4$45 k 'u'7'78 k !))9)9 : k "%(>"?@k ""56k D>k $D>k 'tnk !!1!12k  u(( )k k r>rzl The Dia model consisting of a (byte) text encoder and audio decoder with a prediction head on top. c^\rSrSrSrS\4U4SjjrSrSr\ \ SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \\\4S \ \S\ \S\ \S\ \S\ \ RS\ \ RS\\\44Sjj55rSrU=r$)DiaForConditionalGenerationiIr+r*cv>[TU]U5 Xl[U5UlUR R UlUR RUl[R"UR RUR UR-SS9Ul SUl UR5 g)NFrp ForMaskedLM)rIrJr*rr+rrMrLrrrrN logits_dense loss_typerrxs r?rJ$DiaForConditionalGeneration.__init__Qs   f% "11>> //::II  ! ! - -0A0ADOO0S[` ' r>c6URR5$r)r+rrs r?r'DiaForConditionalGeneration.get_encoder`zz%%''r>c6URR5$r)r+rrs r?r'DiaForConditionalGeneration.get_decodercrr>r,rrrrrrrrArBlabelsrrXc  XUR"S UUUUUUUUU U U S. U D6nUSnURSnURU5RUSURUR 45R SS5R5RUUR-SUR 5nSnU bUR"S UXR S.U D6n[UUURURURURURURUR S9 $) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size * num_codebooks, target_sequence_length) or (batch_size, target_sequence_length, num_codebooks)`, *optional*): 1. (batch_size * num_codebooks, target_sequence_length): corresponds to the general use case where the audio input codebooks are flattened into the batch dimension. This also aligns with the flat- tened audio logits which are used to calculate the loss. 2. (batch_size, sequence_length, num_codebooks): corresponds to the internally used shape of Dia to calculate embeddings and subsequent steps more efficiently. If no `decoder_input_ids` are provided, it will create a tensor of `bos_token_id` with shape `(batch_size, 1, num_codebooks)`. Indices can be obtained using the [`DiaProcessor`]. See [`DiaProcessor.__call__`] for more details. [What are decoder input IDs?](../glossary#decoder-input-ids) decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings`. [What are position IDs?](../glossary#position-ids) labels (`torch.LongTensor` of shape `(batch_size * num_codebooks,)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.decoder_config.vocab_size - 1]` or -100. Tokens with indices set to `-100` are ignored (masked). ) r,rrrrrrrrArBrrrZr!r[N)logitsrrL) lossrrrrrkrr\rr/)r+rbrrarMrLrr loss_functionrrrrrkrr\r)rTr,rrrrrrrrArBrrroutputsrFro audio_logitsrs r?rf#DiaForConditionalGeneration.forwardfsAX**  )/!5#9++/!5)    $AJ&,,Q/    / 0 T:r4#4#4dooF G Yq!_ Z\ T*t000"doo F   %%o\&UdUdohnoD#33")"?"?&99$55&-&G&G")"?"?&99  r>)r*rrr+rMrL) NNNNNNNNNNNN)r0r1r2r3r5r"rJrrrrrrPrrrrr rrrfr=rjrks@r?rrIsj   y ((15598<;?=ACG9=$(,0/3-159R E,,-R !!1!12R $E$4$45 R 'u'7'78 R !))9)9 : R "%(>"?@R ""56R D>R $D>R 'tnR ))*R !!1!12R  uo% &R R r>r)rr(r)Nr!)r)LtypingrrrrPr activationsr cache_utilsr r r integrationsr masking_utilsr modeling_attn_mask_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrrr configuration_diar"r#r$generation_diar%integrations.flex_attentionr& get_loggerr0rr(ModulerArmrrrrrirrrrrrr-r7r.rerr__all__r/r>r?rsP,-, !CC7/gB9 LF&ML. !!J   H % ? ? ?!ryy!8)RYY)$Y'JJ(J(<)ryy>)BG) G)T000BS#Sl8D08DvN&#N&b { !{  { | l "46Hl  l ^ Lr>