tjldZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z m Z dd lmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZddlmZmZmZm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/m0Z0ddl1m2Z2e!j3e4Z5eGddeZ6Gddej7Z8Gdde,Z9Gdde(Z:Gdd e)Z;Gd!d"e'Z<Gd#d$ej7Z=Gd%d&eZ>Gd'd(e6Z?Gd)d*eZ@Gd+d,e6ZAed-.Gd/d0e6ZBed1.Gd2d3e6e2ZCgd4ZDdS)5zPyTorch Dia model.)CallableN)nn)initialization) DynamicCacheEncoderDecoderCache)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging)merge_with_config_defaults)capture_outputs)LlamaAttention LlamaRMSNormLlamaRotaryEmbeddingeager_attention_forward)Phi3MLP) DiaConfigDiaDecoderConfigDiaEncoderConfig)DiaGenerationMixincNeZdZUeed<dZdZdZdZdZ dZ dZ ddgZ fdZ xZS)DiaPreTrainedModelconfigmodelT input_idsDiaEncoderLayerDiaDecoderLayerct|t|trSt j|jjtj|jj z}tj |j |dSdS)Ndtype) super _init_weights isinstanceDiaMultiChannelEmbeddingtorcharanger( num_channelslong vocab_sizeinitcopy_offsets)selfmoduler; __class__s a/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/dia/modular_dia.pyr1z DiaPreTrainedModel._init_weights?sy f%%% f6 7 7 0l4;#;5:NNNQUQ\QggG Jv~w / / / / / 0 0)__name__ __module__ __qualname__r"__annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphmain_input_name_no_split_modulesr1 __classcell__r>s@r?r'r'3sz &*#N!!O*,=>000000000r@r'cLeZdZdZdeffd ZdejdejfdZxZ S)r3aIn order to efficiently compute the audio embedding from the 9 different channels, we vectorize the embedding process by using a single embedding layer and an offset. Example: - num_embeds = 4 - vocab_size = 8 - num_channels = 3 We would have offsets = [0, 8, 16] If audio_codes = [0, 1, 2, 3], [1, 3, 4, 7], [5, 6, 7, 8], then tokens = audio_codes + offsets = [0, 1, 2, 3, 9, 11, 12, 15, 21, 22, 23, 24] This allows us to use a single embedding layer for all channels. r(cZttj|j|jz|j|_|j|_|j|_tj |jtj |jz}| d|ddS)Nr.r;F) persistent) r0__init__r Embeddingr8r6 hidden_sizeembedr4r5r7register_buffer)r<r(r;r>s r?rRz!DiaMultiChannelEmbedding.__init__Ts \&"3f6I"I6K]^^ !-"/,v2%*EEEHYY YEBBBBBr@ audio_codesreturncX||j|jzd|jd|jdz}|||jd|jdd|j}|dS)Nr!rr)dim)r;todeviceviewshaperUrTsum)r<rWtokensembedss r?forwardz DiaMultiChannelEmbedding.forward\s  0B C CCII  !!${'8';;  F##((a+:KA:NPRTXTdeezzaz   r@) rArBrC__doc__r#rRr4TensorrcrMrNs@r?r3r3Fs|  C/CCCCCC!5<!EL!!!!!!!!r@r3ceZdZdS)DiaMLPNrArBrCr@r?rgrgdDr@rgceZdZdS) DiaRMSNormNrhrir@r?rlrlhrjr@rlceZdZdS)DiaRotaryEmbeddingNrhrir@r?rnrnlrjr@rnc.eZdZdZddeezdedefdZdS) DiaSelfAttention=Multi-headed attention from 'Attention Is All You Need' paperFr( layer_idx is_causalctj|||_||_|j|_|jj|_|jjp|j|_|j|jz|_ t|d|j|jz|_ d|_ d|_ ||_tj|j|j|j zd|_tj|j|j|j zd|_tj|j|j|j zd|_tj|j|j z|jd|_dS)Nhead_dimr!Fbias)rModulerRr(rrrTnum_attention_heads num_headsnum_key_value_headsnum_key_value_groupsgetattrruscalingattention_dropoutrsLinearq_projk_projv_projo_proj)r<r(rrrss r?rRzDiaSelfAttention.__init__ss0 4    "!-8#';#B#Tdn $(Nd6N$N! F4F$.4XYY  !$"i 0$.4=2PW\]]] i 0$2JT]2Zafggg i 0$2JT]2Zafggg i >@PW\]]] r@N)F) rArBrCrdr$r#intboolrRrir@r?rprppsZGG^^/2BB^s^_c^^^^^^r@rpceZdZdZdedeffd Z d dejdejdejdzd e dzd e e d e ejejdzff d Z xZS)DiaCrossAttentionrqr(rrct||_||_|j|_|j|_|jj|_|jj|_ |j|j z|_ |j |_ d|_ d|_d|_t!j|j|j|j zd|_t!j|j|j |j zd|_t!j|j|j |j zd|_t!j|j|j z|jd|_dS)Nr!rvFrw)r0rRr(rrrTcross_hidden_sizecross_num_attention_headsr{cross_num_key_value_headsr|r}cross_head_dimrurrrsrrrrrrr<r(rrr>s r?rRzDiaCrossAttention.__init__s&  "!-!'!9>#';#H $(Nd6N$N!-  !$i 0$.4=2PW\]]] i 68PSWS`8`glmmm i 68PSWS`8`glmmm i >@PW\]]] r@N hidden_statescross_attention_statesattention_maskpast_key_valueskwargsrXc |jdd}g|d|jR}g|jddd|jR}|||dd} ||j|jnd} |;| r9|jj |jj } |jj |jj } n| ||dd} | ||dd} |3|j| | |j\} } d|j|j<tj|jjt&} | || | | |fd|ji|\}}|g|dR}||}||fS)NrZr!rFTr)r_rurr^ transpose is_updatedgetrrcross_attention_cachelayerskeysvaluesrrupdater get_interfacer(_attn_implementationrrreshape contiguousr)r<rrrrr input_shape hidden_shape cross_shape query_statesr key_states value_statesattention_interface attn_output attn_weightss r?rczDiaCrossAttention.forwards.$)#2#. 88b8$-88 M.4SbS9M2Mt}MM {{=1166|DDNNqRSTT GVGb_/33DNCCChm  &: &(>EdnUZJ*@GW^LL%;<<AA+NNXXYZ\]^^J;;'=>>CCKPPZZ[\^_``L*+:+P+W+W N,,( L >B*4>:(?(M K ,.E) ) %8$7      % % L % % % ! \"))*>r@Nrposition_embeddingsrrrXc |}||}|j|f||d|\}}||z}|}||}||} || z}|S)N)rr)rrrr) r<rrrrresidual normed_statesself_attn_output_mlp_outs r?rczDiaEncoderLayer.forwards!((77 1d1  3)     ! !#33  ))-88 ((=)) 7* r@r) rArBrCr$rrRr4rerrr rcrMrNs@r?r+r+s"/"C""""""IM.2 |#5<#=>E t+  -.  u|U\D00 1 r@r+c eZdZeedZdeffd Zee e d de j de j dzde edefd ZxZS) DiaEncoder)r attentionsr(ct|_tjjj|_tjfdtj D|_ tjj |_t|_|dS)Nc0g|]}t|Sri)r+.0rrr(s r? z'DiaEncoder.__init__..# a a aI_VY / / a a ar@rr()r0rRr(rrSr8rT embedding ModuleListrangenum_hidden_layersrrlrnormrn rotary_emb post_initr<r(r>s `r?rRzDiaEncoder.__init__s     f&79KLLm a a a avG_A`A` a a a   v1vGGG ,F;;; r@Nr*rrrXc `||}tj|jd|jdddf}t |j||}|||}|jD]}||f|||d|}| |}t|S)NrZr])r( inputs_embedsr position_ids)rrr)last_hidden_state) rr4r5r_r]r r(rrrr )r<r*rrrrr encoder_layers r?rczDiaEncoder.forwardsy11 |IOB$7 @PQQQRVXYXYXYRYZ 2;')    #oom,oWW![  M)M-)$7   MM -00 ????r@N)rArBrCr+rp_can_record_outputsr$rRrrrr4rerrr rcrMrNs@r?rrs(& /       /3@@<@ t+@+, @  @@@^_ @@@@@r@rceZdZdedeffd Z d dejdeejejfdzdejdzdejdzd ejdzd e dzd eejejdzejdzffd Z xZ S)r,r(rrct|j|_t ||d|_t |||_t|j|j |_ t|j|j |_ t|j|j |_ t||_dS)NTrr)r0rRrT embed_dimrprrcross_attentionrlrr pre_ca_norm pre_mlp_normrgrrs r?rRzDiaDecoderLayer.__init__*s +.vyDQQQ0CC%f&8foNNN%f&8foNNN&v'9vOOO&>>r@Nrrrencoder_hidden_statesencoder_attention_maskrrXc f|}t|tr|j}|} ||} |j| |||fi|\} } | | z}|} ||} |j| |f||d|\} } | | z}|} ||} || }| |z}|S)N)rr) r2rself_attention_cacherrrrrr)r<rrrrrrrself_attn_cacherrrr cross_statesrs r?rczDiaDecoderLayer.forward4s* o': ; ; C-BO ((77 1d1         !!#33  ((77 .$.  ! 2+       a!</  ))-88 ((=)) 7* r@NNNNN) rArBrCr#rrRr4rerrrcrMrNs@r?r,r,)s"/"C""""""IM.2596:6:++|+#5<#=>E+ t+ + %|d2 + !& t 3 +-t3+ u|U\D0%,2EE F++++++++r@r,ceZdZdZeeedZdeffd Z e e e dde jde jdzde jdzd e jdzd e jdzd edzd eed eezfdZxZS) DiaDecoderz-Transformer Decoder Stack using DenseGeneral.)rrcross_attentionsr(ctj|_j|_t |_t jfdtj D|_ tj j |_t|_|dS)Nc0g|]}t|Sri)r,rs r?rz'DiaDecoder.__init__..qrr@rr)r0rRr6r8r3 embeddingsrrrrrrlrTrrrnrrrs `r?rRzDiaDecoder.__init__ks    "/ +26::m a a a avG_A`A` a a a   v1vGGG ,F;;; r@Nr*rrrrrrrXc t|dd\}} ||nd} |3tj| |j| z}|d}||} |/ts!| | z} tj|| |j}t|j | ||}t|j | ||}| | |} |j D]}|| | ||f|||d|} || } t| | S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks)`): The original `decoder_input_ids` in 3D shape to facilitate more efficient computations. [What are input IDs?](../glossary#input-ids) NrZrr)r(rrr)r(rrrr)rrr)rr)sizeget_seq_lengthr4r5r] unsqueezerronesr r(r rrrr)r<r*rrrrrr batch_size seq_lengthpast_key_values_lengthrmask_seq_lengthrlayers r?rczDiaDecoder.forwardxs("+!1!1#2#!6 JETE`!?!?!A!A!Afg   < 9;KLLLOeeL'11!44L 22  !*B*D*D !4zAO"Z OIL\]]]N+;')+     ";;'1"7 " " "  #oom,oWW[  E!E$% (> /)    MM -00 8++    r@r)rArBrCrdr,rprrr#rRrrrr4re LongTensor FloatTensorrrrrrrcrMrNs@r?rrbs>77)&- /       15.2:>:>6:A A <A &-A  t+ A %047 A !& 04 7 A -t3A +,A  3U :A A A ^_ A A A A A r@rz[ The bare Dia model outputting raw hidden-states without any specific head on top. ) custom_introceZdZdeffd Zee ddejdzdejdzdejdzdejdzdejdzd e e zdzd e dzd e dzd e e zfd ZxZS)DiaModelr(ct|||_t|j|_t |j|_| dSr) r0rRr(rencoder_configencoderrdecoder_configdecoderrrs r?rRzDiaModel.__init__s\     !&"788 !&"788  r@Nr*rdecoder_input_idsdecoder_position_idsdecoder_attention_maskencoder_outputsr use_cacherXc ||td|jr%|jr|rtdd}|r8|6t t |jt |j}||jd||d| }nct|tsNt|dt|dkr|dndt|d kr|d nd }|dj dd |jj j} } } |.tj| d| f|jj j|j }|jd kr+|| | | dd }|jd||||d|||d | } t/| j| j| j| j| j|d|j|jS)a\ decoder_input_ids (`torch.LongTensor` of shape `(batch_size * num_codebooks, target_sequence_length) or (batch_size, target_sequence_length, num_codebooks)`, *optional*): 1. (batch_size * num_codebooks, target_sequence_length): corresponds to the general use case where the audio input codebooks are flattened into the batch dimension. This also aligns with the flat- tened audio logits which are used to calculate the loss. 2. (batch_size, sequence_length, num_codebooks): corresponds to the internally used shape of Dia to calculate embeddings and subsequent steps more efficiently. If no `decoder_input_ids` are provided, it will create a tensor of `bos_token_id` with shape `(batch_size, 1, num_codebooks)`. Indices can be obtained using the [`DiaProcessor`]. See [`DiaProcessor.__call__`] for more details. [What are decoder input IDs?](../glossary#decoder-input-ids) decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings`. [What are position IDs?](../glossary#position-ids) NzXYou should either provide text ids or the cached text encodings. Neither has been found.zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr)r*rrr!r)rrrrZ)r fill_valuer])r*rrrrrr)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentionsri) ValueErroris_gradient_checkpointingtraininglogger warning_oncerrr(rr2r lenr_rr6r4full bos_token_idr]ndimrrrrrrrrr)r<r*rrrrrrrrbszseq_lenchannelsdecoder_outputss r?rczDiaModel.forwardsdH  !8j   ) "dm " "##p"  v01,dk2R2R2RT`hlhsTtTtTtuuO  "*dl#-OO O_== -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO#2!"4":1"=r4;C]CjhW  $ % 1h'DK4N4[dhdo!!!   !Q & & 1 9 9#x Q Q [ [\]_` a a &$,  '-1"1!"4#1+      "-?+;"1"?.9,=&5a&8"1"?.9    r@)NNNNNNNN)rArBrCr"rRrrr4rr rrrrrcrMrNs@r?rrs9 y.226598<:>:>6:!%] ] #d*] (4/] !+d2 ] $.5 ] !& 04 7 ] )5047] -t3] $;]  # #] ] ] ^] ] ] ] ] r@rzl The Dia model consisting of a (byte) text encoder and audio decoder with a prediction head on top. ceZdZdZdZdeffd Zee dde j dzde j dzde j dzd e j dzd e j dzd e e zdzd e dzd edzde j dzde ezfdZxZS)DiaForConditionalGenerationr))audior(c`t|||_t||_|jj|_|jj|_tj |jj |j|jzd|_ d|_ | dS)NFrw ForMaskedLM)r0rRr(rr)rr6r8rrrT logits_dense loss_typerrs r?rRz$DiaForConditionalGeneration.__init__7s     f%% "1> /:I  ! -0ADO0S[`   ' r@Nr*rrrrrrrlabelsrXc |jd ||||||||d| } | d} | jd} || | d|j|jfdd| |jzd|j}d}| |jd || |jd| }t||| j | j | j | j | j| j| j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size * num_codebooks, target_sequence_length) or (batch_size, target_sequence_length, num_codebooks)`, *optional*): 1. (batch_size * num_codebooks, target_sequence_length): corresponds to the general use case where the audio input codebooks are flattened into the batch dimension. This also aligns with the flat- tened audio logits which are used to calculate the loss. 2. (batch_size, sequence_length, num_codebooks): corresponds to the internally used shape of Dia to calculate embeddings and subsequent steps more efficiently. If no `decoder_input_ids` are provided, it will create a tensor of `bos_token_id` with shape `(batch_size, 1, num_codebooks)`. Indices can be obtained using the [`DiaProcessor`]. See [`DiaProcessor.__call__`] for more details. [What are decoder input IDs?](../glossary#decoder-input-ids) decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings`. [What are position IDs?](../glossary#position-ids) labels (`torch.LongTensor` of shape `(batch_size * num_codebooks,)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.decoder_config.vocab_size - 1]` or -100. Tokens with indices set to `-100` are ignored (masked). )r*rrrrrrrrrZr!rN)logitsrr8) lossr!rrrrr rr ri)r)r_rr^r6r8rr loss_functionrrrrrr rr )r<r*rrrrrrrrroutputsrr audio_logitsr"s r?rcz#DiaForConditionalGeneration.forwardFs<R$*  )/!5#9++      $AJ&,Q/    / 0 0 T:r4#4doF G G Yq!__ Z\\ T*t00"do F F   %4%o\&UYUdoohnooD#3")"?&9$5&-&G")"?&9    r@) NNNNNNNNN)rArBrCrEoutput_modalitiesr"rRrrr4rr rrrrrcrMrNs@r?rr.s\  " y      .226598<:>:>6:!%*.L L #d*L (4/L !+d2 L $.5 L !& 04 7 L )5047L -t3L $;L  4'L   L L L ^L L L L L r@r)rr'r)Erdcollections.abcrr4rrr9 cache_utilsrr masking_utilsr r modeling_flash_attention_utilsr modeling_layersr modeling_outputsr rrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrutils.output_capturingrllama.modeling_llamarrrrphi3.modeling_phi3r configuration_diar"r#r$generation_diar% get_loggerrArr'ryr3rgrlrnrprr+rr,rrr__all__rir@r?r9s$$$$$$ &&&&&&<<<<<<<<JJJJJJJJBBBBBB999999 GFFFFFFF&&&&&&llllllllllllll777777555555 )(((((LLLLLLLLLL......  H % %00000000$!!!!!ry!!!<     W                -   ^^^^^~^^^,G)G)G)G)G) G)G)G)T0B5@5@5@5@5@#5@5@5@p666660666rZ Z Z Z Z #Z Z Z z g g g g g !g g  g T a a a a a "46Ha a  a H L K Kr@