Phh<dZddlZddlmZddlmZmZmZddlZddlm Z ddl m Z ddl m Z dd lmZmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZddlmZddlm Z m!Z!ddl"m#Z#m$Z$Gdde jJZ&Gdde jJZ'GddeZ(Gdde Z)Gdde jJZ*Gdde Z+eGdd eZ,ed!"Gd#d$e,Z-eGd%d&eZ.ed'"Gd(d)e,Z/gd*Z0y)+zPyTorch Parakeet model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputCausalLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple)check_model_inputs)%FastSpeech2ConformerConvolutionModule)LlamaAttentioneager_attention_forward)ParakeetCTCConfigParakeetEncoderConfigceZdZUdZej ed<ddeffd Zejdej fdZ xZ S)$ParakeetEncoderRelPositionalEncodingz*Relative positional encoding for Parakeet.inv_freqconfigc6t||j|_d}d|tjd|j dtj j|tj|j z zz }|jd|d y) Ng@?rrdtype)devicer#rF) persistent) super__init__max_position_embeddingstorcharange hidden_sizeint64tofloatregister_buffer)selfrr$baser __class__s g/var/www/html/saasai/venv/lib/python3.12/site-packages/transformers/models/parakeet/modular_parakeet.pyr'z-ParakeetEncoderRelPositionalEncoding.__init__)s '-'E'E$  Q 2 2AU[[ILLTZbgbmbmLn$$%   ZeD hidden_statesc|jd}||jkDrtd|d|jdtj|dz | d|j }|j ddddfjj|jdddj|j }|ddddfj}t|j jtr/|j jdk7r|j jnd }tj|d 5|j|jzjdd }|j}|j!} tj"|| gd } | j$g| jddd} ddd j|j&S#1swY%xYw)NrzSequence Length: z= has to be less or equal than config.max_position_embeddings .r$rmpscpuF) device_typeenabledrdimr")shaper( ValueErrorr)r*r$rr.expandr- isinstancetypestrautocast transposesincosstackreshaper#) r0r5 seq_length position_idsinv_freq_expandedposition_ids_expandedr<freqsrIrJ pos_embeds r3forwardz,ParakeetEncoderRelPositionalEncoding.forward7s"((+ 44 4#J<02262N2N1OqR  ||JNZKML`L`a MM$4- ( . . 0 7 7 8K8KA8NPRTU V Y YZgZnZn o !-T4] ; A A C-..33S9m>R>R>W>W[`>`  % %  ^^ U C&,,.1F1L1L1NNYYZ[]^_E))+C))+C S#JB7I) ))D9??3B+?DDI D||-"5"5|66D Cs ?B G//G8N) __name__ __module__ __qualname____doc__r)Tensor__annotations__rr'no_gradrS __classcell__r2s@r3rr$sF4ll E4 EU]]_7U\\77r4rc*eZdZdeffd ZdZxZS)ParakeetEncoderFeedForwardrc`t|tj|j|j |j |_t|j|_ tj|j |j|j |_ |j|_ y)Nbias) r&r'rLinearr+intermediate_sizeattention_biaslinear1r hidden_act activationlinear2activation_dropoutr0rr2s r3r'z#ParakeetEncoderFeedForward.__init__Ws| yy!3!3V5M5MTZTiTij  !2!23yy!9!96;M;MTZTiTij "(";";r4c|j|j|}tjj ||j |j }|j|}|S)Nptraining)rhrfr functionaldropoutrjrori)r0r5s r3rSz"ParakeetEncoderFeedForward.forward^sU ](CD  --mt?V?Vaeanan-o  ]3 r4)rUrVrWrr'rSr\r]s@r3r_r_Vs<4<r4r_c&eZdZddeffd ZxZS) ParakeetEncoderConvolutionModulerc&t|||yrT)r&r')r0r module_configr2s r3r'z)ParakeetEncoderConvolutionModule.__init__fs /r4rT)rUrVrWrr'r\r]s@r3rsrses0400r4rsc eZdZdZdedeffd Z d dejde ejde ejde e d e ejejff d Z d ZxZS) ParakeetEncoderAttentionztMulti-head attention with relative positional encoding. See section 3.3 of https://huggingface.co/papers/1901.02860.r layer_idxct|||d|_tj|j |j |jzd|_tjtj|j |j|_ tjtj|j |j|_ y)N)rxFra)r&r' is_causalrrcr+num_attention_headshead_dimrelative_k_proj Parameterr)zerosbias_ubias_vr0rrxr2s r3r'z!ParakeetEncoderAttention.__init__ms 95!yy););V=W=WZ^ZgZg=gnstll5;;v/I/I4==#YZ ll5;;v/I/I4==#YZ r4r5position_embeddingsattention_maskkwargsreturnc |jdd}|\}}||d|jf}|j|j|j dd} |j |j|j dd} |j |j|j dd} t} |jjdk7rt|jj} | |jjd|jjd|jz} | |jjd|jjd|jz}|j|}|j|d|jj|j}||jddddz}|j!|}|dd|f}||j"z}|)|j%|j't)d}| |f| | | ||j*sd n |j,|j"d |\}}|j.g|dj1}|j3|}||fS) Nr8rreagerrr.z-inf)querykeyvaluerrqscaling)rAr|q_projviewrHk_projv_projrr_attn_implementationr rr{rr}permute _rel_shiftr masked_fill_ logical_notr.roattention_dropoutrL contiguouso_proj)r0r5rrr input_shape batch_sizerM hidden_shape query_states key_states value_statesattention_interfacequery_states_with_bias_uquery_states_with_bias_vrelative_key_states matrix_bd attn_output attn_weightss r3rSz ParakeetEncoderAttention.forwardwst$))#2. !, J"JDMMB {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (? ;; + +w 6"9$++:Z:Z"[ #/$++2B2B t{{..4==3 $  $0$++2B2B t{{..4==3 $  #223FG166z2t{{GfGfhlhuhuv-/B/J/J1aQRTU/VV OOI. c;J;./  ,  %"..~/I/I/KUSY][I%8  % *$#}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r4c|j\}}}}tjj|d}|j ||d|}|ddddddfj ||||}|S)ztRelative position shift for Shaw et al. style attention. See appendix B of https://huggingface.co/papers/1901.02860.)rr)padr8Nr)rArrprr)r0attention_scoresr num_heads query_lengthposition_lengths r3rz#ParakeetEncoderAttention._rel_shiftst?O?U?U< I|_==,,-=6,J+00YLY+Aq!"H5:::yR^`opr4rT)rUrVrWrXrintr'r)rYrrrtuplerSrr\r]s@r3rwrwjs~[4[[26 7)||7)&ell37)!. 7) +, 7) u||U\\) * 7)r r4rwceZdZdeffd Zdej dejfdZ d dej dej fdZ xZ S) ParakeetEncoderSubsamplingConv2Drc t||j|_|j|_|j |_|jdz dz|_ttj|j|_ tj|_|j j#tj$d|j|j|j |j|j j#tj&t)|jdz D]}|j j#tj$|j|j|j|j |j|j|j j#tj$|j|jd|j j#tj&|j*|j |jzz}tj,|j |z|j.d|_y)Nrr) kernel_sizestridepadding)rrrgroupsrTra)r&r'subsampling_conv_kernel_sizersubsampling_conv_stridersubsampling_conv_channelschannelsrrmathlog2subsampling_factor num_layersr ModuleListlayersappendConv2dReLUrange num_mel_binsrcr+linear)r0ri out_lengthr2s r3r'z)ParakeetEncoderSubsamplingConv2D.__init__s !>>44 88 ((1,2 dii(A(ABCmmo   IIaD4D4DT[[bfbnbn o  2779%t*+A KK   MMMM $ 0 0;; LL==   KK  ryy STU V KK  rwwy ),"((T[[$//-IJ ii @ @: MvOaOahlm r4 input_lengths conv_layerct|drR|jdk7rC|j}|jd}|jd}||dz|dz|z |zdz}|S|S)Nr)rrrr)hasattrrrr)r0rrrrroutput_lengthss r3_get_output_lengthz3ParakeetEncoderSubsamplingConv2D._get_output_lengthsx :x (Z->->&-H ((G$003K&&q)F+gaj871:E SX^^abbN! !r4input_featuresrc|jd}||jdnd}|jD]}||}t|tj s&|)|j ||}|jd}tj||j|dddfk}||ddddddfz}|jddj|jd|jdd}|j|}|S)Nrr8rr9r) unsqueezesumrrDrrrrAr)r*r$rHrLr)r0rrr5current_lengthslayercurrent_seq_length channel_masks r3rSz(ParakeetEncoderSubsamplingConv2D.forwards&003 4B4N.,,R0TX[[E!-0M%+0J"&"9"9/5"Q%2%8%8%;"LL!3N!?? !&//15==m>Q>QRS>TVcViVijkVlnpq  M2 r4rT) rUrVrWrr'r)rYrrrrSr\r]s@r3rrsI!n4!nF  ")) ellELLr4rc eZdZd dedeeffd Z d dejdeejdeejde e dejf d Z xZ S) ParakeetEncoderBlockrrxct|d|_t||_t |||_t||_t||_ tj|j|_ tj|j|_tj|j|_tj|j|_tj|j|_yNF)r&r'gradient_checkpointingr_ feed_forward1rw self_attnrsconv feed_forward2r LayerNormr+norm_feed_forward1 norm_self_att norm_convnorm_feed_forward2norm_outrs r3r'zParakeetEncoderBlock.__init__s &+#7?1&)D4V< 7?"$,,v/A/A"B\\&*<*<=f&8&89"$,,v/A/A"B V%7%78 r4r5rrrrc x|}|j|j|}|d|zz}|j|}|jd|||d|\}}||z}|j |j ||} || z}|j |j|} |d| zz}|j|}|S)Ng?)r5rr)r) rrrrrrrrr) r0r5rrrresidualnormalized_hidden_statesr_ conv_output ff2_outputs r3rSzParakeetEncoderBlock.forwards!**4+B+B=+QR  3#66 #'#5#5m#D ' 2) 3    Q & 3 ii} =ni] % 3 ''(?(? (NO %j(88  m4 r4rTNN) rUrVrWrrrr'r)rYrrrSr\r]s@r3rrsx 94 9# 9$266: ||!.&ell3  +,   r4rceZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ dZeedZfdZd ej(fd Zdd ej(d eefd ZxZS)ParakeetPreTrainedModelrmodelrTrF)r5 attentionsct||t|jdr|jj}n%t |jj dd}t|trO|jjjd||jjjd|yy)Ninitializer_rangeg{Gz?r)meanstd) r& _init_weightsrrrgetattrget_text_configrDrwrdatanormal_r)r0modulerr2s r3rz%ParakeetPreTrainedModel._init_weightsBs f% 4;; 3 4++//C$++5579LdSC f6 7 MM   & &CS & 9 MM   & &CS & 9 8r4rct|jtr|jjn |j}|j}|j }t tj|j}|dz dzdz}||z }|}t|D]Q} tj|jtj|z|dz}tj|}S|jtj S)Nrrr"r!)rDrrencoder_configrrrrrrrr)divr-r.floor) r0rrrrr all_paddingsadd_padlengthsrs r3_get_subsampling_output_lengthz6ParakeetPreTrainedModel._get_subsampling_output_lengthPs7A$++O`7a33gkgrgr$AA 77>#D#DEF #aA-1 ,z"Aii  = GPSVVGkk'*G#zz z**r4r target_lengthc|j|jd}||n|j}tj||j |dddfk}|S)z Convert the input attention mask to its subsampled form. `target_length` sets the desired output length, useful when the attention mask length differs from `sum(-1).max()` (i.e., when the longest sequence in the batch is padded) r8Nr9)rrmaxr)r*r$)r0rrr max_lengths r3_get_output_attention_maskz2ParakeetPreTrainedModel._get_output_attention_maskasc <<^=O=OPR=ST&3&?]^EWEWEY j9N9NOR`abdhahRiir4rT)rUrVrWrrZbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_flat_attention_mask_supports_sdpa_supports_flex_attn_supports_flash_attn_can_compile_fullgraph_supports_attention_backendrrw_can_record_outputsrr)rYrrrrr\r]s@r3rr-s &O&*#/0$(!N!!"&-. :+ELL+"  V^_bVc r4rz{ The Parakeet Encoder model, based on the [Fast Conformer architecture](https://huggingface.co/papers/2305.05084). ) custom_introc eZdZUeed<dZdeffd Zeee d de jde e jde edefdZxZS) ParakeetEncoderrencoderc t||||_d|_|j|_|j |_|j |_|jrtj|jnd|_ t||_ t||_t!j"t%|j&Dcgc]}t)||c}|_|j-ycc}w)NFr!)r&r'rrrqdropout_positions layerdrop scale_inputrsqrtr+ input_scaler subsamplingrencode_positionsrrrnum_hidden_layersrr post_initrs r3r'zParakeetEncoder.__init__vs   &+#~~ !'!9!9))>> from transformers import AutoProcessor, ParakeetEncoder >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> encoder = ParakeetEncoder.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> encoder_outputs = encoder(**inputs) >>> print(encoder_outputs.last_hidden_state.shape) ``` rmrrr8rFT)rr)last_hidden_state)rrrrrprqrorrrArrCrHrr)randrr ) r0rrrr5r encoder_layerto_dropdropout_probabilitys r3rSzParakeetEncoder.forwardsp<((H %(8(88 "33MB --mt||VZVcVc-d  mm33 4#9#9DMM4   %!<<^[h[n[nop[q<rN+55a8??MDWDWXYDZ\^_N+n.F.Fq!.LLN+55a8N![[MG}}&+jjn#&7"G -!!#1(;! ! ) ??r4rT)rUrVrWrrZr r'rrrr)rYrrrr rSr\r]s@r3rrms "!!4&26:@ :@!.:@+, :@  :@:@r4rceZdZUdZej ed<dZee ejed<dZ ee e ejed<dZ ee e ejed<y)ParakeetGenerateOutputal Outputs of Parakeet models. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. sequencesNlogitsrr5) rUrVrWrXr) LongTensorrZr,rr FloatTensorrr5rr4r3r*r*sm&15FHU5,,- .5<@JuU%6%6789@?CM8E%(9(9":;<Cr4r*zS Parakeet Encoder with a Connectionist Temporal Classification (CTC) head. cPeZdZUeed<deffd Zee d dejde ejde ejde e de f dZej d dejde ejd ede e deeej&ff d ZxZS) ParakeetForCTCrct||t|j|_t j |jj|jd|_ |jy)Nrr) r&r'rrrrConv1dr+ vocab_sizectc_headr!rks r3r'zParakeetForCTC.__init__sS  &v'<'<=  &"7"7"C"CVEVEVdef  r4rrlabelsrrc |jd ||d|}|j}|j|jddjdd}d}|Y||n$t j |tj }|j|jd} ||jjk7} | jd} |j| } tjj|dtjjdd} tj j"j%d 5tjj'| | | | |jj|jj(|jj* }dddt-|||j.|j0 S#1swY,xYw)a Example: ```python >>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> outputs = model(**inputs) >>> print(outputs.loss) ```rrrrNr"r8)r?r#rF)r=)blank reduction zero_infinity)lossr,r5rr)rr$r4rHr) ones_likelongrrr pad_token_id masked_selectrrp log_softmaxfloat32backendscudnnflagsctc_lossctc_loss_reductionctc_zero_infinityr r5r)r0rrr5rencoder_outputsr5r,r;r labels_masktarget_lengthsflattened_targets log_probss r3rSzParakeetForCTC.forwards:'$,, ))   (99 }66q!<=GG1M  #1"<%//R`hmhrhrBs !??@R@RSU@VWM!DKK$<$<>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> predicted_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> print(transcription) ``` T return_dictr7r8r>rr#)r+r,rr5r) rSr,argmaxrrArr>r*rr5)r0rrrMroutputsr+s r3generatezParakeetForCTC.generate=s:!%}".$,,# ))# # NN))b)1   %!<<^[d[j[jkl[m<nN)-)A)AI~o & ")#~~"--%33  r4rr)rUrVrWrrZr'rrr)rYrrrr rSr[boolrr*r-rRr\r]s@r3r0r0s 026)- E  E !.E & E +, E  E E NU]]_26(- 3 3!.3"& 3 +, 3 %u'7'77 8 33r4r0)r0rr)1rXr dataclassesrtypingrrrr)r activationsr modeling_layersr modeling_outputsr r modeling_utilsr rprocessing_utilsrutilsrrrr utils.genericr4fastspeech2_conformer.modeling_fastspeech2_conformerrllama.modeling_llamarrconfiguration_parakeetrrModulerr_rsrwrrrrr*r0__all__rr4r3rbs4 !,, !9?F&VV/hJL/7299/7d  0'L0 L ~L ^BryyBJ,5,^<o<<~ T@-T@  T@n D[D D4 H,H  HV Kr4