tj9dZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l mZmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6e!j7e8Z9edeGddeZ:Gddej;Z<Gddej;Z=Gd d!e*Z>Gd"d#e,Z?Gd$d%ej;Z@Gd&d'eZAeGd(d)eZBed*Gd+d,eBZCeGd-d.eZDeGd/d0eDZEed1Gd2d3eBeZFGd4d5ej;ZGGd6d7ej;ZHeGd8d9eZIed:Gd;deHZKed?Gd@dAe6eJZLgdBZMdS)CzPyTorch Parakeet model.N)Callable) dataclass)nn)initialization)ACT2FN) CompileConfigGenerationMixinGenerationMode)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingCausalLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)capture_outputs) AutoModel)%FastSpeech2ConformerConvolutionModule)LlamaAttentioneager_attention_forward)ParakeetCTCConfigParakeetEncoderConfigParakeetRNNTConfigParakeetTDTConfig)ParakeetRNNTDecoderCacheParakeetRNNTGenerationMixinParakeetTDTGenerationMixinz Extends [~modeling_outputs.BaseModelOutputWithPooling] to include the output attention mask since sequence length is not preserved in the model's forward. ) custom_introc2eZdZUdZdZejdzed<dS)ParakeetEncoderModelOutputa attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Nattention_mask)__name__ __module__ __qualname____doc__r+torchTensor__annotations__k/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/parakeet/modular_parakeet.pyr*r*5s5+/NEL4'.....r4r*ceZdZUejed<d deffd Ze d dedzdejfdZ ej dejfdZ xZ S) $ParakeetEncoderRelPositionalEncodinginv_freqNconfigct|j|_||_|||}|d|ddS)Ndevicer8F) persistent)super__init__max_position_embeddingsr9.compute_default_relative_positional_parametersregister_buffer)selfr9r<r8 __class__s r5r?z-ParakeetEncoderRelPositionalEncoding.__init__Lse '-'E$ FFvV\F]] ZeDDDDDr4returncd}d|tjd|jdtj|tj|jz zz }|S)Ng@?rrdtype)r<rI)r0arange hidden_sizeint64tofloat)r9r<baser8s r5rAzSParakeetEncoderRelPositionalEncoding.compute_default_relative_positional_parametersSs`   Q 2AU[IIILLTZbgbmLnn$%  r4 hidden_statesc|jd}tj|dz | d|j}|jddddf|jddd|j}|ddddf}t|jj tr|jj dkr |jj nd}t|d5||z dd }| }|} tj|| gd } | jg| jdd dR} dddn #1swxYwY| |j S) Nr r;rmpscpuF) device_typeenabledrdimrH)shaper0rJr<r8rNexpandrM isinstancetypestrr transposesincosstackreshaperI) rCrP seq_length position_idsinv_freq_expandedposition_ids_expandedrUfreqsr`ra pos_embeds r5forwardz,ParakeetEncoderRelPositionalEncoding.forwardbs "(+ |JNZKML`aaa M$4- ( . . 0 0 7 7 8KA8NPRTU V V Y YZgZn o o !-T4] ; A A C C-.3S99 >K>R>W[`>`>`  % %   U C C C E E&,,..1F1L1L1N1NNYYZ[]^__E))++C))++C S#JB777I) )D9?3B3+?DDDDI  E E E E E E E E E E E E E E E||-"5|666s8BFF"%F"NNN) r,r-r.r0r1r2r"r? staticmethodrAno_gradrj __classcell__rDs@r5r7r7IslEE4EEEEEE/3  %,     \ U]__7U\777_77777r4r7c*eZdZdeffd ZdZxZS)ParakeetEncoderFeedForwardr9c:ttj|j|j|j|_t|j |_ tj|j|j|j|_ |j |_ dS)Nbias) r>r?rLinearrKintermediate_sizeattention_biaslinear1r hidden_act activationlinear2activation_dropoutrCr9rDs r5r?z#ParakeetEncoderFeedForward.__init__|s} y!3V5MTZTijjj  !23y!96;MTZTijjj "(";r4c|||}tj||j|j}||}|S)Nptraining)r{ryr functionaldropoutr}rr|)rCrPs r5rjz"ParakeetEncoderFeedForward.forwardsY ](C(CDD  --mt?Vaean-oo  ]33 r4)r,r-r.r"r?rjrorps@r5rrrr{sT<4<<<<<<r4rrc&eZdZddeffd ZxZS) ParakeetEncoderConvolutionModuleNr9cLt||dSrk)r>r?)rCr9 module_configrDs r5r?z)ParakeetEncoderConvolutionModule.__init__s# /////r4rk)r,r-r.r"r?rorps@r5rrsJ0040000000000r4rc eZdZdZdedeffd Z d dejdejdzdejdzd e e d e ejejff d Z d Z xZS)ParakeetEncoderAttentionztMulti-head attention with relative positional encoding. See section 3.3 of https://huggingface.co/papers/1901.02860.r9 layer_idxct||d|_tj|j|j|jzd|_tj tj |j|j|_ tj tj |j|j|_ dS)N)rFrt)r>r? is_causalrrvrKnum_attention_headshead_dimrelative_k_proj Parameterr0zerosbias_ubias_vrCr9rrDs r5r?z!ParakeetEncoderAttention.__init__s 9555!y);V=WZ^Zg=gnstttl5;v/I4=#Y#YZZ l5;v/I4=#Y#YZZ r4NrPposition_embeddingsr+kwargsrEc |jdd}|\}}||d|jf}|||dd} |||dd} |||dd} tj|j j t} | |j d|j j d|jz} | |jd|j j d|jz}||}||d|j j |j}||ddddz}||}|dd|f}||jz}|5||t+d}| |f| | | ||jsdn|j|jd |\}}|jg|dR}||}||fS) NrRr rrr.z-inf)querykeyvaluer+rscaling)rZrq_projviewr_k_projv_projr get_interfacer9_attn_implementationrrrrrpermute _rel_shiftr masked_fill_ logical_notrNrattention_dropoutrc contiguouso_proj)rCrPrr+r input_shape batch_sizerd hidden_shape query_states key_states value_statesattention_interfacequery_states_with_bias_uquery_states_with_bias_vrelative_key_states matrix_bd attn_output attn_weightss r5rjz ParakeetEncoderAttention.forwards$)#2#. !, J"JDMB {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT (?(M K ,.E) ) $0$+2B2B t{.4=3 3 $  $0$+2B2B t{.4=3 3 $  #223FGG166z2t{Gfhlhuvv-/B/J/J1aQRTU/V/VV OOI.. c;J;./  ,  %"..~/I/I/K/KUSY]][[I%8$7  % *$#}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r4c|j\}}}}tj|d}|||d|}|ddddddf||||}|S)ztRelative position shift for Shaw et al. style attention. See appendix B of https://huggingface.co/papers/1901.02860.)r r)padrRNr )rZrrrr)rCattention_scoresr num_heads query_lengthposition_lengths r5rz#ParakeetEncoderAttention._rel_shifts?O?U< I|_=,,-=6,JJ+00YLYY+AAAqqq!""H5:::yR^`oppr4rk)r,r-r.r/r"intr?r0r1rrtuplerjrrorps@r5rrs~~[4[[[[[[[/3 7)7)|7)#\D07) t+ 7) +, 7) u|U\) * 7)7)7)7)r       r4rcneZdZdeffd ZdejdejfdZ d dejdejfd Z xZ S) ParakeetEncoderSubsamplingConv2Dr9c 6t|j|_|j|_|j|_|jdz dz|_ttj |j |_ tj|_|jtjd|j|j|j|j|jtjt)|j dz D]}|jtj|j|j|j|j|j|j|jtj|j|jd|jtj|j|j|j zz}tj|j|z|jd|_dS)Nr r) kernel_sizestridepadding)rrrgroupsrTrt)r>r?subsampling_conv_kernel_sizersubsampling_conv_stridersubsampling_conv_channelschannelsrrmathlog2subsampling_factor num_layersr ModuleListlayersappendConv2dReLUrange num_mel_binsrvrKlinear)rCr9i out_lengthrDs r5r?z)ParakeetEncoderSubsamplingConv2D.__init__s !>4 8 (1,2 di(ABBCCmoo   IaD4DT[bfbn o o o    2799%%%t*++ * *A K   MM $ 0; L=     K  ry STUUU V V V K  rwyy ) ) ) )(T[$/-IJ i @: MvOahlmmm r4 input_lengths conv_layerct|drK|jdkr@|j}|jd}|jd}||dz|dz|z |zdz}|S|S)Nr)r r rr )hasattrrrr)rCrrrrroutput_lengthss r5_get_output_lengthz3ParakeetEncoderSubsamplingConv2D._get_output_lengthsw :x ( ( "Z->&-H-H (G$03K&q)F+gaj871:E SX^^abbN! !r4Ninput_featuresr+c.|d}||dnd}|jD]}||}t|tjra|_|||}|jd}tj ||j |dddfk}||ddddddfz}| dd |jd|jdd}| |}|S)Nr rRrr;r) unsqueezesumrr\rrrrZr0rJr<r_rcr)rCrr+rPcurrent_lengthslayercurrent_seq_length channel_masks r5rjz(ParakeetEncoderSubsamplingConv2D.forward s<&0033 4B4N.,,R000TX[ @ @E!E-00M%++ @0J"&"9"9/5"Q"Q%2%8%;"L!3N!?? %//155==m>QRS>TVcVijkVlnpqq  M22 r4rk) r,r-r.r"r?r0r1rrrrjrorps@r5rrs!n4!n!n!n!n!n!nF  ")    elELr4rc eZdZd dededzffd Z d dejdejdzdejdzdee d ejf d Z xZ S) ParakeetEncoderBlockNr9rc$td|_t||_t |||_t||_t||_ tj |j |_ tj |j |_tj |j |_tj |j |_tj |j |_dS)NF)r>r?gradient_checkpointingrr feed_forward1r self_attnrconv feed_forward2r LayerNormrKnorm_feed_forward1 norm_self_att norm_convnorm_feed_forward2norm_outrs r5r?zParakeetEncoderBlock.__init__$s &+#7??1&)DD4V<< 7??"$,v/A"B"B\&*<==f&899"$,v/A"B"B V%788 r4rPr+rrrEc |}|||}|d|zz}||}|jd|||d|\}}||z}||||} || z}|||} |d| zz}||}|S)Ng?)rPr+r)r+r3) rrrrrrrrr) rCrPr+rrresidualnormalized_hidden_statesr_ conv_output ff2_outputs r5rjzParakeetEncoderBlock.forward3s!**4+B+B=+Q+QRR  3#66 #'#5#5m#D#D ' 2) 3      Q & 3 ii} = =ni]] % 3 ''(?(? (N(NOO %j(88  m44 r4rkrl) r,r-r.r"rr?r0r1rrrjrorps@r5rr#s 9 94 9t 9 9 9 9 9 9$/337 | t+#\D0  +,   r4rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ dZdZeedZejfd Zd ejfd Zdd ejded zfdZxZS)ParakeetPreTrainedModelr9modelraudioTrF)rP attentionsct|t|jdd}t |t r:t j|jd|t j|j d|dSt |tr6| |j}t j |j |dSdS)Ninitializer_rangeg{Gz?r)meanstd)r> _init_weightsgetattrr9r\rinitnormal_rrr7rAcopy_r8)rCmoduler buffer_valuerDs r5rz%ParakeetPreTrainedModel._init_weightshs f%%%dk#6== f6 7 7 6 LSc : : : : LSc : : : : : :  D E E 6!PPQWQ^__L Jv 5 5 5 5 5 6 6r4rct|jd|j}|j}|j}t t j|j}|dz dzdz}||z }|}t|D]O} tj | tj |z|dz}tj |}P| tjS)Nencoder_configr rrHrG)rr9rrrrrrrr0divrMrNfloor) rCrr rrr all_paddingsadd_padlengthsrs r5_get_subsampling_output_lengthz6ParakeetPreTrainedModel._get_subsampling_output_lengthts .> LL$A 7>#DEEFF #aA-1 ,z"" + +Ai  = = GPPSVVGk'**GGzz z***r4Nr+ target_lengthc||d}||n|}tj||j|dddfk}|S)z Convert the input attention mask to its subsampled form. `target_length` sets the desired output length, useful when the attention mask length differs from `sum(-1).max()` (i.e., when the longest sequence in the batch is padded) rRNr;)rrmaxr0rJr<)rCr+rr max_lengths r5_get_output_attention_maskz2ParakeetPreTrainedModel._get_output_attention_masksu <<^=O=OPR=S=STT&3&?]]^EWEWEYEY j9NOOOR`abababdhahRiir4rk)r,r-r.r!r2base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_flat_attention_mask_supports_sdpa_supports_flex_attn_supports_flash_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr0rnrr1rrrrorps@r5rrRs &O&*#/0$(!N!!"&-. U]__ 6 6 6 6_ 6+EL++++"   VY\`V`        r4rz{ The Parakeet Encoder model, based on the [Fast Conformer architecture](https://huggingface.co/papers/2305.05084). ceZdZUeed<dZdeffd Zeee e d de j de j dzde d eed ef d ZxZS) ParakeetEncoderr9encoderct|_d|_j|_j|_j|_jrtj j nd|_ t|_ t|_t!jfdt%jD|_|dS)NFrGc0g|]}t|Sr3)r).0rr9s r5 z,ParakeetEncoder.__init__..s$ f f f !&) 4 4 f f fr4)r>r?r9rrdropout_positions layerdrop scale_inputrsqrtrK input_scaler subsamplingr7encode_positionsrrrnum_hidden_layersr post_initr~s `r5r?zParakeetEncoder.__init__s     &+#~ !'!9)>> from transformers import AutoProcessor, ParakeetEncoder >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> encoder = ParakeetEncoder.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> encoder_outputs = encoder(**inputs) >>> print(encoder_outputs.last_hidden_state.shape) ``` rNr rrRrFT)r+r)last_hidden_stater+)r1r0r2rrrrr,rrZrr[r_rr0randr-r*r) rCrr+r5rrPr output_mask encoder_layerto_dropdropout_probabilitys r5rjzParakeetEncoder.forwardsF((HH %(88 "33MBB --mt|VZVc-dd  m33 4#9DM4    %99.XeXklmXn9ooK(22155<0JOd0J;??,,,jn    r4)NT)r,r-r.r"r2rr?rrrrr0r1boolrrr rjrorps@r5r&r&s "!!!!4&/3&* B B  B  t+B  $ B +, B  B B B _ ^B B B B B r4r&ceZdZUdZejed<dZeej dzed<dZ eeej dzed<dZ eeej dzed<dS)ParakeetCTCGenerateOutputaz Outputs of Parakeet CTC model generation. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. sequencesNlogitsrrP) r,r-r.r/r0 LongTensorr2rBr FloatTensorrrPr3r4r5r@r@s&.2FE%# $t +2229=JeE-./$6===<@M5u012T9@@@@@r4r@c"eZdZdZfdZxZS)ParakeetGenerateOutputz` Deprecated alias for ParakeetCTCGenerateOutput. Use ParakeetCTCGenerateOutput instead. cntj|i|tddS)Nz`ParakeetGenerateOutput` is deprecated and removed starting from version 5.11.0; please use `ParakeetCTCGenerateOutput` instead.)r>r?logger warning_once)rCargsrrDs r5r?zParakeetGenerateOutput.__init__sG$)&))) O     r4)r,r-r.r/r?rorps@r5rFrFsB         r4rFzS Parakeet Encoder with a Connectionist Temporal Classification (CTC) head. c6eZdZUeed<deffd Zee ddej dej dzdej dzde e de f d Z ej ddej dej dzd ed edzde e deejzf d ZxZS)ParakeetForCTCr9ct|tj|j|_t j|jj|j d|_ | dS)Nr r) r>r?r from_configr r'rConv1drK vocab_sizectc_headr4r~s r5r?zParakeetForCTC.__init__&sh     ,V-BCC  &"7"CVEVdefff  r4Nrr+labelsrrEc H||dd|jd||d|}|j}||dddd}d}||jd} ||jjk} | d} | | } tj |dtjd d} tjjd 5tj | | | | |jj|jj|jj }dddn #1swxYwYt+|||j|j S)a Example: ```python >>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> outputs = model(**inputs) >>> print(outputs.loss) ```Nr5Trr+r rrR)rXrIrF)rV)blank reduction zero_infinity)lossrBrPrr3) setdefaultr'r8rQr_r+rr9 pad_token_id masked_selectrr log_softmaxr0float32backendscudnnflagsctc_lossctc_loss_reductionctc_zero_infinityrrPr)rCrr+rRrencoder_outputsrPrBrXencoder_lengths labels_masktarget_lengthsflattened_targets log_probss r5rjzParakeetForCTC.forward.s:     5t < < <&$, ))     (9 }66q!<<==GG1MM  -<@@DDO!DK$<>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> predicted_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> print(transcription) ``` NT return_dictrTrRrWr r7)rArBrrPr3) get_compiled_call__call__rBargmaxrrZr9rZr@rrP) rCrr+rjrkr model_forwardoutputsrAs r5generatezParakeetForCTC.generatetsBCQB\..~>>>bfbo $}"/-# ))# # # # N))b)11   %!<<^[d[jkl[m<nnN)-)AI~o & " ,#~"-%3  r4rl)NFN)r,r-r.r!r2r?rrr0r1rrrrjrnr>r r@rCrsrorps@r5rLrLsl 0/3&* B B  B  t+B  t# B +, B  B B B ^B HU]__/3(-/3 99 9 t+9"& 9 &, 9 +, 9 #U%5 5999_99999r4rLcZeZdZdZdeffd Z d dejdedzdej fdZ xZ S) ParakeetRNNTDecoderz'LSTM-based prediction network For RNN-Tr9cHt|j|_tj|j|j|_tj|j|j|j d|_ tj |j|j|_ dS)NT) input_sizerKr batch_first) r>r?blank_token_idr EmbeddingrPdecoder_hidden_size embeddingLSTMnum_decoder_layerslstmrvdecoder_projectorr~s r5r?zParakeetRNNTDecoder.__init__s $3f&79STTG120     "$6+EvGa!b!br4N input_idscacherEc|7|dddf|jk}|jr|r|jS||}|-|j}|s|||j|jf}nd}|||\}\}} | |} |'|r|nd} | | || | |jS| S)NrR)mask) ryis_initializedallrr|lazy_initialization hidden_state cell_staterrupdate) rCrr blank_mask embeddingswas_initializedhidden_cell_states lstm_outputrrdecoder_outputrs r5rjzParakeetRNNTDecoder.forwards  "111b5)T-@@J# # (8(8 #{"^^I..   #2O" 6))*555"'"4e6F!G  !% 26))JHZ2[2[/ /lJ// <<  "1;J;;tD LLzL M M M; r4rk) r,r-r.r/r#r?r0rCr%r1rjrorps@r5rurus11 c1 c c c c c c26#($.  r4ructeZdZdZdeffd ZdejdejdeejejffdZ xZ S)ParakeetRNNTJointNetworkzPJoint network that combines encoder and decoder outputs to predict token logits.r9ctt|j|_t j|j|j|_ |j|_dSrk) r>r?rrzr{rrvr{rPheadr~s r5r?z!ParakeetRNNTJointNetwork.__init__sO  !23If8&:KLL  +r4decoder_hidden_statesencoder_hidden_statesrEc\|||z}||Srk)r{r)rCrr joint_outputs r5rjz ParakeetRNNTJointNetwork.forwards. 'r?rrNr r'rrvrKr{encoder_projectorrudecoderrjointmax_symbols_per_stepr4r~s r5r?zParakeetForRNNT.__init__s     ,V-BCC !#6+@+LfNh!i!i*622 -f55 $*$?! r4Nrr+rrEc d|jd||d|}||j|_|S)NrTr3)r'rr8 pooler_output)rCrr+rrds r5get_audio_featuresz"ParakeetForRNNT.get_audio_featuressR'$, ))     )-(>(>?`(a(a%r4decoder_input_idsruse_decoder_cacherdrRc ||jd ||d|}|r|t|j}|||} ||jdddddddf| dddddddfd} d} ||jd} |j d | dddt| f|| ||jj kd|jj d|} t| | |j|j|j|j|S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForRNNT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-rnnt-0.6b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForRNNT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` NrTrrrrrR)rBrR logit_lengths label_lengthsryrXrBr8rrPrrr3)rr%r9rrrsqueezer+r loss_functionrrryrr8rPr) rCrr+rrrrdrRrrrBrXrs r5rjzParakeetForRNNT.forward*sX  "5d5--O  B!64T[AAM $ ->m T T"1"?111dAAA "N"74AAA "F   '!**    +:>>rBBM%4%aaa!;3}'8'8':':#;#;!;;<+%)CCHHLL#{9   D"-?)7)7&1'    r4rkNNNNNNN)r,r-r.r#r2rr GREEDY_SEARCH_supported_generation_modesr?rr0r1rrr*rrrCr%r>rrDrrjrorps@r5rrs ./#1#?"@1/3     t+ +,  $    /3.2599=)-X\&*N N  t+N  t+N !+d2 N 0$6 N  $; N 4eEr?rrvr{rPlen durationsrr~s r5r?z ParakeetTDTJointNetwork.__init__sI    If8&:KcRXRbNcNc:cdd r4)r,r-r.r/r$r?rorps@r5rr}sZ e0eeeeeeeeeer4rzG Parakeet Encoder with a TDT (Token Duration Transducer) head. ceZdZUeed<deffd Zee ddej dzdej dzdej dzde dzde dzd e eejzdzd ej dzd eed efd ZxZS)ParakeetForTDTr9ct|t||_|dSrk)r>r?rrr4r~s r5r?zParakeetForTDT.__init__s@    ,V44  r4Nrr+rrrrdrRrrEc ||jd ||d|}|r|t|j}|||} ||jdddddddf| dddddddfd} d} ||jd | dd|jjf| d|jjdf||j d||jj k d|jj |jj d|} t| | |j|j|j|j| S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForTDT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-tdt-0.6b-v3" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForTDT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` NrTrrr.rR) token_logitsduration_logitsrRrrryrrr3)rr%r9rrrrrrPr+rrZryrrr8rPr) rCrr+rrrrdrRrrrBrXs r5rjzParakeetForTDT.forwardsX  "5d5--O  B!64T[AAM $ ->m T T"1"?111dAAA "N"74AAA "F   '!**    %4% #C)A4;+A)A$AB &sDK,B,D,D'D E-<@@DD%)AAFFrJJ#{9+/    D"-?)7)7&1'    r4r)r,r-r.r$r2r?rrr0r1rCr%r>r*rrDrrrrjrorps@r5rrsE 0 /3.2599=)-X\&*O O  t+O  t+O !+d2 O 0$6 O  $; O 4eErsj $$$$$$!!!!!! &&&&&&!!!!!!HHHHHHHHHH999999[[[[[[[[[[FFFFFFFF&&&&&&HGGGGGGG555555hhhhhhJJJJJJJJssssssssssss  H % %   / / / / /!; / /   //7/7/7/7/729/7/7/7d        00000'L000 L L L L L ~L L L ^BBBBBryBBBJ,,,,,5,,,^;;;;;o;;;| ] ] ] ] ] -] ]  ] @ AAAAA AA A4       6       KKKKK,oKK  K\-----")---`'''''ry'''$ :::::3:: :& n n n n n -/Jn n  n b e e e e e6 e e e Z Z Z Z Z /Z Z  Z z p o or4