tj|ddlZddlmZddlmZddlZddlmZddlmZ ddl m Z ddl m Z mZmZdd lmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3e#j4e5Z6e!deGddeZ7Gddej8Z9Gddej8Z:Gddej8Z;d Z<ed!dRd"Z=d#ej>d$e?d%ej>fd&Z@ dSd(ej8d)ej>d*ej>d+ej>d,ej>dzd-eAd.eAd/ee fd0ZBee=Gd1d2ej8ZCGd3d4ej8ZDGd5d6eZEe!Gd7d8eZFe!d9Gd:d;eFZGeGd<d=eZHeGd>d?eHZIe!d@GdAdBeFeZJGdCdDej8ZKGdEdFej8ZLeGdGdHeZMe!dIGdJdKeFe2ZNGdLdMeLZOe!dNGdOdPe3eNZPgdQZQdS)TN)Callable) dataclass)nn)initialization)ACT2FN) CompileConfigGenerationMixinGenerationMode)use_kernel_func_from_hubuse_kernelized_func)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingCausalLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)capture_outputs) AutoModel)ParakeetCTCConfigParakeetEncoderConfigParakeetRNNTConfigParakeetTDTConfig)ParakeetRNNTDecoderCacheParakeetRNNTGenerationMixinParakeetTDTGenerationMixinz Extends [~modeling_outputs.BaseModelOutputWithPooling] to include the output attention mask since sequence length is not preserved in the model's forward. ) custom_introc2eZdZUdZdZejdzed<dS)ParakeetEncoderModelOutputa attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Nattention_mask)__name__ __module__ __qualname____doc__r*torchTensor__annotations__l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/parakeet/modeling_parakeet.pyr)r)/s5+/NEL4'.....r3r)ceZdZUejed<d deffd Ze d dedzdejfdZ ej dejfdZ xZ S) $ParakeetEncoderRelPositionalEncodinginv_freqNconfigct|j|_||_|||}|d|ddS)Ndevicer7F) persistent)super__init__max_position_embeddingsr8.compute_default_relative_positional_parametersregister_buffer)selfr8r;r7 __class__s r4r>z-ParakeetEncoderRelPositionalEncoding.__init__Fse '-'E$ FFvV\F]] ZeDDDDDr3returncd}d|tjd|jdtj|tj|jz zz }|S)Ng@?rrdtype)r;rH)r/arange hidden_sizeint64tofloat)r8r;baser7s r4r@zSParakeetEncoderRelPositionalEncoding.compute_default_relative_positional_parametersMs`   Q 2AU[IIILLTZbgbmLnn$%  r3 hidden_statesc|jd}tj|dz | d|j}|jddddf|jddd|j}|ddddf}t|jj tr|jj dkr |jj nd}t|d5||z dd }| }|} tj|| gd } | jg| jdd dR} dddn #1swxYwY| |j S) Nrr:rmpscpuF) device_typeenabledrdimrG)shaper/rIr;r7rMexpandrL isinstancetypestrr transposesincosstackreshaperH) rBrO seq_length position_idsinv_freq_expandedposition_ids_expandedrTfreqsr_r` pos_embeds r4forwardz,ParakeetEncoderRelPositionalEncoding.forward\s "(+ |JNZKML`aaa M$4- ( . . 0 0 7 7 8KA8NPRTU V V Y YZgZn o o !-T4] ; A A C C-.3S99 >K>R>W[`>`>`  % %   U C C C E E&,,..1F1L1L1N1NNYYZ[]^__E))++C))++C S#JB777I) )D9?3B3+?DDDDI  E E E E E E E E E E E E E E E||-"5|666s8BFF"%F"NNN) r+r,r-r/r0r1r!r> staticmethodr@no_gradri __classcell__rCs@r4r6r6CslEE4EEEEEE/3  %,     \ U]__7U\777_77777r3r6c*eZdZdeffd ZdZxZS)ParakeetEncoderFeedForwardr8c:ttj|j|j|j|_t|j |_ tj|j|j|j|_ |j |_ dS)Nbias) r=r>rLinearrJintermediate_sizeattention_biaslinear1r hidden_act activationlinear2activation_dropoutrBr8rCs r4r>z#ParakeetEncoderFeedForward.__init__vs} y!3V5MTZTijjj  !23y!96;MTZTijjj "(";r3c|||}tj||j|j}||}|S)Nptraining)rzrxr functionaldropoutr|rr{)rBrOs r4riz"ParakeetEncoderFeedForward.forward}sY ](C(CDD  --mt?Vaean-oo  ]33 r3r+r,r-r!r>rirnros@r4rqrqusT<4<<<<<<r3rqc.eZdZddeffd ZddZxZS) ParakeetEncoderConvolutionModuleNr8c >t|j}|)|j}tt |dd|_n.|d}t|dd|_|dz dz|_tj |d|zddd|j |_ tj |||d|j||j |_ tj||_tj ||ddd|j |_dS) z Args: config (ParakeetEncoderConfig): Configuration for the model. module_config (dict): Configuration for the module (e.g., encoder or decoder). Nrysilu kernel_sizerzrrr)rstridepaddingrt)rrgroupsrt)r=r>rJconv_kernel_sizergetattrrzgetrrConv1dconvolution_biaspointwise_conv1depthwise_conv BatchNorm1dnormpointwise_conv2)rBr8 module_configchannelsrrCs r4r>z)ParakeetEncoderConvolutionModule.__init__s% %   1K$WV\6%J%JKDOO' 6K$]%6%6|V%L%LMDO#aA- !y a(l!QVMd   !i   L(   N8,, !y hAaI`   r3c.|dd}||}tj|d}|^|jt jkrt j|d}nt j|dkd}| |d}| |}| |}| |}| |}|ddS)aY Compute convolution module. Args: hidden_states (`torch.Tensor` of shape `(batch, time, channels)`): Input tensor. attention_mask (`torch.Tensor` of shape `(batch, 1, time, time)`): Attention mask. Returns: `torch.Tensor`: Output tensor of shape `(batch, time, channels)`. rrrVN)r^rrrglurHr/boolall masked_fillrrrzr)rBrOr*all_masked_rowss r4riz(ParakeetEncoderConvolutionModule.forwards &//155 ,,];;  ))-Q)??   %#uz11"')^O"C"C"C"')n.C,D!"L"L"L)55osKKM++M::  -00  66 ,,];; &&q!,,,r3rjrros@r4rrs_   4       D"-"-"-"-"-"-"-"-r3rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..NrQrrV)rYr/cat)xx1x2s r4 rotate_halfrs] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r3rotary_pos_embc||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkr`r_ unsqueeze_dimq_embedk_embeds r4apply_rotary_pos_embrsc& -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r3rOn_reprDc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rYrZrb)rOrbatchnum_key_value_headsslenhead_dims r4 repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr3rmodulequerykeyvaluer*scalingrkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)NrrrQrWrHrr)rnum_key_value_groupsr/matmulr^rrsoftmaxfloat32rLrHrr contiguous) rrrrr*rrr key_states value_states attn_weights attn_outputs r4eager_attention_forwardrs3 ;<r8rrrJnum_attention_headsrrrrattention_dropout is_causalrrurwq_projk_projv_projo_projrelative_k_proj Parameterr/zerosbias_ubias_vrBr8rrCs r4r>z!ParakeetEncoderAttention.__init__s  " F4F&Jd4dee $*$>&B\$\!}d* !'!9i   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf     "y);V=WZ^Zg=gnstttl5;v/I4=#Y#YZZ l5;v/I4=#Y#YZZ r3NrOposition_embeddingsr*rrDc |jdd}|\}}||d|jf}|||dd} |||dd} |||dd} tj|j j t} | |j d|j j d|jz} | |jd|j j d|jz}||}||d|j j |j}||ddddz}||}|dd|f}||jz}|5||t+d}| |f| | | ||jsdn|j|jd |\}}|jg|dR}||}||fS) NrQrrrr.z-infr)rrrr*rr)rYrrviewr^rrr get_interfacer8_attn_implementationrrrrrpermute _rel_shiftr masked_fill_ logical_notrMrrrbrr)rBrOrr*r input_shape batch_sizerc hidden_shape query_statesrrattention_interfacequery_states_with_bias_uquery_states_with_bias_vrelative_key_states matrix_bdrrs r4riz ParakeetEncoderAttention.forward3s$)#2#. !, J"JDMB {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT (?(M K ,.E) ) $0$+2B2B t{.4=3 3 $  $0$+2B2B t{.4=3 3 $  #223FGG166z2t{Gfhlhuvv-/B/J/J1aQRTU/V/VV OOI.. c;J;./  ,  %"..~/I/I/K/KUSY]][[I%8$7  % *$#}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r3c|j\}}}}tj|d}|||d|}|ddddddf||||}|S)ztRelative position shift for Shaw et al. style attention. See appendix B of https://huggingface.co/papers/1901.02860.)rr)padrQNr)rYrrrr)rBattention_scoresr num_heads query_lengthposition_lengths r4rz#ParakeetEncoderAttention._rel_shiftls?O?U< I|_=,,-=6,JJ+00YLYY+AAAqqq!""H5:::yR^`oppr3rj)r+r,r-r.r!intr>r/r0rrtuplerirrnros@r4rrs~~[4[[[[[[[B/3 7)7)|7)#\D07) t+ 7) +, 7) u|U\) * 7)7)7)7)r       r3rcneZdZdeffd ZdejdejfdZ d dejdejfd Z xZ S) ParakeetEncoderSubsamplingConv2Dr8c 6t|j|_|j|_|j|_|jdz dz|_ttj |j |_ tj|_|jtjd|j|j|j|j|jtjt)|j dz D]}|jtj|j|j|j|j|j|j|jtj|j|jd|jtj|j|j|j zz}tj|j|z|jd|_dS)Nrr)rrr)rrrrrTrs)r=r>subsampling_conv_kernel_sizersubsampling_conv_stridersubsampling_conv_channelsrrrmathlog2subsampling_factor num_layersr ModuleListlayersappendConv2dReLUrange num_mel_binsrurJlinear)rBr8i out_lengthrCs r4r>z)ParakeetEncoderSubsamplingConv2D.__init__vs !>4 8 (1,2 di(ABBCCmoo   IaD4DT[bfbn o o o    2799%%%t*++ * *A K   MM $ 0; L=     K  ry STUUU V V V K  rwyy ) ) ) )(T[$/-IJ i @: MvOahlmmm r3 input_lengths conv_layerct|drK|jdkr@|j}|jd}|jd}||dz|dz|z |zdz}|S|S)Nr)rrrr)hasattrrrr)rBrr rrroutput_lengthss r4_get_output_lengthz3ParakeetEncoderSubsamplingConv2D._get_output_lengthsw :x ( ( "Z->&-H-H (G$03K&q)F+gaj871:E SX^^abbN! !r3Ninput_featuresr*c.|d}||dnd}|jD]}||}t|tjra|_|||}|jd}tj ||j |dddfk}||ddddddfz}| dd |jd|jdd}| |}|S)NrrQrr:r)rsumrr[rrr rYr/rIr;r^rbr)rBrr*rOcurrent_lengthslayercurrent_seq_length channel_masks r4riz(ParakeetEncoderSubsamplingConv2D.forwards<&0033 4B4N.,,R000TX[ @ @E!E-00M%++ @0J"&"9"9/5"Q"Q%2%8%;"L!3N!?? %//155==m>QRS>TVcVijkVlnpqq  M22 r3rj) r+r,r-r!r>r/r0rrr rirnros@r4rrus!n4!n!n!n!n!n!nF  ")    elELr3rc eZdZd dededzffd Z d dejdejdzdejdzdee d ejf d Z xZ S) ParakeetEncoderBlockNr8rc$td|_t||_t |||_t||_t||_ tj |j |_ tj |j |_tj |j |_tj |j |_tj |j |_dS)NF)r=r>gradient_checkpointingrq feed_forward1r self_attnrconv feed_forward2r LayerNormrJnorm_feed_forward1 norm_self_att norm_convnorm_feed_forward2norm_outrs r4r>zParakeetEncoderBlock.__init__s &+#7??1&)DD4V<< 7??"$,v/A"B"B\&*<==f&899"$,v/A"B"B V%788 r3rOr*rrrDc |}|||}|d|zz}||}|jd|||d|\}}||z}||||} || z}|||} |d| zz}||}|S)Ng?)rOr*r)r*r2) rrrrrr rr!r") rBrOr*rrresidualnormalized_hidden_statesr_ conv_output ff2_outputs r4rizParakeetEncoderBlock.forwards!**4+B+B=+Q+QRR  3#66 #'#5#5m#D#D ' 2) 3      Q & 3 ii} = =ni]] % 3 ''(?(? (N(NOO %j(88  m44 r3rjrk) r+r,r-r!rr>r/r0rrrirnros@r4rrs 9 94 9t 9 9 9 9 9 9$/337 | t+#\D0  +,   r3rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ dZdZeedZejfd Zd ejfd Zdd ejded zfdZxZS)ParakeetPreTrainedModelr8modelraudioTrF)rO attentionsct|t|jdd}t |t r:t j|jd|t j|j d|dSt |tr6| |j}t j |j |dSdS)Ninitializer_rangeg{Gz?r)meanstd)r= _init_weightsrr8r[rinitnormal_rrr6r@copy_r7)rBrr1 buffer_valuerCs r4r2z%ParakeetPreTrainedModel._init_weightss f%%%dk#6== f6 7 7 6 LSc : : : : LSc : : : : : :  D E E 6!PPQWQ^__L Jv 5 5 5 5 5 6 6r3rct|jd|j}|j}|j}t t j|j}|dz dzdz}||z }|}t|D]O} tj | tj |z|dz}tj |}P| tjS)Nencoder_configrrrGrF)rr8rrrrrrrr/divrLrMfloor) rBrr8rrr all_paddingsadd_padlengthsr&s r4_get_subsampling_output_lengthz6ParakeetPreTrainedModel._get_subsampling_output_length s .> LL$A 7>#DEEFF #aA-1 ,z"" + +Ai  = = GPPSVVGk'**GGzz z***r3Nr* target_lengthc||d}||n|}tj||j|dddfk}|S)z Convert the input attention mask to its subsampled form. `target_length` sets the desired output length, useful when the attention mask length differs from `sum(-1).max()` (i.e., when the longest sequence in the batch is padded) rQNr:)r>rmaxr/rIr;)rBr*r?r  max_lengths r4_get_output_attention_maskz2ParakeetPreTrainedModel._get_output_attention_masksu <<^=O=OPR=S=STT&3&?]]^EWEWEYEY j9NOOOR`abababdhahRiir3rj)r+r,r-r r1base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_flat_attention_mask_supports_sdpa_supports_flex_attn_supports_flash_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr/rmr2r0r>rrCrnros@r4r*r*s &O&*#/0$(!N!!"&-. U]__ 6 6 6 6_ 6+EL++++"   VY\`V`        r3r*z{ The Parakeet Encoder model, based on the [Fast Conformer architecture](https://huggingface.co/papers/2305.05084). ceZdZUeed<dZdeffd Zeee e d de j de j dzde d eed ef d ZxZS) ParakeetEncoderr8encoderct|_d|_j|_j|_j|_jrtj j nd|_ t|_ t|_t!jfdt%jD|_|dS)NFrFc0g|]}t|Sr2)r).0rr8s r4 z,ParakeetEncoder.__init__..?s$ f f f !&) 4 4 f f fr3)r=r>r8rrdropout_positions layerdrop scale_inputrsqrtrJ input_scaler subsamplingr6encode_positionsrrrnum_hidden_layersr post_initr}s `r4r>zParakeetEncoder.__init__1s     &+#~ !'!9)>> from transformers import AutoProcessor, ParakeetEncoder >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> encoder = ParakeetEncoder.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> encoder_outputs = encoder(**inputs) >>> print(encoder_outputs.last_hidden_state.shape) ``` rNrr?rQrFT)r*r)last_hidden_stater*)r\r[r]rrrrrWrCrYrrZr^rr/randrXr)r) rBrr*r`rrOr output_mask encoder_layerto_dropdropout_probabilitys r4rizParakeetEncoder.forwardDsF((HH %(88 "33MBB --mt|VZVc-dd  m33 4#9DM4    %99.XeXklmXn9ooK(22155<0JOd0J;??,,,jn    r3)NT)r+r,r-r!r1rDr>rrrrr/r0rrrrrirnros@r4rQrQ(s "!!!!4&/3&* B B  B  t+B  $ B +, B  B B B _ ^B B B B B r3rQceZdZUdZejed<dZeej dzed<dZ eeej dzed<dZ eeej dzed<dS)ParakeetCTCGenerateOutputaz Outputs of Parakeet CTC model generation. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. sequencesNlogitsr-rO) r+r,r-r.r/ LongTensorr1rlr FloatTensorr-rOr2r3r4rjrjs&.2FE%# $t +2229=JeE-./$6===<@M5u012T9@@@@@r3rjc"eZdZdZfdZxZS)ParakeetGenerateOutputz` Deprecated alias for ParakeetCTCGenerateOutput. Use ParakeetCTCGenerateOutput instead. cntj|i|tddS)Nz`ParakeetGenerateOutput` is deprecated and removed starting from version 5.11.0; please use `ParakeetCTCGenerateOutput` instead.)r=r>logger warning_once)rBargsrrCs r4r>zParakeetGenerateOutput.__init__sG$)&))) O     r3)r+r,r-r.r>rnros@r4rprpsB         r3rpzS Parakeet Encoder with a Connectionist Temporal Classification (CTC) head. c6eZdZUeed<deffd Zee ddej dej dzdej dzde e de f d Z ej ddej dej dzd ed edzde e deejzf d ZxZS)ParakeetForCTCr8ct|tj|j|_t j|jj|j d|_ | dS)Nrr) r=r>r from_configr8rRrrrJ vocab_sizectc_headr_r}s r4r>zParakeetForCTC.__init__sh     ,V-BCC  &"7"CVEVdefff  r3Nrr*labelsrrDc H||dd|jd||d|}|j}||dddd}d}||jd} ||jjk} | d} | | } tj |dtjd d} tjjd 5tj | | | | |jj|jj|jj }dddn #1swxYwYt+|||j|j S)a Example: ```python >>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> outputs = model(**inputs) >>> print(outputs.loss) ```Nr`Trr*rrrQrrF)rU)blank reduction zero_infinity)lossrlrOr-r2) setdefaultrRrcrzr^r*rr8 pad_token_id masked_selectrr log_softmaxr/rbackendscudnnflagsctc_lossctc_loss_reductionctc_zero_infinityrrOr-)rBrr*r{rencoder_outputsrOrlrencoder_lengths labels_masktarget_lengthsflattened_targets log_probss r4rizParakeetForCTC.forwards:     5t < < <&$, ))     (9 }66q!<<==GG1MM  -<@@DDO!DK$<>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> predicted_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> print(transcription) ``` NT return_dictr}rQrVrrb)rkrlr-rOr2) get_compiled_call__call__rlargmaxrCrYr8rrjr-rO) rBrr*rrr model_forwardoutputsrks r4generatezParakeetForCTC.generate sBCQB\..~>>>bfbo $}"/-# ))# # # # N))b)11   %!<<^[d[jkl[m<nnN)-)AI~o & " ,#~"-%3  r3rk)NFN)r+r,r-r r1r>rrr/r0rrrrirmrr rjrmrrnros@r4rvrvsl 0/3&* B B  B  t+B  t# B +, B  B B B ^B HU]__/3(-/3 99 9 t+9"& 9 &, 9 +, 9 #U%5 5999_99999r3rvcZeZdZdZdeffd Z d dejdedzdej fdZ xZ S) ParakeetRNNTDecoderz'LSTM-based prediction network For RNN-Tr8cHt|j|_tj|j|j|_tj|j|j|j d|_ tj |j|j|_ dS)NT) input_sizerJr batch_first) r=r>blank_token_idr Embeddingrydecoder_hidden_size embeddingLSTMnum_decoder_layerslstmrudecoder_projectorr}s r4r>zParakeetRNNTDecoder.__init__Ks $3f&79STTG120     "$6+EvGa!b!br3N input_idscacherDc|7|dddf|jk}|jr|r|jS||}|-|j}|s|||j|jf}nd}|||\}\}} | |} |'|r|nd} | | || | |jS| S)NrQ)mask) ris_initializedrrrlazy_initialization hidden_state cell_staterrupdate) rBrr blank_mask embeddingswas_initializedhidden_cell_states lstm_outputrrdecoder_outputrs r4rizParakeetRNNTDecoder.forwardWs  "111b5)T-@@J# # (8(8 #{"^^I..   #2O" 6))*555"'"4e6F!G  !% 26))JHZ2[2[/ /lJ// <<  "1;J;;tD LLzL M M M; r3rj) r+r,r-r.r"r>r/rmr$r0rirnros@r4rrHs11 c1 c c c c c c26#($.  r3rcteZdZdZdeffd ZdejdejdeejejffdZ xZ S)ParakeetRNNTJointNetworkzPJoint network that combines encoder and decoder outputs to predict token logits.r8ctt|j|_t j|j|j|_ |j|_dSrj) r=r>rryrzrrurryheadr}s r4r>z!ParakeetRNNTJointNetwork.__init__{sO  !23If8&:KLL  +r3decoder_hidden_statesencoder_hidden_statesrDc\|||z}||Srj)rzr)rBrr joint_outputs r4riz ParakeetRNNTJointNetwork.forwards. 'r/r0rrirnros@r4rrxsZZ,1,,,,,, '$|' %|' u|U\) * ''''''''r3rcdeZdZUdZdZejdzed<dZejdzed<dZ e dzed<dS)ParakeetRNNTOutputa Output of the Parakeet RNN-T forward pass. Args: loss (`torch.FloatTensor`, *optional*): RNN-T loss, returned when `labels` are provided. logits (`torch.FloatTensor`): Joint token logits. Shape is `(batch, T, U+1, vocab)` for training or `(batch, 1, 1, vocab)` for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache containing hidden state, cell state, and last output. Nrrl decoder_cache) r+r,r-r.rr/rnr1rlrr$r2r3r4rrsd  &*D% d ")))'+FE  $+++59M+d299999r3rz? Parakeet Encoder with an RNN-T (RNN Transducer) head. cneZdZUeed<dgZejgZdeffd Z e dde j de j dzde edefd Zee dde j dzde j dzd e jdzd edzd edzd eee jzdzde j dzde edefdZxZS)ParakeetForRNNTr8rcht|tj|j|_t j|jj|j |_ t||_ t||_|j|_|dSrj)r=r>rrxr8rRrrurJrencoder_projectorrdecoderrjointmax_symbols_per_stepr_r}s r4r>zParakeetForRNNT.__init__s     ,V-BCC !#6+@+LfNh!i!i*622 -f55 $*$?! r3Nrr*rrDc d|jd||d|}||j|_|S)Nr}r2)rRrrc pooler_output)rBrr*rrs r4get_audio_featuresz"ParakeetForRNNT.get_audio_featuressR'$, ))     )-(>(>?`(a(a%r3decoder_input_idsruse_decoder_cacherr{c ||jd ||d|}|r|t|j}|||} ||jdddddddf| dddddddfd} d} ||jd} |j d | dddt| f|| ||jj kd|jj d|} t| | |j|j|j|j|S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForRNNT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-rnnt-0.6b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForRNNT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` Nr}rrrrrQ)rlr{ logit_lengths label_lengthsrrrlrcrrOr-rr2)rr$r8rrrsqueezer*r loss_functionrrArrrcrOr-) rBrr*rrrrr{rrrlrrs r4rizParakeetForRNNT.forwardsX  "5d5--O  B!64T[AAM $ ->m T T"1"?111dAAA "N"74AAA "F   '!**    +:>>rBBM%4%aaa!;3}'8'8':':#;#;!;;<+%)CCHHLL#{9   D"-?)7)7&1'    r3rjNNNNNNN)r+r,r-r"r1rHr GREEDY_SEARCH_supported_generation_modesr>rr/r0rrr)rrrmr$rrrnrrirnros@r4rrs ./#1#?"@1/3     t+ +,  $    /3.2599=)-X\&*N N  t+N  t+N !+d2 N 0$6 N  $; N 4eErrurrylen durationsrr}s r4r>z ParakeetTDTJointNetwork.__init__sI    If8&:KcRXRbNcNc:cdd r3)r+r,r-r.r#r>rnros@r4rrsZ e0eeeeeeeeeer3rzG Parakeet Encoder with a TDT (Token Duration Transducer) head. ceZdZUeed<deffd Zee ddej dzdej dzdej dzde dzde dzd e eejzdzd ej dzd eed efd ZxZS)ParakeetForTDTr8ct|t||_|dSrj)r=r>rrr_r}s r4r>zParakeetForTDT.__init__(s@    ,V44  r3Nrr*rrrrr{rrDc ||jd ||d|}|r|t|j}|||} ||jdddddddf| dddddddfd} d} ||jd | dd|jjf| d|jjdf||j d||jj k d|jj |jj d|} t| | |j|j|j|j| S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForTDT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-tdt-0.6b-v3" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForTDT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` Nr}rrr.rQ) token_logitsduration_logitsr{rrrrrr2)rr$r8rrrrrryr*rrrrrrcrOr-) rBrr*rrrrr{rrrlrs r4rizParakeetForTDT.forward.sX  "5d5--O  B!64T[AAM $ ->m T T"1"?111dAAA "N"74AAA "F   '!**    %4% #C)A4;+A)A$AB &sDK,B,D,D'D E-<@@DD%)AAFFrJJ#{9+/    D"-?)7)7&1'    r3r)r+r,r-r#r1r>rrr/r0rmr$rr)rrnrrrrirnros@r4rr sE 0 /3.2599=)-X\&*O O  t+O  t+O !+d2 O 0$6 O  $; O 4eErs3* $$$$$$!!!!!! &&&&&&!!!!!!HHHHHHHHHHIIIIIIII999999[[[[[[[[[[FFFFFFFF&&&&&&______________GGGGGGGG555555ssssssssssssrrrrrrrrrr  H % %   / / / / /!; / /   //7/7/7/7/729/7/7/7d        E-E-E-E-E-ryE-E-E-P(((*++,+2 UU\ U# U%, U U U U&%% I% <% % < % L4' %  %%'(%%%%2)**_ _ _ _ _ ry_ _ +*_ DBBBBBryBBBJ,,,,,5,,,^;;;;;o;;;| ] ] ] ] ] -] ]  ] @ AAAAA AA A4       6       KKKKK,oKK  K\-----")---`'''''ry'''$ :::::3:: :& n n n n n -/Jn n  n b e e e e e6 e e e Z Z Z Z Z /Z Z  Z z p o or3