tjDdZddlZddlmZddlZddlZddlmZddlm Z ddl m Z ddl mZdd lmZmZmZdd lmZdd lmZdd lmZdd lmZddlmZmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0ddl1m2Z2ddl3m4Z4e+j5e6Z7dZ8dLde9de9de:dej;fdZe9e9fd"e:d#e9d$ej?dzd%e9dej@f d&ZAGd'd(ejBZC dNd*ejDd+ej;d,ej;d-ej;d$ej;dzd.e:dzd/e:fd0ZEGd1d2ejDZFGd3d4eZGGd5d6eZHe)Gd7d8e$ZIGd9d:eIZJGd;deIZLe)d?@GdAdBe4eIZMGdCdDeIZNe)dE@GdFdGeIeZOe)dH@GdIdJeIZPgdKZQdS)OzPyTorch Whisper model.N)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs) WhisperConfig)WhisperGenerationMixin'lengthchannels max_timescalereturnc|dzdkrtd|dtj||dzdz z }tj| tj|dzz}tj|dd|ddz}tj|| gdS)z*Returns sinusoids for positional embeddingrzVNumber of channels has to be divisible by 2 for sinusoidal positional embeddings, got z channels.r dim) ValueErrormathlogtorchexparangeviewcatsincos)r$r%r&log_timescale_incrementinv_timescales scaled_times j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/whisper/modeling_whisper.py sinusoidsr;7s!|q yem y y y   #h}55Q9JKY 77%,xST}:U:UUVVN,v&&++B22^5H5HB5O5OOK 9koo''):):; C C CC input_ids pad_token_iddecoder_start_token_idc||j}|ddddf|ddddf<||dddf<|td||dk||S)z1 Shift input ids one token to the right. Nr*r rz1self.model.config.pad_token_id has to be defined.i) new_zerosshapecloner- masked_fill_)r=r>r?shifted_input_idss r:shift_tokens_rightrFDs"++IO<<(CRC06688aaae4aaadLMMM""#4#.compute_num_masked_span{s~i,6DwNOOoy99 [ (? : :-z)_compute_mask_indices..s 9 9 9!o 9 9 9r<)dtyperF)replace)r-nprandomranditemdetachsumtolistrangezerosboolchoicer2len concatenateonesint32appendarray broadcast_toreshaperOput_along_axis)rBrGrHrIrJ batch_sizerT input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrPrQspec_aug_mask_idxdummy_mask_idxoffsetsrRrSs `` ` @@r:_compute_mask_indicesrxUsP0#(JQABBB_$$ :^i : :'6 : : :   innQ$$&&G         $  % ##B''..000 9 9 9 9uZ'8'8 9 9 9Hj/:$GGGM11/BBa%55 11,??I,, IlkAo6 7 7RW-    ! !Q & &-q0NN.q1NN (;o(MUWU] ^ ^ ^ao o p   !!"34444"455111aaa:&5H+(V,33J@SVa@abbi $$T4]3Gog 4G'UVV^^'+5G,g5/A"555GVYZGZ-!0CCDm%7B??? r<c<eZdZddedededzffd Zd dZxZS) WhisperPositionalEmbeddingN num_positions embedding_dim padding_idxcLt||dSN)super__init__)selfr{r|r} __class__s r:rz#WhisperPositionalEmbedding.__init__s#  66666r<rcZ||j|||jdzS|j|SNr )weightrB)rr=past_key_values_length position_idss r:forwardz"WhisperPositionalEmbedding.forwards8  ;58NQZQ`abQc8ccd d;|, ,r<r)rN)__name__ __module__ __qualname__rNrr __classcell__rs@r:rzrzsp77c7#7CRVJ777777--------r<rzmodulequerykeyvaluescalingdropoutc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nr*r)rr+ptrainingr ) sizer0matmul transposer functionalsoftmaxrr contiguous) rrrrrIrrkwargs attn_weights attn_outputs r:eager_attention_forwardrs**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r<ceZdZdZ ddededed ed ed ed edzd edzffd Z dde j de j dzde dzde j dzdede e dee j e j dzee j dzffdZxZS)WhisperAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderbias is_causal layer_idxconfigc pt||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ |*|r(t d|j jd||_t!j||d|_t!j||||_t!j||||_t!j||||_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr)rrrrrhead_dimrr-rrrlogger warning_oncerrrrLineark_projv_projq_projout_proj) rrrrrrrrrrs r:rzWhisperAttention.__init__sW "" !Y.  MI %$. 8 83dn33%.333 }d* $"      ,4>+B,,,    #i 95AAA i 94@@@ i 94@@@  )YTBBB r< hidden_stateskey_value_statespast_key_valuesrIoutput_attentionsrr'c |du}|jdd}g|d|jR} |||jz| dd} |Tt|tr?|j |j } |rd|j |j <|j }n|j }||n|} |r3|r1| r/|j|j j} |j|j j}n|dd|j|jf}|| |dd} || |dd}||| ||j \} }t+j|jjt2}||| | ||f|jsdn|jd|d |\}}|jg|dR}||}||fS) z#Input shape: Batch x Time x ChannelNr*r r)Trr?)rrr)rBrrrr3rr isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrrupdater get_interfacer_attn_implementationrrrrnr)rrrrrIrris_cross_attention input_shape hidden_shape query_statesrcurrent_states key_states value_stateskv_shapeattention_interfacerrs r:rzWhisperAttention.forwards.T9#)#2#. 88b8$-88  M22T\AGG UU__`acdeepprr   &:oGZ+[+[ &(377GGJ! G=A*4>:"1"G"1"F.>-I))}  l/ lj l(/?DJ*1$.AHLL $ADNDMJH^4499(CCMMaQRSS^^``J;;~66;;HEEOOPQSTUU``bbL*+:+A+A*l\`\j+k+k( L(?(M K ,.E) ) %8$7      % $}>CC$,/ % %  % % ! \*k);;;;;;FFHH mmK00 L((r<)rFTFNN)NNNF)rrr__doc__rNfloatrer!rr0Tensorr rrtuplerrrs@r:rrsyGG   $'+&C&C&C&C &C  &C  &C&C:&C$&C&C&C&C&C&CV15(,.2"' H)H)|H) ,-H) H)  t+ H)  H)-.H) u|U\D0% 2E2LL MH)H)H)H)H)H)H)H)r<rcfeZdZdeffd ZdejdejdeedejfdZ xZ S)WhisperEncoderLayerrct|j|_t |j|j|j||_tj |j|_ |j |_ t|j |_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)rrrr)rrd_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrractivation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normrrrs r:rzWhisperEncoderLayer.__init__is )n4,     %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r<rrIrr'c |}||}|jd||d|\}}tj||j|j}||z}|}||}|||}tj||j |j}| |}tj||j|j}||z}|j tj kr9tj|j jdz }tj|| |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rrIri)minrOrV)rrrrrrrrrrrrZr0float16finforOclamp)rrrIrresidualrX clamp_values r:rzWhisperEncoderLayer.forward{s[!11-@@ )4> ')     q  --mt|VZVc-dd  =0  --m<< **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0  %- / /+m&9::>EK!K K<[YYYMr<) rrrr!rr0rrrrrrs@r:rrhs=}======$"|" "+, "  """"""""r<rceZdZddededzffd Z ddejdejdzdejdzd ejdzd edzd e dzd e e d ejfdZ xZ S)WhisperDecoderLayerNrrc t|j|_t |j|j|jdd|||_|j|_t|j |_ |j |_ tj|j|_t |j|j|jd|||_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rrrrrrr)rrrr)rrrrrdecoder_attention_headsrrrrrrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrr)rrrrs r:rzWhisperDecoderLayer.__init__s  )n4,   ~ #F$>?"(";$&L$@$@!, N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r<TrrIencoder_hidden_statesencoder_attention_maskr use_cacherr'c |}||}|j|f||d|\}} tj||j|j}||z}|]|}||}|j|f|||d|\}} tj||j|j}||z}|}||}| | |}tj||j |j}| |}tj||j|j}||z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )rrIrN)rrIr) rrrrrrrrrrrrr) rrrIrrrrrrrXs r:rzWhisperDecoderLayer.forwards*!11-@@ *4>  +)      q  --mt|VZVc-dd  =0  ! ,$H 88GGM0t0 !65 /        M1M11-4 > `{1 `v3SDKNr<rceZdZdZeedZdZdeffd Z dZ de j fdZ d e j fd Zee dd eedefd ZxZS)rz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`WhisperEncoderLayer`]. Args: config: WhisperConfig )r attentions)rrctj|_j|_j}j|_j|_j |_ j rtj |nd|_ tj|j|dd|_tj||ddd|_tj|j ||_|jdtjfdt-jD|_tjj|_d|_|dS) Nrrr ) kernel_sizepaddingr))rstriderFc.g|]}tSrV)r)rWrXrs r:rYz+WhisperEncoder.__init__..<s"$g$g$gQ%8%@%@$g$g$gr<)rrrencoder_layerdrop layerdropr num_mel_binsr>r}max_source_positionsscale_embeddingr.sqrt embed_scalerConv1dconv1conv2 Embeddingrrequires_grad_ ModuleListrcencoder_layersrr layer_normgradient_checkpointing post_init)rrrrs ` r:rzWhisperEncoder.__init__+sB    ~ 1N "/!.$*$?!393IR49Y///sYt0)TUVVV Yy)1VWXXX !|D,EyQQ ++E222m$g$g$g$g%PVPeJfJf$g$g$ghh ,v~66&+# r<cP|D] }d|_ d|_dSNF) parameters requires_grad_requires_grad)rparams r:_freeze_parametersz!WhisperEncoder._freeze_parametersCs4__&& ( (E"'E  #r<r'c|jSrr'rs r:get_input_embeddingsz#WhisperEncoder.get_input_embeddingsHs zr<rc||_dSrr8rrs r:set_input_embeddingsz#WhisperEncoder.set_input_embeddingsKs  r<Nrc |jj|jjdz|jjdz}|jd|kr$t d|d|jdd|dtj ||}tj ||}| ddd}tj |j j|j }|| |z}tj||j|j }t%|jD];\}} d } |jr!tjg} | |jkrd } | s | |d fi|}<||}t/|S)a0 Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Whisper does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rr*z7Whisper expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .r)r devicerFTN)last_hidden_state)rr"r'rr(rBr-rrgelupermuter0r2rnum_embeddingsrArr enumeraterr^r r-r) rrrIrexpected_seq_length inputs_embeds all_positionsridx encoder_layerto_dropdropout_probabilitys r:rzWhisperEncoder.forwardNs,#k>ARSTAUUX\XbXijkXll   #': : :IJ]IIkykAClDIIsFIII  **4::n+E+EFF  **4::m+D+DEE %--aA66  T%9%HQ^Qefff %(<(<](K(KK  --mt|VZVc-dd "+DK"8"8   CG} #&+jnn#&77"G  - !!!!!  66 +    r<r)rrrrrr_can_record_outputsrr!rr6rModuler:r=rrrrrrrrs@r:rrs-&"}0$$$ bi") 6 6 +, 6  6 6 6 _ 6 6 6 6 6 r<rceZdZdZeeeddeedddZdZdZ d e ffd Z e e dd eed efdZxZS)WhisperDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`WhisperDecoderLayer`] Args: config: WhisperConfig r r)index layer_namer)rrcross_attentionsr=)rrctj|_j|_j|_j|_j|_j rtj j nd|_ tjjj |j|_t%|jj |_tjfdt+jD|_tjj |_d|_|dS)Nrc0g|]}t|SrV)r)rWrrs r:rYz+WhisperDecoder.__init__..s$ b b b  3 3 b b br<F)rrrdecoder_layerdropr r>r}max_target_positionsr"r#r.r$rr%rr) vocab_size embed_tokensrzrr+rcdecoder_layersrrr-r.r/rs `r:rzWhisperDecoder.__init__s    ~ 1!.$*$?!$*$?!8>8NW49V^444TWL):FNDL\]]9$:SU[Ucddm b b b bU6K`EaEa b b b   ,v~66&+# r<Nrr'c &|du|duz rtd|||}|r[|Y| |jjr6t t |jt |jnt |j}||nd} |]tj|j d|j | z}| d |j dd}|| || |} n| || |} || |j z} tj| |j|j} t'|j|||| } t)|jD]?\} }|jr tjg}||jkr,|| | |fd|r|nd|d |} @|| } t3| | S) ax Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`WhisperTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`EncoderDecoderCache` or `tuple(tuple(torch.FloatTensor))`, *optional*): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time)rrr r@)rrr)rrHrIrr)rrr)rBr)r-rZris_encoder_decoderr r get_seq_lengthr0r2rBrA unsqueezerepeatrtorrrrr rFrr^r r-r)rr=rIrrrHrrrr positionsr causal_maskrJ decoder_layerrMs r:rzWhisperDecoder.forwardsZ  -t"; < ustt t   --i88M  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  FUE`!?!?!A!A!Afg   < (;A(>}G[\\\_uuL'11!44;;M~G\didn o o o  1!!!T' : A A"kSU V V 01N, - ; (1 , , ,#8[)+7 K;+< $$$ $)<0D^Mbjojt#u#u#u 34N/ 0r<decoder_input_idsdecoder_attention_maskencoder_outputsrdecoder_inputs_embedsdecoder_position_idsrr'c |&|||}|j|fi| }nct|tsNt|dt |dkr|dndt |dkr|dnd}|jd |||d|||| d| } t | j| j| j | j | j |j|j |j S) ay decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`WhisperTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Whisper uses the `decoder_start_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_whisper._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the BART paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, WhisperModel >>> from datasets import load_dataset >>> model = WhisperModel.from_pretrained("openai/whisper-base") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("openai/whisper-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_features = inputs.input_features >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 512] ```N)rIrr r)rBrr)r=rIrrrHrr)rBrdecoder_hidden_statesdecoder_attentionsrTencoder_last_hidden_staterencoder_attentionsrV) rrirrrgrjrrBrrrrT) rrrIrrrrrrrrdecoder_outputss r:rzWhisperModel.forward`sDp  "!66~Vd6eeN*dlOOO_== -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO'$,  '1"1!"4+/-      "-?+;"1"?.9,=&5&G"1"?.9    r<r) NNNNNNNNN)rrrr!rr:r=rsr0 FloatTensorrrrrrr rerrrrrs@r:rgrgs})))******37))))(4/))))V482659:>BF(,AE?C!%Y Y )D0Y (4/Y !+d2 Y !& 04 7 Y uU%6784? Y Y  %U%67$>Y $E$45<Y $;Y  u| 1 1Y Y Y ^Y Y Y Y Y r<rgzh The Whisper Model with a language modeling head. Can be used for automatic speech recognition. ) custom_introceZdZdZddiZdeffd ZdZdZde j fd Z d Z e e dd ejd zd ejd zdejd zdejd zdeeejd zded zdeejd zdeejd zdejd zded zdeejezfdZxZS)WhisperForConditionalGenerationrproj_out.weight!model.decoder.embed_tokens.weightrct|t||_t j|j|jd|_|j |_ | dSNFr) rrrgrrrrrYproj_outrXr/rs r:rz(WhisperForConditionalGeneration.__init__sj    !&))  &.&2C%PPP $*$?! r<c|jSrrr9s r:get_output_embeddingsz5WhisperForConditionalGeneration.get_output_embeddings }r<c||_dSrrrnew_embeddingss r:set_output_embeddingsz5WhisperForConditionalGeneration.set_output_embeddings & r<r'c4|jSrrr:r9s r:r:z4WhisperForConditionalGeneration.get_input_embeddingsz..000r<cB|jjdSrq)rrir6r9s r:rsz.WhisperForConditionalGeneration.freeze_encoders! --/////r<NrrIrrrrrrlabelsrc z| e| jd|jkr&td| jdd|jd|'|%t| |jj|jj}|j|f|||||||| d| } || j } d}| et}| | j } || d|jj| d}t!|| | j| j| j| j| j| j| j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`WhisperTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Whisper uses the `decoder_start_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_whisper._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the BART paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. `sequence_length` should be smaller than or equal to `config.max_target_positions`. Example: ```python >>> import torch >>> from transformers import AutoProcessor, WhisperForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_features = inputs.input_features >>> generated_ids = model.generate(inputs=input_features) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription ' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```Nr zLabels' sequence length z- cannot exceed the maximum allowed length of z tokens.)rIrrrrrrrr*) losslogitsrrrrTrrr)rBrXr-rFrr>r?rrrBrrarAr3rYrnrrrrrTrrr)rrrIrrrrrrrrroutputs lm_logitsrloss_fcts r:rz'WhisperForConditionalGeneration.forwardsD  |A!::: Qv|AQQmqnGQQQ!(-B-J$6DK4dk6X%%!'1dj  ' )/+#9+"7!5 ' '  ' ' MM'";<<   '))HYYy/00F8INN2t{/EFFWYHZHZ[[D#3")"?&9$5&-&G")"?&9    r<) NNNNNNNNNN)rrrr _tied_weights_keysr!rrrrrOr:rsrrr0rrrr rerrrrrs@r:rrs  +-PQ}'''1bi1111000482659:>BF(,AE?C*.!%i i )D0i (4/i !+d2 i !& 04 7 i uU%6784? i i  %U%67$>i $E$45<i  4'i $;i  u|  .i i i ^i i i i i r<rc4eZdZdZfdZdZdZdZxZS)WhisperDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ct|d|_t||_|dSr1)rrr]rQrjr/rs r:rzWhisperDecoderWrapper.__init__TsH    $)!%f--  r<c|jjSrrlr9s r:r:z*WhisperDecoderWrapper.get_input_embeddingsZrmr<c||j_dSrrlr<s r:r=z*WhisperDecoderWrapper.set_input_embeddings]ror<c|j|i|Sr)rj)rargsrs r:rzWhisperDecoderWrapper.forward`st|T,V,,,r<) rrrrrr:r=rrrs@r:rrNso  )))***-------r<rzx Whisper decoder with a language modeling head on top (linear layer with weights tied to the input embeddings). ceZdZddiZdZfdZdZdZdej fdZ d Z e e ddejd zd ejd zd eejd zd ed zdejd zdejd zded zdeezfdZxZS)WhisperForCausalLMrrr=ct|d|_t||_t j|j|jd|_ | dSr) rrr]rrrrrrYrr/rs r:rzWhisperForCausalLM.__init__msh    $)!*622  &"4f6GeTTT  r<c|jSrrr9s r:rz(WhisperForCausalLM.get_output_embeddingswrr<c||_dSrrrs r:rz(WhisperForCausalLM.set_output_embeddingszrr<r'c4|jSrrr9s r:r:z'WhisperForCausalLM.get_input_embeddings}rr<c:|j|dSr)rr=r<s r:r=z'WhisperForCausalLM.set_input_embeddingss ''.....r<NrIrrrHrrc t|tttfr|d}|jjd||||||d|} || d} d} |e|| j}t} | | d|j j | d} t| | | j| j| j| jS)aF encoder_outputs (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import WhisperForCausalLM, WhisperForConditionalGeneration, WhisperProcessor >>> import torch >>> from datasets import load_dataset >>> processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2") >>> assistant_model = WhisperForCausalLM.from_pretrained("distil-whisper/distil-large-v2") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> sample = ds[0]["audio"] >>> input_features = processor( ... sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt" ... ).input_features >>> predicted_ids = model.generate(input_features, assistant_model=assistant_model) >>> # decode token ids to text >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0] >>> transcription ' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.' ```r)r=rIrrrHrNr*)rrrrrrTrV)rrrlistrrjrrarArr3rrYrrrrrT) rr=rIrrrHrrrrrrrs r:rzWhisperForCausalLM.forwards ` o'E F F 1-a0O%$*$ )"1+'     wqz**  YYv}--F'))H8FKKDK,BCCV[[QS__UUD0#3!/)$5     r<re)rrrrrrrrrrOr:r=rrr0rrrrr rerrrrs@r:rrdsu ,-PQ!O'''1bi1111///.2.2;?(,26*.!%K K #d*K  t+K u01D8 K  K (4/ K  4'K $;K  2 2K K K ^K K K K K r<rz Whisper Encoder Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c eZdZfdZdZdejfdZdejfdZe e d de j dzd e e e jdzd e j dzde e jezfd ZxZS) rct|t||_|jdz}|jr.t jtj ||z |_ t j |j |j |_t j |j |j|_|dSr)rrrrir rr Parameterr0rirrrclassifier_proj_size projector num_labels classifierr/)rr num_layersrs r:rz&WhisperForAudioClassification.__init__s    %f-- -1  ( S!#ej.D.Dz.Q!R!RD 6#5v7RSS)F$?ARSS r<c8|jdS)z Calling this function will disable the gradient computation for the Whisper encoder so that its parameters will not be updated during training. Only the projection layers and classification head will be updated. Nrrr9s r:rsz,WhisperForAudioClassification.freeze_encoderrtr<r'c4|jSr)rir:r9s r:r:z2WhisperForAudioClassification.get_input_embeddingss|00222r<rc:|j|dSr)rir=r<s r:r=z2WhisperForAudioClassification.set_input_embeddingss ))%00000r<Nrrrc |jjrd|d<||j|fi|}nct|tsNt |dt |dkr|dndt |dkr|dnd}|jjrx|t }tj|d}tj |j d }|| d ddzd}n|d}||}|d}||}d} |et%} ||j}| | d |jj| d } t-| ||j|j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, WhisperForAudioClassification >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("sanchit-gandhi/whisper-medium-fleurs-lang-id") >>> model = WhisperForAudioClassification.from_pretrained("sanchit-gandhi/whisper-medium-fleurs-lang-id") >>> ds = load_dataset("google/fleurs", "all", split="validation", streaming=True) >>> sample = next(iter(ds)) >>> inputs = feature_extractor( ... sample["audio"]["array"], sampling_rate=sample["audio"]["sampling_rate"], return_tensors="pt" ... ) >>> input_features = inputs.input_features >>> with torch.no_grad(): ... logits = model(input_features).logits >>> predicted_class_ids = torch.argmax(logits).item() >>> predicted_label = model.config.id2label[predicted_class_ids] >>> predicted_label 'Afrikaans' ```Toutput_hidden_statesNrr r)rr+r*)rrrr)rrrirrrg_HIDDEN_STATES_START_POSITIONr0stackrrrrr3rarmeanrrrarArrrr) rrrrrr norm_weights pooled_outputrrrs r:rz%WhisperForAudioClassification.forwardsT ; - 2-1F) *  "*dlOOO_== -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO ; - /+,IJM!K 1===M=001C0LLL*\->->r1a-H-HHMMRSMTTMM+A.M}55 %**q*11 //  '))HYYv}--F8FKKDK,BCCV[[QS__UUD')7&1     r<)NNN)rrrrrsrrOr:r=rrr0rrrrrrrrs@r:rrs     ***3bi33331")111137BF*. P P (4/P uU%6784?P  4' P u| 7 7 P P P ^P P P P P r<r)rrrgrr)r#)Nr)Nr)Rrr.collections.abcrnumpyr\r0rtorch.nnrrr activationsr cache_utilsr r r generationr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrrconfiguration_whisperr!generation_whisperr" get_loggerrrrrNrrr;rFrrndarrayrxr)rzrOrrrrrrrQrgrrrr__all__rVr<r:rsJ $$$$$$ %%%%%%&&&&&&!!!!!!CCCCCCCCCC))))))//////:99999GFFFFFFF&&&&&&RRRRRRRRRRRR777777EEEEEEEE000000666666  H % % ! D Dc DS D D5< D D D D %, c [^    */3 tt c?ttt$t+ t  t Z ttttn--------"!%% I% <% % < % L4' % T\ %%%%%4s)s)s)s)s)rys)s)s)n555554555pYYYYY4YYYx_>j j j j j +j j j ZR R R R R +R R R j\ \ \ \ \ )\ \ \ ~ H H H H H &<>TH H  H V-----2---, g g g g g /g g  g T m m m m m $:m m  m `   r<