tjԪ`ddlZddlZddlZddlmZmZddlZddlZddl m cm Z ddlm Z ddl mZddlmZmZddlmZmZmZmZmZddlmZdd lmZdd lmZd d lmZm Z ej!e"Z#d ej$de%dej$fdZ&dej'fdZ(dZ) ddZ*GddeZ+dS)N)CallableIterator)nn)EncoderDecoderCache)GenerationConfigGenerationMixin)LogitsProcessorList$SuppressTokensAtBeginLogitsProcessorSuppressTokensLogitsProcessorWhisperNoSpeechDetectionWhisperTimeStampLogitsProcessor)StoppingCriteriaList)BaseModelOutput)logging)TASK_IDSTO_LANGUAGE_CODEinputs filter_widthreturnc,|dks |dzdkrtd|dz}|jd|kr|Stj|||ddfd}|d|ddd|f}|S) z Applies a median filter of width `filter_width` along the last dimension of the input. The `inputs` tensor is assumed to be 3- or 4-dimensional. rrz&`filter_width` should be an odd numberreflect)mode.) ValueErrorshaper functionalpadunfoldsort)rr pad_widthresults l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/whisper/generation_whisper.py_median_filterr&+s qL1,11ABBB!I |B9$$ ]  v 9a'C)  T TF]]2|Q / / 4 4 6 6q 9#y. IF Mmatrixc8|j\}}tj|dz|dzftjtjz}tj|dz|dzftj }d|d<t d|dzD]}t d|dzD]q}||dz |dz f}||dz |f}|||dz f} ||kr || kr|d} } n||kr || kr|d} } n| d} } ||dz |dz f| z|||f<| |||f<r|jddz }|jddz }d|dddf<d|dddf<g} g} |dks|dkr| |dz | |dz |||fdkr |dz}|dz}n>|||fdkr|dz}n*|||fdkr|dz}ntd|d|d |dk|dktj| ddd } tj| ddd } | | fS) z Measures similarity between two temporal sequences: the input audio and the output tokens. Used to generate token-level timestamps. rdtyper)rrrNz9Internal error in dynamic time warping. Unexpected trace[, z]. Please file a bug report.r) rnponesfloat32infrangeappend RuntimeErrorarray)r( output_length input_lengthcosttracejic0c1c2ct text_indices time_indicess r%_dynamic_time_warpingrB@s #),M< 7MA%|a'78 K K Kbf TD Wma')9:"* M M M MEDJ 1lQ& ' 'q-!+,,  Aa!eQUl#Ba!eQhBaQhBBww27711bR"WW1111Aq1u -1DAJE!Q$KK   AA AAE!QQQ$KE!!!Q$KLL a%%1q55AE"""AE""" A;!   FA FAA 1a4[A   FAA 1a4[A   FAApAppQRppp  a%%1q558L))$$B$/L8L))$$B$/L  %%r'cj|/tfd|Dd}|rt||dSdS)Nc3<K|]}t||VdSN) isinstance).0clslogit_processor_classs r% z2_get_attr_from_logit_processors..xs3jj:cShCiCijjjjjjjr')nextgetattr)logits_processorrIattribute_namelogit_processors ` r%_get_attr_from_logit_processorsrPvsO#jjjj/?jjjlpqq  B?NDAA A 4r'rightlongestFc d} g} g} |dvrtd||dvrtd||dkr|td|rg}g}|D]d}|dd }|t|tjr|n|d |r||d etj|d } |rtj|d }| |fS| S|D]}|{t d |Ddkr]g}|D]n}| rOt |ddkr6|dd| kr$||dddS||dotj|d }|r tjd|Dd }||| d}|r || d}|Etj||g}|r-tjtj||dz|g}| ||r| |t| t | d} |E| ||r,| tj||dz| tj g||r)| tj g||dkr|dzn| } tt |D]}| t | |z }|dkrd|fn|df}tj | |||| |<|rGtj | ||t | |dkr| |dnd| |<tj| d } |rtj| d }| |fS| S)a skip_ending_double_timestamps: when the segment ended with two timestamp tokens, whether to ignore the last timestamp token see https://github.com/huggingface/transformers/pull/35750 _pad_to_max_length is used in different contexts: 1. At the end of generation: we need to keep both ending timestamp tokens in the segment (see https://github.com/huggingface/transformers/pull/34537). 2. In the middle of generation, e.g. when condition_on_prev_tokens=True and we want to use the last generated tokens as decoder_input_ids: we must skip one of the double ending timestamp tokens (see https://github.com/huggingface/transformers/pull/35750). r)rQleftz5`padding_side` must be either 'right' or 'left', not )rR max_lengthz8`padding` must be either 'longest' or 'max_length', not rUNz>`cut_off_length` must be specified when `padding='max_length'`r$ sequencestoken_timestampsdimcg|] }|d StokensrGds r% z&_pad_to_max_length..s4_4_4_QQx[4_4_4_r'r\rrcjg|]0}|dd|dd|dd1S)r$rWidxsrrr]r^s r%r`z&_pad_to_max_length..s=pppVWQx[!34QvYq\AfIaL5PQpppr'devicerrQ)r value)rr2rFtorchTensorstacklencat ones_likemaxtensorr1Fr )current_segments pad_token_idre padding_sidepaddingbos_token_tensorcut_off_lengthreturn_token_timestampsforce_unique_generate_callskip_ending_double_timestampstimestamp_beginmax_total_lengthrVtoken_timestamps_listsequences_listtimestamps_listsegmentsr$rWcurrent_segment_listr_sequencer: pad_lengthr s r%_pad_to_max_lengthr~s,I,,,_Q]__```///]T[]]^^^ L ^%;YZZZ! ( C CHa[*F  ! !Jvu|,L,L"e&&RXYdRe f f f& C&&v.@'ABBBKA666 " /${?BBB .. . 0)N)N  +4_4_J^4_4_4_0`0`cd0d0dN) 7 707S85E5E5I5IaPXkZ\oapNpNp#))!H+crc*:;;;;"))!H+6666yR888H& #(9pp[oppp$$$ )#^O$4$45*J'78H8H'I$+ 9&6%ABB*',y)9&IIICOQab(($   X & & && ?%,,-=>>>"#3S25G5GHH    )   - . . .& e%,,U_=MV\-]-]-]`c-cddd   U\"V<<< = = =& N%,,U\"V-L-L-LMMM-4 -D-D~))JZ 3'(( ) )  %IaL(9(99 !-!8!8q*ooz1ouYq\s,GGG ! " '(u%a(69:OPQ:R6S6SVW6W6W+A.r22]`((( !! $  I1---I ;'Z-ed?Z.ed@Z/xZ0S)DWhisperGenerationMixin{Gz?Nc T gt|jjD]<t jfd|jDd=t jfd|Dgdd}d|vr|j dk d }|j ddd|f}|k|d kre||d z z }t j |j d |d z |jtj |ddd d fz}t j||gdn|dkd t jfd tj d Dd|pd j d} |jj d } t j| | d zftj|jj } |Kt)|t*rdd|dzfn%t)|t,t.t0jfr;t5t1j|d krdd|d dzfnt)|tjr;t5t j|d krdd|d dzfnnt)|t*r| n| t5|z} t)|tjr|n|}t1j|| }|dddd|dddfj dd kr| S|t)|t*rgt jddd} t j dd}|z | z tC|jj" d t| D]}|t)|t.t,t0jtjfr~|dd||dzf}t j|ddd} t j |dd}||z | z }tC||jj"}| d }n|}tG|$% \}}t1j&t1j'|dd (tR}|||z}t jt j|t j*|t j*|dgg| |<| S)a Calculates token-level timestamps using the encoder-decoder cross-attentions and dynamic time-warping (DTW) to map each output token to a position in the input audio. If `num_frames` is specified, the encoder-decoder cross-attentions will be cropped before applying DTW. Returns: tensor containing the timestamps in seconds for each predicted token c g|] }| Sr]r])rGxr:s r%r`zDWhisperGenerationMixin._extract_token_timestamps..s._._._qt._._._r'rrXc:g|]\}}|dd|fSrEr])rGlhcross_attentionss r%r`zDWhisperGenerationMixin._extract_token_timestamps..s0UUUTQ/2111a48UUUr')rrrrN beam_indicesrrrrer+c rg|]3}tjdddd|ddfddd|f4S)Nr)rYindex)rh index_select)rGr:unrolled_beam_indicesweightss r%r`zDWhisperGenerationMixin._extract_token_timestamps..(sg&wqqq!!!Qz':I^_`_`_`bc_cIdeeer'r+re.raTF)rYkeepdimunbiased)rYr)rr)constant_values)+r1configdecoder_layersr2rhrlrrjpermutersumrnr.rrelong masked_fillrVzerosr/rFintlisttupler-ndarrayrkuniquericpurepeatstdmeanr&median_filter_widthrBdoublenumpyr diffastypeboolro)selfgenerate_outputsalignment_headstime_precision num_frames num_input_ids weight_lengthr beam_indices_first_step_unrolledr6 batch_size timestamps repeat_timerr batch_idxr(r@rAjumps jump_timesrr:rrs @@@@r%_extract_token_timestampsz0WhisperGenerationMixin._extract_token_timestampsst{122 i iA  # #EI._._._._=M=^._._._ef$g$g$g h h h h+UUUU_UUUVV//,,,// - - -.:b@EEbIIMMOOM+8N]N9JKL (]Q->->!22 J|1!4ma6GP\Pckpkuvvv#AAAqsF+-1). 3SUa2bhj(k(k(k%%(4%%:$E$EF[_aFacd$e$e !k"#8#>q#ABB G%D(8(;(A!(D %/5a8 [ ) *%-HXHbHi     ! *c** @!#'8q'8"89Jubj(ABB @s29U_K`K`GaGaefGfGf!#';A!);';";<J77 @C Z@X@X @S@S@U@U@[@[@]@]?])^)^ &L,F27<00&!LLLSSTXYYE%e,~=J%*I]++U\*-E-Eu|U_`bUcTdGeGef%%Jy ! !r'F{Gz?input_featuresgeneration_configrMstopping_criteriaprefix_allowed_tokens_fn synced_gpusreturn_timestampstasklanguageis_multilingual prompt_idsprompt_condition_typecondition_on_prev_tokens temperature.compression_ratio_thresholdlogprob_thresholdno_speech_thresholdnum_segment_framesattention_maskrtime_precision_featuresrwreturn_segmentsreturn_dict_in_generaterxmonitor_progressc @A|j|fi|\}}|jjjjd|jjjjdz}||jjz}||||\}}||k}| ||||}| |||} | | || || ||||| ||||| ||| |||||j|| }!|| |rd |jj_d |vr|d djn|j}"|!jd }#|dt+|dr|j|jnd }$d|vrd|_||||#|$|"}|| |t5|t6t8fs|gn|}%|%d}|||||\}&}'|j@|||'|&|!|| |\}(})}}'}&}!}*| | |)|}+d|vr|d},d|,j!_"|}?|>|?d.An|=}>d/|>iA|r| Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`. For an overview of generation strategies and code examples, check out the [following guide](../generation_strategies). Parameters: input_features (`torch.Tensor` of shape `(batch_size, feature_size, sequence_length)`, *optional*): Float values of log-mel features extracted from the raw speech waveform. The raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] for details. generation_config ([`~generation.GenerationConfig`], *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which had the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complement the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], list[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments: the batch ID `batch_id` and `input_ids`. It has to return a list with the allowed tokens for the next generation step conditioned on the batch ID `batch_id` and the previously generated tokens `inputs_ids`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://huggingface.co/papers/2010.00904). synced_gpus (`bool`, *optional*, defaults to `False`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). return_timestamps (`bool`, *optional*): Whether to return the timestamps with the text. This enables the `WhisperTimestampsLogitsProcessor`. For audios longer than 30 seconds, it is necessary to set `return_timestamps=True`. task (`str`, *optional*): Task to use for generation, either "translate" or "transcribe". language (`str` or list of `str`, *optional*): Language token to use for generation, can be either in the form of `<|en|>`, `en` or `english`. For batched generation, a list of language tokens can be passed. You can find all the possible language tokens in the `model.generation_config.lang_to_id` dictionary. is_multilingual (`bool`, *optional*): Whether or not the model is multilingual. prompt_ids (`torch.Tensor`, *optional*): Rank-1 tensor of token IDs created by passing text to [`~WhisperProcessor.get_prompt_ids`] that is provided as a prompt to each chunk. This can be used to provide or "prompt-engineer" a context for transcription, e.g. custom vocabularies or proper nouns to make it more likely to predict those words correctly. It cannot be used in conjunction with `decoder_start_token_id` as it overwrites this value. prompt_condition_type (`str`, *optional*): Only relevant for long-form transcription. Condition type of `prompt_ids`. 'first-segment' means only the first segment is conditioned on `prompt_ids`. 'all-segments' means each segment is conditioned on `prompt_ids`. Make sure to enable `condition_on_prev_tokens` for 'all-segments'. Defaults to 'first-segment'. For short-term transcription only 'first-segment' is possible. condition_on_prev_tokens (`bool`, *optional*): Only relevant for long-form transcription. Whether to condition each segment on the previous segment. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. temperature (`float` or list of `float`, *optional*): The temperature to be used for generation. Passing a single `float` value and `do_sample=True` activates generation using sampling. For long-form transcription, temperature fallback can be activated by passing a list of float values such as (0.0, 0.2, 0.4, 0.6, 0.8, 1.0). As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. compression_ratio_threshold (`float`, *optional*): Only relevant for long-form transcription. If defined, the zlib compression rate of each segment will be computed. If the compression rate of a segment is higher than `compression_ratio_threshold`, temperature fallback is activated: the generated segment is discarded and the generation is repeated using a higher temperature. The intuition behind this feature is that segments with very high compression rates suffer from a lot of repetition. The unwanted repetition can be reduced by injecting more randomness by increasing the temperature. If `compression_ratio_threshold` is defined make sure that `temperature` is a list of values. A common value for `compression_ratio_threshold` is 1.35. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. logprob_threshold (`float`, *optional*): Only relevant for long-form transcription. If defined, the average log-probability of each segment will be computed. If the log-probability of a given segment is lower than `logprob_threshold`, temperature fallback is activated: the generated segment is discarded and the generation is repeated using a higher temperature. The intuition behind this feature is that segments of low log-probability can be improved by injecting more randomness by increasing the temperature. If `logprob_threshold` is defined make sure that `temperature` is a list of values. A common value for `logprob_threshold` is -1.0. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. no_speech_threshold (`float`, *optional*): Only relevant for long-form transcription. If defined, the "no-speech" token combined with the `logprob_threshold` is used to determine whether a segment contains only silence. In this case, the transcription for this segment is skipped. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. num_segment_frames (`int`, *optional*): The number of frames a single segment is made of. If not defined, `num_segment_frames` defaults to the model's stride times the maximum input length. attention_mask (`torch.Tensor`, *optional*): `attention_mask` needs to be passed when doing long-form transcription using a batch size > 1. time_precision (`int`, *optional*, defaults to 0.02): The duration of output token in seconds. *E.g.* 0.02 means that a generated token on average accounts for 20 ms. time_precision_features (`int`, *optional*, defaults to 0.01): The duration represented by a feature frame in seconds. return_token_timestamps (`bool`, *optional*): Whether to return token-level timestamps with the text. This can be used with or without the `return_timestamps` option. To get word-level timestamps, use the tokenizer to group the tokens into words. return_segments (`bool`, *optional*, defaults to `False`): Whether to additionally return a list of all segments. Note that this option can only be enabled when doing long-form transcription. return_dict_in_generate (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~utils.ModelOutput`] instead of just returning the generated tokens. Note that when doing long-form transcription, `return_dict_in_generate` can only be enabled when `return_segments` is set True. In this case the generation outputs of each segment is added to each segment. force_unique_generate_call (`bool`, *optional*): Whether to force a unique call to the underlying GenerationMixin's [`~generation.GenerationMixin.generate`] method. This is useful for assisted decoding and testing purposes to ensure that only one call to [`~generation.GenerationMixin.generate`] is made and therefore decoder input token ids and eos token ids are returned. monitor_progress (`Callable[[torch.Tensor], None]`, *optional*): If provided, this function can be called to report the progress of the audio transcription. The function takes a tensor argument `p` of shape `(n, 2)`, where `n` is the batch size. `p[i, 0]` contains the index of the audio frame that is currently being transcribed for batch item `i`. `p[i, 1]` contains the total number of frames for batch item `i`. No return value is expected. kwargs (`dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*. Return: [`~utils.ModelOutput`] or `dict[str, Any]` or `torch.LongTensor`: One of the following: - [`~utils.ModelOutput`] when `return_dict_in_generate=True` and (`return_timestamps=False` or `force_unique_generate_call=True`), including the decoder input ids and end of sequence id. - `dict[str, Any]` when (`return_dict_in_generate=True` and `return_timestamps=True`) or `return_segments=True` or `return_token_timestamps=True`. - `torch.LongTensor` in all other cases, excluding the decoder input ids and end of sequence id. The possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateEncoderDecoderOutput`] - [`~generation.GenerateBeamEncoderDecoderOutput`] `segments` is a list of lists (one list per batch element) of `segment`. A `segment` is a dictionary with keys `start`, `end`, `tokens`, `idxs`, and `result`. - `start`: the start timestamp of the segment. - `end`: the end timestamp of the segment. - `tokens`: the tokens of the segment, excluding the decoder input ids and end of sequence id. - `idxs`: the start (included) and end (excluded) indices of the `tokens` of the segment in the underlying call to GenerationMixin's [`~generation.GenerationMixin.generate`] (present in `result`). - `result`: the result of the underlying call to GenerationMixin's [`~generation.GenerationMixin.generate`]. When `return_timestamps=True`, `return_dict_in_generate=True` applies to each call of the underlying GenerationMixin's [`~generation.GenerationMixin.generate`], with outputs stored in `result` of each `segment`. Example: - *Longform transcription*: To transcribe or translate audios longer than 30 seconds, process the audio files without truncation and pass all mel features at once to generate. It is necessary to set `return_timestamps=True`. Indeed, long-form transcription uses a sequential algorithm based on timestamps predictions, with heuristics like compression ratio threshold, log probability threshold and temperature fallback. This algorithm is described in the [the Whisper original paper](https://cdn.openai.com/papers/whisper.pdf), section *3.8. Long-form Transcription*. ```python >>> import torch >>> from transformers import AutoProcessor, WhisperForConditionalGeneration >>> from datasets import load_dataset, Audio >>> processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en") >>> model.cuda() # doctest: +IGNORE_RESULT >>> # load audios > 30 seconds >>> ds = load_dataset("distil-whisper/meanwhile", "default")["test"] >>> # resample to 16kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=16000)) >>> # take first 8 audios and retrieve array >>> audio = ds[:8]["audio"] >>> audio = [x["array"] for x in audio] >>> # make sure to NOT truncate the input audio, to return the `attention_mask` and to pad to the longest audio >>> inputs = processor(audio, return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000) >>> inputs = inputs.to("cuda", torch.float32) >>> # transcribe audio to ids >>> generated_ids = model.generate(**inputs, return_timestamps=True) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> transcription[0] " Folks, if you watch the show, you know, I spent a lot of time right over there. Patiently and astutely scrutinizing the boxwood and mahogany chest set of the day's biggest stories developing the central headline pawns, definitely maneuvering an oso topical night to F6, fainting a classic Sicilian, nade door variation on the news, all the while seeing eight moves deep and patiently marshalling the latest press releases into a fisher's shows in Lip Nitsky attack that culminates in the elegant lethal slow-played, all-passant checkmate that is my nightly monologue. But sometimes, sometimes, folks, I. CHEERING AND APPLAUSE Sometimes I startle away, cubside down in the monkey bars of a condemned playground on a super fun site. Get all hept up on goofballs. Rummage that were discarded tag bag of defective toys. Yank out a fist bowl of disembodied doll limbs, toss them on a stained kid's place mat from a defunct dennies. set up a table inside a rusty cargo container down by the Wharf and challenged toothless drifters to the godless bughouse blitz of tournament that is my segment. Meanwhile." ``` The `monitor_progress` callback can be used to monitor the progress of the transcription: ```python >>> from tqdm import tqdm >>> # prepare inputs like above >>> # define a callback to monitor the progress of the transcription. >>> with tqdm(desc="Progress") as pbar: >>> def monitor_progress(p_batch): >>> i = torch.argmax(p_batch[:, 1]) >>> p = p_batch[i].detach().cpu() >>> pbar.total = int(p[1]) >>> pbar.n = int(p[0]) >>> pbar.update() >>> # transcribe audio to ids >>> generated_ids = model.generate(**inputs, return_timestamps=True, monitor_progress=monitor_progress) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> transcription[0] Progress: 95%|█████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8497/8901 [00:04<00:00, 2052.79it/s] " Folks, if you watch the show, you know, I spent a lot of time right over there. Patiently and astutely scrutinizing the boxwood and mahogany chest set of the day's biggest stories developing the central headline pawns, definitely maneuvering an oso topical night to F6, fainting a classic Sicilian, nade door variation on the news, all the while seeing eight moves deep and patiently marshalling the latest press releases into a fisher's shows in Lip Nitsky attack that culminates in the elegant lethal slow-played, all-passant checkmate that is my nightly monologue. But sometimes, sometimes, folks, I. CHEERING AND APPLAUSE Sometimes I startle away, cubside down in the monkey bars of a condemned playground on a super fun site. Get all hept up on goofballs. Rummage that were discarded tag bag of defective toys. Yank out a fist bowl of disembodied doll limbs, toss them on a stained kid's place mat from a defunct dennies. set up a table inside a rusty cargo container down by the Wharf and challenged toothless drifters to the godless bughouse blitz of tournament that is my segment. Meanwhile." ``` - *Shortform transcription*: If passed mel input features are <= 30 seconds, there are two possibilities: - `return_timestamps=False`: the whole audio will be transcribed with a single call to GenerationMixin's [`~generation.GenerationMixin.generate`]. - `return_timestamps=True`: the audio will be transcribed using the same logic as long-form transcription. ```python >>> import torch >>> from transformers import AutoProcessor, WhisperForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_features = inputs.input_features >>> generated_ids = model.generate(inputs=input_features) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription ' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ``` r)r input_stridekwargsrrwrr)r is_shortformrrrrrrwrrrrrrrr)rr)rrrrr)reagerencoder_outputsr num_beamsNassistant_model)rrM begin_indexrrerr)rrtotal_input_framesr)rseek max_frames init_tokensrrr)rrrTrxFrX)rrrcur_bsz batch_idx_mapmps)rn)rrseek_num_framesrrrsuppress_tokens) rrrqrdo_condition_on_prev_tokensrrrrerrzr)rdecoder_input_idsrset_begin_indexr) segment_inputrrrr temperaturesrrMrrrrwrrrrr) seek_sequence seek_outputs time_offsetrzrrrrprev_idxidxrwr first-segmentc"g|] }|dd S)rNr])rGrs r%r`z3WhisperGenerationMixin.generate..s - - -qQqrrU - - -r'encoder_attentionsc3DK|]}j|ddVdSrE)rrGr:num_return_sequencesoutputss r%rJz2WhisperGenerationMixin.generate..sN77 2156L6L8L6LM777777r'encoder_hidden_statesc3DK|]}j|ddVdSrE)rrs r%rJz2WhisperGenerationMixin.generate..sN:: 5a89O9O;O9OP::::::r'rQ)rqrrrersrwrxzYou have passed `return_dict_in_generate=True` and `return_timestamps=True`, this automatically sets `return_segments=True` to access the results of the underlying calls to GenerationMixin's generate in the returned `segments`.)rVrWrVr)@_prepare_generation_configmodelencoderconv1strideconv2rmax_source_positions_retrieve_total_input_frames_set_return_outputs_set_return_timestamps_set_language_and_task_set_num_frames_set_thresholds_and_condition_set_prompt_condition_type_retrieve_init_tokens_check_decoder_input_ids_attn_implementationrergethasattrrbegin_suppress_tokens_retrieve_logit_processors_set_condition_on_prev_tokensrFrr_retrieve_max_frames_and_seekr _expand_variables_for_generation_prepare_segmentsrrxanyrhrj_maybe_reduce_batchtotyper/float64clamp_get_input_segmentrPr _prepare_decoder_input_ids_set_max_new_tokens_and_lengthrgenerate_with_fallback enumerate_retrieve_segmentrr_stack_split_outputsrr1rkrrrrlogger warning_once)BrrrrMrrrrrrrrrrrrrrrrrrrwrrrxrrrrrrrzrrerrrrrrrrrqrrrrrrprocseek_sequencesr should_skipmodel_output_typer:rprev_irsegment_offsetfinal_segmentspadded_outputsrVrWrrsB @@r%generatezWhisperGenerationMixin.generatesw V%DD$CDU$`$`Y_$`$`!6z)/6q9DJ   D 1E  1 1)!#!%=/ 2         ' 11!/2    & &$%67OKOO - H % -(*FGG 3->-Y**/1MNN 3-1-C-^**-2*j %%''v +  dJ-?Q!G!G!GHHH 6:5M5M-%+ 6N66 2NG])=)= 5=QQTbbeqq  *D077 "?ARO)-(G(G'!1+,G%"3{") / /)H ) ) % v  / /{"3"3 0    +,JJDt%677J,,->-DR-HIII+++"3+)"3!1"3)A'(?,G)%-#,  +!,%.n$=$= 5 5 =&q)q>LLLOF$;;LLL+/+A+A"/!- +$3$3#1,C!-#,C&7,B , ,(.V .  (((H4(((() mj %%''v x&+<+RVe+e+e . -, - - - -!  $ !9 , 4E  // >OQUQ\^deeG#a''7$899g>X>d1677777!&s7+E'F'F!G!G77722G.7$;<<A^Aj49:::::!&s7+H'I'I!J!J:::55G1N++*7; $;'A     # "'8'P "   v   #OO  ")@ "! ! " *8 'I'&$4GG 'IYG  1"0GJ r'c tj|}dt|D}dt|D}dt|D}dt|D}tt|}|j|||||t |D]~\}}|duo|dk|_|jr|nd|_|jrd|_tj|}dD] }||vr||= |j d }|j d kr||krtj |d d d d d ||z fd }tj |d d d ||z f|j }|d 'tj |d d d d ||z fd |d <|d)tj |dd d d d d ||z fd |d<tj|f||| | | ||d|}t#|}|||| ||||\}}||kr|d|}|d|}g}g}g} g}!t |D]Y\}"}#|#d|j krC|#|j k}$|j |jkr|$dz}$|$d kr |#d|$ }#||#||"|||jj|\||"<||"<|#d|jkr |#dd}#|#|||"<||"|||"<|dup|dk}%|jo|%| ||"<||"rv|||"|||"| ||"d |vr!|!|d |"[|}t5|d ks|t5|dz kr|}|}nEt7j| }t7j|}d |vrt7j|!|d <|||| |fS)Ncg|]}dSrEr]rG_s r%r`zAWhisperGenerationMixin.generate_with_fallback..s;;;qd;;;r'cg|]}dSrEr]r2s r%r`zAWhisperGenerationMixin.generate_with_fallback..s:::aT:::r'cg|]}dSFr]r2s r%r`zAWhisperGenerationMixin.generate_with_fallback..s888A%888r'cg|]}dSr6r]r2s r%r`zAWhisperGenerationMixin.generate_with_fallback..s555u555r'rfg?r) do_samplerrrstatic)rgdecoder_attention_maskTr)rrMrrrrr)rrrwrrrrrg?)copyr1rr_setup_no_speech_detectionr"r8rrrcache_implementationrpr rrrsuperr/r_postprocess_outputsr eos_token_id_need_fallbackr vocab_sizerr2rkrhrj)'rrrrrrrrrMrrrrwrrrrrseek_sequence_listseek_outputs_listneeds_fallbackr)fallback_index_map fallback_idxrgenerate_kwargskeyrr*r(new_fallback_index_mapnew_segment_inputnew_decoder_input_idsnew_decoder_attention_maskr:r num_paddingsis_low_temperature __class__s' r%r!z-WhisperGenerationMixin.generate_with_fallbacks(6""<;E'NN;;;::5>>:::88w88855eGnn555 !%..11  0 <  + +,!$(9(FFF$1#2#$6MUCU,,>q,AB"!$_*112DQ2GHHH%,,]1-=>>>)001B11EFFF/699299&AY:Z[\:]^^^!7 %&&!++|s.cs!TTTQ(J!7 8TTTr'cg|]}gSr]r]r2s r%r`z.es>>>q>>>r')rrLr1)rrrrRrqs` r%rz(WhisperGenerationMixin._prepare_segments^s  !&7&MQ`&`&` '(9;NPT U U +5a=>E*,=,=>>> r'c |jd}ttjrdd|dffS|r[t |drKt |d} | | |z } | |} |j| |jdd<d fd ddd|df}  fdt| jd D| fS) Nrrr)rrrWc T|;|dkr5dt||dt|DS|dvrfd|DS|dvrtfd|DS|dkr|sdSg}tjjD]}|jj|jd |jj|j d |j j|jd |j j|j d f}| |t|S| S)NscorescHg|]\}}|| Sr]r)rGvbeam_idxs r%r`z]WhisperGenerationMixin._postprocess_outputs..split_by_batch_index..s*sssmq(( ))sssr'rWrrlogitscDg|]}|Sr]rYrGrZrs r%r`z]WhisperGenerationMixin._postprocess_outputs..split_by_batch_index..s);;;q) ((**;;;r'decoder_attentionsdecoder_hidden_statesrc3NK|]}tfd|DV dS)c3XK|]$}|dV%dSrErY)rGwrs r%rJzfWhisperGenerationMixin._postprocess_outputs..split_by_batch_index...s8"G"G1Y<#5#9#9#;#;"G"G"G"G"G"Gr'N)rr_s r%rJz\WhisperGenerationMixin._postprocess_outputs..split_by_batch_index..s@XXAU"G"G"G"GQ"G"G"GGGXXXXXXr'past_key_values)ziprkrr1rrself_attention_cachelayerskeysrvaluescross_attention_cacher2r) rkrIrrrall_past_key_values layer_idx layer_cachers ` r%split_by_batch_indexzIWhisperGenerationMixin._postprocess_outputs..split_by_batch_indexs'C8OOsss6???)$((** *r'rVcXg|]%fdD&S)c ^i|])\}}|||d*S)r)r)r)rGkrZr:rrrps r% zJWhisperGenerationMixin._postprocess_outputs...sU   Aq''1aLL\L\]kLlLlmmm   r')items)rGr:rrrps @r%r`z?WhisperGenerationMixin._postprocess_outputs..sh           (..00      r'rrE) rrFrhrirrLrrr1) rrrrwrrrr start_idxrsequence_tokensrps `` ` @r%r?z+WhisperGenerationMixin._postprocess_outputsisN&+B/ lEL 1 1 =9:: . < < " w/@BS'T'T  !2LAAJ%'$. ' 6 /3/M/M!1%/5b9 0N00L+ , + + + + + +0'{3AAAyzzMB      ?0344     ,,r'c  i}dD] dvr9tj fdDd| <@ dvrGt fdt t d D| < dkr9tj fdDd| <ʉ d vrHt fd t t d D| < d krd }g}t t d D]1  fd d D\}}} } |||| | f2tt|| <d| <|d} | t}|di|S)Nr)rVrrWc g|] }| Sr]r]rGrZrIs r%r`z?WhisperGenerationMixin._stack_split_outputs..+I+I+IqAcF+I+I+Ir'rXr\c3K|]8tjfdDV9dS)c,g|]}|Sr]r])rGrZr:rIs r%r`zIWhisperGenerationMixin._stack_split_outputs...s! A A Aq3 A A Ar'N)rhrjrrGr:rerIrs @r%rJz>WhisperGenerationMixin._stack_split_outputs..se%%RSEK A A A A AL A A ABBEEfMM%%%%%%r'sequences_scoresc g|] }| Sr]r]rzs r%r`z?WhisperGenerationMixin._stack_split_outputs..r{r'r`c 3K|]NtfdttddDVOdS)c3K|]LtjfdDdVMdS)c8g|]}|Sr]r])rGrZr:r9rIs r%r`zSWhisperGenerationMixin._stack_split_outputs....s%$H$H$HaQsVAYq\$H$H$Hr'rNrhrjsqueezer)rGr9rer:rIrs @r%rJzHWhisperGenerationMixin._stack_split_outputs...s{ $H$H$H$H$H$H<$H$H$HIIQQRSTTWWX^__r'rN)rr1rkr~s @r%rJz>WhisperGenerationMixin._stack_split_outputs..s%%  !&s<?3+?+B'C'C!D!D%%%%%%r'rfc3K|]RdD]MtjfdDdVNSdS))rjrkcng|]1}tt|j2Sr])rLri)rG sub_outputrIrn sub_cachesub_keys r%r`zIWhisperGenerationMixin._stack_split_outputs...sK!"!"!"(2%,GJsOY,O,O,VW`,acj$k$k!"!"!"r'rNr)rGrrrerIrnrs @@r%rJz>WhisperGenerationMixin._stack_split_outputs..s d d!*+= d d!("K!"!"!"!"!"!"!"6B!"!"!" %WQZZRZZ d d d d d d dr')rhrlrWr]) rhrjrrr1rkr2rrdict)rrr*rerrrmself_attention_kself_attention_vcross_attention_kcross_attention_vrWrIrns ` ` @@r%r$z+WhisperGenerationMixin._stack_split_outputss?& (& (CGGG${+I+I+I+IL+I+I+IqQQQTTU[\\ [[[$%%%%%%W\]`amnoapqtau]v]vWwWw%%%   ***${+I+I+I+IL+I+I+IqQQQTTU[\\ [[[$%%%%%% #3|As';#<#<== %%%   )))?3'3*,'%*3|As/C+D+D%E%E  d d d d d d d._ d d d`(*:.sHHHaA01HHHr'rWno_speech_prob)r_retrieve_compression_ratiorr_retrieve_avg_logprobsrrPr )rrrrrMrrBrrEr)compression_ratiologprobsrWrs r%rAz%WhisperGenerationMixin._need_fallbacks  8 D $ @ @PZ [ [  #4#PPP!%  . :|A(:;; HH<HHHO%e,X666! +===!%  0 << ":>>"5),=,QQQ!&" {**r'cR|j|jdkrtt||jz}t|} fdtt|D} ||jd}||jd}||jd}||jd}d|_n9|} tt| }fdt| D} || ||||| fS)Nrcg|]}Sr]r]rGr3rs r%r`zKWhisperGenerationMixin._expand_variables_for_generation..s*g*g*g+C*g*g*gr'rrXcg|]}Sr]r]rs r%r`zKWhisperGenerationMixin._expand_variables_for_generation..s*\*\*\+C*\*\*\r')rrr1rkrepeat_interleave) rrrrrrrrrrrs ` r%rz7WhisperGenerationMixin._expand_variables_for_generation sN  1 =BSBhklBlBl z4E4Z'Z![![\\M-((G*g*g*g*gUSVWdSeSeMfMf*g*g*g '+==>O>djk=llN))*;*PVW)XXD#556G6\bc5ddJ%778I8^de7ffK56  2 2 G w00M*\*\*\*\USZ^^*\*\*\ '       '  r'ct|td}d|D}|||d|dS)N set_inputscBi|]\}}tj|||Sr])rh is_tensor)rGrsrZs r%rtzEWhisperGenerationMixin._setup_no_speech_detection..'s-NNNA5?1;M;MN1NNNr')r input_ids)rPr ru)rMrrrr extra_kwargss r%r<z1WhisperGenerationMixin._setup_no_speech_detection$sU45EG_amnn NNNNN  m:K\\|\]]]]]r'c||jd|jdfSd|vrNt|dtr|ddjn |dj}|d|d|zfStd)NrrrrzPMake sure to provide either `input_features` or `encoder_outputs` to `generate`.)rrFrr)rrrencoder_outputs_shapes r%rz3WhisperGenerationMixin._retrieve_total_input_frames*s  %!'*N,@,DD D  & &f%67II5()!,22-.4 " )+-B1-E -TT Tklllr'cd|d}|0t|d||0t|d||0t|d||2t|d|dSdS)NzAudio input consists of only z@. Short-form transcription is activated.{}, but will be ignored.z#condition_on_prev_tokens is set to z&compression_ratio_threshold is set to zlogprob_threshold is set to zno_speech_threshold is set to )r%warningformat)rrrrrrwarning_prefixs r%_maybe_warn_unused_inputsz0WhisperGenerationMixin._maybe_warn_unused_inputs9s ',> ' ' '  $ / NN>001qWo1q1qrr s s s & 2 NN%%&lOj&l&lmm     ( NN>001cPa1c1cdd e e e  * NN>001gRe1g1ghh i i i i i + *r'c||j}n||_||_|rd|_d|_d|_|d|_d|_|S)NT)rrwoutput_attentions output_scoresrs r%rz*WhisperGenerationMixin._set_return_outputsUsg " *&7&O # #8O  54K1 " 38<  526  /.2  +  (8<  5.2  +&&r'c>|t|dr|j}|s/|durtdtdd}|rt|dstd||_t|dr |jdz}n|jjdz}|S) NrFaYou have passed more than 3000 mel input features (> 30 seconds) which automatically enables long-form generation which requires the model to predict timestamp tokens. Please either pass `return_timestamps=True` or make sure to pass no more than 3000 mel input features.z:Setting `return_timestamps=True` for long-form generation.Tno_timestamps_token_idadYou are trying to return timestamps, but the generation config is not properly set. Make sure to initialize the generation config with the correct attributes that are needed such as `no_timestamps_token_id`. For more details on how to generate the approtiate config, refer to https://github.com/huggingface/transformers/issues/21878#issuecomment-1451902363r)rrrr%inforrrB)rrrrrzs r%rz-WhisperGenerationMixin._set_return_timestampshs  $1BDW)X)X $ 1 C  % E)) v KKT U U U $   W->@X%Y%Y c /@+ $&> ? ? 9/FJOO #k4q8Or'cN|&t|dstd||_t|dr|js||td|&t|dstd||_|(t|dstd||_dSdS)NrzThe generation config is outdated and is thus not compatible with the `is_multilingual` argument to `generate`. Please update the generation config as per the instructions https://github.com/huggingface/transformers/issues/25084#issuecomment-1664398224zCannot specify `task` or `language` for an English-only model. If the model is intended to be multilingual, pass `is_multilingual=True` to generate, or update the generation config. lang_to_idzThe generation config is outdated and is thus not compatible with the `language` argument to `generate`. Please update the generation config as per the instructions https://github.com/huggingface/transformers/issues/25084#issuecomment-1664398224 task_to_idzThe generation config is outdated and is thus not compatible with the `task` argument to `generate`. Please update the generation config as per the instructions https://github.com/huggingface/transformers/issues/25084#issuecomment-1664398224)rrrrrrs r%r z-WhisperGenerationMixin._set_language_and_tasks   &,.?@@  g 1@  - $&7 8 8 ARAb 8#7 n  ,l;;  g *2  &  ,l;;  g &*  " " "  r'cv dttdtdttfd}dtdtffd t dd}t dd} jg|*| 't d d} | t |d d|j} | td | (| d d td | | d d d krd t| d krX| d d krF| d d gz | d d} d z t| d kr| d d kFt| d kr"td| d| d d dtd kptd ko d du} t| ttfratd| Drtdt| |kr#td|dt| d| } n | dg|z} n| g} fd| Dd} | fd| D} nQtdrA| r?|||dd|} | kt'tD]Ntd kr| d <-| O~ t'tD]||t*vrhjjjj}||jntd|dt*d| itdrYtfdjDs&jd jsHtd!r8d"jkr!jnKjrDd"jkr-td#dd"<d$D<t9jt8j|j% |d"S)&NlstnumitrctfdD}|rfdDnS)z/short function to replace num with a itr in lstc3 K|]}|vV dSrEr])rGr:rs r%rJzWWhisperGenerationMixin._retrieve_init_tokens..replace_or_add..s'..QS......r'c g|] }|vrn| Sr]r])rGr:rrs r%r`zXWhisperGenerationMixin._retrieve_init_tokens..replace_or_add..s%;;;!a3hhssA;;;r')rr2)rrrfounds``` r%replace_or_addzDWhisperGenerationMixin._retrieve_init_tokens..replace_or_addsa....#.....E ;;;;;s;;; 3Jr'rrc |}|jvr|}n|tvrdt|d}n|tjvrd|d}nit |dk}t d|d|r t tjnt tjd|jvrt |dj|S)Nz<|z|>rzUnsupported language: z. Language should be one of: .z is not supported by this specific model as it is not in the `generation_config.lang_to_id`. (You should just add it to the generation config))lowerrrrkrkrrrj)rlanguage_tokenis_language_coders r%language_to_idzDWhisperGenerationMixin._retrieve_init_tokens..language_to_ids6~~''H,777!)---!D&6x&@!D!D!D-46666!2h!2!2!2#&x==A#5  rXrr;Kn-466777QUVfVkVmVmQnQnrrr%6%AAA %hhh %/? ?r'rforced_decoder_idszUsing custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.rralTranscription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English. This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`. See https://github.com/huggingface/transformers/pull/28687 for more details.zYou are using token ids in `forced_decoder_ids` that do not seem to correctly follow the prompt pattern of Whisper. Make sure that z) has an entry for all indices >= 1 and < rc3K|]}|duV dSrEr])rGrs r%rJz?WhisperGenerationMixin._retrieve_init_tokens..s&//19//////r'zExpected `language` to be `None`, a single string (e.g. `'en'`), or a list of strings with length equal to the batch size (e.g. `('en', 'fr')` for a batch size of 2). Got a list containing `None`.zgWhen passing a list of languages, the length of the list must match the batch size. Expected length of z , but got z languages.c8g|]}tjSr])r;)rGr3rs r%r`z@WhisperGenerationMixin._retrieve_init_tokens..s#AAA!ty--AAAr'c&g|] }|Sr]r])rGrrs r%r`z@WhisperGenerationMixin._retrieve_init_tokens..s#===aq))===r'rr)rrrrzThe `z4` task is not supported. The task should be one of ``rc3,K|]}|vVdSrEr])rGtir:rs r%rJz?WhisperGenerationMixin._retrieve_init_tokens..4s,``B2Q/``````r' transcriberrzm<|notimestamps|> prompt token is removed from generation_config since `return_timestamps` is set to `'True'`.cg|]}||SrEr])rGr?s r%r`z@WhisperGenerationMixin._retrieve_init_tokens..FsIIIA1=a===r'r)!rrrstrrLdecoder_start_token_idrr%r&rkrrFrr TypeErrorrdetect_languagertolistr1r2rrrrkrrrrh as_tensorrreexpand)rrrrrrrrrrris_lang_id_undefined languageslang_idstask_idr:rrs ` @@@r%r z,WhisperGenerationMixin._retrieve_init_tokenss S   (3-     @S @S @ @ @ @ @ @,(&$77,j$??(?@ RTX.Y.Y.e%+%>"!-##F &16H6KA6N6V''x&16H6KA6NRS6S6SA011A55:LQ:OPQ:RVW:W:W#(:1(=a(@'AA -?-C*Q011A55:LQ:OPQ:RVW:W:W -..22(NM_NN2DQ2G2JNNN #;//14i[9I9IA9M9hR]^_R`dhRh hu . . #//h///// ) 8}} ** [*4[[@CH [[[!II  +II! IBAAAyAAA   ====9===HH & 5 5 :N ++- & +7P=P7P(QRF'-a0JJ  ('9F/9/?/[/[s"(++apqras .G1G J At{%* M M M6 7  ]__ h hTYYFY6GSXYYYY`abababdfafgF h h h h h h h h h h h h h h hq DDD EJ d,7>>@@AAB$&F7qqq- ==$$s C55C9<C9c|dd}|dd}||tddSdS)NrrzQPassing `decoder_input_ids` is deprecated. Consider passing `prompt_ids` instead.)rr)rrrs r%rz/WhisperGenerationMixin._check_decoder_input_idssY"JJ':DAA **%6==  (_-Hc  ) (-H-Hr'cL|rt|dddkrtdt|dst d|.|d|_dStdd|_dSdS)Nr translatez@Token-level timestamps may not be reliable for task 'translate'.rzModel generation config has no `alignment_heads`, token-level timestamps not available. See https://gist.github.com/hollance/42e32852f24243b748ae6bc1f985b13a on how to add this property to the generation config.rzWhen setting `return_token_timestamps` to `True`, make sure to pass an `attention_mask` to get precise token-level timestamps. You can retrieve the `attention_mask` by doing `processor(audio, ..., return_attention_mask=True)` ) rLr%rrrrrrr&rs r%r z&WhisperGenerationMixin._set_num_framess " 4(&$77;FFabbb,.?@@  R)/=/A/A"/E/E/I/I/K/K!,,,##y04!,,, 4 4r'c||nt|dd|_||nt|dd|_||nt|dd|_||nt|dd|_dS)Nrrrr)rLrrrrrs r%r z4WhisperGenerationMixin._set_thresholds_and_conditions!,  *,?FF ++6 ( '*,I4PP 5#. *,A4HH -(3 % $*,FMM 222r'cddg}|p|d}||vr(td|dd||jdur|dkrtd||_dS) Nr all-segmentsrz`prompt_condition_type=zD does not exist. Make sure to set `prompt_condition_type` to one of r,TzeMake sure to set `condition_on_prev_tokens=True` when setting `prompt_condition_type='all-segments'`.)rjoinrr)rrallowed_cond_typess r%r z1WhisperGenerationMixin._set_prompt_condition_types-~>!6 N9KA9N (: : :e*?eeFJFOFOPbFcFcee   5T A AF[_mFmFmw 3H///r'c>||nt|dd}||_dS)NrF)rLrrs r%rz4WhisperGenerationMixin._set_condition_on_prev_tokenss:(3 % $*,FNN ! 6N222r'c|dkr|s|td|dkrh|sf|dtj}t j|ftj}nEt j|ftj|z}t j|ftj}||fS)NrzWhen doing batched long-form audio transcription, make sure to pass an `attention_mask`. You can retrieve the `attention_mask` by doing `processor(audio, ..., return_attention_mask=True)` rr*)rrrrrhrrr.)rrrrrrs r%rz4WhisperGenerationMixin._retrieve_max_frames_and_seeks >>,>>3IO !^^L^'++B//335588DDJ; }EJ???DDZMDDDGYYJ; }EJ???D4r'c|jdurt||}||gn|g|z}|j(t|j|}||gn|g|z}d|_|j)t |j||}||gn|g|z}d|_|j>t|jdz ||dk} || gn| g|z}| ||S)NT)rrd)rrer)no_speech_tokenrscores_is_logprobs) rrrr rr rr r set_model) rrrMrrretimestamp_processorsuppress_tokens_processorbegin_suppress_processorno_speech_detectors r%rz1WhisperGenerationMixin._retrieve_logit_processorss`  .$ 6 6"ABSal"m"m"m )9)A$%%H[G\_oGo   , 8(EFWFgpv(w(w(w %$++++/03CC  15  -  2 >'K!7[Y_((( $ $+***./2BB  7;  3  0 .]s2vvvcd6QRS6T^/22Z^vvvr'rrr9rUrRrTT)rersrtrurvryrzrrXr:)popmax_target_positionsrLrkrrrhr.rr=rrrrlrr)rrrqrrrrrrerrzrrrvprev_start_of_textactive_segmentsprev_ids one_tensorrt prev_tokenss ` ` r%rz1WhisperGenerationMixin._prepare_decoder_input_ids<s( & ( ( & +> ? ? $f, ,49A=' 6$%68KTRR  %*s?/C/Cq/H/H%4R%8""%)" * + +! 74DQ4G0H0H10L0LvvvvvhuvvvO%*;*QUc*c*c%"Z! V5:VVV ASA_- 1 ==ei&7&LPX&X&Xll^gG,!.#!)-.2 /   K!& ;8I*JPR S S S /@DUDb/bF+ , ,  #$T*112C2I!2LaPPK % ;8I*JPR S S S  JJ/ 6 6 6 6 JJ/ 6 6 6 &((r'c ||j|jnd}||jdz|jjkrLt d|jdd|d||jdzd|jjd|jjd t |jd zd z |jdd z }|jL|jEt |j|z|j}td |jd |d dS|j<|j|jdz|jkr |j|jdz }||_dSdSdS)NrrzjThe length of `decoder_input_ids`, including special start tokens, prompt tokens, and previous tokens, is z, and `max_new_tokens` is zL. Thus, the combined length of `decoder_input_ids` and `max_new_tokens` is: z@. This exceeds the `max_target_positions` of the Whisper model: z. You should either reduce the length of your prompt, or reduce the value of `max_new_tokens`, so that their combined length is less than rrrzIncrease max_length from z to z0 since input is conditioned on previous segment.) max_new_tokensrrrrminrUr%r)rrrrrnum_initial_tokensrUs r%r z5WhisperGenerationMixin._set_max_new_tokens_and_lengths=N=]=i*99op -3B7 7$+:Z Z Zb~O~UVX~Ybb,:bb@NQbQhikQl@lbbAE @`bb ?Ck>^ bbb !!<!AA!EGXG^_aGbefGfgg  ' 38I8X8`.9.s'VVV 68 < <VVVr')rmathlog2rrrkzlibcompress)r\rB token_bytesrrs @r%rz2WhisperGenerationMixin._retrieve_compression_ratiosTYz**Q.//!3hhVVVVfmmooVVVWW  ,,s4=3M3M/N/NN  r'c|dkr|nd}tj|j}|jdjdkr|djd}n|jd dt j||zd|jtfdtjdD}|tz }|S)NrfrrrrXc3@K|]}||VdSrEr])rGr:rr\s r%rJz@WhisperGenerationMixin._retrieve_avg_logprobs..s0TTa8A;vay1TTTTTTr') rhrjrrerrp log_softmaxfloatr+rr1rk)rWr\rrescale_temperature sum_logprobs avg_logprobsrs ` @r%rz-WhisperGenerationMixin._retrieve_avg_logprobss-83->->kkAV$$'' 66 <?V\!_ , ,-fl1o-.FFV\!_,../F=&+>">!E!E!G!GRPPPSSTZT`aaTTTTT5PQAR;S;STTTTT #c&kk1 r'c ||} | ddddgk} tj| dd| ddzd}|d| r|| dng}| jd}|j}t|dkr|}g}| r#|t|n|dxxdz cc<d}t|D]\}}|t|dz k}|||}|d|z }|r| rdnd}|||z }|||| |j dkr tj n tj |zz||| |j dkr tj n tj |zz|||z||zf|| d | r"|||z||z||z|dd<|}| r ||}n:||d z |z }||z}n|| }t#|||z|z }|dkrK|d|kr?|d|z |j dkr tj n tj }||||||zz|||t|zf|| d g}| r,|||t|z||z|dd<||}||fS) NraFTrrrrWr)startendr\rcr$r)gerrhwhereadd_rrerkr2r"rrr/ritemnonzeroflattenrnumel)rrrrzrrrrrrrwrtimestamp_tokenssingle_timestamp_endingtimestamp_segment_indicesrW idx_offsetreslicesr last_slicer: current_slice is_last_slice sliced_tokensstart_timestamp_posidx_sliced_tokensend_timestamp_posr,last_timestamp_posrs r%r#z(WhisperGenerationMixin._retrieve_segments"*7)9)9/)J)J"2233"7">">"@"@UDM"Q$)K0@"0EHXYZY[Y[H\0\$]$]^_$`!!&&q)))D[c<,-?@@ac&,R0 % ( ) )A - -.5577FH&  c-001111r a J$-f$5$5 + + = !S[[1_ 4 -j.F G &3A&6&H#.;$^?V$^BB\^!$12C$D$V!!,X!6-00&+QVBVBV\a\ijj()") +84+.. u@T@Tu}}Z_Zghh() )#0!+j!8*}:T U".s"3     +(j)@:P]C])]^almuavvRL!34+ & C!0!: &3:>%B%G%G%I%IO%["!3l!B''7'?'?'A'A'I'I'K'KLJ!$_X%>AX%X[i%i!j!j !!A%%*R.O*K*K&0n&F%J%J%+[E%9%9EMMu}&&" )2&x03E3VV+'c-6H6H)HI*3/ H' $Z*s=?Q?Q2Q%QRU`aiUjj /0-X6N''r')rNN)NNNNNFNNNNNNNNNNNNNrrNFNNN)NNNr)1__name__ __module__ __qualname__rrhrirr rrrrrrrrr/r! staticmethodrr?r$rArr<rrrrr r  FloatTensorrrrr r r rrrrrrr rrr# __classcell__)rPs@r%rrseiLLLL`/3597;9=TX!)-+/'+*.,0048<48*.,0)-.2 $)-/3 %/326BF7I I  t+I ,d2I .4 I  0$6 I  #+C+>S +I"JT"Q I I  $;I DjI S /D(I I L4'I  #TzI #'+I U5#:..5I  &+T\!I "!4<#I $#T\%I & $J'I ( t+)I *+I ,"'-I ."&/I 01I 2"&3I 4%)4K5I 6#EL>4#784?7I I I I VRiRiRiRiRih  \ ?-?-?-B/,/,/,b,+,+,+\   6^^\^  m m\ mjj\j6''\'$!!!F!*!*\!*FYiYiYiz48FJ59"& ??)D0?*_>>6!!\!\$a(a(\a(a(a(a(a(r'r)rQrRNNFFFN),r;rr collections.abcrrrr-rhtorch.nn.functionalrrrptransformers.cache_utilsr generationrr generation.logits_processr r r r rgeneration.stopping_criteriarmodeling_outputsrutilsrtokenization_whisperrr get_loggerr*r%rirr&rrBrPrrr]r'r%r:s ........ 888888;;;;;;;;A@@@@@//////<<<<<<<<  H % %5<su|*3&"*3&3&3&3&l !$"'oooodi(i(i(i(i(_i(i(i(i(i(r'