tj=8dZddlmZddlZddlmZddlmZddlm Z ddl m Z m Z m Z dd lmZdd lmZmZmZdd lmZmZdd lmZd dlmZmZmZmZmZmZm Z m!Z!ddl"m#Z#ej$e%Z&Gddej'Z(GddeZ)GddeZ*GddeZ+GddeZ,Gdde Z-eGdde!Z.eGdd eZ/ed!"Gd#d$e.Z0gd%Z1dS)&zHPyTorch ViViT model - modular file inheriting transformer core from ViT.)IterableN)nn)initialization)create_bidirectional_mask)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)PreTrainedModel ViTAttention ViTEmbeddingsViTLayerViTMLPViTModel ViTPoolerViTPreTrainedModel) VivitConfigcLeZdZdZdeffd ZdejdejfdZxZ S)VivitTubeletEmbeddingsay This class turns `pixel_values` of shape `(batch_size, num_frames, num_channels, height, width)` into the initial `hidden_states` (tubelet embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer encoder. The seq_length equals (num_frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). configcht|j}|j}t |t r|n||f}|j|dz|d|dzz|d|dzz|_||_tj |j |j |||_ dS)Nrrr) kernel_sizestride) super__init__ tubelet_size image_size isinstancer num_frames num_patchesrConv3d num_channels hidden_size projection)selfrr%r& __class__s e/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/vivit/modular_vivit.pyr$zVivitTubeletEmbeddings.__init__5s * & #-j(#C#CaZZ*V`Ia  ,q/ 1!} Q/ 1!} Q/ 1  %)  !3Vb    pixel_valuesreturnc|dd}||dddS)Nrr) transposer-flatten)r.r2s r0forwardzVivitTubeletEmbeddings.forwardEsG#--a33 |,,44Q77AA!QGGGr1) __name__ __module__ __qualname____doc__rr$torchTensorr7 __classcell__r/s@r0rr,s~ {      HELHU\HHHHHHHHr1rceZdZdZdeffd Zdejdededejffd Z d d ejd e dejfd Z xZ S)VivitEmbeddingszY Construct the CLS token, position and tubelet patch embeddings for video input. rcttjt jdd|j|_t||_ |j j }tjt jd|dz|j|_ |j dd|_ |`dS)Nr)r#r$r Parameterr<zerosr, cls_tokenrpatch_embeddingsr)position_embeddingsr% patch_size mask_token)r.rr)r/s r0r$zVivitEmbeddings.__init__Ps ek!Q8J&K&KLL 6v > >+7 #%< A{QPVPb0c0c#d#d  -abb1 OOOr1 embeddingsheightwidthr3ct|||||jdz}||jdz}dS)Nrr)r#interpolate_pos_encodingrH)r.rJrKrL new_height new_widthr/s r0rNz(VivitEmbeddings.interpolate_pos_encoding[sE ((VUCCCtq11 T_Q// r1Fr2rNc |j\}}}}}||}|j|dd} t j| |fd}|r|||||z}n^||jdks||jdkr2td|d|d|jdd|jdd ||j z}| |}|S) Nr)dimrzInput image size (*z) doesn't match model (z).) shaperFrEexpandr<catrNr& ValueErrorrGdropout) r.r2rN batch_sizer(r+rKrLrJ cls_tokenss r0r7zVivitEmbeddings.forwardas3>J>P; J fe**<88 ^**:r2>> Y J7Q??? # ?#d&C&CJPVX]&^&^^JJ+++u8J/J/J EEE%EE+EE.2oa.@EEE$d&>>J\\*-- r1)F) r8r9r:r;rr$r<r=intrNboolr7r>r?s@r0rArAKs {      05<00UX0]b]i000000 ELD]b]ir1rAceZdZdS)VivitAttentionNr8r9r:r1r0r_r_xDr1r_ceZdZdS)VivitMLPNr`rar1r0rdrd|rbr1rdceZdZdS) VivitLayerNr`rar1r0rfrfrbr1rfceZdZdS) VivitPoolerNr`rar1r0rhrhrbr1rhcZeZdZUeed<dZdZddgZej dZ dS)VivitPreTrainedModelrvivit)videorArfctj||t|tr4t j|jt j|jdSdS)zInitialize the weightsN)r _init_weightsr'rAinitzeros_rErG)r.modules r0rnz"VivitPreTrainedModel._init_weightss_ %dF333 fo . . 4 K( ) ) ) K2 3 3 3 3 3 4 4r1N) r8r9r:r__annotations__base_model_prefixinput_modalities_no_split_modulesr<no_gradrnrar1r0rjrjsZ !*L9U]__44_444r1rjceZdZddedeffd Zeede dde j dzd ed e j dzd e e d ef d ZxZS) VivitModelTradd_pooling_layercrt|t||_dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N)r#r$rArJ)r.rryr/s r0r$zVivitModel.__init__s0    )&11r1F)tie_last_hidden_statesNr2rNattention_maskkwargsr3c |||}t|j||}|}|jD] }|||fi|} ||}|j||nd} t || S)a Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```)rN)r inputs_embedsr|N)last_hidden_state pooler_output)rJrrlayers layernormpoolerr ) r.r2rNr|r}embedding_output hidden_stateslayersequence_output pooled_outputs r0r7zVivitModel.forwardsj ??r?s@r0rxrxs22{2t222222 _E22226)..2 ^j^j'$.^j#'^j t+ ^j +, ^j $ ^j^j^j^32 ^j^j^j^j^jr1rxa ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ) custom_introc eZdZdeffd Zee d dejdzdej dzde de e d e f d ZxZS) VivitForVideoClassificationrc:t||j|_t|d|_|jdkrt j|j|jnt j|_ | dS)NF)ryr) r#r$ num_labelsrxrkrLinearr,Identity classifier post_init)r.rr/s r0r$z$VivitForVideoClassification.__init__s     +%@@@ OUN_bcNcNc")F$68IJJJikitiviv r1NFr2labelsrNr}r3c |j|fd|i|}|j}||dddddf}d}||j|||jfi|}t |||j|jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```rNNr)losslogitsr attentions)rkrr loss_functionrr rr) r.r2rrNr}outputsrrrs r0r7z#VivitForVideoClassification.forward!sx$.4: $ $ 3K$ OU$ $ "3Aqqq!9::  %4%ffdkLLVLLD$!/)     r1)NNF)r8r9r:rr$rr r<r LongTensorr]r r r r7r>r?s@r0rrs {      26*.). i i '$.i  4'i #' i +, i  i i i ^i i i i i r1r)rxrjr)2r;collections.abcrr<rrro masking_utilsrmodeling_outputsrr r processing_utilsr utilsr r r utils.genericrrutils.output_capturingrvit.modeling_vitrrrrrrrrconfiguration_vivitr get_loggerr8loggerModulerrAr_rdrfrhrjrxr__all__rar1r0rsSON$$$$$$ &&&&&&666666bbbbbbbbbb&&&&&&@@@@@@@@@@IIIIIIII555555                    -,,,,,  H % %HHHHHRYHHH>*****m***Z     \        v                )    4 4 4 4 4- 4 4 4jjjjjjjjjjjjjjjjZ    x x x x x "6x x   x v P O Or1