ó <±hÕ«ãó8•SrSSKrSSKJr SSKJr SSKJrJ r J r SSKrSSK r SSKr SSK Jr SSKJrJrJr SS KJr SS KJr SSKJrJr SSKJrJr SS KJrJ r SSK!J"r"J#r#J$r$ SSK%J&r&J'r' SSK(J)r) \$RT"\+5r,\\#"SS9"SS\"555r-\\#"SS9"SS\"555r.Sr/"SS\R`5r1"SS\R`5r2SAS\R`S\ RfS \ RfS!\ RfS"\ \ RfS#\4S$\44S%jjr5"S&S'\R`5r6"S(S)\R`5r7"S*S+\R`5r8"S,S-\R`5r9"S.S/\R`5r:"S0S1\5r;"S2S3\R`5r<\#"S4S5\55r=\#"S6S7\=55r>"S8S9\R`5r?\#"S:S9"S;S<\=55r@\#"S=S9"S>S?\=55rA/S@QrBg)Bz,PyTorch VideoMAE (masked autoencoder) model.éN)Údeepcopy)Ú dataclass)ÚCallableÚOptionalÚUnion)Únn)ÚBCEWithLogitsLossÚCrossEntropyLossÚMSELossé)ÚACT2FN)ÚGradientCheckpointingLayer)ÚBaseModelOutputÚImageClassifierOutput)ÚALL_ATTENTION_FUNCTIONSÚPreTrainedModel)Ú find_pruneable_heads_and_indicesÚprune_linear_layer)ÚModelOutputÚauto_docstringÚlogging)ÚIMAGENET_DEFAULT_MEANÚIMAGENET_DEFAULT_STDé)ÚVideoMAEConfigz[ Class for VideoMAEDecoder's outputs, with potential hidden states and attentions. )Úcustom_introcóž•\rSrSr%SrSr\\R\ S'Sr \\\R\ S'Sr\\\R\ S'Sr g)ÚVideoMAEDecoderOutputé-zx logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. NÚlogitsÚ hidden_statesÚ attentions©)Ú__name__Ú __module__Ú__qualname__Ú__firstlineno__Ú__doc__r rÚtorchÚFloatTensorÚ__annotations__r!Útupler"Ú__static_attributes__r#óÚf/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/videomae/modeling_videomae.pyrr-sR‡ñð +/€FˆHU×&Ñ&Ñ'Ó.Ø8<€M8˜E %×"3Ñ"3Ñ4Ñ5Ó<Ø59€J˜˜u×0Ñ0Ñ1Ñ2Ö9r.rzb Class for VideoMAEForPreTraining's outputs, with potential hidden states and attentions. cóÆ•\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr\\\R\ S'Sr \\\R\ S'Srg) ÚVideoMAEForPreTrainingOutputé>zÃ loss (`torch.FloatTensor` of shape `(1,)`): Pixel reconstruction loss. logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. NÚlossr r!r"r#)r$r%r&r'r(r3rr)r*r+r r!r,r"r-r#r.r/r1r1>sg‡ñð)-€Dˆ(5×$Ñ$Ñ %Ó,Ø*.€FˆHU×&Ñ&Ñ'Ó.Ø8<€M8˜E %×"3Ñ"3Ñ4Ñ5Ó<Ø59€J˜˜u×0Ñ0Ñ1Ñ2Ö9r.r1cót^•U4Sjn[R"[U5Vs/sH o2"U5PM sn5n[R"USS2SSS245USS2SSS24'[R"USS2SSS245USS2SSS24'[ R"U5RS5$s snf)z Sinusoid position encoding tablec ó„>•[T5Vs/sH%o[R"SSUS--T-5-PM' sn$s snf)Ni'é)ÚrangeÚnpÚpower)ÚpositionÚhid_jÚd_hids €r/Úget_position_angle_vecÚ;get_sinusoid_encoding_table..get_position_angle_vecXs?ø€ÜRWÐX]ÔR^Ó_ÑR^Èœ2Ÿ8š8 E¨1°¸± Ñ+;¸eÑ+CÓDÔDÑR^Ñ_Ð_ùÒ_s+=Nrr6r)r8Úarrayr7ÚsinÚcosr)r*Ú unsqueeze)Ú n_positionr<r=Úpos_iÚsinusoid_tables ` r/Úget_sinusoid_encoding_tablerFTs°ø€õ`ô—X’XÌ%ÐPZÔJ[Ó\ÑJ[ÀÐ5°eÖ<ÑJ[Ñ\Ó]€NÜ Ÿfšf ^²A°q°t¸!°t°GÑ%<Ó=€N’1ad˜d7ÑÜ Ÿfšf ^²A°q°t¸!°t°GÑ%<Ó=€N’1ad˜d7Ñä×Ò˜^Ó,×6Ñ6°qÓ9Ð9ùò ]s¥B5có2^•\rSrSrSrU4SjrSrSrU=r$)ÚVideoMAEEmbeddingsébz/ Construct the patch and position embeddings. cóÎ>•[TU]5 [U5UlURRUl[URUR5UlXlg©N) ÚsuperÚ__init__ÚVideoMAEPatchEmbeddingsÚpatch_embeddingsÚnum_patchesrFÚhidden_sizeÚposition_embeddingsÚconfig©ÚselfrSÚ __class__s €r/rMÚVideoMAEEmbeddings.__init__hsPø€Ü ‰ÑÔä 7¸Ó ?ˆÔØ×0Ñ0×<Ñ<ˆÔä#>¸t×?OÑ?OÐQW×QcÑQcÓ#dˆÔ Ør.có•URU5nX0RR5RU5R UR SS9-nUb'URupEnX2)nURUSU5nU$)NT©ÚdeviceÚcopyéÿÿÿÿ)rOrRÚdetachÚtype_asÚtorZÚshapeÚreshape)rUÚpixel_valuesÚbool_masked_posÚ embeddingsÚ batch_sizeÚ_Únum_channelss r/ÚforwardÚVideoMAEEmbeddings.forwardqs‘€à×*Ñ*¨<Ó8ˆ ð ×":Ñ":×"AÑ"AÓ"C×"KÑ"KÈJÓ"W×"ZÑ"ZØ×$Ñ$¨4ð#[ð# ñ ˆ ð Ñ&Ø*4×*:Ñ*:Ñ'ˆJ˜<Ø#Ð$4Ñ5ˆJØ#×+Ñ+¨J¸¸LÓIˆJàÐr.)rSrPrOrR© r$r%r&r'r(rMrhr-Ú __classcell__©rVs@r/rHrHbsø†ñõ ÷ðr.rHcó2^•\rSrSrSrU4SjrSrSrU=r$)rNéƒac Video to Patch Embedding. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size) * (height // patch_size) * (width // patch_size). c óŠ>•[T U]5 URnURnURnUR nURnURn[U[RR5(aUOX"4n[U[RR5(aUOX34nX lX0l[U5UlUSUS-USUS--X`R--nX@lX€l [R"UUURUSUS4URUSUS4S9Ulg)Nrr)Úin_channelsÚout_channelsÚkernel_sizeÚstride)rLrMÚ image_sizeÚ patch_sizergrQÚ num_framesÚtubelet_sizeÚ isinstanceÚcollectionsÚabcÚIterableÚintrPrÚConv3dÚ projection) rUrSrtrurgrQrvrwrPrVs €r/rMÚ VideoMAEPatchEmbeddings.__init__s3ø€Ü ‰ÑÔà×&Ñ&ˆ Ø×&Ñ&ˆ Ø×*Ñ*ˆØ×(Ñ(ˆØ×&Ñ&ˆ Ø×*Ñ*ˆä#-¨j¼+¿/¹/×:RÑ:R×#SÑ#S‘ZÐZdÐYqˆ Ü#-¨j¼+¿/¹/×:RÑ:R×#SÑ#S‘ZÐZdÐYqˆ Ø$ŒØ$ŒÜ Ó-ˆÔà ˜‰]˜j¨™mÑ +° ¸1± ÀÈAÁÑ0NÑOÐS]×arÑarÑSrÑsð ð)ÔØ&ÔÜŸ)š)Ø$Ø$Ø×*Ñ*¨J°q©M¸:Àa¹=ÐIØ×%Ñ% z°!¡}°jÀ±mÐDñ ˆr.có•URup#pEnX@R:wa[S5eXPRS:wdX`RS:wa2[SUSUSURSSURSS3 5eUR SSSS S 5nURU5R S5RSS5nU$)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (Ú*z) doesn't match model (z).r6ré)r`rgÚ ValueErrorrtÚpermuter~ÚflattenÚ transpose)rUrbrervrgÚheightÚwidthrds r/rhÚVideoMAEPatchEmbeddings.forward¨sÜ€Ø>J×>PÑ>PÑ;ˆ °eØ×,Ñ,Ó,ÜØwóð ð—_‘_ QÑ'Ó'¨5·O±OÀAÑ4FÓ+FÜØ$ V H¨A¨e¨WÐ4KÈDÏOÉOÐ\]ÑL^ÐK_Ð_`Ðae×apÑapÐqrÑasÐ`tÐtvÐwóð ð$×+Ñ+¨A¨q°!°Q¸Ó:ˆØ—_‘_ \Ó2×:Ñ:¸1Ó=×GÑGÈÈ1ÓMˆ ØÐr.)rtrgrPrur~rwrjrls@r/rNrNƒsø†ñõ ÷6 ð r.rNÚmoduleÚqueryÚkeyÚvalueÚattention_maskÚscalingÚdropoutcó°•[R"XRSS55U-n[RRUS[RS9RUR5n[RRX†URS9nUbX„-n[R"Xƒ5n U RSS5R5n X˜4$)Nr\éþÿÿÿ)ÚdimÚdtype)ÚpÚtrainingrr6)r)Úmatmulr†rÚ functionalÚsoftmaxÚfloat32r_r”rr–Ú contiguous) rŠr‹rŒrrŽrrÚkwargsÚattn_weightsÚattn_outputs r/Úeager_attention_forwardrŸ¹s¶€ô—<’< §}¡}°R¸Ó'<Ó=ÀÑG€Lô—=‘=×(Ñ(¨¸2ÄUÇ]Á]Ð(ÐS×VÑVÐW\×WbÑWbÓc€Lô—=‘=×(Ñ(¨È6Ï?É?Ð(Ð[€LðÑ!Ø#Ñ4ˆä—,’,˜|Ó3€KØ×'Ñ'¨¨1Ó-×8Ñ8Ó:€KàÐ$Ð$r.c óÀ^•\rSrSrS\SS4U4SjjrS S\\RS\ S\ \\R\R4\\R44SjjrS r U=r$)ÚVideoMAESelfAttentioné×rSÚreturnNcó>•[TU]5 URUR-S:wa7[ US5(d&[SURSURS35eXlURUl[URUR-5UlURUR-Ul URUlURS-UlSUl [R"URURSS9Ul[R"URURSS9Ul[R"URURSS9UlUR&(as[R("[*R,"UR55Ul[R("[*R,"UR55UlgSUlSUlg) NrÚembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads Ú.gà¿F©Úbias)rLrMrQÚnum_attention_headsÚhasattrrƒrSr|Úattention_head_sizeÚ all_head_sizeÚattention_probs_dropout_probÚdropout_probrÚ is_causalrÚLinearr‹rŒrÚqkv_biasÚ Parameterr)ÚzerosÚq_biasÚv_biasrTs €r/rMÚVideoMAESelfAttention.__init__ØsŽø€Ü ‰ÑÔØ×Ñ × :Ñ :Ñ:¸aÓ?ÌÐPVÐXh×HiÑHiÜØ" 6×#5Ñ#5Ð"6ð7Ø×3Ñ3Ð4°Að7óð ðŒØ#)×#=Ñ#=ˆÔ Ü#& v×'9Ñ'9¸F×OÜ×#Ñ#ðLõô '>¸d¿k¹k×>^Ñ>^Ñ&_Ð#á)<ØØØØØØ—n‘nØ—L‘LØ#Ÿ}Ÿ}‘C°$×2CÑ2Cñ * Ñ&ˆ ð#0×"4Ñ"4Ó"6°s¸Ð";¸t×?QÑ?QÐ>SÑ"SÐØ%×-Ñ-Ð.EÓFˆ æ6G= /Ð2ˆàˆðO\ÐM]ˆàˆr.)r¬r«rSr®r¯rŒr©r´r‹rrµr©NF)r$r%r&r'rrMrr)ÚTensorÚboolrr,rhr-rkrls@r/r¡r¡×smø†ð˜~ð°$÷ð6bgñ'Ø(0°·±Ñ(>ð'ØZ^ð'à ˆuU—\‘\ 5§<¡<Ð/Ñ0°%¸¿¹Ñ2EÐEÑ F÷'ó'r.r¡cóŠ^•\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r$)ÚVideoMAESelfOutputiz™ The residual connection is defined in VideoMAELayer instead of here (as is the case with other models), due to the layernorm applied before each block. rSr£NcóÌ>•[TU]5 [R"URUR5Ul[R"UR5UlgrK) rLrMrr°rQÚdenseÚDropoutÚhidden_dropout_probrrTs €r/rMÚVideoMAESelfOutput.__init__#sBø€Ü ‰ÑÔÜ—Y’Y˜v×1Ñ1°6×3EÑ3EÓFˆŒ Ü—z’z &×"<Ñ"<Ó=ˆr.r!Úinput_tensorcóJ•URU5nURU5nU$rK©rÛr©rUr!rßs r/rhÚVideoMAESelfOutput.forward(s$€ØŸ ™ =Ó1ˆ ØŸ™ ]Ó3ˆ àÐr.rá)r$r%r&r'r(rrMr)rÖrhr-rkrls@r/rÙrÙsIø†ñð >˜~ð>°$÷>ð U§\¡\ðÀÇÁðÐRW×R^ÑR^÷òr.rÙcóò^•\rSrSrS\SS4U4SjjrS\\SS4SjrS S\ RS \\ RS \S\ \\ R\ R4\\ R44SjjrSrU=r$)ÚVideoMAEAttentioni0rSr£Ncó€>•[TU]5 [U5Ul[ U5Ul[ 5UlgrK)rLrMr¡Ú attentionrÙÚoutputÚsetÚpruned_headsrTs €r/rMÚVideoMAEAttention.__init__1s0ø€Ü ‰ÑÔÜ.¨vÓ6ˆŒÜ(¨Ó0ˆŒÜ›EˆÕr.Úheadscó6•[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URlURR[U5- URlURRURR-URlUR RU5Ulg)Nrr©r“)Úlenrrçr©r«rêrr‹rŒrrèrÛr¬Úunion)rUrìÚindexs r/Úprune_headsÚVideoMAEAttention.prune_heads7s€Üˆu‹:˜‹?ØÜ7Ø—>‘>×5Ñ5°t·~±~×7YÑ7YÐ[_×[lÑ[ló ‰ˆô 2°$·.±.×2FÑ2FÈÓNˆ‰ÔÜ/°·±×0BÑ0BÀEÓJˆ‰ÔÜ1°$·.±.×2FÑ2FÈÓNˆ‰ÔÜ.¨t¯{©{×/@Ñ/@À%ÈQÑOˆ‰Ôð.2¯^©^×-OÑ-OÔRUÐV[ÓR\Ñ-\ˆ‰Ô*Ø'+§~¡~×'IÑ'IÈDÏNÉN×LnÑLnÑ'nˆ‰Ô$Ø ×-Ñ-×3Ñ3°EÓ:ˆÕr.r!r·r¸cóf•URXU5nURUSU5nU4USS-nU$)Nrr)rçrè)rUr!r·r¸Úself_outputsÚattention_outputrÓs r/rhÚVideoMAEAttention.forwardIsC€ð—~‘~ mÐ@QÓRˆàŸ;™; |°A¡¸ ÓFÐà#Ð%¨°Q°RÐ(8Ñ8ˆØˆr.)rçrèrêrÕ)r$r%r&r'rrMrér|ròr)rÖrr×rr,rhr-rkrls@r/rårå0s—ø†ð"˜~ð"°$÷"ð; S¡ð;¨dô;ð*-1Ø"'ñ à—|‘|ðð˜EŸL™LÑ)ðð ð ð ˆuU—\‘\ 5§<¡<Ð/Ñ0°%¸¿¹Ñ2EÐEÑ F÷ór.råcón^•\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) ÚVideoMAEIntermediateiXrSr£Ncó>•[TU]5 [R"URUR 5Ul[UR[5(a[URUlgURUlgrK)rLrMrr°rQÚintermediate_sizerÛrxÚ hidden_actÚstrr Úintermediate_act_fnrTs €r/rMÚVideoMAEIntermediate.__init__Ys`ø€Ü ‰ÑÔÜ—Y’Y˜v×1Ñ1°6×3KÑ3KÓLˆŒ Üf×'Ñ'¬×-Ñ-Ü'-¨f×.?Ñ.?Ñ'@ˆDÕ$à'-×'8Ñ'8ˆDÕ$r.r!cóJ•URU5nURU5nU$rK©rÛrþ)rUr!s r/rhÚVideoMAEIntermediate.forwardas&€ØŸ ™ =Ó1ˆ Ø×0Ñ0°Ó?ˆ àÐr.r©r$r%r&r'rrMr)rÖrhr-rkrls@r/rùrùXs6ø†ð9˜~ð9°$÷9ð U§\¡\ð°e·l±l÷òr.rùcó†^•\rSrSrS\SS4U4SjjrS\RS\RS\R4SjrS r U=r $) ÚVideoMAEOutputiirSr£NcóÌ>•[TU]5 [R"URUR 5Ul[R"UR5Ul grK) rLrMrr°rûrQrÛrÜrÝrrTs €r/rMÚVideoMAEOutput.__init__jsBø€Ü ‰ÑÔÜ—Y’Y˜v×7Ñ7¸×9KÑ9KÓLˆŒ Ü—z’z &×"<Ñ"<Ó=ˆr.r!rßcóR•URU5nURU5nX-nU$rKrárâs r/rhÚVideoMAEOutput.forwardos,€ØŸ ™ =Ó1ˆ ØŸ™ ]Ó3ˆ à%Ñ4ˆ àÐr.rárrls@r/rrisDø†ð>˜~ð>°$÷>ð U§\¡\ðÀÇÁðÐRW×R^ÑR^÷òr.rcóÞ^•\rSrSrSrS\SS4U4SjjrSS\RS\ \RS \ S\\\R\R4\\R44S jjr SrU=r$) Ú VideoMAELayeriyz?This corresponds to the Block class in the timm implementation.rSr£Ncój>•[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nr©Úeps)rLrMÚchunk_size_feed_forwardÚseq_len_dimrårçrùÚintermediaterrèrÚ LayerNormrQÚlayer_norm_epsÚlayernorm_beforeÚlayernorm_afterrTs €r/rMÚVideoMAELayer.__init__|s‡ø€Ü ‰ÑÔØ'-×'EÑ'EˆÔ$ØˆÔÜ*¨6Ó2ˆŒÜ0°Ó8ˆÔÜ$ VÓ,ˆŒÜ "§¢¨V×-?Ñ-?ÀV×EZÑEZÑ [ˆÔÜ!Ÿ|š|¨F×,>Ñ,>ÀF×DYÑDYÑZˆÕr.r!r·r¸cóÔ•URURU5UUS9nUSnUSSnXQ-nURU5nURU5nUR Xq5nU4U-nU$)N)r¸rr)rçrrrrè)rUr!r·r¸Úself_attention_outputsrörÓÚlayer_outputs r/rhÚVideoMAELayer.forward†s’€ð"&§¡Ø×!Ñ! -Ó0ØØ/ð"0ð" Ðð 2°!Ñ4ÐØ(¨¨Ð,ˆð)Ñ8ˆ ð×+Ñ+¨MÓ:ˆØ×(Ñ(¨Ó6ˆð—{‘{ <Ó?ˆà/ GÑ+ˆàˆr.)rçrrrrrèrrÕ)r$r%r&r'r(rrMr)rÖrr×rr,rhr-rkrls@r/rrys†ø†ÙIð[˜~ð[°$÷[ð-1Ø"'ñ à—|‘|ðð˜EŸL™LÑ)ðð ð ð ˆuU—\‘\ 5§<¡<Ð/Ñ0°%¸¿¹Ñ2EÐEÑ F÷ór.rcóš^•\rSrSrS\SS4U4SjjrS S\RS\\RS\ S \ S \ S\ \\44Sjjr SrU=r$)ÚVideoMAEEncoderi¤rSr£NcóÔ>•[TU]5 Xl[R"[UR5Vs/sHn[U5PM sn5UlSUl gs snfrÕ) rLrMrSrÚ ModuleListr7Únum_hidden_layersrÚlayerÚgradient_checkpointing)rUrSrfrVs €r/rMÚVideoMAEEncoder.__init__¥sRø€Ü ‰ÑÔØŒÜ—]’]Ä5È×IaÑIaÔCbÓ#cÑCb¸a¤M°&Ö$9ÑCbÑ#cÓdˆŒ Ø&+ˆÕ#ùò$ds½A%r!r·r¸Úoutput_hidden_statesÚreturn_dictcó6•U(aSOSnU(aSOSn[UR5H9up‰U(aXa4-nUbX(OSn U "XU5nUSnU(dM1X{S4-nM; U(aXa4-nU(d[SXU455$[UUUS9$)Nr#rrc3ó,# •UHocMUv• M g7frKr#©Ú.0Úvs r/Ú Ú*VideoMAEEncoder.forward..Çsé€ÐmÑ$[˜qŸ™Ò$[ùó‚‹ ©Úlast_hidden_stater!r")Ú enumerater r,r)rUr!r·r¸r#r$Úall_hidden_statesÚall_self_attentionsÚiÚlayer_moduleÚlayer_head_maskÚ layer_outputss r/rhÚVideoMAEEncoder.forward«s¹€ö#7™B¸DÐÞ$5™b¸4Ðä(¨¯©Ö4‰OˆAÞ#Ø$5Ð8HÑ$HÐ!à.7Ñ.C˜išlÈˆOá(¨ÐIZÓ[ˆMà)¨!Ñ,ˆMç Ð Ø&9È1Ñ=MÐ•[TU]U5 Xl[U5Ul[U5UlUR(aSUlO.[R"URURS9UlUR5 g)Nr )rLrMrSrHrdrÚencoderÚuse_mean_poolingÚ layernormrrrQrÚ post_initrTs €r/rMÚVideoMAEModel.__init__ésgø€Ü ‰Ñ˜Ô ØŒä,¨VÓ4ˆŒÜ& vÓ.ˆŒà×"×"Ø!ˆDNäŸ\š\¨&×*<Ñ*<À&×BWÑBWÑXˆDŒNð ‰Õr.có.•URR$rK)rdrO)rUs r/Úget_input_embeddingsÚ"VideoMAEModel.get_input_embeddingsøs€Ø‰×/Ñ/Ð/r.cóš•UR5H7up#URRURR U5 M9 g)z… Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)ÚitemsrNr rçrò)rUÚheads_to_pruner rìs r/Ú_prune_headsÚVideoMAEModel._prune_headsûs<€ð +×0Ñ0Ö2‰LˆEØL‰L×Ñ˜uÑ%×/Ñ/×;Ñ;¸EÖBò3r.rbrcr·r¸r#r$r£cóÚ•UbUOURRnUbUOURRnUbUOURRnUR X0RR 5nUR X5nURUUUUUS9nUSn URbURU 5n U(d U 4USS-$[U URURS9$)a‹ bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. If `None`, then all patches are considered. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> import av >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1568, 768] ```N©r·r¸r#r$rrr-)rSr¸r#Úuse_return_dictÚ get_head_maskrrdrNrPrr!r") rUrbrcr·r¸r#r$Úembedding_outputÚencoder_outputsÚsequence_outputs r/rhÚVideoMAEModel.forwardsþ€ðr2CÑ1NÑ-ÐTX×T_ÑT_×TqÑTqÐà$8Ñ$DÑ È$Ï+É+×JjÑJjð ð&1Ñ%<‘kÀ$Ç+Á+×B]ÑB]ˆð×&Ñ& y·+±+×2OÑ2OÓPˆ àŸ?™?¨<ÓIÐàŸ,™,ØØØ/Ø!5Ø#ð'ð ˆð*¨!Ñ,ˆØ>‰>Ñ%Ø"Ÿn™n¨_Ó=ˆOæØ#Ð%¨¸¸Ð(;Ñ;Ð;äØ-Ø)×7Ñ7Ø&×1Ñ1ñ ð r.)rSrdrNrP)NNNNN)r$r%r&r'rMrTrYrr)r*rÚ BoolTensorrÖr×rr,rrhr-rkrls@r/rLrLçsµø†õ ò0òCðð7;Ø,0Ø,0Ø/3Ø&*ñy à×'Ñ'ðy ð" %×"2Ñ"2Ñ3ðy ð˜EŸL™LÑ)ð y ð $ D™>ðy ð' t™nð y ð˜d‘^ðy ð ˆuoÐ%Ñ &ôy óöy r.rLcó8^•\rSrSrU4SjrSSjrSrU=r$)ÚVideoMAEDecoderi€có>•[TU]5 URUR-URS--n[U5nURUlURUl URUlURUl [R"[!UR5Vs/sHn[#U5PM sn5Ul[R&"UR5UlUS:”a![R*"URU5O[R,"5UlSUlXlgs snf)Nr6rF)rLrMrgrwrurÚdecoder_hidden_sizerQÚdecoder_num_hidden_layersrÚdecoder_num_attention_headsr©Údecoder_intermediate_sizerûrrr7rÚdecoder_layersrÚnormr°ÚIdentityÚheadr!rS)rUrSrPÚdecoder_num_labelsÚdecoder_configrfrVs €r/rMÚVideoMAEDecoder.__init__s ø€Ü ‰ÑÔà#×0Ñ0°6×3FÑ3FÑFÈ×IZÑIZÐ\]ÑI]Ñ]Ðä! &Ó)ˆØ%+×%?Ñ%?ˆÔ"Ø+1×+KÑ+KˆÔ(Ø-3×-OÑ-OˆÔ*Ø+1×+KÑ+KˆÔ(Ü ŸmšmÜ49¸&×:ZÑ:ZÔ4[Ó\Ñ4[¨qŒ]˜>Ö *Ñ4[Ñ\ó ˆÔô—L’L ×!;Ñ!;Ó<ˆŒ àI[Ð^_ÓI_ŒBIŠIf×0Ñ0Ð2DÔEÔeg×epÒepÓerð Œ ð',ˆÔ#Øùò ]sÂ/Ecóˆ•U(aSOSnU(aSOSn[UR5H/up‰U(aXa4-nU "USUS9n U SnU(dM'XzS4-nM1 U(aXa4-nUS:”aUSS2U*S24nURU5nURU5nU(d[ SX¶U455$[X¶US9$)Nr#)r·r¸rrc3ó,# •UHocMUv• M g7frKr#r's r/r*Ú*VideoMAEDecoder.forward..¸sé€ÐfÑ$T˜qŸ™Ò$Tùr,)r r!r")r/rkrlrnr,r)rUr!Úreturn_token_numr¸r#r$r0r1r2r3r5r s r/rhÚVideoMAEDecoder.forward—sã€ö#7™B¸DÐÞ$5™b¸4ÐÜ(¨×)<Ñ)<Ö=‰OˆAÞ#Ø$5Ð8HÑ$HÐ!á(¨À$ÐZkÑlˆMà)¨!Ñ,ˆMç Ð Ø&9È1Ñ=MÐö Ø 1Ð4DÑ DÐà˜aÓØ)ª!Ð.>Ð->Ñ-?Ð*?Ñ@ˆMðŸ ™ -Ó0ˆ Ø—‘˜=Ó)ˆæÜÑf VÐ@SÑ$TÓfÓfÐfÜ$¨FÐ`sÑtÐtr.)rSrkr!rnrl)FFT)r$r%r&r'rMrhr-rkrls@r/rere€sø†õð4 Ø"Ø÷ "uò"ur.rezb The VideoMAE Model transformer with the decoder on top for self-supervised pre-training. cóÂ^•\rSrSrU4Sjr\SS\RS\RS\ \RS\ \S\ \S\ \S \\ \44S jj5rSrU=r$) ÚVideoMAEForPreTrainingi¼có>•[TU]U5 Xl[U5Ul[ R"URURSS9Ul [ R"[R"SSUR55Ul [URRR UR5Ul[%XRRR S9UlUR)5 g)NFr§r)rP)rLrMrSrLr9rr°rQrgÚencoder_to_decoderr²r)r³Ú mask_tokenrFrdrPrRreÚdecoderrQrTs €r/rMÚVideoMAEForPreTraining.__init__Âs¸ø€Ü ‰Ñ˜Ô ØŒä% fÓ-ˆŒ ä"$§)¢)¨F×,>Ñ,>À×@ZÑ@ZÐafÑ"gˆÔÜŸ,š,¤u§{¢{°1°a¸×9SÑ9SÓ'TÓUˆŒÜ#>ØM‰M×$Ñ$×0Ñ0°&×2LÑ2Ló$ ˆÔ ô' v¿=¹=×;SÑ;S×;_Ñ;_Ñ`ˆŒð ‰Õr.rbrcr·r¸r#r$r£cót•UbUOURRnURUUUUUUS9nUSnURU5nURupšnUc[S5eURRU SS5RU5nUR5RURSS9nXÂ)RU SU5n XÂRU SU5n[R"X-URU-/SS 9nUR!XþRS5nUR"nSn[R$"5 URR&S :waUnO„URnUR(n[R*"[,5RUUS9SSSS2SS4n[R*"[.5RUUS9SSSS2SS4nUU-U-nURun nnnnURR0URR2nnURR4(aÒUR7U UU-UUUU-UUU-U5nUR9SSSS SSSS 5R;5nUR7U UU-U-U-U-U-UU-U-U5nUUR=SSS9- UR?SSSS9RA5S--nUR7U UU-U-U-U-U-UU-U-U-5nO—URR&S :wa[S5eUR7U UU-UUUU-UUU-U5nUR9SSSS SSSS 5R;5nUR7U UU-U-U-U-U-UU-U-U-5nURun nnUURU SU5n SSS5 [C5n!U!"UW 5nU(dU4USS-n"UbU4U"-$U"$[EUUURFURHS9$!,(df N\=f)ae bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> from transformers import AutoImageProcessor, VideoMAEForPreTraining >>> import numpy as np >>> import torch >>> num_frames = 16 >>> video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224))) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base") >>> pixel_values = image_processor(video, return_tensors="pt").pixel_values >>> num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2 >>> seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame >>> bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss = outputs.loss ```N)rcr·r¸r#r$rz!One must provided a boolean mask r\TrYrrîr)rZr”r‚ér6éér’)r“Úkeepdim)r“Úunbiasedr‚gíµ ÷Æ°>zQCan't unnormalize non-RGB images. Consider setting config.norm_pix_loss to False.©r3r r!r")%rSr]r9rzr`rƒrRÚexpandr^r]r_rZrar)Úcatr{r|r Úno_gradrgr”Ú as_tensorrrrwruÚ norm_pix_lossrÂr„r›r;ÚvarÚsqrtrr1r!r")#rUrbrcr·r¸r#r$rÓrareÚseq_lenrgÚexpanded_position_embeddingsÚpos_emb_visibleÚpos_emb_maskÚx_fullÚdecoder_outputsr r3ÚframesrZr”r;r<Útimer‡rˆrwruÚframes_normÚvideos_patchrfÚlabelsÚloss_fctrès# r/rhÚVideoMAEForPreTraining.forwardÓsÇ€ðJ&1Ñ%<‘kÀ$Ç+Á+×B]ÑB]ˆà—-‘-ØØ+ØØ/Ø!5Ø#ð ð ˆð" !™*ˆØ×1Ñ1Øó ˆð-<×,AÑ,AÑ)ˆ ˜\ðÑ"ÜÐ@ÓAÐAØ'+×'?Ñ'?×'FÑ'FÀzÐSUÐWYÓ'Z×'bÑ'bÐcoÓ'pÐ$Ø'C×'JÑ'JÓ'L×'OÑ'OÐWc×WjÑWjÐquÐ'OÐ'vÐ$Ø6Ð7GÑH×PÑPÐQ[Ð]_ÐamÓnˆØ3ÑD×LÑLÈZÐY[Ð]iÓjˆô—’˜OÑ=¸t¿¹ÐQ]Ñ?]Ð^ÐdeÑfˆðŸ,™, v×/AÑ/AÀ!Ñ/DÓEˆØ ×'Ñ'ˆàˆÜ ]Š]_à{‰{×'Ñ'¨1Ó,à%‘ð&×,Ñ,Ø$×*Ñ*Ü—’Ô'<Ó=×@Ñ@ÈÐV[Ð@Ð\Ð]aÐcgÒijÐlpÐrvÐ]vÑwÜ—o’oÔ&:Ó;×>Ñ>ÀfÐTYÐ>ÐZÐ[_ÐaeÒghÐjnÐptÐ[tÑuØ%¨Ñ+¨dÑ2àð[ ð' t™nð [ ð˜d‘^ð[ ð ˆuÐ2Ð2Ñ 3ô[ óö[ r.rxz VideoMAE Model transformer with a video classification head on top (a linear layer on top of the average pooled hidden states of all tokens) e.g. for ImageNet. cóÒ^•\rSrSrU4Sjr\SS\\RS\\RS\\RS\\ S\\ S\\ S \ \\44S jj5r SrU=r$) ÚVideoMAEForVideoClassificationircó >•[TU]U5 URUl[U5UlUR (a [R"UR5OSUl URS:”a+[R"URUR5O[R"5UlUR5 g)Nr)rLrMÚ num_labelsrLr9rOrrrQÚfc_normr°rmÚ classifierrQrTs €r/rMÚ'VideoMAEForVideoClassification.__init__ys‘ø€Ü ‰Ñ˜Ô à ×+Ñ+ˆŒÜ% fÓ-ˆŒ ð 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import torch >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) eating spaghetti ```Nr\rrÚ regressionÚsingle_label_classificationÚmulti_label_classificationr\r„)rSr]r9rr;ržÚproblem_typerœr”r)Úlongr|rÚsqueezer rÂr rr!r") rUrbr·r–r¸r#r$rÓrar r3r—rès r/rhÚ&VideoMAEForVideoClassification.forward†sü€ðx&1Ñ%<‘kÀ$Ç+Á+×B]ÑB]ˆà—-‘-ØØØ/Ø!5Ø#ð ð ˆð" !™*ˆà<‰<Ñ#Ø"Ÿl™l¨?×+?Ñ+?ÀÓ+BÓC‰Oà-ªa°¨dÑ3ˆOà—‘ Ó1ˆàˆØÒØ{‰{×'Ñ'Ñ/Ø—?‘? aÓ'Ø/;D—K‘KÕ,Ø—_‘_ qÓ(¨f¯l©l¼e¿j¹jÓ.HÈFÏLÉLÔ\a×\eÑ\eÓLeØ/LD—K‘KÕ,à/KD—K‘KÔ,à{‰{×'Ñ'¨<Ó7Ü"›9Ø—?‘? aÓ'Ù# F§N¡NÓ$4°f·n±nÓ6FÓG‘Dá# FÓ3‘DØ—‘×)Ñ)Ð-JÓJÜ+Ó-Ù §¡¨B°·±Ó @À&Ç+Á+ÈbÃ/ÓR‘Ø—‘×)Ñ)Ð-IÓIÜ,Ó.Ù Ó/æØY ¨¨ Ñ,ˆFØ)-Ñ)9TG˜fÑ$ÐE¸vÐEä$ØØØ!×/Ñ/Ø×)Ñ)ñ ð r.)ržrrœr9)NNNNNN)r$r%r&r'rMrrr)rÖr×rr,rrhr-rkrls@r/ršršrs¯ø†õðð04Ø,0Ø)-Ø,0Ø/3Ø&*ñN à˜uŸ|™|Ñ,ðN ð˜EŸL™LÑ)ðN ð˜Ÿ™Ñ&ð N ð $ D™>ðN ð' t™nð N ð˜d‘^ðN ð ˆuÐ+Ð+Ñ ,ôN óöN r.rš)rxrLr8rš)r¿)Cr(Úcollections.abcryr[rÚdataclassesrÚtypingrrrÚnumpyr8r)Útorch.utils.checkpointrÚtorch.nnr r rÚactivationsr Úmodeling_layersrÚmodeling_outputsrrÚmodeling_utilsrrÚ pytorch_utilsrrÚutilsrrrÚutils.constantsrrÚconfiguration_videomaerÚ get_loggerr$rÄrr1rFÚModulerHrNrÖÚfloatrŸr¡rÙrårùrrrr8rLrerxršÚ__all__r#r.r/Úrºsbðñ3ãÝÝ!ß,Ñ,ãÛÛÝßAÑAå!Ý9ßFßFßQ÷ñ÷ KÝ2ð × Ò ˜HÓ %€ðÙðñô :˜Kó:óóð:ðÙðñô : ;ó:óóð:ò :ô˜Ÿ™ôôB2˜bŸi™iô2ðzñ%ØI‰Ið%à<‰<ð%ð ‰ð%ð<‰<ð %ð ˜UŸ\™\Ñ*ð%ðð %ðõ%ô