"S1S2\ Rd5r?\$"S3S4\55r@\$"S5S6\@55rA"S7S8\ Rd5rB\$"S9S9"S:S;\@55rC\$"S\@55rD"S?S@\ Rd5rE"SASB\ Rd5rF"SCSD\ Rd5rG"SESF\ Rd5rH"SGSH\ Rd5rI\$"SISJ\@55rJ\$"SKS9"SLSM\@\(55rK/SNQrLg)PzPyTorch BEiT model.N) dataclass)OptionalUnion)Tensornn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)PreTrainedModel)#compile_compatible_method_lru_cache find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) BackboneMixin) BeitConfigz- Class for outputs of [`BeitModel`]. ) custom_introc\rSrSrSrSrg)BeitModelOutputWithPooling0a2 pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__r"^/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/beit/modeling_beit.pyr r 0sr)r input drop_probtrainingreturncUS:XdU(dU$SU- nURS4SURS- --nU[R"X@RUR S9-nUR 5 URU5U-nU$)a* Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rr)r)dtypedevice)shapendimtorchrandr1r2floor_div)r+r,r- keep_probr3 random_tensoroutputs r* drop_pathr<?sCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr)c^\rSrSrSrS S\\SS4U4SjjjrS\RS\R4Sjr S\ 4S jr S r U=r$) BeitDropPathSzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nr,r.c.>[TU]5 XlgN)super__init__r,)selfr, __class__s r*rCBeitDropPath.__init__Vs "r) hidden_statescB[XRUR5$rA)r<r,r-rDrGs r*forwardBeitDropPath.forwardZs FFr)c SUR3$)Nzp=r,rDs r* extra_reprBeitDropPath.extra_repr]sDNN#$$r)rMrA)r#r$r%r&r'rfloatrCr5rrJstrrOr( __classcell__rEs@r*r>r>SsQb#(5/#T##GU\\GellG%C%%r)r>c ^\rSrSrSrS\SS4U4SjjrS\RS\ S \ S\R4S jr SS \RS \ \RS \ \ S\R4SjjrSrU=r$)BeitEmbeddingscz[ Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configr.Nc^>[TU]5 [R"[R "SSUR 55UlUR(a<[R"[R "SSUR 55Ul OSUl [U5Ul URUl [UR[R R"5(a UROURUR4UlURR$nUR&(a?[R"[R "SUS-UR 55UlOSUl[R*"UR,5Ulg)Nr)rBrCr Parameterr5zeros hidden_size cls_tokenuse_mask_token mask_tokenBeitPatchEmbeddingspatch_embeddings patch_size isinstance image_size collectionsabcIterable num_patches use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)rDrXrhrEs r*rCBeitEmbeddings.__init__is' ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO 3F ; ++&++[__-E-EFF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r) embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rr bicubicFsizemode align_cornersdim)r3rjr5jit is_tracingrbrreshapepermuter functional interpolateviewcat) rDrorprqrh num_positionsclass_pos_embedpatch_pos_embedr{ new_height new_widthsqrt_num_positionss r*interpolate_pos_encoding'BeitEmbeddings.interpolate_pos_encodingsS!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr) pixel_valuesbool_masked_posrc,URbUb[R"S5 URu pEnUR U5unupUR 5upnUbIUR RXS5n URS5RU 5n USU - -X--nURRU SS5n[R"X4SS9nURbXpRXuU5-nURU5nXxU 44$)Nz`interpolate_pos_encoding` argument has no effect for BEiTEmbeddings, embeddings are always interpolated to the input image size. The argument will be removed in transformers v4.51.0.rsrrz)rjwarningswarnr3rarwr_expand unsqueezetype_asr]r5rrrm)rDrrr_rprqro patch_height patch_width batch_sizeseq_len mask_tokensw cls_tokenss r*rJBeitEmbeddings.forwards  # # /4L4X MMn  +001e262G2G 2U/ /\!+!2 Q  &//00bIK))"-55kBA#q1u- ?J^^**:r2> YY 7Q?  # # /#&C&CJX]&^^J\\*- +666r))r]rmrdr_rarbrjNN)r#r$r%r&r'rrCr5rintrr BoolTensorboolrJr(rSrTs@r*rVrVcs >z>d>.&D5<<&D&DUX&D]b]i]i&DV7;37 7ll7"%"2"237#+4. 7  77r)rVcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)r`z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c>[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nUSUS-USUS-4nX lX0lX@lX`l Xpl [R"XEX3S9Ulg)Nrr kernel_sizestride)rBrCrdrb num_channelsr\rcrerfrgrh patch_shaperConv2d projection) rDrXrdrbrr\rhrrEs r*rCBeitPatchEmbeddings.__init__s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L:ir)rr.cURup#pEX0R:wa [S5eURU5nURSURSpUR S5R SS5nXgU44$)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rtr r)r3r ValueErrorrflatten transpose) rDrrrrprqrorrs r*rJBeitPatchEmbeddings.forwards2>2D2D/ & ,, ,w __\2 $.$4$4Q$79I9I!9Lk''*44Q: +666r))rdrrhrrbr) r#r$r%r&r'rCr5rrJr(rSrTs@r*r`r`s. j" 7ELL 7U\\ 7 7r)r`c ^\rSrSrSS\S\\SS4U4SjjjrSS\RS\\RS \ S \\RS \ S \\\ S\ \\R\\R\R444S jjr SrU=r$)BeitSelfAttentionNrX window_sizer.cJ>[TU]5 XlURUR-S:wa7[ US5(d&[ SURSURS35eURUl[URUR- 5UlURUR-Ul [R"URUR5Ul [R"URURSS9Ul [R"URUR5Ul[R"UR 5Ul[%U5UlUR&(a[)XS9Ulgg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .F)biasr)rBrCrXr\num_attention_headshasattrrrattention_head_size all_head_sizerLinearquerykeyvaluerkattention_probs_dropout_probrmrhas_relative_position_biasBeitRelativePositionBiasrelative_position_biasrDrXrrEs r*rCBeitSelfAttention.__init__sN     : : :a ?PVXhHiHi"6#5#5"67334A7  $*#=#= #&v'9'9FS"S%**,CD 6G=/2O\M]r)) rrrXrmrrrrrrrANFNFN)r#r$r%r&rrtuplerCr5rrrrrJr(rSrTs@r*rrsdzddSWdd4-1"'9=).+/>||>ELL)> > !) 6 > #' >U3Z(> uU\\"E%,, *D$EE F>>r)rc^\rSrSrS S\R S\\R S\S\\R S\S\\\ S\ \\R \\R \R 444U4S jjjr S r U=r $) BeitSdpaSelfAttentioniHrGrrrrrr.c >U(dUb'[RS5 [TU] UUUUUUS9$URupxn UR U5R USURUR5RSS5n URU5R USURUR5RSS5n URU5R USURUR5RSS5n Sn UR(aQUupXRR-XRR-4nURUXQRSS9n Ub U cUn OX- n S[ R""UR5- n[$R&R(R+U U U U UR,(aURR.OSSUS 9nUR1S SSS 5R35nUR55SS UR64-nUR "U6nUS4$) Na`BeitSdpaSelfAttention` is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rGrrrrrrsrrtrr0F) attn_mask dropout_p is_causalscalerr r)logger warning_oncerBrJr3rrrrrrrrrXrbrrrr5rrscaled_dot_product_attentionr-rrrrwr)rDrGrrrrrrrrrrr attn_biasrprqrscalingrrrEs r*rJBeitSdpaSelfAttention.forwardIsH  5   w  7?+#"3'=)A% # %2$7$7!  JJ} % T*b$":":D>UXI &--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD d""r)r"r)r#r$r%r&r5rrrrrrrJr(rSrTs@r*rrHs-1"'9=).+/F#||F#ELL)F# F# !) 6 F# #' F#U3Z(F# uU\\"E%,, *D$EE FF#F#r)rc^\rSrSrSrS\SS4U4SjjrS S\RS\RS\R4S jjr S r U=r $) BeitSelfOutputiz The residual connection is defined in BeitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rXr.Nc>[TU]5 [R"URUR5Ul[R "UR5UlgrA) rBrCrrr\denserkrlrmrDrXrEs r*rCBeitSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r)rG input_tensorcJURU5nURU5nU$rArrm)rDrGrgammas r*rJBeitSelfOutput.forward$ =1  ]3 r)rrA) r#r$r%r&r'rrCr5rrJr(rSrTs@r*rrsI >z>d> U\\^c^j^jr)r)eagersdpac&^\rSrSrSS\S\\SS4U4SjjjrSrSS\ RS \\ RS \ S \\ RS \ S \\\ S\ \\ R\\ R\ R444SjjrSrU=r$) BeitAttentioniNrXrr.c>[TU]5 [UR"XS9Ul[ U5Ul[5Ulg)Nr) rBrCBEIT_SELF_ATTENTION_CLASSES_attn_implementation attentionrr;set pruned_headsrs r*rCBeitAttention.__init__s< 4V5P5PQRXr$V, Er)c6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrrz)lenrrrrrrrrrr;rrunion)rDheadsindexs r* prune_headsBeitAttention.prune_headss u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r)rGrrrrrchURXX4XV5nURUSU5nU4USS-n U $)Nrr)rr;) rDrGrrrrr self_outputsattention_outputrs r*rJBeitAttention.forwardsK~~ &7Qi  ;;|A F#% QR(88r))rr;rrAr)r#r$r%r&rrrrCr r5rrrrrJr(rSrTs@r*rrs"z""SW"" ;*-1"'9=).+/||ELL)  !) 6  #' U3Z( uU\\"E%,, *D$EE Fr)rcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) BeitIntermediateirXr.Nc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl grA) rBrCrrr\intermediate_sizerrc hidden_actrRr intermediate_act_fnrs r*rCBeitIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r)rGcJURU5nURU5nU$rArrrIs r*rJBeitIntermediate.forwards& =1 00? r)r r#r$r%r&rrCr5rrJr(rSrTs@r*rrs69z9d9U\\ellr)rcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) BeitOutputirXr.Nc>[TU]5 [R"URUR 5Ul[R"UR5Ul grA) rBrCrrrr\rrkrlrmrs r*rCBeitOutput.__init__sB YYv779K9KL zz&"<"<= r)rGcJURU5nURU5nU$rArrIs r*rJBeitOutput.forwardrr)rrrTs@r*rrs6>z>d> U\\ellr)rc,^\rSrSrSrSS\S\\S\SS4U4Sjjjr SS \ RS \\ RS \ S \\ RS \ S\\\ \ 4S\\\ R\\ R\ R444SjjrSrU=r$) BeitLayeriz?This corresponds to the Block class in the timm implementation.NrXrdrop_path_rater.c>[TU]5 URUlSUl[ XS9Ul[ U5Ul[U5Ul [R"URURS9UlUS:a [U5O[R "5Ul[R"URURS9UlUR&nUS:aw[R("U[*R,"UR5-SS9Ul[R("U[*R,"UR5-SS9UlgSuUlUlg) Nrrepsr0rT) requires_gradr)rBrCchunk_size_feed_forward seq_len_dimrrr intermediaterr;r LayerNormr\layer_norm_epslayernorm_beforer>Identityr<layernorm_afterlayer_scale_init_valuerZr5oneslambda_1lambda_2)rDrXrr" init_valuesrEs r*rCBeitLayer.__init__s '-'E'E$&vG,V4 ( " V-?-?VEZEZ [9G#9Mn5SUS^S^S`!||F,>,>FDYDYZ33 ?LLuzz&BTBT7U)UeijDMLLuzz&BTBT7U)UeijDM+5 (DM4=r)rGrrrrrc URURU5UUUUUS9nUSnUSSn URbURU-nURU5U-nUR U5n UR U 5n UR U 5n URbURU -n URU 5U-n U 4U -n U $)N)rrrrrr)rr,r1r<r.r)r;r2) rDrGrrrrrself_attention_outputsr r layer_outputs r*rJBeitLayer.forwards"&  ! !- 0 /#9%=! "0" 2!4(, == $#}}/?? '78=H ++M: ((6 {{<0 == $==<7L~~l3mC /G+r)) rr'r<r)r1r2r.r,r;r()Nr0r)r#r$r%r&r'rrrrQrCr5rrrrrJr(rSrTs@r*r!r!sI6z66`e6pt66*-1"'9=).04)||)ELL)) ) !) 6 ) #' )U38_-) uU\\"E%,, *D$EE F))r)r!c^\rSrSrS\S\SS4U4Sjjr\"SS9S\\\4S\ R4S j5r S S \ S\ R4S jjr S rU=r$)ri4rXrr.Nc>[TU]5 X lSUS-S- SUS-S- -S-Ul[R "[ R"URUR55Ul g)Nrtrrr ) rBrCrnum_relative_distancerrZr5r[rrelative_position_bias_tablers r*rC!BeitRelativePositionBias.__init__5sp &&'+a.&81&<[QR^ASVWAW%X[\%\",.LL KK22F4N4N O- )r) )maxsizecSUS-S- SUS-S- -S-nUSUS-n[R"[R"US5[R"US5SS9n[R"U5n[R"US5nUSS2SS2S4USS2SSS24- nUR SSS5R 5nUSS2SS2S4==USS- - ss'USS2SS2S4==USS- - ss'USS2SS2S4==SUS-S- -ss'[R"US-4S-URS9nURS 5USS2SS24'US- USSS24'US- USS2S4'US- US 'U$) z This method creates the relative position index, modified to support arbitrary window sizes, as introduced in [MiDaS v3.1](https://huggingface.co/papers/2307.14460). rtrrr ij)indexingN)rwr1rs)rr) r5meshgridarangestackrrrr[r1sum) rDrr; window_areagridcoordscoords_flattenrelative_coordsrelative_position_indexs r* generate_relative_position_index9BeitRelativePositionBias.generate_relative_position_index>s "#[^!3a!7A A:ELLUV)B12&)>)BA&(=(A%&&r)rcSURS-S- nSURS-S- nSUS-S- nSUS-S- nURnURn Xg-S-n USU S- n U RSXTS5R SSSS5n [ R RU [U5[U54SS9n U R SSSS5RU S- S5n [R"XU S- S/5n URU5nXRS5nURUSUS-S-USUS-S-S5nUR SSS5R5nU(a?[ R RURS5X34SS S 9RS5nURS5$) ze Modification of timm.models.beit.py: Attention._get_rel_pos_bias to support arbitrary window sizes. rtrrr Nrsbilinear)rwrxFrv)rr<r;r~rrrrrr5rrMrrrsqueeze)rDrrr old_height old_widthrr old_relative_position_bias_tableold_num_relative_distancenew_num_relative_distance old_sub_table new_sub_table new_relative_position_bias_tablerLrs r*rJ BeitRelativePositionBias.forwardWs!))!,,q0 ((++a/ Q'!+  A&* +/+L+L($($>$>!$.$:Q$>!89X;TWX;XY %--aKSSTUWXZ[]^_  11 :!6 )8L MT^2 &--aAq9AAB[^_B_acd +099 =VYZ=Z=\] ^, (#'"G"G "T!AB^B^_aBb!c"8!%>&003)# &?& gaj #&//22r))r;r<r)FN)r#r$r%r&rrrCrrr5rrMrrJr(rSrTs@r*rr4sr z  $ )4'E#s(O'PUP\P\'5'0-3T-3]b]i]i-3-3r)rc^\rSrSrSS\S\\SS4U4SjjjrSS\RS\\RS \ S \ S \ S \\\ \ 4S \ S\ \\ 44SjjrSrU=r$) BeitEncoderiNrXrr.c >[TU]5 XlURUlUR(a[ XS9Ul[R"SURURSS9Vs/sHo3R5PM nn[R"[UR5Vs/sH#n[UUR (aUOSXES9PM% sn5UlSUlgs snfs snf)Nrrcpu)r2)rr"F)rBrCrX!use_shared_relative_position_biasrrrr5linspacer"num_hidden_layersitemr ModuleListranger!use_relative_position_biaslayergradient_checkpointing)rDrXrxdprirEs r*rCBeitEncoder.__init__s  *0*R*R'  * **B6*cD '"'63H3H&JbJbkp!qr!qAvvx!qr]]v778   9A /5/P/P VZ#&6 9     ',#s s 3C14)C6rGrroutput_hidden_statesrr return_dictc U(aSOSnU(aSOSn [UR5HupU(aX4-nUR(aRUupXRR-XRR-4nUR XUR SS9nOSnUbX*OSnU "UUUUUUS9nUSnU(dMU US4-n M U(aX4-nU(d[SXU 455$[UUU S9$)Nr"r)rr)rrrrrrc3,# UH ocMUv M g7frAr").0vs r* &BeitEncoder.forward..sm$[q$[s )last_hidden_staterG attentions) enumeraterfrrXrbrr3rr)rDrGrrrlrrrmall_hidden_statesall_self_attentionsrj layer_modulerprqrrlayer_head_mask layer_outputss r*rJBeitEncoder.forwards1#7BD$5b4(4OA#$58H$H!.. * %)?)??++J`J`A`a )-)D)D]j]p]pqr]s*E*&*.&.7.CilO()"3'=)A% M*!,M  &9]1=M.  - - KK   " " $ MM   $ $S )  / /    ! ! ' ' )  ,!!&&,,.))5**//5576  8 9 9  / / 4 4 : : <  * **$$**4;;+M+MN$$**4;;+M+MN++r)r"N)r#r$r%r&r__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules"_keys_to_ignore_on_load_unexpected_supports_sdparr(r"r)r*r~r~s5 $O&*#$ *H)I&NOr)r~c^\rSrSrSS\S\SS4U4SjjjrSrSr\ SS \ RS \ \ RS \ \ RS \ \S \ \S\S\ \S\\\44Sjj5rSrU=r$) BeitModelirXadd_pooling_layerr.Nc>[TU]U5 Xl[U5Ul[ XRR RS9UlUR(a[R"5O([R"URURS9UlU(a [!U5OSUlUR%5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rr$N)rBrCrXrVror\rarencoderuse_mean_poolingrr-r*r\r+ layernorm BeitPoolerpooler post_init)rDrXrrEs r*rCBeitModel.__init__s   (0"67W7W7c7cd $44BKKM",,vGYGY_e_t_t:u ->j(4  r)c.URR$rArorarNs r*get_input_embeddingsBeitModel.get_input_embeddings///r)cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrfrr )rDheads_to_prunerfrs r* _prune_headsBeitModel._prune_headss< +002LE LL  u % / / ; ;E B3r)rrrrrlrrmc 2UbUOURRnUbUOURRnUbUOURRnUR X0RR 5nUR XS9upURSSn URUUUUU UUS9n U Sn URU 5n URbURU 5OSn U(dU bX4OU 4nXSS-$[U U U RU RS9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)rrt)rrrlrrmrrr)rt pooler_outputrGru)rXrrluse_return_dict get_head_maskraror3rrrr rGru)rDrrrrrlrrmembedding_outputrrencoder_outputssequence_output pooled_output head_outputss r*rJBeitModel.forwards;2C1N-TXT_T_TqTq$8$D $++JjJj &1% 4 'tn 4 #'4 d^4  u00 14 4 r)rcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) riRrXr.Nc>[TU]5 UR(a/[R"UR UR S9UlgSUlg)Nr$)rBrCrrr*r\r+rrs r*rCBeitPooler.__init__SsA KQKbKbBLL++1F1F G hl r)rGcURb0USS2SS2SS24nURURS55nU$USS2S4nU$)Nrr)rr)rDrG patch_tokensrs r*rJBeitPooler.forwardYsU >> %(AB2L NN<+<+[TU]U5 URUl[USS9Ul[ R "URURS9Ul [ R"URUR5Ul UR5 g)NFrr$)rBrC num_labelsrrrr*r\r+rr vocab_sizelm_headrrs r*rC#BeitForMaskedImageModeling.__init__nsu   ++f> f&8&8f>S>STyy!3!3V5F5FG  r)cgrAr"rNs r*get_output_embeddings0BeitForMaskedImageModeling.get_output_embeddings{sr)rrrlabelsrrlrrmc lUbUOURRnURUUUUUUUS9n U Sn URU 5n UR U SS2SS245n Sn Ub[ 5n U "XU5n U(dU 4U SS-nU bU 4U-$U$[ U U U RU RS9$)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```N)rrrrlrrmrrlosslogitsrGru) rXrrrrr rrGru)rDrrrrrrlrrmrrprediction_scoresmasked_lm_lossloss_fctr;s r*rJ"BeitForMaskedImageModeling.forwardsX&1% I 'tnI #'I d^I  un$ %I I r)rz Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. c^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \ S \\ S\ \ \ 44S jj5rSrU=r$)BeitForImageClassificationirXr.Nc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 g)NTrr) rBrCrrrrrr\r- classifierrrs r*rC#BeitForImageClassification.__init__ss   ++f= OUN_N_bcNc"))F$6$68I8IJikititiv r)rrrrrlrrmc PUbUOURRnURUUUUUUS9nU(a UROUSn UR U 5n Sn UGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5n UR S:Xa&U "U R5UR55n OU "X5n OURR S:Xa=[5n U "U RSUR 5URS55n O,URR S:Xa[5n U "X5n U(dU 4USS-n U bU 4U -$U $[!U U UR"UR$S 9$) ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrlrrmr regressionsingle_label_classificationmulti_label_classificationrsrtr)rXrrrr problem_typerr1r5longrr rQr rrrrGru)rDrrrrrlrrmrrrrrr;s r*rJ"BeitForImageClassification.forwards"&1% = 'tn = #'= d^=  u++ ,= = r)rc^\rSrSrSrSS\S\S\\\\\44S\\\\\4\4S\ S\\\\\44S S 4U4S jjjr S \ RS \ R4S jr SrU=r$)BeitConvModulei"a4 A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. in_channels out_channelsrpaddingrdilationr.Nc >[TU]5 [R"UUUUUUS9Ul[R "U5Ul[R"5Ulg)N)rrrrrr) rBrCrrconv BatchNorm2dbnReLU activation)rDrrrrrrrEs r*rCBeitConvModule.__init__*sQ II#%#   ...'')r)r+clURU5nURU5nURU5nU$rA)rrr)rDr+r;s r*rJBeitConvModule.forward?s05!( r))rrr)rFr)r#r$r%r&r'rrrrRrrCr5rrJr(rSrTs@r*rr"s5601$$$3c3h/0 $ sE#s(OS01 $  $U38_,-$ $$*U\\ellr)rcv^\rSrSrS\S\S\SS4U4SjjrS\RS\R4S jrS r U=r $) BeitPyramidPoolingBlockiG pool_scalerchannelsr.Nc>[TU]5 [R"U5[ X#SS9/Ul[ UR 5H upEUR[U5U5 M" g)Nrr) rBrCrAdaptiveAvgPool2drlayersrv add_modulerR)rDrrrrjrfrEs r*rC BeitPyramidPoolingBlock.__init__HsX    , ;a @  "$++.HA OOCFE */r)r+c@UnURH nU"U5nM U$rAr)rDr+ hidden_staterfs r*rJBeitPyramidPoolingBlock.forwardQs% [[E .L!r)r) r#r$r%r&rrCr5rrJr(rSrTs@r*rrGsD+3+S+C+D+U\\ellr)rc ^\rSrSrSrS\\S4S\S\S\SS 4 U4S jjrS \ RS\ \ R4S jr S r U=r$)BeitPyramidPoolingModuleiXa Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. align_corners (bool): align_corners argument of F.interpolate. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. pool_scales.rrryr.Nc>[TU]5 XlX@lX lX0l/Ul[U5HEupV[XbUS9nUR RU5 UR[U5U5 MG g)N)rrr) rBrCrryrrblocksrvrappendrrR) rDrrrryrjrblockrEs r*rC!BeitPyramidPoolingModule.__init__fsn &*&  &{3MA+zemnE KK  u % OOCFE *4r)rhc/nURHVnU"U5n[RRXAR 5SSSUR S9nUR U5 MX U$)NrtrPrv)rrrrrwryr)rDrhppm_outsppmppm_outupsampled_ppm_outs r*rJ BeitPyramidPoolingModule.forwardrsg;;C!fG " 9 9ffhqrl4K]K]!:!  OO- .  r))ryrrrr)r#r$r%r&r'rrrrCr5rlistrJr(rSrTs@r*rrXs`  +E#s(O +# +QT +ei +nr +$u||*<r)rcx^\rSrSrSrS\SS4U4SjjrSrS\RS\R4S jr S r U=r $) BeitUperHeadi}z Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://huggingface.co/papers/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rXr.Nc>[TU]5 URUlUR/S-UlURUlSUl[R"UR URSS9Ul [URURSUR UR S9Ul [URS[UR5UR --UR SSS9Ul[R "5Ul[R "5UlURSSHmn[X R SS9n[UR UR SSS9nUR"R'U5 UR$R'U5 Mo [[UR5UR -UR SSS9Ulg) NFrrrs)ryr rr)rBrCrr\rrryrrrrr psp_modulesrr bottleneckrc lateral_convs fpn_convsrfpn_bottleneck)rDrXrl_convfpn_convrEs r*rCBeitUperHead.__init__s !--"../!3** "))DMM63D3DRST4      R MM,,   )   R 3t'7'7#84==#H H MM   ]]_++CR0K#KANF%dmmT]]PQ[\]H    % %f - NN ! !( + 1 -   !DMM 1 MM  r)cUSnU/nURURU55 [R"USS9nUR U5nU$)Nrsrrz)extendrr5rr)rDinputsrhpsp_outsr;s r* psp_forwardBeitUperHead.psp_forwardsL 2J3((+,99X1-* r)encoder_hidden_statesc <[UR5VVs/sHup#U"X5PM nnnURURU55 [ U5n[ US- SS5HPnXBS- R SSnXBS- [RRXBUSURS9-XBS- 'MR [ US- 5Vs/sHo RU"XB5PM nnURUS5 [ US- SS5HAn[RRXrUSR SSSURS9Xr'MC [R"USS9nURU5nURU5nU$s snnfs snf)NrrrsrtrPrvrz)rvrrrrrdr3rrrryrr5rrr) rDrrj lateral_convlateralsused_backbone_levels prev_shapefpn_outsr;s r*rJBeitUperHead.forwardsR[\`\n\nRopRoqL!6!9:Rop(()>?@ #8}+a/B7A!a%..qr2J&1uo 0I0I *:TM_M_1J1HUO8=BBVYZBZ<[\<[qNN1%hk2<[\ %+a/B7A--33 (1+"3"3AB"7jX\XjXj4HK899X1-$$X.( 3q]s FF) ryrrrrrrrrr) r#r$r%r&r'rrCrr5rrJr(rSrTs@r*r r }sA$ z$ d$ LU\\ellr)r c ^\rSrSrSrS S\S\S\S\\\\\44SS4 U4S jjjr S \ RS\ R4S jr S r U=r$) BeitFCNHeadia Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://huggingface.co/papers/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rXin_indexrrr.Nc 2>[TU]5 URUlURUlUR UlURUl X l US-U-n/nUR[URUR X5US95 [URS- 5H2nUR[UR UR X5US95 M4 URS:Xa[R"5UlO[R""U6UlUR(a4[URUR -UR X3S-S9Ul[R&"UR UR(SS9Ulg)Nrt)rrrrrr r)rBrCr\rauxiliary_channelsrauxiliary_num_convs num_convsauxiliary_concat_input concat_inputr&rrrdrr-convs Sequentialconv_catrrr) rDrXr&rr conv_paddingr-rjrEs r*rCBeitFCNHead.__init__sH !--11 33"99 #q(H4    $--[iq  t~~)*A LLMM4==kjr + >>Q DJ.DJ   *  4==0$--[qrbrDM))DMM63D3DRSTr)rcXRnURU5nUR(a%UR[R "X#/SS95nUR U5nU$)Nrrz)r&r-r,r/r5rr)rDrrGr;s r*rJBeitFCNHead.forwardsT-mm< M*   ]]599m-D!#LMF( r))rrr,r/r-rr&r*)rtr r)r#r$r%r&r'rrrrrCr5rrJr(rSrTs@r*r%r%s tu U  U,/ UBE UUZ[^`efiknfn`o[oUp U  U UDU\\ellr)r%c^\rSrSrS\SS4U4SjjrSr\SS\\ RS\\ RS \\ RS \\ S \\ S \ S \\ S\ \ \44Sjj5rSrU=r$)BeitForSemanticSegmentationi rXr.Nc x>[TU]U5 URUl[USS9Ul[ UR R5S:wa [S5e[R"[R"URURSSS9[R"UR5[R"5[R"URURSSS95Ul[R"[R"URURSSS95Ul[R""5Ul[R&"SSS9Ul[+U5UlUR.(a [1U5OSUlUR55 g)NFrr zBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.rtr)rBrCrrrrrX out_indicesrrr.rr\rGELUfpn1fpn2r-fpn3 MaxPool2dfpn4r  decode_headuse_auxiliary_headr%auxiliary_headrrs r*rC$BeitForSemanticSegmentation.__init__sO   ++f>  t{{&& '1 ,-  MM   v1163E3EST]^ _ NN6-- . GGI   v1163E3EST]^ _   MM   v1163E3EST]^ _  KKM LLQq9 (/5;5N5Nk&1TX r)cX[RRXRSSSSS9nUb,[RRX#RSSSSS9n[ UR R S9nU"XC5nUnUb$U"WU5n XR RU -- nU$)NrrPFrv) ignore_index)rrrr3r rXsemantic_loss_ignore_indexauxiliary_loss_weight) rDrauxiliary_logitsrupsampled_logitsupsampled_auxiliary_logitsr main_lossrauxiliary_losss r* compute_loss(BeitForSemanticSegmentation.compute_loss/s==44 bc*55   ')+)B)B ||BC'8zY^*C* &$1W1WX-6   '%&@&IN KK55F FD r)rrrrrlrrmc UbUOURRnUbUOURRnUb%URRS:Xa [ S5eUR UUUSUUS9nU(a UR OUSn [U 5V V s/sH&upU S-URR;dM$U PM( n n n URSn URRURR-nU Vs/sH2oSS2SS2SS24RSSS5RU SX5PM4 n nURURUR UR"/n[%['U 55HnUU"U U5U U'M UR)U 5nSnUR*bUR+U 5nSnUbUR-UUU5nU(d%U(a U4USS-nO U4USS-nUbU4U-$U$[/UUU(a UR OSUR0S 9$s sn n fs snf) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nrz/The number of labels should be greater than oneTrrrtrsr)rXrrlrrrrGrvr7r3rdrbrr~r9r:r;r=rdrr>r@rKrru)rDrrrrrlrrmrridxfeaturefeaturesrpatch_resolutionrhopsrjrrFrr;s r*rJ#BeitForSemanticSegmentation.forwardBs[D&1%r9r:r;r=rr)r#r$r%r&rrCrKrrr5rrrrrrJr(rSrTs@r*r5r5 szd@&04,0)-,0/3).&*X u||,X ELL)X & X $D> X 'tn X #'X d^X  u-- .X X r)r5zM BEiT backbone, to be used with frameworks like DETR and MaskFormer. c r^\rSrSrU4SjrSr\S S\S\\ S\\ S\\ S\ 4 S jj5r S r U=r $) BeitBackboneic >[TU]U5 [TU] U5 [URS-5Vs/sHo!R PM snUl[U5Ul[XRRRS9Ul UR(Ga[URR 5S:wa [#S5eUR n[$R&"[$R("X3SSS9[$R*"X1R,S9[$R."5[$R("X3SSS95Ul[$R&"[$R("X3SSS95Ul[$R4"5Ul[$R8"SSS9UlUR=5 gs snf)Nrrr zBeitBackbone requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.rtrr$)rBrC_init_backbonerdrar\ num_featuresrVror\rarradd_fpnrrXr7rrr.rrbatch_norm_epsr8r9r:r-r;r<r=r)rDrXrr\rEs r*rCBeitBackbone.__init__sW   v&9>v?W?WZ[?[9\]9\A//9\](0"67W7W7c7cd >>>4;;**+q0 1 !,,K "";STU{0E0EF "";STU DI b&8&8_`ij&klDI DI 1=DI 1^sGc.URR$rArrNs r*r!BeitBackbone.get_input_embeddingsrr)rrlrrmr.cUbUOURRnUbUOURRnUbUOURRnURSnUR U5unupxURSSn UR USUU US9n U(a U ROU Sn Sn [URU 5HjupXR;dMURR(a4USS2SS2SS24nURSSS5nURUSXx5nX4- n Ml URR(aYURU S5UR!U S5UR#U S5UR%U S 5/n ['U 5n U(dU(a U 4U SS-nU$U 4U SS-nU$[)U U(a U ROSU R*S 9$) a Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NrrtT)rlrrrmrr"rsr ) feature_mapsrGru)rXrrlrr3rorrGzip stage_names out_featuresreshape_hidden_statesrr~rYr9r:r;r=rrru)rDrrlrrmrrrrrrrGr_stagerr;s r*rJBeitBackbone.forwards@&1% Q d^ Q  Q Q r)rU)rrr5rr~rU)r0F)Mr'collections.abcrerr dataclassesrtypingrrr5torch.utils.checkpointrrtorch.nnrr r activationsr modeling_layersr modeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrutils.backbone_utilsrconfiguration_beitr get_loggerr#rr rQrr<Moduler>rVr`rrrrrrrr!rr\r~rrrrrrrr r%r5rU__all__r"r)r*rvs !" AA!9.vv771*   H %  !;  U\\eTV[VbVb( %299 % c7RYYc7L#7"))#7LV VrG#-G#TRYY& ! )BII)X ryy   >*>BP3ryyP3fI "))I X#O/#O#OLT #T T n&^ !4^ ^ B K !4K  K \"RYY"Jbii""ryy"JR299Rj8"))8vM "5M M ` t & t  t n r)