tjddlZddlmZddlmZddlZddlZddlm Z ddl m cm Z ddlm Z ddlmZddlmZddlmZdd lmZmZdd lmZmZdd lmZdd lmZdd lm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2e#j3e4Z5e!deGddeZ6e!deGdde Z7Gdde j8Z9Gdde j8Z:Gdd e j8Z; dTd"e j8d#ej d$ej d%ej d&ej dzd'edzd,ej fd-Z?Gd.d/e j8Z@Gd0d1e j8ZAd2ZBd3ZCGd4d5eZDe!d6eGd7d8e ZEe!Gd9d:eZFGd;dd?eFZHGd@dAe j8ZIGdBdCe j8ZJGdDdEe j8ZKGdFdGe j8ZLGdHdIeZMGdJdKe j8ZNGdLdMe jOZPGdNdOe j8ZQe!dPGdQdReFZRgdSZSdS)VN)Callable) dataclass)Tensor)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)#compile_compatible_method_lru_cache) ModelOutputauto_docstringcan_return_tuplelogging)TransformersKwargsis_flash_attention_requestedmerge_with_config_defaults)OutputRecordercapture_outputs) AutoModel) Sam2ConfigSam2HieraDetConfigSam2MaskDecoderConfigSam2PromptEncoderConfigSam2VisionConfigz,Base class for the vision encoder's outputs.) custom_introcPeZdZUdZdZejdzed<dZejdzed<dS)Sam2VisionEncoderOutputa last_hidden_state (`torch.FloatTensor` of shape `(batch_size, height, width, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. fpn_hidden_states (`tuple(torch.FloatTensor)`): Tuple of `torch.FloatTensor` (one for each feature level, from high to low resolution) of shape `(batch_size, hidden_size, height, width)`. Feature maps from the Feature Pyramid Network neck. fpn_position_encoding (`tuple(torch.FloatTensor)`): Tuple of `torch.FloatTensor` (one for each feature level, from high to low resolution) of shape `(batch_size, hidden_size, height, width)`. Positional encodings corresponding to the `fpn_hidden_states`. Nfpn_hidden_statesfpn_position_encoding) __name__ __module__ __qualname____doc__r$torch FloatTensor__annotations__r%d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/sam2/modeling_sam2.pyr#r#6sP  37u(4/6666:5,t3:::::r.r#z'Base class for the Sam2 model's output.c eZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdfed<dZ e ejdfdzed<dZ e ejdfdzed <dZe ejdfdzed <dS) Sam2ImageSegmentationOutputa iou_scores (`torch.FloatTensor` of shape `(batch_size, point_batch_size, num_masks)`): The Intersection over Union (IoU) scores of the predicted masks. pred_masks (`torch.FloatTensor` of shape `(batch_size, point_batch_size, num_masks, height, width)`): The predicted low-resolution masks. This is an alias for `low_res_masks`. These masks need to be post-processed by the processor to be brought to the original image size. object_score_logits (`torch.FloatTensor` of shape `(batch_size, point_batch_size, 1)`): Logits for the object score, indicating if an object is present. image_embeddings (`tuple(torch.FloatTensor)`): The features from the FPN, which are used by the mask decoder. This is a tuple of `torch.FloatTensor` where each tensor has shape `(batch_size, channels, height, width)`. vision_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(batch_size, height, width, hidden_size)`. Hidden-states of the vision model at the output of each stage. vision_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the vision model. mask_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the mask decoder. N iou_scores pred_masksobject_score_logits.image_embeddingsvision_hidden_statesvision_attentionsmask_decoder_attentions)r&r'r(r)r2r*r+r,r3r4r5tupler6r7r8r-r.r/r1r1Hs,,0J!D(///+/J!D(///48*T18886:eE-s23:::AE% 13 67$>EEE>BuU.34t;BBBDHU5#4c#9:TAHHHHHr.r1c.eZdZdZdeffd ZdZxZS)Sam2PatchEmbeddingsa Turns pixel values into patch embeddings for transformer consumption. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`Sam2ImageProcessor.__call__`] for details. Returns: embeddings (`torch.FloatTensor`): Patch embeddings depend on image_size, patch_kernel_size, patch_stride and patch_padding configct|j}|j}t j|||j|j|j|_ dS)N) kernel_sizestridepadding) super__init__ num_channels hidden_sizennConv2dpatch_kernel_size patch_stride patch_padding projection)selfr<rCrD __class__s r/rBzSam2PatchEmbeddings.__init__xs] * ( )  0&(    r.c|j\}}}}|||jjjdddd}|S)Nrrrr)shaperJtoweightdtypepermute)rK pixel_values_rCheightwidth embeddingss r/forwardzSam2PatchEmbeddings.forwardsV)5);&<__\__T_5K5Q%R%RSS[[\]_`bcefgg r.)r&r'r(r)rrBrX __classcell__rLs@r/r;r;js^    1       r.r;cPeZdZdZ ddededed edzffd Zee d  dd e j de j e zde jdeded edzdede jdzde jfdZ dd e j de j e zde jde jdzde jf dZxZS)Sam2SinePositionEmbeddingz This is a more standard version of the position embedding, very similar to the one used by the Attention is all you need paper, generalized to work on images. @'FNnum_position_features temperature normalizescalect||durtd||_||_||_|dt jzn||_dS)NFz+normalize should be True if scale is passedr) rArB ValueErrorr_r`ramathpirb)rKr_r`rarbrLs r/rBz"Sam2SinePositionEmbedding.__init__sj   e!3!3JKK K%:"&"$)MQ[[u r.r)maxsizerNdevicerQmaskreturnc  |\}} } } |wtjd| dz||ddddf|| | } tjd| dz||ddddf|| | } n?||}|d} |d} |r6d}| | ddddddf|zz |z} | | ddddddf|zz |z} tj|tj||}|dtj|ddz|z z}| dddddddf|z }| dddddddf|z }tj|dddddddddf|dddddddddf fd  d }tj|dddddddddf|dddddddddf fd  d }tj ||fd  dd dd}|S) NrrQrhrư>floor) rounding_moderdimr) r*arangeexpandrOcumsumint64divstacksincosflattencatrR)rNrhrQr_rarbr`ri batch_sizerTrUrVy_embedx_embed embed_maskepsdim_tpos_xpos_yposs r/build_sine_position_embeddingz7Sam2SinePositionEmbedding.build_sine_position_embeddings(-$ Avu < l1fqjfMMMdTUTUTUW[m\ccFEGl1eaiuVLLLTSWYZYZYZ][bbFEGGJ ''**G ''**G  CCBCC!3c!9:UBGAAArss!3c!9:UBG 2%+fUUUXXY^__EIeQg$N$N$N NQf fg111aaa &.111aaa &. U111aaaADqD=15577qqq!!!QQQ1}9M9Q9Q9S9STZ[\\\ddefgg U111aaaADqD=15577qqq!!!QQQ1}9M9Q9Q9S9STZ[\\\ddefggiA...66q!QBB r.c b|||||j|j|j|j|SN)rr_rarbr`)rKrNrhrQris r/rXz!Sam2SinePositionEmbedding.forwards911 65$"r?r@)rArBr<r\fpn_hidden_sizeposition_encodingrE ModuleListconvsbackbone_channel_listappendrFfpn_kernel_size fpn_stride fpn_paddingfpn_top_down_levels)rKr<rrLs r/rBzSam2VisionNeck.__init__s  !:"("8A"=" " " ]__ !7  K J   +!'!7 & 6!,".     $*#=   r. hidden_statesrj.cd}d}t|jdz }t|ddD]}||dddd}|j||z ||j|jj}||jvs||kr|}nTtj |tj dd dd |j}||z}| |j |j|j|j} ||fz }|| fz }||fS) Nr-rrnrrr)rQg@nearestF) scale_factormode align_corners antialias)lenrrangerRrOrPrQrF interpolater*float32rrNrh) rKrr$r%nilateral_features prev_featurestop_down_featuresprev_position_encodings r/rXzSam2VisionNeck.forwardsq "  OOa q"b!! ? ?A,Q/771aCC 0tz!a%01A1D1DTZPQ]EYE_1`1`aa 000AFF 0 $%M!$$5=$99!$""&# %%% "%+,, "!13D D %)%;%;#]%9=;N&&b$%% # -!1 1  !&<%> > ! ! "777r.) r&r'r(r rBr*rr9rXrYrZs@r/rrs>/>>>>>>(8U\8eE%,PSBSr? ceil_mode)rRrEr max_pool2d)rrs r/do_poolr"s_ !Q1A    \]b ccA !Q1A Hr.c reZdZ d dededededeeefdzf fd Zdejd ejfd Z xZ S) Sam2MultiScaleAttentionNr<rsdim_outnum_attention_headsrc(t||_||_||_||_||_||z}|dz|_tj ||dz|_ tj |||_ d|_ dS)NrF) rArBr<rsrrrrbrELinearqkvproj is_causal)rKr<rsrrrhead_dimrLs r/rBz Sam2MultiScaleAttention.__init__.s   (#6 11t^ 9S'A+..Igw// r.rrjc |j\}}}}|||||zd|jd}t j|d\}} } ||jz| ddz} tjj | tj d |j } |jr]t||||d|j}|jdd\}}||||z|jd}|dd}| dd} | dd} t!j|jjt(} | ||| | fd|j|jd|\} }| |||d} || } | S)Nrrnr)rQrsr)rrr)rNrreshaperr*unbindrbrrErrrrOrQrrr get_interfacer<_attn_implementationrrr)rKrrr~rUrVrTrrrrrattention_interfacers r/rXzSam2MultiScaleAttention.forwardFs'4':$ FE1hh}%%--j&5.!TMegijj!La00sE *cmmB.C.CC x*22VXZ[[E1%%mmAq!!1%%(?(M K ,.E) ) -,       nJ       Q"))*feRHH ii ,, r.r) r&r'r(rrr9rBr*rrXrYrZs@r/rr-s04 "  !  CHo, 0&U\& &&&&&&&&r.rc DeZdZ d dedededededef fd Zd ZxZS) Sam2FeedForwardreluF input_dim hidden_dim output_dim num_layers activationsigmoid_outputc\t||_t||_t j||_t j||_t j fdt|dz D|_ ||_ dS)Nc:g|]}tjSr-)rEr).0rTrs r/ z,Sam2FeedForward.__init__..~s%$f$f$f1RYz:%F%F$f$f$fr.r) rArBrrrrErproj_inproj_outrrlayersr)rKrrrrrrrLs ` r/rBzSam2FeedForward.__init__ps $ ,yJ77  *j99 m$f$f$f$fPUV`cdVdPePe$f$f$fgg ,r.c ||}||}|jD] }|||}!||}|jrt j|}|Sr)rrrrrrsigmoid)rKrlayers r/rXzSam2FeedForward.forwards ]33  66 [ B BE OOEE-,@,@AAMM m44   5Im44Mr.)rF) r&r'r(rrrrBrXrYrZs@r/rros!$---- -  -  -------"       r.rc j|j\}}}}| |z}| |z}tj|ddd|d|f}||z||z} }||z} | |z} ||| || ||}|ddddddd|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rrrrrqrn)rNrErpadviewrRr) hidden_state window_sizer~rUrVrC pad_height pad_width padded_height padded_widthn_hn_wwindowss r/window_partitionrs/;.@+J|'[(J;&I=$$\Aq!Y:3VWWL"(:"5uy7Hq+AqA$4]K$P$P !M6 di  '   $   ,*d.?.BBK>!A#&DAqR;&ER;&E%iU+F  a  .}k6TUWXSYZZM =0 ++M::%1A(B(BB r.) r&r'r(rrrBr*rrrr+rXrYrZs@r/rrs-:"-:-: -:  -:-:-:-:-:-:^)|)+,)   ))))))))r.rzW Hiera model's outputs that also contains a pooling of the last hidden states. ceZdZUdZdZejdzed<dZe ejdfdzed<dZ e ejdfdzed<dZ e ejdfdzed<dS)Sam2HieraDetModelOutputat last_hidden_state (`torch.FloatTensor` of shape `(batch_size, height, width, hidden_size)`): hidden-states at the output of the last layer of the model. intermediate_hidden_states (`tuple[torch.FloatTensor]` of shape `(batch_size, height, width, hidden_size)`): Sequence of hidden-states at the output of the intermediate layers of the model. Nlast_hidden_state.intermediate_hidden_statesr attentions) r&r'r(r)rr*r+r,r r9rr!r-r.r/rr%s37u(4/666GKe&7&< = DKKK:>M5*C/047>>>7;Je',-4;;;;;r.rcleZdZeZdZdZdZdZdZ dZ gdZ e j fdZxZS)Sam2PreTrainedModelsam2rS)imageT)z ^memory_.*z^mask_downsample.*z^object_pointer_proj.*z0^temporal_positional_encoding_projection_layer.*no_memory_positional_encodingno_object_pointer%occlusion_spatial_embedding_parameterct|t|trD|jt j|j|jt j|jdSdSt|tr"t j |j |j dSt|tr"|j t j|j dSdSdS)N)std)rA _init_weights isinstanceSam2HieraDetModel pos_embedinitzeros_pos_embed_windowSam2PositionalEmbeddingnormal_positional_embeddingrb Sam2Modelno_memory_embedding)rKrrLs r/r+z!Sam2PreTrainedModel._init_weightsLs f%%% f/ 0 0 8+ F,---&2 F34444432  7 8 8 8 L4&, G G G G G G  * * 8)5 F677777 8 855r.)r&r'r(r config_classbase_model_prefixmain_input_nameinput_modalities_supports_sdpa_supports_flash_attn_supports_attention_backend"_keys_to_ignore_on_load_unexpectedr*no_gradr+rYrZs@r/r#r#9sL$O!N"&***&U]__ 8 8 8 8_ 8 8 8 8 8r.r#c eZdZeZdZeedZdeffd Z dZ de e e fde jfdZee d de jd zd eede ezfd ZxZS) r-rSrr!r<c t|t||_t jt jd|jg|j R|_ t jt jd|j|j d|j d|_ tj|jdz |_t j|_d}t)|jD]I\}}t+|D]4}t-||||}|j||dz }5J|dS)Nrr)r<rrr)rArBr; patch_embedrE Parameterr*zerosrD+window_positional_embedding_background_sizer.r r1nprvblocks_per_stagetolist stage_endsrblocks enumeraterrr post_init)rKr<rrrHrblockrLs r/rBzSam2HieraDetModel.__init__csi    .v66 K6- c0b c c c  !# K6-v/KA/NPVPlmnPo p p! ! 9V%<==AIIKKmoo +4V5L+M+M % % 'I'"#344 % % +!Y)]l ""5)))1$  % r.c|jSr)rCrKs r/get_input_embeddingsz&Sam2HieraDetModel.get_input_embeddings{s r.hwrjc|\}}|j}tj|j||fd}||dt |j|jDz}|dddd}|S)Nbicubic)sizercg|] \}}||z Sr-r-)rrys r/rz4Sam2HieraDetModel._get_pos_embed..s 2o2o2oda162o2o2or.rrrr)r1rrr.tileziprNrR)rKrRhw window_embedr.s r/_get_pos_embedz Sam2HieraDetModel._get_pos_embed~s1, M$.1vINNN  1 12o2oc)/[g[mFnFn2o2o2o p pp %%aAq11 r.Nrc "|td||}|||jddz}d}t |jD]\}}||fi|}||jvr||fz}t||S)N You have to specify pixel_valuesrrr-)rr )rdrCr]rNrLrKrJr)rKrSrrr r block_modules r/rXzSam2HieraDetModel.forwards  ?@@ @((66 %(;(;M%DEE !D$== "%i+- y%)K00%)K2H2HIrRRRRr.rr&r'r(rrBrXrYrZs@r/r2r2shK6KKKKKK SSSSSSSSr.r2c*eZdZdeffd ZdZxZS)Sam2MaskEmbeddingr<ct|jdz|_t|j|_t jd|jdd|_t j|j|jdd|_ t j|j|j d|_ t|j|j d|_t|jdz|j d|_dS)Nrqrrr>r?)r>channels_firstr data_format)rArBmask_input_channelsrrrrErFconv1conv2rDconv3 Sam2LayerNormr r rrjs r/rBzSam2MaskEmbedding.__init__s #)#=#B  !23Yq$":RSTTT Yt79Sabklmmm Yv96;M[\]]] (  $&*?M]   )  $q (f.CQa   r.c,||}||}||}||}||}||}||}|Sr)rr rrrr)rKmasksrdense_embeddingss r/rXzSam2MaskEmbedding.forwards 5)) ((77  66  =11 ((77  66 ::m44r.rwrZs@r/ryrysT  6               r.ryc eZdZdeffd ZdejdejdedejfdZdejdejfd Z d e ejejfd zd ejd zd ejd zdejd zde ejejff dZ xZ S)Sam2PromptEncoderr<c$tt||_t ||_t jd|j|_ |j |j z|j |j zf|_ d|j z|j zd|j z|j zf|_ |j |_t j|j|j|_|j|_t jd|j|_dS)Nrrq)rArBr2shared_embeddingry mask_embedrE EmbeddingrD no_mask_embed image_size patch_sizeimage_embedding_sizemask_input_sizeinput_image_sizenum_point_embeddings point_embednot_a_point_embedrjs r/rBzSam2PromptEncoder.__init__ s  7 ? ?+F33\!V-?@@%+%6&:K%KVM^bhbsMs$t! !F$5 59J JAPVPaLaekevLvw & 1<(CVEWXX!-!#a1C!D!Dr.pointslabelsrrjc@|dz}|rPtjj|ddd}tjj|ddd}|j|jf}|||}tj|ddk|jj|}tj|dd k|tj |}|| | d |dk dzz}|S) zEmbeds point prompts.?rrrrconstantrrr)rrrn).Ni)min) r*rErrrrwhererrP zeros_likerclamp unsqueeze)rKrrrrupoint_embeddings r/ _embed_pointszSam2PromptEncoder._embed_pointss#  XX(,,V\ Z[,\\FX(,,VV*TV,WWF,d.CD // DD +fY&72&=t?U?\^mnn + 9  $   _ - -  *D,<,WXY>Z   !"222r.) r&r'r(rrBr*rrrrr9rXrYrZs@r/rr s$ E6 E E E E E EEL%,TV[Vb2  %,  5<     (3EL%,67$>(3lT)(3\D( (3 \D( (3 u|U\) * (3(3(3(3(3(3(3(3r.rceZdZdZd fd Z d dejdejdejdejdzdeed e ejejff d Z xZ S) Sam2Attentionz SAM2's attention layer that allows for downscaling the size of the embedding after projection to queries, keys, and values. Nc.t||jn|}||_|j|_|j|z|_|j|_|j|jz|_|jdz|_d|_ tj |j|j|_ tj |j|j|_ tj |j|j|_tj |j|j|_dS)NrF)rArBattention_downsample_rater<rD internal_dimrrrrrErq_projk_projv_projo_proj)rKr<downsample_raterLs r/rBzSam2Attention.__init__ps >M>U&::[j !-"./A#)#= )V-GG }d* i 0$2CDD i 0$2CDD i 0$2CDD i 143CDD r.rrrattention_similarityrrjc |jdd\}}||zd|j|jf}||j|dd}||j|dd}||j|dd}tj |j j t} t|j r)|'td} td| ||||f|d|j|jd|\} } | ||d|j|jz} || } | | fS)NrrnrsdpaznFalling back to SDPA for target-guided attention because Flash Attention does not support additive bias masks.r)rrrr)rNrrrrrrrr rr<rrrlogger warning_oncerrrrr) rKrrrrrr~point_batch_size new_shaperrrs r/rXzSam2Attention.forwards(-{2A2$ $"22B8PRVR_` ' E""'3==aCC#dkk##Y/99!Q??' E""'3==aCC(?(M K ,.E) )  ( 4 4 9M9Y#:&"A    H    %8$7     % 0Ln % %  % % ! \")) ("d.F.V  *,, kk+.. L((r.r) r&r'r(r)rBr*rrrr9rXrYrZs@r/rrjs EEEEEE*59 .).)|.)\.)| .) $lT1 .) +, .) u|U\) *.).).).).).).).)r.rc VeZdZd dedeffd Zdedededed ed eef d Z xZ S) Sam2TwoWayAttentionBlockFr<skip_first_layer_pectt|d|_t j|j|_t||_t j|j|_ t|j|j |j|j |_ t j|j|_t j|j|_t||_||_dS)a A transformer block with four layers: (1) self-attention of sparse inputs (2) cross attention of sparse inputs -> dense inputs (3) mlp block on sparse inputs (4) cross attention of dense inputs -> sparse inputs Arguments: config (`Sam2MaskDecoderConfig`): The configuration file used to instantiate the block attention_downsample_rate (*optionalk*, int, defaults to 2): The downsample ratio of the block used to reduce the inner dim of the attention. skip_first_layer_pe (*optional*, bool, defaults to `False`): Whether or not to skip the addition of the query_point_embedding on the first layer. r)r)rN)rArBr self_attnrEr rDr cross_attn_token_to_imagerrmlp_dimnum_hidden_layersr layer_norm3 layer_norm4cross_attn_image_to_tokenr)rKr<rrLs r/rBz!Sam2TwoWayAttentionBlock.__init__s &vqAAA<(:;;)6v)>)>&<(:;;"  0BvOg   <(:;;<(:;;)6v)>)>   r.querieskeysquery_point_embeddingkey_point_embeddingrrc "|jr||||\}}n%||z}||||\} }|| z}||}||z}||z} ||| ||\} }|| z}||}||} || z}||}||z}||z} || ||\} }|| z}||}||| fS)Nrrr)rrrr) rrr rrrrrr) rKrrrrrrrTrattn_outrmlp_outs r/rXz Sam2TwoWayAttentionBlock.forwards^  # )g7'RRJGQQ33E..u%w.OOKHa(G""7++//((44SCW5  !H$""7++((7##G#""7++//((443EQX4YY !h%%h&&r.)F) r&r'r(rrrBrrrrXrYrZs@r/rrs77474777777>*'*'*' & *' $ *' % *'+,*'*'*'*'*'*'*'*'r.rcZeZdZdeffd Z d dededededeed ee zf d Z xZ S) Sam2TwoWayTransformerr<ct||_|j|_t j|_t|jD]/}|jt||dk0t||_ t j |j |_dS)Nr)r)rArBr<rrErrrrrrfinal_attn_token_to_imager rDlayer_norm_final_attn)rKr<rrLs r/rBzSam2TwoWayTransformer.__init__s  !'!9moo t-.. _ _A K  7UVZ[U[]]] ^ ^ ^ ^)6v)>)>&%'\&2D%E%E"""r.Nrr5image_positional_embeddingsrrrjc |td|dddd}|dddd}|}|}|jD]} |||z }| d|||||d|\}}} ||z} ||z} || | |\} } || z}||}||fS)Nz&You have to specify an image_embeddingrr)rrrrrrr-)rdr|rrrrr)rKrr5rrtarget_embeddingrrrrrTrrrs r/rXzSam2TwoWayTransformer.forward sL  #EFF F+33A66@@AFFPPQRSS&A&I&I!&L&L&V&VWXZ[&\&\&f&fgh&i&i##[  E+++$u &6$?%9        GT11**00445cQU4VV !H$,,W55}r.r) r&r'r(rrBrrrr9r rXrYrZs@r/rrs F4 F F F F F F& (( (!(&, ( % (+,(  ((((((((r.rcReZdZdZdddfd Zdejdejffd ZxZS) raALayerNorm that supports two data formats: channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch_size, height, width, channels) while channels_first corresponds to inputs with shape (batch_size, channels, height, width). rm channels_lastr}c ztj|fd|i||dvrtd|||_dS)Nr)rr|zUnsupported data format: )rArBNotImplementedErrorr~)rKnormalized_shaperr~rrLs r/rBzSam2LayerNorm.__init__=sY)==s=f=== A A A%&O+&O&OPP P&r.featuresrjc|jdkrR|dddd}t|}|dddd}n!t|}|S)z Args: features: Tensor of shape (batch_size, channels, height, width) OR (batch_size, height, width, channels) r|rrrr)r~rRrArX)rKrrLs r/rXzSam2LayerNorm.forwardCsw  / / /''1a33Hwwx00H''1a33HHwwx00Hr.) r&r'r(r)rBr*rrXrYrZs@r/rr7s 15/'''''''            r.rc eZdZdeffd Z ddejdejdejdejded eejd ejdzd ejdzd e e d e ejejejejffdZ dZ dZxZS)Sam2MaskDecoderr<ct||_|j|_|j|_|jdz|_t jd|j|_t j|j|j|_ t||_ t j |j|jdzdd|_ t j |jdz|jdzdd|_t|jdzd|_t j|_g}t'|jD]*}|t)|j|j|jdzdgz }+t j||_t)|j|j|j|jd |_t j|j|jdzdd|_t j|j|jdzdd|_t jd|j|_t)|j|jdd|_|j|_|j |_ |j!|_!dS) Nrrqrr{r|)r~rT)r)"rArBr<rDnum_multimask_outputsnum_mask_tokensrEr iou_token mask_tokensr transformerConvTranspose2d upscale_conv1 upscale_conv2rupscale_layer_normGELUrrrroutput_hypernetworks_mlpsiou_head_hidden_dimiou_head_depthiou_prediction_headrFconv_s0conv_s1obj_score_tokenpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)rKr< mlps_listrTrLs r/rBzSam2MaskDecoder.__init__RsJ  !-%+%A"%;a?a)9::<(NOO088 /0@$BRVWBWefopqqq/0@A0EtGW[\G\jktuvvv"/0@A0ESc"d"d"d')) t+,, i iA /$*:DNTU=VV&A&S&STdfg&h&h#-=T-=. --(C!5- . .  . . **)AAAq!!!4 *111aaaa$:N6N1OQRQRQR+RS,55a;;@@ ) )<  4++,:-aaaABBh7 <(<"EEE$4$>$>r$B$B$L$LR$P$P$Z$Z[]$^$^!$=$D$D A',,R002B2G2G2K2K% % !!& -=qB[ \ \$)L1EqJZJdJdegJhJh$i$i!,AAAqqq!A#qqq!!!O< .qqq!!!QqSy 9556GHH$(OO  + o & 0 01B C C  !       5 6 6 ! %   ..r.)NN)r&r'r(rrBr*rrlistrrr9rXrrrYrZs@r/rrQsF'\4'\'\'\'\'\'\b5904mDmD,mD&+\mD#(, mD "' mD  mD#'u|"4mD$lT1mD ,-mD+,mD u|U\5<E FmDmDmDmD^    #/#/#/#/#/#/#/r.rz Segment Anything Model 2 (SAM 2) for generating segmentation masks, given an input image and input points and labels, boxes, or masks. ceZdZdZdeediZiZdeffd Z dZ de j fd Z e jd e jd eedee j fd Ze j dde jd zde jd zde jd zde jd zfdZeee dd e jd zde jd zde jd zde jd zde jd zde jd zdede jd zde jd zd eedefdZeed e jd eedeezfdZxZ S)r5)r%textr8r)indexr<cFt|t|j|_t j|j|_t|j|_ |j |j _ t|j |_|jj|_|jj|_|jj|_t&jt'jdd|j|_|dS)Nr)rArBr2prompt_encoder_configshared_image_embeddingrre vision_configvision_encoderrprompt_encoderrmask_decoder_configr mask_decoderribackbone_feature_sizesrrr*rErDrEr6rMrjs r/rBzSam2Model.__init__'s    &=f>Z&[&[#'3F4HII/0LMM:@:U"7+F,FGG"("6"I&,&:&Q# .>#(8#5#5ek!Q6X6X#Y#Y  r.c4|jSr)r5rQrPs r/rQzSam2Model.get_input_embeddings8s"77999r.rjc|jj}|jjj}|jjj}t j|||}|ddz }|ddz }||dz }||dz }|t j ||gd}| ddd dS)N)rhrQrrrrrrnr) r6rr3r4rhrQr*onesrvryrRr)rKrU target_device target_dtypegridrrr4s r/$get_image_wide_positional_embeddingsz.Sam2Model.get_image_wide_positional_embeddings;s"73HO 2GM z$}LIII++!+$$s*++!+$$s*DG#DG##::5;QXGY_a;b;b;bcc#++Aq!44>>qAAAr.rSrc |jd|j|fddi|}|j}|d|jz|d<fdt ||jD}|S)z Returns the image embeddings by passing the pixel values through the vision encoder. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Input pixel values r return_dictTrnc^g|])\}}|dddjdg|R*SrrrrnrRrrfeat feat_sizer~s r/rz2Sam2Model.get_image_embeddings..]sT   i 'DLLAq ! ! &z2 B B B B   r.)rNget_image_featuresr$r6rYr9)rKrSr image_outputs feature_mapsr5r~s @r/get_image_embeddingszSam2Model.get_image_embeddingsHs"'* // YY$YRXYY $6 (+d.FF R    #&|T5P#Q#Q     r.Nrrrrc8|||||}|S)a Returns the prompt embeddings by passing the input points, labels, boxes and masks through the prompt encoder. Args: input_points (`torch.FloatTensor` of shape `(batch_size, point_batch_size, num_points_per_image, 2)`): Optional input points for the prompt encoder. The padding of the point is automatically done by the processor. `point_batch_size` refers to the number of masks that we want the model to predict per point. The model will output `point_batch_size` times 3 masks in total. input_labels (`torch.LongTensor` of shape `(batch_size, point_batch_size, num_points_per_image)`): Optional input labels for the prompt encoder. The padding of the labels is automatically done by the processor, or can be fed by the user. input_boxes (`torch.FloatTensor` of shape `(batch_size, num_boxes_per_image, 4)`): Optional input boxes for the prompt encoder. The padding of the boxes is automatically done by the processor. users can also pass manually the input boxes. input_masks (`torch.LongTensor` of shape `(batch_size, image_size, image_size)`): Optional input masks for the prompt encoder. rrrr)r6)rKrrrr prompt_outputs r/get_prompt_embeddingszSam2Model.get_prompt_embeddingsds22++%%## ,  r.Tr5rrrc |du|duz std|J|H|jd|jdkr,td|jdd|jdd|} | |jdn|djd| ddd} d} d} |Y|j|fd d i| }|j}|j} |j} |d|jz|d<fd t||j D}|8|6tj |dddddddftj |j }|a|_tjddd |dj|dj }tjddtj|dj  }|j|jdd|jjkrMt+j||jjddd |j}|||||\}}|jd|d| ||||dd|| d| \}}}}t5||||| | S)a input_points (`torch.FloatTensor` of shape `(batch_size, num_points, 2)`): Input 2D spatial points, this is used by the prompt encoder to encode the prompt. Generally yields to much better results. The points can be obtained by passing a list of list of list to the processor that will create corresponding `torch` tensors of dimension 4. The first dimension is the image batch size, the second dimension is the point batch size (i.e. how many segmentation masks do we want the model to predict per input point), the third dimension is the number of points per segmentation mask (it is possible to pass multiple points for a single mask), and the last dimension is the x (vertical) and y (horizontal) coordinates of the point. If a different number of points is passed either for each image, or for each mask, the processor will create "PAD" points that will correspond to the (0, 0) coordinate, and the computation of the embedding will be skipped for these points using the labels. input_labels (`torch.LongTensor` of shape `(batch_size, point_batch_size, num_points)`): Input labels for the points, this is used by the prompt encoder to encode the prompt. According to the official implementation, there are 3 types of labels - `1`: the point is a point that contains the object of interest - `0`: the point is a point that does not contain the object of interest - `-1`: the point corresponds to the background We added the label: - `-10`: the point is a padding point, thus should be ignored by the prompt encoder The padding labels should be automatically done by the processor. input_boxes (`torch.FloatTensor` of shape `(batch_size, num_boxes, 4)`): Input boxes for the points, this is used by the prompt encoder to encode the prompt. Generally yields to much better generated masks. The boxes can be obtained by passing a list of list of list to the processor, that will generate a `torch` tensor, with each dimension corresponding respectively to the image batch size, the number of boxes per image and the coordinates of the top left and bottom right point of the box. In the order (`x1`, `y1`, `x2`, `y2`): - `x1`: the x coordinate of the top left point of the input box - `y1`: the y coordinate of the top left point of the input box - `x2`: the x coordinate of the bottom right point of the input box - `y2`: the y coordinate of the bottom right point of the input box input_masks (`torch.FloatTensor` of shape `(batch_size, image_size, image_size)`): SAM model also accepts segmentation masks as input. The mask will be embedded by the prompt encoder to generate a corresponding embedding, that will be fed later on to the mask decoder. These masks needs to be manually fed by the user, and they need to be of shape (`batch_size`, `image_size`, `image_size`). image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_channels, window_size, window_size)`): Image embeddings, this is used by the mask decoder to generate masks and iou scores. For more memory efficient computation, users can first retrieve the image embeddings using the `get_image_embeddings` method, and then feed them to the `forward` method instead of feeding the `pixel_values`. multimask_output (`bool`, *optional*): In the original implementation and paper, the model always outputs 3 masks per image (or per point / per bounding box if relevant). However, it is possible to just output a single mask, that corresponds to the "best" mask, by specifying `multimask_output=False`. attention_similarity (`torch.FloatTensor`, *optional*): Attention similarity tensor, to be provided to the mask decoder for target-guided attention in case the model is used for personalization as introduced in [PerSAM](https://huggingface.co/papers/2305.03048). target_embedding (`torch.FloatTensor`, *optional*): Embedding of the target concept, to be provided to the mask decoder for target-semantic prompting in case the model is used for personalization as introduced in [PerSAM](https://huggingface.co/papers/2305.03048). Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoModel, AutoProcessor >>> model = AutoModel.from_pretrained("danelcsb/sam2.1_hiera_tiny") >>> processor = AutoProcessor.from_pretrained("danelcsb/sam2.1_hiera_tiny") >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/sam-car.png" >>> with httpx.stream("GET", url) as response: ... raw_image = Image.open(BytesIO(response.read())).convert("RGB") >>> input_points = [[[400, 650]]] # 2D location of a window on the car >>> inputs = processor(images=raw_image, input_points=input_points, return_tensors="pt") >>> # Get segmentation mask >>> outputs = model(**inputs) >>> # Postprocess masks >>> masks = processor.post_process_masks( ... outputs.pred_masks, inputs["original_sizes"] ... ) ``` NzAExactly one of pixel_values or image_embeddings must be provided.rzGYou should provide as many bounding boxes as input points per box. Got z and .rrnrBTc^g|])\}}|dddjdg|R*SrDrErFs r/rz%Sam2Model.forward..sT   #D)+ Q1%%*:rFIFFF   r.rlrrFbilinear)rUrrrrN)r5rrrrrrr)r2r3r4r5r6r7r-)rdrNr@rrIr$rr!r6rYr9r* ones_likerrhrErQr<int32r6rrrrrOr8r1)rKrSrrrrr5rrrrrr7r6rJrKrrlow_res_multimasksr2rTr4r~s @r/rXzSam2Model.forwards@%*:d*BC b`aa a  # (?!!$ (9!(<<< R^j^pqr^sRR{F{LMN{ORRR'+&O&O&Q&Q#.:.F\'**L\]_L`LfghLi &A&H&HUVXY[\&]&]# #  #5LT5L\5v5vgk5vou5v5vM(:L#0#> - 8  ,B/$2JJL     '*<9T'U'U      # (< ?<111aaa +C59]i]pqqqL  K$7 ;Aq!+;B+?+EN^_aNbNiL"Jz1au{ScdfSgSnooooL  " %)<)LLLm%%'',<"'#"  "[&'' /3.A.A%%## /B/ / ++ BSAR B -b1(C%6$4-%5crc%:!5- B B  B B >J+>+!) 3-!5/     r.c 8|j|fddi|}|j}|j}t|}|j|d|d<|j|d|d<d|D}d|D}||_||_|S)z pixel_values (`torch.FloatTensor`): Input pixel values of shape `(batch_size, num_channels, height, width)`. rBTrrcbg|],}|dddd-Srrrr|rR)r feature_maps r/rz0Sam2Model.get_image_features..Ls8```K ++A..66q!Q??```r.cbg|],}|dddd-SrZr[)r feature_maps_position_embeddingss r/rz0Sam2Model.get_image_features..MsH, , , 0 - 4 4Q 7 7 ? ?1a H H, , , r.)r5r$r%r-r8rr)rKrSrvision_outputsrKr^s r/rIzSam2Model.get_image_features5s3F$2El2o2o`d2ohn2o2o%7 +9+O(L)) +33LODD Q+33LODD Qa`S_``` , , 4T, , , (,8(/O,r.)NNNN) NNNNNNTNN)!r&r'r(r:rrra_tied_weights_keysrrBrQr*rr@r?r+rrr-rL LongTensorrPrrrrr1rXrr9r#rIrYrZs@r/r5r5s)4nnE]ef6g6g6ghz"::: Bel B B B BU]__ ' +,  el     _ 6U]__260404/3 '$.&-&-  %, _@ 26150404/359!%9=59k k '$.k '$.k &- k &- k %, k  +d2k k $/$6k  +d2k +,k  %k k k ^_ k Z'+, ( ( ^r.r5)r5rcr#r-)rr)Trecollections.abcr dataclassesrnumpyrGr*torch.nnrEtorch.nn.functionalrrrrr/ activationsrmodeling_layersr modeling_outputsr r modeling_utilsr r processing_utilsr pytorch_utilsrutilsrrrr utils.genericrrrutils.output_capturingrrautorconfiguration_sam2rrrrr get_loggerr&rr#r1Moduler;r\rrrrrrrrrrrr#r-rcr2ryrrrrr rrr5__all__r-r.r/rvso* $$$$$$!!!!!! &&&&&&!!!!!!999999KKKKKKKKFFFFFFFF&&&&&&@@@@@@KKKKKKKKKKKKiiiiiiiiiiEEEEEEEE  H % %KLLL  ; ; ; ; ;8 ; ; ML ; FGGG IIIII+II HGI@")BJ J J J J  J J J Z1818181818RY181818v%% I% <% % < % L4' %  %%%%%,  u| 3:     ?????bi???Dbi<222><<<:YYYYY4YYYx   < < < <