tj(0ddlZddlZddlmZddlmZddlZddlZddl m cm Z ddlm Z m Z ddlmZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'erddl(m)Z)e!r ddl*m+Z+ddl,m-Z-e deGddeZ. dNdej dej dej fdZ/de de de fdZ0dej dej dej fd Z1Gd!d"e j2Z3de de d#e4de fd$Z5dej dej d#e4dej fd%Z6Gd&d'e j2Z7Gd(d)e j2Z8Gd*d+e j2Z9 dOd-e j2d.ej d/ej d0ej d1ej dzd2e:d3e:fd4Z;Gd5d6e j2Z<Gd7d8e j2Z=Gd9d:e j2Z>Gd;de j2Z@Gd?d@eZAGdAdBe jBZCGdCdDe j2ZDGdEdFe j2ZEGdGdHe j2ZFe GdIdJeZGe dKGdLdMeGZHdJdMgZIdS)PN)Callable) dataclass)Tensornn)initialization)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)merge_with_config_defaults)capture_outputs) EomtConfig)linear_sum_assignment) PartialState)reducea Class for outputs of [`EomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~EomtImageProcessor.post_process_semantic_segmentation`] or [`~EomtImageProcessor.post_process_instance_segmentation`] or [`~EomtImageProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~EomtImageProcessor] for details regarding usage. ) custom_introc eZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dZeejdzed <dS) "EomtForUniversalSegmentationOutputa* loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_state hidden_states attentions patch_offsets)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rr r!r"tupler#r$listrd/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/eomt/modeling_eomt.pyrr3s*&*D% d ")))59%+d299959%+d299926u(4/66659M5*+d299926Je'(4/666/3M4 %,33333r/rFinput_featurespoint_coordinatesreturnc |dkrd}|d}tjjj|d|zdz fi|}|r|d}|S)a( A wrapper around `torch.nn.functional.grid_sample` to support 3D point_coordinates tensors. Args: input_features (`torch.Tensor` of shape (batch_size, channels, height, width)): A tensor that contains features map on a height * width grid point_coordinates (`torch.Tensor` of shape (batch_size, num_points, 2) or (batch_size, grid_height, grid_width,: 2)): A tensor that contains [0, 1] * [0, 1] normalized point coordinates add_dim (`bool`): boolean value to keep track of added dimension Returns: point_features (`torch.Tensor` of shape (batch_size, channels, num_points) or (batch_size, channels, height_grid, width_grid): A tensor that contains features for points in `point_coordinates`. rTg@?)dim unsqueezer)r functional grid_samplesqueeze)r1r2add_dimkwargspoint_featuress r0 sample_pointr?^s(!##-77::X(4^SK\E\_bEbmmflmmN3'//22 r/inputslabelsc(|d}dtj||jz}|ddddf|ddddfz}d|dz|dzz z }|S)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. rr5N)sigmoidflattenr)matmulTsum)r@rA numerator denominatorrs r0pair_wise_dice_lossrK~s^^   % %a ( (FEL222I**R..D)FJJrNN47,CCK A +/2 2D Kr/cF|jd}tjd}||tj|}||tj|}tj||z |j}tj||z d|z j}||z}|S)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnone reduction)shaperBCEWithLogitsLossr) ones_like zeros_likerFrG) r@rAheight_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negrs r0$pair_wise_sigmoid_cross_entropy_lossrZs|A$v666I&Yvuv/F/FGG&Yvu/?/G/GHH|25EEvxPPH|25EEF ~VVH h D Kr/c eZdZdZ ddedededeffd Zejd ej d ej d ej d ej d e e e f dZ xZ S)EomtHungarianMatcheraqThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). r61 cost_class cost_mask cost_dice num_pointsct|dkr|dkr|dkrtd||_||_||_||_dS)aHCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)super__init__ ValueErrorrar^r_r`)selfr^r_r`ra __class__s r0rdzEomtHungarianMatcher.__init__sc"  ??yA~~)q..344 4$$""r/r r mask_labels class_labelsr3cHg}|jd}t|D]}||d}||} |dd||f } ||| } | dddf} | dddf} t jd|jd| j} | | jddd} t| | d d} | | jddd}t| |d d} t| | }t| | }|j |z|j| zz|j|zz}t j|t jd }t j|t jd }t j|d}t)|}||d |D}|S) ao Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rrCNrr5deviceF align_cornersg _Bg _cg|]E\}}tj|tjtj|tjfFS)dtype)r) as_tensorint64).0ijs r0 z0EomtHungarianMatcher.forward..sR   _c_`bcU_Qek 2 2 2EOAU[4Y4Y4Y Z   r/)rPrangesoftmaxtor)randrarlrepeatr?r;rZrKr_r^r`minimumtensormaximum nan_to_numrcpuappend)rfr rrhriindices batch_sizeru pred_probs pred_maskr^ target_maskr2target_coordinatespred_coordinatesr_r` cost_matrixassigned_indicesmatched_indicess r0forwardzEomtHungarianMatcher.forwards68*,*/2 z"" - -A-a088<.zs"%O%O%OVd6<&8&8%O%O%Or/rrqrlrr5F) rlenrqrlr)zerosrboolziprPcopy_)rfrmax_size batch_shaper_heightwidthrqrlpadded_tensors padding_masksr~ padded_tensor padding_masks r0_pad_images_to_max_in_batchz$EomtLoss._pad_images_to_max_in_batchxs1$$%O%Ow%O%O%OPP7||nx/ '2$ Avu  "[fMMM J#>ejY_``` 36wP]3^3^ G G /FM< +FLO+->v|A->@Q&,q/@QQ R X XY_ ` ` `AFL*6<?*,=fl1o,== > >},,r/rrirc|}|j\}}}tj|j}||} t jdt||D} t j||f|j t j |j } | | | <| dd} || | } d| i}|S)aCompute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )weightc*g|]\}\}}||Sr.r.)rttargetrrvs r0rwz(EomtLoss.loss_labels..s$ H H H>66AqVAY H H Hr/) fill_valuerqrlrr5loss_cross_entropy) rPrCrossEntropyLossr$_get_predictions_permutation_indicesr)catrfullrrsrl transpose)rfrrir pred_logitsr num_queriesrrUidxtarget_classes_otarget_classespred_logits_transposedloss_celossess r0 loss_labelszEomtLoss.loss_labelss"+ %0%6" K't/@AAA 77@@ 9 H HSw-G-G H H H    %$/]h]o   /s!,!6!6q!!z%EomtLoss.loss_masks..st99&AAr/Frmr) loss_mask loss_dice) r _get_targets_permutation_indicesrr)rsample_points_using_uncertaintyrarrr?r;rr)rfr rhrrsrc_idxtgt_idx pred_masks target_masksrr2 point_labels point_logitsrs` r0 loss_maskszEomtLoss.loss_maskss4;;GDD77@@)'2 ::;GG a#G,  4( #AAAtG, ]__ i i $ D DAAAA%, !! ( 6GW\]]]eefghhL i i i i i i i i i i i i i i i$J0AQVWWW__`abb 4L,PYZZ"<yII     s?ACC #C ctjdt|D}tjd|D}||fS)NcDg|]\}\}}tj||Sr.r) full_like)rtrusrcrs r0rwzAEomtLoss._get_predictions_permutation_indices..s,"a"a"a{q(35?3#:#:"a"a"ar/cg|]\}}|Sr.r.)rtrrs r0rwzAEomtLoss._get_predictions_permutation_indices..s(E(E(E#q(E(E(Er/r)rr)rfr batch_indicespredictions_indicess r0rz-EomtLoss._get_predictions_permutation_indicessT "a"aiX_N`N`"a"a"abb #i(E(EW(E(E(EFF111r/ctjdt|D}tjd|D}||fS)NcDg|]\}\}}tj||Sr.r)rtrurtgts r0rwz=EomtLoss._get_targets_permutation_indices..s,"a"a"a{q(1c5?3#:#:"a"a"ar/cg|]\}}|Sr.r.)rtrrs r0rwz=EomtLoss._get_targets_permutation_indices..s#@#@#@HQC#@#@#@r/r)rfrrtarget_indicess r0rz)EomtLoss._get_targets_permutation_indicessR "a"aiX_N`N`"a"a"abb #@#@#@#@#@AAn,,r/rc0tj| }|S)a In Eomt paper, uncertainty is estimated as L1 distance between 0.0 and the logit prediction in 'logits' for the foreground class in `classes`. Args: logits (`torch.Tensor`): A tensor of shape (R, 1, ...) for class-specific or class-agnostic, where R is the total number of predicted masks in all images and C is: the number of foreground classes. The values are logits. Returns: scores (`torch.Tensor`): A tensor of shape (R, 1, ...) that contains uncertainty scores with the most uncertain locations having the highest uncertainty score. )r)abs)rfruncertainty_scoress r0rzEomtLoss.calculate_uncertaintys %y001!!r/rarrc |jd}t||z}tj||d|j}t ||d} || } t||z} || z } tj| dddddf| dd} |tj|tj|j z}| |dddfz } | d d| d ddf || d}| dkr3tj |tj|| d|jgd }|S) a This function is meant for sampling points in [0, 1] * [0, 1] coordinate space based on their uncertainty. The uncertainty is calculated for each point using the passed `uncertainty function` that takes points logit prediction as input. Args: logits (`float`): Logit predictions for P points. uncertainty_function: A function that takes logit predictions for P points and returns their uncertainties. num_points (`int`): The number of points P to sample. oversample_ratio (`int`): Oversampling parameter. importance_sample_ratio (`float`): Ratio of points that are sampled via importance sampling. Returns: point_coordinates (`torch.Tensor`): Coordinates for P sampled points. rr5rkFrmNr)kr7rrCr7) rPrr)r{rlr?topkarangelongviewr)rfruncertainty_functionrarr num_boxesnum_points_sampledr2rpoint_uncertaintiesnum_uncertain_pointsnum_random_pointsrshifts r0rz(EomtLoss.sample_points_using_uncertaintys<LO  .>!>??"Jy2DaPVP]^^^#F,=USSS 22<@@"#:Z#GHH&)==j,QQQ111W59MSTUUUVWX"U\)5:V\Vc%d%d%dd uQQQW~-222q99#((2,,/JOOPY[oqrss q % "EJy:KQW]Wd$e$e$ef!!! ! r/Nauxiliary_predictionsc |||||}|||dj}i|||||||||}|rt |D]b\ } | d}| d}|||||} fd| D} || c|S)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], then it contains the logits from the inner layers of the EomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrkNr rc&i|] \}}|d|S)rr.)rtkeyvaluers r0 z$EomtLoss.forward..os)WWWzsE^^c^^UWWWr/) r get_num_masksrlrrrritemsupdate) rfr rrhrirrrr aux_outputs loss_dictrs @r0rzEomtLoss.forward=sH,,35I;Xdee&&|LOz)EomtLoss.get_num_masks..xs(AAG AAAAAAr/rr)min) rHr)rrrrr _shared_stater num_processesclamp)rfrirlr world_sizes r0rzEomtLoss.get_num_maskstsAALAAAAA OIU[PPP  " $ $ :)R//"9-- )^^9 K J 6A>>> r/r)r%r&r'rdictstrrrdr-rrrr,rnparrayrr)rrrrrrrlrrrs@r0rrLs! z! S%Z8H! ! ! ! ! ! F$tCy/d3i-4<-E&RX.DY----" $* :>v, QVWYW_Q`  c6k     D<#l<%,'<rx <  < c5< <<<<|222 --- "EL"U\"""""5! 5! 5!  5! "' 5! 5!5!5!5!zAE 55#l5$l5%,' 5 5<( 5 $C$56= 5 c5< 5555n %,  QVQ]        r/rcFeZdZdZfdZdejdejfdZxZS)EomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct|j|j}}|j|j}}t |tjj r|n||f}t |tjj r|n||f}|d|dz|d|dzz}||_||_||_||_ tj |||||_ dS)Nrr kernel_sizestride)rcrd image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)rfrr,r-r.r/r4rgs r0rdzEomtPatchEmbeddings.__init__s !'!2F4EJ $*$79Kk #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq !!} 1 5*Q-:VW=:XY $$(&)L+:^hiiir/ pixel_valuesr3c|jd}||jkrtd|jd|d||ddd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r5)rPr.rer6rEr)rfr7r. embeddingss r0rzEomtPatchEmbeddings.forwards#)!, 4, , ,I!.II9EIII __\22::1==GG1MM r/) r%r&r'r(rdr)rrrrs@r0r'r'sm j j j j jELU\r/r'cPeZdZdZdeddffd ZdejdejfdZxZ S)EomtEmbeddingszM Construct the CLS token, mask token, position and patch embeddings. rr3Nct||_|j|_t jt jdd|j|_ t jt j d|j |j|_ t||_|jj}t j|j|_d|j z|_t j||j|_|dt j|dddS)Nr position_idsrrCF) persistent)rcrdrr-r Parameterr)randnr/ cls_tokenrnum_register_tokensregister_tokensr'patch_embeddingsr4Dropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsrrexpand)rfrr4rgs r0rdzEomtEmbeddings.__init__s   +ek!Q8J&K&KLL!|EK6;UW]Wi,j,jkk 3F ; ;+7 z&"<== !"V%?!?#%< V=O#P#P  ^U\+-F-F-M-Mg-V-Vchiiiiir/r7c|j\}}}}|jjjj}|||}|j|dd}|j|dd}|| |j z}tj |||gd}| |}|S)NrprCrr)rPrFr6rrqrzrCrMrErLr>r)rrI)rfr7rr target_dtyper: cls_tokensrEs r0rzEomtEmbeddings.forwards*0 Aq!,7>D **> .55j"bII$":":4;L"M"MM Y OZHaPPP \\*-- r/ r%r&r'r(rrdr)rrrrs@r0r<r<sjzjdjjjjjj EL U\        r/r<modulequeryrrattention_maskscalingrIc tj||dd|z}|||z}tj|dtj|j}tj |||j }tj||} | dd } | |fS)NrC)r7rq)ptrainingrr5) r)rFrrr9ryfloat32rzrqrIrZ contiguous) rSrTrrrUrVrIr= attn_weights attn_outputs r0eager_attention_forwardr_s<s}}R'<'<==GL!#n4 =((2U](SSVVW\WbccL=((6?([[L,|U33K''1--88::K  $$r/c ~eZdZdZfdZ ddejdejdzdeejejdzffdZxZ S) EomtAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ |j |_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)rcrdrr/ embed_dimnum_attention_heads num_headshead_dimrescaleattention_dropoutrI is_causalrLineark_projv_projq_projout_projrfrrgs r0rdzEomtAttention.__init__s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( / i?? i?? i??  $.$.AA r/Nr"rUr3c |jdd}g|d|jR}|||dd}|||dd}|||dd}tj|j j t} | ||||||j |j |jsdn|j\} } | jg|dR} || } | | fS)z#Input shape: Batch x Time x ChannelNrCrr5rR)rirVrI)rPrfrmrrrkrlr get_interfacer_attn_implementationr_rirgrZrIreshaper\rn) rfr"rUr= input_shape hidden_shapequerieskeysvaluesattention_interfacer^r]s r0rzEomtAttention.forwardsg$)#2#. 88b8$-88 ++m,,11,??II!QOO{{=))..|<<FFq!LL]++00>>HHANN(?(M K ,.E) ) %8$7     nJ#}>CC$, % % % ! \*k);;;;;;FFHH mmK00 L((r/r) r%r&r'r(rdr)rr,rrrs@r0rarasGGBBBBB./3!)!)|!) t+!) u|U\D00 1 !)!)!)!)!)!)!)!)r/racDeZdZdfd ZdejdejfdZxZS)EomtLayerScaler3Ncttj|jt j|jz|_dSr) rcrdrrAlayerscale_valuer)rr/lambda1ros r0rdzEomtLayerScale.__init__sC |F$;ejI[>\>\$\]] r/ hidden_statec||jzSr)r~rfrs r0rzEomtLayerScale.forward!sdl**r/r3Nr%r&r'rdr)rrrrs@r0r{r{si^^^^^^+EL+U\++++++++r/r{cDeZdZdfd ZdejdejfdZxZS)EomtMLPr3Nc~t|jx}}t|j|jz}t j||d|_t|j trt|j |_ n |j |_ t j||d|_ dS)NTbias)rcrdr/r mlp_ratiorrjfc1r0 hidden_actr#r activationfc2rfr in_features out_featureshidden_featuresrgs r0rdzEomtMLP.__init__&s %+%77 lf063CCDD9[/EEE f' - - 0$V%67DOO$/DO9_lFFFr/rc||}||}||}|Sr)rrrrs r0rzEomtMLP.forward1s;xx -- |44 xx -- r/rrrs@r0rr%si G G G G G GELU\r/rcDeZdZdfd ZdejdejfdZxZS) EomtSwiGLUFFNr3NcDt|jx}}t|j|jz}t|dzdz dzdzdz}t j|d|zd|_t j||d|_dS)Nr5rTr) rcrdr/rrrrj weights_in weights_outrs r0rdzEomtSwiGLUFFN.__init__9s %+%77 lf063CCDD2Q677!;AAE)K_1D4PPP9_lNNNr/rc||}|dd\}}tj||z}||S)Nr5rCr)rchunkrr9silur)rfrx1x2hiddens r0rzEomtSwiGLUFFN.forwardBs]|44 ##A2#..B##B''",'''r/rrrs@r0rr8siOOOOOO(EL(U\((((((((r/rc^eZdZdZd deddffd ZdejdejfdZde fd Z xZ S) EomtDropPathzStochastic depth (DropPath) per sample, for residual blocks. Identity when ``drop_prob`` is 0 or outside training. See `Deep Networks with Stochastic Depth `_. rR drop_probr3NcVt||_dSr)rcrdr)rfrrgs r0rdzEomtDropPath.__init__Ps$ "r/r"c|jdks|js|Sd|jz }|jdfd|jdz zz}t j||j|j}t j||z}| ||zS)NrRrr)rr) rrZrPndimr)r{rqrlfloordiv)rfr" keep_probrP random_tensors r0rzEomtDropPath.forwardTs >S   & $Q')DM4F4J,KK 5 0CML`aaa  MI$=>>   ++m;;r/cd|jS)Nzp=)rrfs r0 extra_reprzEomtDropPath.extra_repr]s$DN$$$r/rR) r%r&r'r(rrdr)rrr#rrrs@r0rrIs ##%#$######>2Dn#U#U q $ 1 12G H H'<== M zz-00 xx -- ((66 ~~l33mC r/rrQrs@r0rrasMM3z3d333333&/3| t+  r/rcDeZdZdfd ZdejdejfdZxZS)EomtLayerNorm2dư>TcPt|||dS)N)relementwise_affine)rcrd)rfr.raffinergs r0rdzEomtLayerNorm2d.__init__s( 36JJJJJr/rr3c|dddd}tj||j|j|j|j}|dddd}|S)Nrr5rr)permuteF layer_normnormalized_shaperrrrs r0rzEomtLayerNorm2d.forwards^#++Aq!Q77 |L$2GVZV_aeaijj #++Aq!Q77 r/)rTrrs@r0rrsiKKKKKKELU\r/rcHeZdZdeffd ZdejdejfdZxZS)EomtScaleLayerrc$t|j}tj||dd|_t |j|_tj ||dd|d|_ t||_ dS)Nr5r)rrF)r*paddinggroupsr) rcrdr/rConvTranspose2dconv1r rrr5conv2r layernorm2drfrr/rgs r0rdzEomtScaleLayer.__init__s ( ' [aXYZZZ  !23Y       +;77r/r"r3c||}||}||}||}|Sr)rrrrrfr"s r0rzEomtScaleLayer.forwardsN =11  66  =11 ((77 r/ r%r&r'rrdr)rrrrs@r0rrsj8z888888 U\elr/rcHeZdZdeffd ZdejdejfdZxZS)EomtScaleBlockrctj|_t jfdt |jD|_dS)Nc.g|]}tSr.)rrtrrs r0rwz+EomtScaleBlock.__init__..s!#[#[#[qN6$:$:#[#[#[r/)rcrdnum_upscale_blocks num_blocksr ModuleListrxblockros `r0rdzEomtScaleBlock.__init__sX  3]#[#[#[#[E$/DZDZ#[#[#[\\ r/r"r3c0|jD] }||}|Sr)r)rfr"rs r0rzEomtScaleBlock.forwards*Z 1 1E!E-00MMr/rrs@r0rrsq]z]]]]]] U\elr/rcHeZdZdeffd ZdejdejfdZxZS) EomtMaskHeadrc t|j}tj|||_tj|||_tj|||_t|j |_ dSr) rcrdr/rrjrrfc3r rrrs r0rdzEomtMaskHead.__init__sm ( 9[+669[+669[+66 !23r/r"r3c|||}|||}||}|Sr)rrrrrs r0rzEomtMaskHead.forwardsS(?(?@@ (?(?@@ // r/rrs@r0rrsj4z444444U\elr/rceZdZUdZeed<dZdZdZdZ dgZ dZ e e d Zejd ejd d ffd ZxZS)EomtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. reomtr7)imageFrT)r"r#rSr3Nct||jj}t |t jt jt jfrtj |j tj d|jitjj |j \}}|dkrdtj |z nd}tj|j| |dSdSt |t jrctj|j dd|jrrPrMrrrrrEomtForUniversalSegmentationones_attn_mask_probs)rfrSrfan_inrboundrrgs r0rz!EomtPreTrainedModel._init_weightss f%%%k+ fry")R5GH I I /  !&-49Q<< @ @ @ @{&!HMGG VV 17!DIf---- fkE6599999'& - - / LSa 8 8 8 8!-gfmMach6i6i- FM&*<=>>>>>.---  / / /vy)) Mv~t{/KLLLLL M M  / / /  v/cs C C C C K. / / / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h h h  ) ) / :f&7!&;<. s!$`$`$`1Yv%6%6$`$`$`r/r)rrr)rrr)&rcrdrnum_hidden_layersr<r:rrr/r layernormrKrrTrrxlayersr upscale_blockr mask_headrjrclass_predictorr,r- grid_sizerrrrrrUrr)rr post_initros `r0rdz%EomtForUniversalSegmentation.__init__sy     !'!9(00f&8f>STTT\&"4f6HII m$`$`$`$`fF^@_@_$`$`$`aa +F33%f--!y);V=NQR=RSS +v/@@&BSW]WhBhi"("5++. .  "T=MNNN . 6;L0M0MNNN r/r rrhrirr3c||||||}|jD](\}}|D]\} } || vr| |z} )|S)Nr rrhrir)rUrr) rfr rrhrirrrrloss_keyrs r0 get_loss_dictz*EomtForUniversalSegmentation.get_loss_dict!s(,~~!5!5#%"7 (6( (  +1133 # #KC"+//"3"3 # #$(??FND #r/rcDt|Sr)rHrx)rfrs r0get_lossz%EomtForUniversalSegmentation.get_loss9s9##%%&&&r/Nr$r=c d\}}d}|td||} t|jD]w\} } | |j|jjz kri|jjdddddf | j ddd | j } tj| | fd} | |j|jjz kr|js'|j| |jz |jjzdkr|| } || \}}||fz }||fz }tj| j d| j d| j d| j tj}t+j||jd }||d|dd}|jj}||jjz}|dk|ddd||df<|||j| |jz |jjz|||j }|dddd f d|jjdd}||d }| | |} y|| }||\}}||fz }||fz }d}|L|Jd}tA||D]7\}}|!||||d}||"|z }8tG|||||S)ag mask_labels (`list[torch.Tensor]`, *optional*): list of mask labels of shape `(num_labels, height, width)` to be fed to a model class_labels (`list[torch.LongTensor]`, *optional*): list of target class labels of shape `(num_labels, height, width)` to be fed to a model. They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. )r.r.Nz You have to specify pixel_valuesrrCrr)rlrqbilinear)sizemode)probnum_query_tokensencoder_start_tokensrl.gerRr)rr rr!r$)$rer:rrrrrrTrrMrPrzrlr)rrZrrpredictrrr interpolaterrr"rrJ_disable_attention_maskrdr masked_fillrrrr)rfr7rhrir$r=masks_queries_logits_per_layerclass_queries_logits_per_layerrUr"r layer_modulerTnorm_hidden_statesr rinterpolated_logitsr%r&sequence_outputrrs r0rz$EomtForUniversalSegmentation.forward<s*JPF&(F  ?@@ @ 55 !*4;!7!7. H. H Cd,t{/EEEE )$111*5<<]=PQR=SUWY[\\__`m`tuu % 5-*@a H H H d,t{/EEEE F!%!5cD&>',,Q//1D1I1I!1L1Lb''#$(;#: '7$/:['[$ObdeNeqqq"3#3"35I5J5JJK"&!=!="-cD4J.JT[Mc.cd%5)=)0 ">"""04 !=!D!DRIhjlnp!q!q!/!5!5!7!7!C!C^OUY!Z!Z(LGGMM..7759\\/5R5R22&+?*AA&&+?*AA&  "|'?D>A.0N?? 1 1:$&:!..)=)= +!-*. /  i0001!5!5-'     r/c|jjSr)r:rFrs r0get_input_embeddingsz1EomtForUniversalSegmentation.get_input_embeddingss //r/rc|ddd|jjddf}||}|dd|jj|jjzdddf}|dd}|j|jddg|jR}| |}| |}tj d||}||fS)Nrr5rrCzbqc, bchw -> bqhw) rrrr:rJrrsrPrrrr)einsum)rfr query_tokens class_logits prefix_tokens mask_logitss r0r'z$EomtForUniversalSegmentation.predictsaaa!:4;#:!:AAA=> ++L99 qqq$+"9DO<]"]"_"_abababbc %//155 - -m.A!.DbZ4>ZZZ ~~l33 **=99 l#6 mTT L((r/c|dkr:tj|jd|||k}d|ddd||df|<|S)Nrrrk)r)r{rP) attn_maskr$r%r&rlrandom_queriess r0r)z4EomtForUniversalSegmentation._disable_attention_masksb !88"Z (: Rr/)NNN)r%r&r'rrrdrr"r#rrrrrr-rrrrr2r)r' staticmethodr)rrs@r0rrs %Oz8$%    $CK0  c6k 0'$sF{"3''''' ,0,0-1 e e e &\D(e 6lT) e F|d* e +, e  ,e e e ^_ e N000)el)))) \r/r)Fr)Jcollections.abcr1rr dataclassesrnumpyr$r)torch.nn.functionalrr9rrrr activationsr file_utilsr r r modeling_layersr modeling_utilsrrprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrconfiguration_eomtrscipy.optimizer accelerateraccelerate.utilsrrr?rKrZrr\rrrrr'r<rr_rar{rrrrrrrrrrr__all__r.r/r0rOs* $$$$$$!!!!!! &&&&&&!!!!!!LLLLLLLLLL999999FFFFFFFF&&&&&&PPPPPPPPPP777777555555******5444444(''''''''''''    4444444   4BLQL5:\ \@6,u|X]Xd8ggggg29gggTff<u|U\VY^c^j(uuuuuryuuup ")B"""""RY"""X%% I% <% % < % L4' %  %%%%%.8)8)8)8)8)BI8)8)8)v+++++RY+++bi&(((((BI((("%%%%%29%%%0'''''*'''TblRY2     RY   29"-/-/-/-/-//-/-/-/` @@@@@#6@@  @F !"@ Ar/