tjdZddlZddlmZddlmZddlZddlmZddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*e#j+e,Z-e"deGddeZ.e"deGddeZ/e"deGdde Z0e"deGdd e Z1Gd!d"ej2Z3d#Z4Gd$d%ej2Z5Gd&d'ej2Z6Gd(d)ej2Z7 dXd+ej2d,ej8d-ej8d.ej8d/ej8dzd0e9dzd1e9d2ee!fd3Z:Gd4d5ej2Z;Gd6d7ej2Z<Gd8d9ej2Z=Gd:d;eZ>Gd<d=eZ?Gd>d?ej2Z@Gd@dAej2ZAGdBdCej2ZBGdDdEej2ZCe"GdFdGeZDGdHdIeDZEGdJdKeDZFe"dLGdMdNeDZGGdOdPej2ZHe"dQGdRdSeDZIe"dTGdUdVeDZJgdWZKdS)YzPyTorch DETR model.N)Callable) dataclass)initialization)ACT2FN) load_backbone)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutput"BaseModelOutputWithCrossAttentionsSeq2SeqModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)#compile_compatible_method_lru_cache) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) DetrConfigaU Base class for outputs of the DETR decoder. This class adds one attribute to BaseModelOutputWithCrossAttentions, namely an optional stack of intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. This is useful when training the model with auxiliary decoding losses. ) custom_introc2eZdZUdZdZejdzed<dS)DetrDecoderOutputa< intermediate_hidden_states (`torch.FloatTensor` of shape `(config.decoder_layers, batch_size, num_queries, hidden_size)`, *optional*, returned when `config.auxiliary_loss=True`): Intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. Nintermediate_hidden_states__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/detr/modeling_detr.pyrr28 <@ 1D 8?????r(raS Base class for outputs of the DETR encoder-decoder model. This class adds one attribute to Seq2SeqModelOutput, namely an optional stack of intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. This is useful when training the model with auxiliary decoding losses. c2eZdZUdZdZejdzed<dS)DetrModelOutputa@ intermediate_hidden_states (`torch.FloatTensor` of shape `(config.decoder_layers, batch_size, sequence_length, hidden_size)`, *optional*, returned when `config.auxiliary_loss=True`): Intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. Nrrr'r(r)r,r,Dr*r(r,z4 Output type of [`DetrForObjectDetection`]. ceZdZUdZdZejdzed<dZe dzed<dZ ejdzed<dZ ejdzed<dZ e e dzed<dZejdzed<dZeejdzed <dZeejdzed <dZeejdzed <dZejdzed <dZeejdzed <dZeejdzed<dS)DetrObjectDetectionOutputaB loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` are provided)): Total loss as a linear combination of a negative log-likelihood (cross-entropy) for class prediction and a bounding box loss. The latter is defined as a linear combination of the L1 loss and the generalized scale-invariant IoU loss. loss_dict (`Dict`, *optional*): A dictionary containing the individual losses. Useful for logging. logits (`torch.FloatTensor` of shape `(batch_size, num_queries, num_classes + 1)`): Classification logits (including no-object) for all queries. pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_queries, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual image in the batch (disregarding possible padding). You can use [`~DetrImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. auxiliary_outputs (`list[Dict]`, *optional*): Optional, only returned when auxiliary losses are activated (i.e. `config.auxiliary_loss` is set to `True`) and labels are provided. It is a list of dictionaries containing the two above keys (`logits` and `pred_boxes`) for each decoder layer. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the decoder of the model. Nloss loss_dictlogits pred_boxesauxiliary_outputslast_hidden_statedecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_stateencoder_hidden_statesencoder_attentions)r r!r"r#r/r$r%r&r0dictr1r2r3listr4r5tupler6r7r8r9r:r'r(r)r.r.Vs\,&*D% d ")))!Itd{!!!'+FE  $++++/J!D(///+/tDzD(///26u(4/666=A5!23d:AAA:>e/047>>>8<eE-.5<<<:>u047>>>=A5!23d:AAA:>e/047>>>>>r(r.z1 Output type of [`DetrForSegmentation`]. ceZdZUdZdZejdzed<dZe dzed<dZ ejdzed<dZ ejdzed<dZ ejdzed<dZ ee dzed<dZejdzed <dZeejdzed <dZeejdzed <dZeejdzed <dZejdzed <dZeejdzed<dZeejdzed<dS)DetrSegmentationOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` are provided)): Total loss as a linear combination of a negative log-likelihood (cross-entropy) for class prediction and a bounding box loss. The latter is defined as a linear combination of the L1 loss and the generalized scale-invariant IoU loss. loss_dict (`Dict`, *optional*): A dictionary containing the individual losses. Useful for logging. logits (`torch.FloatTensor` of shape `(batch_size, num_queries, num_classes + 1)`): Classification logits (including no-object) for all queries. pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_queries, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual image in the batch (disregarding possible padding). You can use [`~DetrImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. pred_masks (`torch.FloatTensor` of shape `(batch_size, num_queries, height/4, width/4)`): Segmentation masks logits for all queries. See also [`~DetrImageProcessor.post_process_semantic_segmentation`] or [`~DetrImageProcessor.post_process_instance_segmentation`] [`~DetrImageProcessor.post_process_panoptic_segmentation`] to evaluate semantic, instance and panoptic segmentation masks respectively. auxiliary_outputs (`list[Dict]`, *optional*): Optional, only returned when auxiliary losses are activated (i.e. `config.auxiliary_loss` is set to `True`) and labels are provided. It is a list of dictionaries containing the two above keys (`logits` and `pred_boxes`) for each decoder layer. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the decoder of the model. Nr/r0r1r2 pred_masksr3r4r5r6r7r8r9r:)r r!r"r#r/r$r%r&r0r;r1r2r@r3r<r4r5r=r6r7r8r9r:r'r(r)r?r?st8&*D% d ")))!Itd{!!!'+FE  $++++/J!D(///+/J!D(///+/tDzD(///26u(4/666=A5!23d:AAA:>e/047>>>8<eE-.5<<<:>u047>>>=A5!23d:AAA:>e/047>>>>>r(r?c2eZdZdZfdZfdZdZxZS)DetrFrozenBatchNorm2dz BatchNorm2d where the batch statistics and the affine parameters are fixed. Copy-paste from torchvision.misc.ops with added eps before rqsrt, without which any other models than torchvision.models.resnet[18,34,50,101] produce nans. ct|dtj||dtj||dtj||dtj|dS)Nweightbias running_mean running_var)super__init__register_bufferr$oneszeros)selfn __class__s r)rIzDetrFrozenBatchNorm2d.__init__s  Xuz!}}555 VU[^^444 ^U[^^<<< ]EJqMM:::::r(c n|dz}||vr||=t|||||||dS)Nnum_batches_tracked)rH_load_from_state_dict) rM state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsnum_batches_tracked_keyrOs r)rRz+DetrFrozenBatchNorm2d._load_from_state_dicts[#)+@"@ "j 0 023 %%  oWa     r(cB|jdddd}|jdddd}|jdddd}|jdddd}d}|||zz}|||zz }||z|zS)Nrgh㈵>)rDreshaperErGrFrsqrt)rMxrDrErGrFepsilonscales r)forwardzDetrFrozenBatchNorm2d.forwards$$QAq11y  B1--&..q"a;; (00B1== +/66888lU**5y4r()r r!r"r#rIrRrb __classcell__rOs@r)rBrBsj;;;;;              r(rBcl|D]\}}t|tjrt |j}|jjtjdkr||j |j|j |j |j |j |j |j ||j |<tt|dkrt#|dS)z Recursively replace all `torch.nn.BatchNorm2d` with `DetrFrozenBatchNorm2d`. Args: model (torch.nn.Module): input model metarN)named_children isinstancenn BatchNorm2drB num_featuresrDdevicer$copy_rErFrG_moduleslenr<childrenreplace_batch_norm)modelnamemodule new_modules r)rqrqs,,.. ' ' f fbn - - ..v/BCCJ}#u|F';';;;!'' 666%%fk222'--f.ABBB&,,V-?@@@#-EN4 tFOO%%&& ' '! + + v & & & ' 'r(cFeZdZdZfdZdejdejfdZxZS)DetrConvEncoderz Convolutional backbone, using either the AutoBackbone API or one from the timm library. nn.BatchNorm2d layers are replaced by DetrFrozenBatchNorm2d as defined above. c0t||_t|}|j|_t j5t|dddn #1swxYwYd}t|dr |j }d}||_ |j j }d|vrd|j D]L\}}|r"d|vrd|vrd|vr|d)d|vrd |vrd |vr|dKdSdS) NF _backboneTresnetlayer2layer3layer4zstage.1zstage.2zstage.3)rHrIconfigrchannelsintermediate_channel_sizesr$no_gradrqhasattrryrrbackbone_config model_typenamed_parametersrequires_grad_)rMr~backbone is_timm_modelbackbone_model_typers parameterrOs r)rIzDetrConvEncoder.__init__s   ((*2*;']__ ) ) x ( ( ( ) ) ) ) ) ) ) ) ) ) ) ) ) ) )  8[ ) ) !)H M $4? * * *#':#>#>#@#@ 8 8i 8t++0D0DY]I]I]!00777 ,,$1F1F9\`K`K`!00777 + * 8 8sA33A7:A7 pixel_values pixel_maskcv||}t|tr|j}g}|D]}tj|d|jdd tj d}| ||f|S)Nsizer) rrrhr; feature_mapsri functional interpolatefloatshapetor$boolappend)rMrrfeaturesout feature_mapmasks r)rbzDetrConvEncoder.forwards::l++ h % % -,H# , ,K=,,Z-=-C-C-E-EKL]^`^a^aLb,ccffglgqrrstuD JJ T* + + + + r() r r!r"r#rIr$Tensorrbrcrds@r)rwrwsh88888< EL el        r(rwcPeZdZdZ ddededed edzffd Zee d  dd e j de j e zde jdeded edzdede jdzde jfdZ dd e j de j e zde jde jdzde jf dZxZS)DetrSinePositionEmbeddingz This is a more standard version of the position embedding, very similar to the one used by the Attention is all you need paper, generalized to work on images. @'FNnum_position_features temperature normalizeract||durtd||_||_||_|dt jzn||_dS)NFz+normalize should be True if scale is passed) rHrI ValueErrorrrrmathpira)rMrrrrarOs r)rIz"DetrSinePositionEmbedding.__init__,sj   e!3!3JKK K%:"&"$)MQ[[u r(rmaxsizerrldtyperreturnc  |\}} } } |wtjd| dz||ddddf|| | } tjd| dz||ddddf|| | } n?||}|d} |d} |r6d}| | ddddddf|zz |z} | | ddddddf|zz |z} tj|tj||}|dtj|ddz|z z}| dddddddf|z }| dddddddf|z }tj|dddddddddf|dddddddddf fd  d }tj|dddddddddf|dddddddddf fd  d }tj ||fd  dd dd}|S) Nr)rrlrgư>r\floor) rounding_moderdimr) r$arangeexpandrcumsumint64divstacksincosflattencatpermute)rrlrrrrarr batch_size_heightwidthy_embedx_embed embed_maskepsdim_tpos_xpos_yposs r)build_sine_position_embeddingz7DetrSinePositionEmbedding.build_sine_position_embedding;s(-$ Avu < l1fqjfMMMdTUTUTUW[m\ccFEGl1eaiuVLLLTSWYZYZYZ][bbFEGGJ ''**G ''**G  CCBCC!3c!9:UBGAAArss!3c!9:UBG 2%+fUUUXXY^__EIeQg$N$N$N NQf fg111aaa &.111aaa &. U111aaaADqD=15577qqq!!!QQQ1}9M9Q9Q9S9STZ[\\\ddefgg U111aaaADqD=15577qqq!!!QQQ1}9M9Q9Q9S9STZ[\\\ddefggiA...66q!QBB r(c b|||||j|j|j|j|SN)rrrrar)rMrrlrrs r)rbz!DetrSinePositionEmbedding.forwardgs911 65$">L=((6?([[L,|U33K''1--88::K  $$r(c eZdZdZ ddededededef fd Z dd e j d e j d zd e j d zde e de e j e j ff dZxZS)DetrSelfAttentionz Multi-headed self-attention from 'Attention Is All You Need' paper. In DETR, position embeddings are added to both queries and keys (but not values) in self-attention. rTr~ hidden_sizenum_attention_headsrrEct||_||z|_|jdz|_||_d|_tj||||_ tj||||_ tj||||_ tj||||_ dSNrFrE rHrIr~head_dimrattention_dropout is_causalriLineark_projv_projq_projo_projrMr~rrrrErOs r)rIzDetrSelfAttention.__init__  #':: }d* !(i [tDDD i [tDDD i [tDDD i [tDDD r(N hidden_statesrposition_embeddingsrrc |jdd}g|d|jR}|||zn|}|||dd}|||dd} |||dd} tj|j j t} | ||| | |f|j sdn|j |jd|\} } | jg|dR} || } | | fS)zZ Position embeddings are added to both queries and keys (but not values). Nr\rrrrrrrrviewrrrr get_interfacer~_attn_implementationrrrrr]rr)rMr rr r input_shape hidden_shapequery_key_input query_states key_states value_statesattention_interfacerrs r)rbzDetrSelfAttention.forwards$)#2#. 88b8$-88 ATA`-*===fs{{?3388FFPPQRTUVV [[1166|DDNNqRSTT {{=1166|DDNNqRSTT (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r(rTNNr r!r"r#rrrrrIr$rrrr=rbrcrds@r)rrs EEEE! E  E  EEEEEE./337 $)$)|$) t+$)#\D0 $) +, $) u|U\) * $)$)$)$)$)$)$)$)r(rceZdZdZ ddededededef fd Z dd e j d e j d e j d zde j d zde j d zde e de e j e j ffdZxZS)DetrCrossAttentionz Multi-headed cross-attention from 'Attention Is All You Need' paper. In DETR, queries get their own position embeddings, while keys get encoder position embeddings. Values don't get any position embeddings. rTr~rrrrEct||_||z|_|jdz|_||_d|_tj||||_ tj||||_ tj||||_ tj||||_ dSrrrs r)rIzDetrCrossAttention.__init__rr(Nr key_value_statesrr encoder_position_embeddingsrrc |jdd}g|d|jR}|jdd} g| d|jR} |||zn|} |||zn|} || |dd} || | dd}||| dd}tj|j j t}||| |||f|j sdn|j |jd|\}}|jg|dR}||}||fS)z Position embeddings logic: - Queries get position_embeddings - Keys get encoder_position_embeddings - Values don't get any position embeddings Nr\rrrr r )rMr rrr rrquery_input_shapequery_hidden_shapekv_input_shapekv_hidden_shape query_input key_inputrrrrrrs r)rbzDetrCrossAttention.forward s*/4D0D"DdmDD)/4>N>B> >>=P=\m&999bo +6 : : :!  {{;//445GHHRRSTVWXX [[++00AAKKAqQQ {{#34499/JJTTUVXYZZ (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k)A+<AbAAALLNN kk+.. L((r(rNNNrrds@r)rrs% EEEE! E  E  EEEEEE0/337;? 1)1)|1) ,1) t+ 1) #\D0 1) &+\D%8 1)+,1) u|U\) *1)1)1)1)1)1)1)1)r(rcPeZdZdededeffd ZdejdejfdZxZ S)DetrMLPr~rintermediate_sizecttj|||_tj|||_t |j|_|j |_ |j |_ dSr) rHrIrirfc1fc2ractivation_function activation_fnactivation_dropoutr)rMr~rr*rOs r)rIzDetrMLP.__init__Asi 9[*;<<9. <<#F$>?"(";~ r(r rc0|||}tj||j|j}||}tj||j|j}|S)Nr)r/r,rirrr0rr-)rMr s r)rbzDetrMLP.forwardIs}**488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd r() r r!r"rrrIr$rrbrcrds@r)r)r)@sy&z&&PS&&&&&&U\elr(r)c ~eZdZdeffd Z d dejdejdejdzdeedejf d Z xZ S) DetrEncoderLayerr~c~t|j|_t ||j|j|j|_tj |j|_ |j |_ t||j|j |_tj |j|_dSN)r~rrr)rHrId_modelrrencoder_attention_headsr self_attnri LayerNormself_attn_layer_normrr)encoder_ffn_dimmlpfinal_layer_normrMr~rOs r)rIzDetrEncoderLayer.__init__Rs !>*( & >,     %'L1A$B$B!~ 64#3V5KLL " T-= > >r(Nr rspatial_position_embeddingsrrc |}|jd|||d|\}}tj||j|j}||z}||}|}||}||z}||}|jr_tj | s9tj |j j dz }tj|| |}|S)a[ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, target_len, source_len)` where padding elements are indicated by very large negative values. spatial_position_embeddings (`torch.FloatTensor`, *optional*): Spatial position embeddings (2D positional encodings of image locations), to be added to both the queries and keys in self-attention (but not to values). )r rr ri)minmaxr')r8rirrrr:r<r=r$isfiniteallfinforrBclamp)rMr rr?rresidualr clamp_values r)rbzDetrEncoderLayer.forward`s"!)4> ') ;      q --mt|VZVc-dd  =0 11-@@  //  =0 --m<< = ^>-004466 ^#k-*=>>BTI % M |Q\ ] ] ] r(r r r!r"rrIr$rrrrbrcrds@r)r3r3Qs ?z ? ? ? ? ? ?$<@ ''|' '&+\D%8 ' +, '  ''''''''r(r3ceZdZdeffd Z d dejdejdzdejdzdejdzdejdzd ejdzd eed ejfd Z xZ S)DetrDecoderLayerr~ct|j|_t ||j|j|j|_|j|_tj |j|_ t||j|j|j|_ tj |j|_t||j|j|_tj |j|_dSr5)rHrIr6rrdecoder_attention_headsrr8rrir9r:r encoder_attnencoder_attn_layer_normr)decoder_ffn_dimr<r=r>s r)rIzDetrDecoderLayer.__init__s !>*( & >,     ~ $&L1A$B$B!.( & >,     (*|D4D'E'E$64#3V5KLL " T-= > >r(Nr rr?"object_queries_position_embeddingsr9encoder_attention_maskrrc |}|jd|||d|\}} tj||j|j}||z}||}|^|}|jd|||||d|\}} tj||j|j}||z}||}|}||}||z}| |}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, target_len, source_len)` where padding elements are indicated by very large negative values. spatial_position_embeddings (`torch.FloatTensor`, *optional*): Spatial position embeddings (2D positional encodings from encoder) that are added to the keys only in the cross-attention layer (not to values). object_queries_position_embeddings (`torch.FloatTensor`, *optional*): Position embeddings for the object query slots. In self-attention, these are added to both queries and keys (not values). In cross-attention, these are added to queries only (not to keys or values). encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, hidden_size)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, target_len, source_len)` where padding elements are indicated by very large negative values. )r r rrN)r rrr rr') r8rirrrr:rNrOr<r=) rMr rr?rQr9rRrrGrs r)rbzDetrDecoderLayer.forwardsF8!*4> ' B)      q --mt|VZVc-dd  =0 11-@@  ! ,$H0t0 +!65$F,G        M1M11-4 GroupNorm -> Activation.relu in_channels out_channels activationcttj||dd|_tjt d|||_t||_ dS)Nrr kernel_sizepadding) rHrIriConv2dconv GroupNormrAnormrrY)rMrWrXrYrOs r)rIzDetrConvBlock.__init__sa Ik[[\\ \">>CVWW ">;!3CEXYY -"<?K14DkUVFVXkll"<?K14DkUVFVXkll"<?K14DkUWFWYlmm   9[B%6qRSTTTr(rattention_masksrorc|jd|ddddddd}|dd}fd|D}t j||gd}||}||}t|j |D]\}}|||}| |S)a Args: features: Encoder output features, shape (batch_size, hidden_size, H, W) attention_masks: Cross-attention maps from decoder, shape (batch_size, num_queries, num_heads, H, W) fpn_features: List of 3 FPN features from low to high resolution, each (batch_size, C, H, W) Returns: Predicted masks, shape (batch_size * num_queries, 1, output_H, output_W) rr\rc g|]B}|dddddddCS)rr\r)rrr).0fpn_feat num_queriess r) z1DetrMaskHeadSmallConv.forward..>s[   X`H  q ! ! ( (["b" E E M MaQR S S   r(r) rrrrr$rrxryzipr{r|)rMrr}ror  fpn_stagerrs @r)rbzDetrMaskHeadSmallConv.forward*s&+A. %%a((//KRLLTTUVXYZZ)11!Q77    dp    8_"=1EEE  =11  =11 #&t #E#E ? ? Ix%ImX>>MM ...r(rd) r r!r"r#rr<rrIr$rrbrcrds@r)rtrt s$* UUU3iU U ! UUUUUU2/,//5<( /  ////////r(rtc veZdZdZ ddedededeffd Z dd ej d ej d ej d zfd Z xZ S)DetrMHAttentionMapzdThis is a 2D attention module, which only returns the attention softmax (no multiplication by value)rTrrrrEct||z|_|jdz|_||_t j||||_t j||||_dS)Nrr) rHrIrrrrirrr)rMrrrrErOs r)rIzDetrMHAttentionMap.__init__Osr #':: }d* !(i [tDDD i [tDDD r(Nrrrcg|jddd|jR}|jdd|jg|jddR}|||}tj||jj d d|jj |}|j\}}}} |j\} } } } } ||z|| f} ||z| | z| f}|||| | f}| dd | }| ddddd |}tj||jz| dd| dd}|||z}tj|dd|}tj||j|j }|S) Nr\rrrrrrrr)rrrrrirconv2drrDrrErrrr$rrrrrrrr)rMrrrr"key_hidden_shaperr num_headsrrrr query_shape key_shapeattn_weights_shaperrrs r)rbzDetrMHAttentionMap.forward^sZK|1#2#6JJDMJJ&,Q/T]EUVXVYVYEZ[[{{<00556HII ]))  *44R88BB2FF HX  $  8D7I4 KH!+!11a!I-{HE )+Ve^XF ()[&%P&&q!,,7799>>{KK  Aq!Q//::<<AA)LL\%$,. a0C0C D D J JK] ^ ^ h hijlm n n   %'.8L},,\-A-A!-D-D",MMRRS_SdSdSfSfgg },,\T=S^b^k,ll r(rr) r r!r"r#rrrrIr$rrbrcrds@r)rrLsnn  E E E! E E  E E E E E E ko!L6;lTYT`cgTgr(rceZdZUeed<dZdZdZgdZdZ dZ dZ dZ dZ dgZejfdZxZS) DetrPreTrainedModelr~rrr)image)rwr3rKTzMdetr\.model\.backbone\.model\.layer\d+\.0\.downsample\.1\.num_batches_trackedc6t||jj}t |t ro|D]X}t |tjrtj|jj tj|jj dSdS)Nr)ar)gain)rH _init_weightsr~init_xavier_stdrhrtmodulesrir_initkaiming_uniform_rDrE constant_rzeros_rrxavier_uniform_runiform_rr)rMrt xavier_stdmrOs r)rz!DetrPreTrainedModel._init_weightssn f%%%[0 f3 4 4 ;^^%% 2 2a++2)!(a8888v)qvq111  2 2  2 3 3 ; K * + + + K * + + +  !5J G G G G  !5J G G G G G G  < = = ; M&/6 7 7 7 M&29 : : : : : ; ;r()r r!r"rr&base_model_prefixmain_input_nameinput_modalities_no_split_modulessupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_supports_attention_backend_supports_flex_attn"_keys_to_ignore_on_load_unexpectedr$rrrcrds@r)rrs $O!VVV&*#N"&X*&U]__;;;;_;;;;;r(rcveZdZdZeedZdeffd Ze e d de e de fdZxZS) DetrEncoderz Transformer encoder that processes a flattened feature map from a vision backbone, composed of a stack of [`DetrEncoderLayer`] modules. Args: config (`DetrConfig`): Model configuration object. )r  attentionsr~ctj|_tjfdt jD|_|dS)Nc.g|]}tSr')r3rrr~s r)rz(DetrEncoder.__init__.."$d$d$d!%5f%=%=$d$d$dr() rHrIrrirzrangeencoder_layerslayers post_initr>s `r)rIzDetrEncoder.__init__sn    ~ m$d$d$d$duVMbGcGc$d$d$dee  r(Nrrc |}tj||j|j}t |j||}|jD]}|||fd|i|}t|S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Flattened feature map (output of the backbone + projection layer) that is passed to the encoder. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding pixel features. Mask values selected in `[0, 1]`: - 1 for pixel features that are real (i.e. **not masked**), - 0 for pixel features that are padding (i.e. **masked**). [What are attention masks?](../glossary#attention-mask) spatial_position_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Spatial position embeddings (2D positional encodings) that are added to the queries and keys in each self-attention layer. rr~ inputs_embedsrr?)r4)rirrrr r~rr )rMrrr?rr  encoder_layers r)rbzDetrEncoder.forwards.&  --mt|VZVc-dd 2;')    "[  M)M~KfjpMM????r(r')r r!r"r#r3r_can_record_outputsrrIrrrrr rbrcrds@r)rrs-=L]^^z $( $@$@ +, $@  $@$@$@_ $@$@$@$@$@r(rc~eZdZdZeeedZdeffd Z e e d de e defdZxZS) DetrDecodera  Transformer decoder that refines a set of object queries. It is composed of a stack of [`DetrDecoderLayer`] modules, which apply self-attention to the queries and cross-attention to the encoder's outputs. Args: config (`DetrConfig`): Model configuration object. )r rr7r~c6tj|_tjfdt jD|_tjj |_ | dS)Nc.g|]}tSr')rKrs r)rz(DetrDecoder.__init__..rr() rHrIrrirzrdecoder_layersrr9r6 layernormrr>s `r)rIzDetrDecoder.__init__s    ~ m$d$d$d$duVMbGcGc$d$d$dee fn55 r(Nrrc ||}|t|j||}||t|j|||}|jjrdnd} t|jD];\} } | |||||fd|i|}|jjr||}| |fz } <||}|jjrt j| } t|| S)a  Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): The query embeddings that are passed into the decoder. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on certain queries. Mask values selected in `[0, 1]`: - 1 for queries that are **not masked**, - 0 for queries that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding pixel_values of the encoder. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). spatial_position_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Spatial position embeddings (2D positional encodings from encoder) that are added to the keys in each cross-attention layer. object_queries_position_embeddings (`torch.FloatTensor` of shape `(batch_size, num_queries, hidden_size)`, *optional*): Position embeddings for the object query slots that are added to the queries and keys in each self-attention layer. Nr)r~rrr9r'rR)r4r) r r~auxiliary_loss enumeraterrr$rr) rMrrr9rRr?rQrr  intermediateidx decoder_layers r)rbzDetrDecoder.forwardsGP  $)M  %6{+-N ! ,1G1S%>{+5&; &&& ""[7ArrT #,DK"8"8 1 1 C)M+2%  (> M{) 1 $} = =  00 }55  ; % 5 ;|44L =]ijjjjr(NNNNNN)r r!r"r#rKrrrrrIrrrrrrbrcrds@r)rrs*'. z       "#$(+/TkTk+,Tk TkTkTk_ TkTkTkTkTkr(rz The bare DETR Model (consisting of a backbone and encoder-decoder Transformer) outputting raw hidden-states without any specific head on top. ceZdZdeffd ZdZdZee dde j dzde j dzde j dzd e j dzd e j dzd e j dzd e e d ee j ezfdZxZS) DetrModelr~cft|t||_|jdkrt |jdzd|_n?|jdkrt|jdz|_ntd|jtj |j |j|_ tj|jjd|jd |_t#||_t'||_|dS) NsinerT)rlearnedzNot supported r\rrl)rHrIrwrposition_embedding_typerr6position_embeddingrrrirrquery_position_embeddingsr_rinput_projectionrencoderrdecoderrr>s r)rIzDetrModel.__init__\s    '//  )V 3 3&?RS@S_c&d&d&dD # #  +y 8 8&B6>UVCV&W&WD # #Nf.LNNOO O)+f6H&.)Y)Y& " $-*RSU*VX^Xftu v v v"6** "6**  r(cx|jjD]\}}|ddS)NFrrrrrrMrparams r)freeze_backbonezDetrModel.freeze_backbonepsG +<<>> ( (HAu   ' ' ' ' ( (r(cx|jjD]\}}|ddS)NTrrs r)unfreeze_backbonezDetrModel.unfreeze_backbonetsG +<<>> ' 'HAu   & & & & ' 'r(Nrrdecoder_attention_maskencoder_outputsrdecoder_inputs_embedsrrc ||td||j\}} } } |j} |tj|| | f| }|||} | d\}}||}|ddd}| |j| |j |ddd}|d}n2|jd}|j} |}|jd}t|d z}| tj ||j j||g| |j ddd}|ttj|d||f tjd}|d}n#tj||f| tj }||jd|||d |}|jjd|dd}||}ntj|}|jd|||||j|d|}t=|j|j|j |j!|j|j|j |j"S)a decoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, num_queries)`, *optional*): Mask to avoid performing attention on certain object queries in the decoder. Mask values selected in `[0, 1]`: - 1 for queries that are **not masked**, - 0 for queries that are **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing the flattened feature map (output of the backbone + projection layer), you can choose to directly pass a flattened representation of an image. Useful for bypassing the vision backbone. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_queries, hidden_size)`, *optional*): Optionally, instead of initializing the queries with a tensor of zeros, you can choose to directly pass an embedded representation. Useful for tasks that require custom query initialization. Examples: ```python >>> from transformers import AutoImageProcessor, DetrModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50") >>> model = DetrModel.from_pretrained("facebook/detr-resnet-50") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> # the last hidden states are the final query embeddings of the Transformer decoder >>> # these are of shape (batch_size, num_queries, hidden_size) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 100, 256] ```Nz8You have to specify either pixel_values or inputs_embedsrr\rrrrlrrrg?)rrlrrrlrrrr?rrr?rQr9rR)r4r5r6r7r8r9r:rr')#rrrlr$rKrrrrrrrrr~r6rirrrrrlongrrrDrr zeros_likerr4r,r rr7r)rMrrrrrrrr num_channelsrrrlvision_featuresrrprojected_feature_mapflattened_featuresr?flattened_maskseq_lenfeat_dimrQqueriesdecoder_outputss r)rbzDetrModel.forwardxs;h  M$9WXX X  6B6H 3J fe!(F!"Z*fe)DfUUU "mmL*EEO / 3 K%)$9$9+$F$F !!6!>!>q!A!A!K!KAq!Q!Q ''k.?VbVhos'tt1a ( "\\!__NN&,Q/J")F!. $)!,G7C<((H''*j$+2ExQY%Z[[!'-( 1a (%}00D1A1G1G1I1IQY[cPd0eehhinisttuvw!%a"'Z,A&X]Xb!c!c!c  "*dl0-,G O.2-K-R-\-\]^-_-_-f-f 1. . * ! ,+GG&'IJJG'$, !1(C/Q"1"C#1     -?"1"?.9,=&5&G"1"?.9'6'Q    r(r)r r!r"rrIrrrrr$r% LongTensorrrr=r,rbrcrds@r)rrUsHz(((('''26.2;?4826:>G G '$.G $t+G !& 1D 8 G *T1 G (4/ G  %047G +,G  u !O 3G G G ^G G G G G r(rc(eZdZdZfdZdZxZS)DetrMLPPredictionHeadz Very simple multi-layer perceptron (MLP, also called FFN), used to predict the normalized center coordinates, height and width of a bounding box w.r.t. an image. ct||_|g|dz z}tjdt |g|z||gzD|_dS)Nrc3FK|]\}}tj||VdSr)rir)rrNks r) z1DetrMLPPredictionHead.__init__..s0#g#g1BIaOO#g#g#g#g#g#gr()rHrI num_layersrirzrr)rM input_dim hidden_dim output_dimrhrOs r)rIzDetrMLPPredictionHead.__init__ so $ LJN +m#g#gYKRSOUVZdYeUe@f@f#g#g#ggg r(ct|jD]F\}}||jdz kr(tj||n ||}G|S)Nr)rrrrirrV)rMr_ilayers r)rbzDetrMLPPredictionHead.forwardsd!$+.. V VHAu01DOa4G0G0G ""5588,,,UUSTXXAAr()r r!r"r#rIrbrcrds@r)rrsV hhhhh r(rz DETR Model (consisting of a backbone and encoder-decoder Transformer) with object detection heads on top, for tasks such as COCO detection. ceZdZdeffd Zee ddejdej dzdejdzdejdzdejdzd ejdzd e e dzd e e d eejezfd ZxZS)DetrForObjectDetectionr~c,t|t||_t j|j|jdz|_t|j|jdd|_ | dS)Nrrr)rrrr) rHrIrrrrirr6 num_labelsclass_labels_classifierrbbox_predictorrr>s r)rIzDetrForObjectDetection.__init__s    v&& (*y NF-1( ( $4nAZ[    r(Nrrrrrrlabelsrrc 8|j|f|||||d|} | d} || } || } d\} }}|}d\}}|jjrC| j}||}||}|| ||j| |j||\} }}t| || | || j | j | j | j | j| j| j S)aB decoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, num_queries)`, *optional*): Mask to avoid performing attention on certain object queries in the decoder. Mask values selected in `[0, 1]`: - 1 for queries that are **not masked**, - 0 for queries that are **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing the flattened feature map (output of the backbone + projection layer), you can choose to directly pass a flattened representation of an image. Useful for bypassing the vision backbone. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_queries, hidden_size)`, *optional*): Optionally, instead of initializing the queries with a tensor of zeros, you can choose to directly pass an embedded representation. Useful for tasks that require custom query initialization. labels (`list[Dict]` of len `(batch_size,)`, *optional*): Labels for computing the bipartite matching loss. List of dicts, each dictionary containing at least the following 2 keys: 'class_labels' and 'boxes' (the class labels and bounding boxes of an image in the batch respectively). The class labels themselves should be a `torch.LongTensor` of len `(number of bounding boxes in the image,)` and the boxes a `torch.FloatTensor` of shape `(number of bounding boxes in the image, 4)`. Examples: ```python >>> from transformers import AutoImageProcessor, DetrForObjectDetection >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50") >>> model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> target_sizes = torch.tensor([image.size[::-1]]) >>> results = image_processor.post_process_object_detection(outputs, threshold=0.9, target_sizes=target_sizes)[ ... 0 ... ] >>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]): ... box = [round(i, 2) for i in box.tolist()] ... print( ... f"Detected {model.config.id2label[label.item()]} with confidence " ... f"{round(score.item(), 3)} at location {box}" ... ) Detected remote with confidence 0.998 at location [40.16, 70.81, 175.55, 117.98] Detected remote with confidence 0.996 at location [333.24, 72.55, 368.33, 187.66] Detected couch with confidence 0.995 at location [-0.02, 1.15, 639.73, 473.76] Detected cat with confidence 0.999 at location [13.24, 52.05, 314.02, 470.93] Detected cat with confidence 0.999 at location [345.4, 23.85, 640.37, 368.72] ```)rrrrrrr'Nr) r/r0r1r2r3r4r5r6r7r8r9r:)rrrrsigmoidr~rr loss_functionrlr.r4r5r6r7r8r9r:)rMrrrrrrrroutputssequence_outputr1r2r/r0r3 outputs_class outputs_coordrs r)rbzDetrForObjectDetection.forward/seL$*  !#9+'"7     "!*--o>>((99AACC -=*i*  +5 (M={) L&A $ < <\ J J $ 3 3L A A I I K K 151C1C ZmUb22 .D).)!/%7")"?&9$5&-&G")"?&9    r(r)r r!r"rrIrrr$r%rr<r;rrr=r.rbrcrds@r)rrs:z"/3;?4826:>$(l l 'l $t+l !& 1D 8 l *T1 l (4/ l  %047l T T!l +,l  u !$= =l l l ^l l l l l r(rz DETR Model (consisting of a backbone and encoder-decoder Transformer) with a segmentation head on top, for tasks such as COCO panoptic. ceZdZdeffd Zee ddejdej dzdejdzdejdzdejdzd ejdzd e e dzd e e d eejezfd ZxZS)DetrForSegmentationr~crt|t||_|j|j}}|jjjj}t||z|ddddd||j |_ t||d|_ |dS)Nr\)rurhrr.r)r)rHrIrdetrr6r7rrrrrtr. mask_headrbbox_attentionr)rMr~rnumber_of_headsrrOs r)rIzDetrForSegmentation.__init__s    +622 (.~v7U_ %)Y_%=%X".&83DDbD9"##># & :    1oWZ[[[ r(Nrrrrrrrrrc Z|j\} } } } |j} |tj| | | f| }|jj||}|d\}}|jj|}|d dd}|jj |j| |j |d dd}|d}||jjj d|||d|}|jjj jd| dd}||}ntj|}|jjjd|||||j|d |}|d}|j|}|j|}|jd d\} } |j dd| |jj| | }|| | | }|ytj|j j}tj|ddtjd |j|j |}|||| }||||dd|dd|ddg}|| |jjj |jd |jd} d\}!}"}#|d\}$}%|jj!rM|j"}&|j|&}$|j|&}%|#||| || |j|$|%\}!}"}#tI|!|"||| |#|j|j%|j&|j'|j|j%|j& S)aN decoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, num_queries)`, *optional*): Mask to avoid performing attention on certain object queries in the decoder. Mask values selected in `[0, 1]`: - 1 for queries that are **not masked**, - 0 for queries that are **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Kept for backward compatibility, but cannot be used for segmentation, as segmentation requires multi-scale features from the backbone that are not available when bypassing it with inputs_embeds. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_queries, hidden_size)`, *optional*): Optionally, instead of initializing the queries with a tensor of zeros, you can choose to directly pass an embedded representation. Useful for tasks that require custom query initialization. labels (`list[Dict]` of len `(batch_size,)`, *optional*): Labels for computing the bipartite matching loss, DICE/F-1 loss and Focal loss. List of dicts, each dictionary containing at least the following 3 keys: 'class_labels', 'boxes' and 'masks' (the class labels, bounding boxes and segmentation masks of an image in the batch respectively). The class labels themselves should be a `torch.LongTensor` of len `(number of bounding boxes in the image,)`, the boxes a `torch.FloatTensor` of shape `(number of bounding boxes in the image, 4)` and the masks a `torch.FloatTensor` of shape `(number of bounding boxes in the image, height, width)`. Examples: ```python >>> import io >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> import numpy >>> from transformers import AutoImageProcessor, DetrForSegmentation >>> from transformers.image_transforms import rgb_to_id >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50-panoptic") >>> model = DetrForSegmentation.from_pretrained("facebook/detr-resnet-50-panoptic") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> # Use the `post_process_panoptic_segmentation` method of the `image_processor` to retrieve post-processed panoptic segmentation maps >>> # Segmentation results are returned as a list of dictionaries >>> result = image_processor.post_process_panoptic_segmentation(outputs, target_sizes=[(300, 500)]) >>> # A tensor of shape (height, width) where each value denotes a segment id, filled with -1 if no segment is found >>> panoptic_seg = result[0]["segmentation"] >>> panoptic_seg.shape torch.Size([300, 500]) >>> # Get prediction score and segment_id to class_id mapping of each segment >>> panoptic_segments_info = result[0]["segments_info"] >>> len(panoptic_segments_info) 5 ```Nrr\rrrrrrrrr)r)rr}ror'r) r/r0r1r2r@r3r4r5r6r7r8r9r:r')(rrlr$rKrrrrrrrrrrrrDrrrrr4rrrrr~r6rErAwheretensorrrrrrrr?r rr7)'rMrrrrrrrrrrrrrlrrrrrr?rrQrrrr1r2memoryr min_dtype bbox_mask seg_masksr@r/r0r3rr rs' r)rbzDetrForSegmentation.forwardsdR3?2D/ L&%$  Z$?OOOJ)/22<LL+B/ T!%  @ @ M M2::1==GG1MM IO . .!'l>PW[ /  WQZZ Yq!__ $a  "5dio50-,G O.2Y_-V-]-g-ghi-j-j-q-q 1. . * ! ,+GG&'IJJG1$)/1 !1(C/Q"1"C#1     *!,22?CCY--o>>FFHH #)"##.  2<$(l l 'l $t+l !& 1D 8 l *T1 l (4/ l  %047l T T!l +,l  u !$: :l l l ^l l l l l r(r )rr rr)Nr)Lr#rcollections.abcr dataclassesrr$torch.nnrirr activationsrbackbone_utilsr masking_utilsr modeling_layersr modeling_outputsr r r modeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_detrr get_loggerr loggerrr,r.r?ModulerBrqrwrrrrrrrr)r3rKrUrgrtrrrrrrrr __all__r'r(r)r-s $$$$$$!!!!!! &&&&&&!!!!!!++++++666666999999 GFFFFFFF&&&&&&@@@@@@ JIIIIIII555555******  H % % @@@@@:@@ @ @@@@@(@@ @  "?"?"?"?"? "?"?   "?J  )?)?)?)?)?[)?)?   )?X$ $ $ $ $ BI$ $ $ N'''011111bi111hJ J J J J  J J J Z29J!%% I% <% % < % L4' % T\ %%'(%%%%8?)?)?)?)?) ?)?)?)DM)M)M)M)M)M)M)M)`bi"666661666rYYYYY1YYYx 8 8 8 8 8BI 8 8 844444444,@/@/@/@/@/BI@/@/@/F00000000f";";";";";/";";";J:@:@:@:@:@%:@:@:@zpkpkpkpkpk%pkpkpkf f f f f f #f f  f RBI& @ @ @ @ @ 0@ @  @ F D D D D D -D D  D N   r(