tj5dZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZmZmZd d lmZejeZedeGddeZGddejZGddejZGddejZGddejZGddejZ GddejZ!GddejZ"dDd!Z#Gd"d#ejZ$Gd$d%ejZ%Gd&d'ejZ&ej'j(dEd*e)d+e*fd,Z+Gd-d.ejZ,Gd/d0ejZ-Gd1d2ejZ.Gd3d4ejZ/Gd5d6ejZ0Gd7d8ejZ1Gd9d:ejZ2Gd;dejZ4eGd?d@eZ5edAGdBdCe5Z6dCd@gZ7dS)FzPyTorch ZoeDepth model.N) dataclass)nn)initialization)ACT2FN) load_backbone)DepthEstimatorOutput)PreTrainedModel) ModelOutputauto_docstringlogging)ZoeDepthConfigz[ Extension of `DepthEstimatorOutput` to include domain logits (ZoeDepth specific). ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdfdzed<dZ e ejdfdzed<dS) ZoeDepthDepthEstimatorOutputaT loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. domain_logits (`torch.FloatTensor` of shape `(batch_size, num_domains)`): Logits for each domain (e.g. NYU and KITTI) in case multiple metric heads are used. Nlosspredicted_depth domain_logits. hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrtuplerl/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/zoedepth/modeling_zoedepth.pyrr"s&*D% d ")))04OU&-444.2M5$t+222:>M5*C/047>>>7;Je',-4;;;;;r!rc^eZdZdZfdZdeejdeejfdZxZ S)ZoeDepthReassembleStageaE This class reassembles the hidden states of the backbone into image-like feature representations at various resolutions. This happens in 3 stages: 1. Map the N + 1 tokens to a set of N tokens, by taking into account the readout ([CLS]) token according to `config.readout_type`. 2. Project the channel dimension of the hidden states according to `config.neck_hidden_sizes`. 3. Resizing the spatial dimensions (height, width). Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. c 6t|j|_tj|_t |j|jD]/\}}|j t|||0|jdkr|tj|_ |j }|jD]W}|j tj tjd|z|t|jVdSdS)N)channelsfactorproject)super__init__ readout_typer ModuleListlayerszipneck_hidden_sizesreassemble_factorsappendZoeDepthReassembleLayerreadout_projectsbackbone_hidden_size SequentialLinearr hidden_act)selfconfigneck_hidden_sizer' hidden_size_ __class__s r"r+z ZoeDepthReassembleStage.__init__Gs "/moo (+F,DfF_(`(` j j $ f K  6vHXaghhh i i i i  ) + +$&MOOD ! 5K-  %,,M")A O["I"I6RXRcKdee , +  r!rreturnc|djd}tj|d}|dddf|ddddf}}|j\}}}|||||}|dddd}|jdkri|dd}|d |} tj|| fd }n#|jd kr||d z}g} t| |dD]~\} } |jdkr|j | | } | ddd|d ||} |j | | } | | | S) z Args: hidden_states (`list[torch.FloatTensor]`, each of shape `(batch_size, sequence_length + 1, hidden_size)`): List of hidden states from the backbone. rdimNrrr)r()rr)radd)shapercatreshapepermute contiguousr,flatten unsqueeze expand_as enumeratesplitr4r.r2) r9r patch_height patch_width batch_size cls_tokentotal_batch_sizesequence_length num_channelsreadoutout stage_idx hidden_states r"forwardzZoeDepthReassembleStage.forwardXs #1%+A.  -Q777 #0A#6 aaae8L= :G:M7/<%--.> k[ghh %--aAq99DDFF   ) ))11!44<CCMM  % ' ')I,?,?,C,CCM'01D1DZUV1D1W1W'X'X % % #I| I--?t4Y? MM (//1a88@@RQ]_jkkL14;y1,??L JJ| $ $ $ $ r! rrrrr+listrTensorrZ __classcell__r>s@r"r$r$7su  "&T%,%7&W[\a\hWi&&&&&&&&r!r$c$eZdZfdZdZxZS)r3c t|j}tj||d|_|dkr tj||||d|_dS|dkrtj|_dS|dkr0tj||dtd|z d|_dSdS)Nr) in_channels out_channels kernel_sizerrdstridepaddingr) r*r+r5rConv2d projectionConvTranspose2dresizeIdentityint)r9r:r&r'r<r>s r"r+z ZoeDepthReassembleLayer.__init__s 1 ) (`abbb A::,XxV\blmnnnDKKK q[[+--DKKK aZZ)HhAcRSV\R\ooghiiiDKKKZr!cZ||}||}|SN)rirkr9rYs r"rZzZoeDepthReassembleLayer.forwards*|44 {{<00 r!rrrr+rZr^r_s@r"r3r3sL j j j j j r!r3c*eZdZdeffd ZdZxZS)ZoeDepthFeatureFusionStager:cttj|_t t |jD])}|jt|*dSro) r*r+rr-r.rangelenr0r2ZoeDepthFeatureFusionLayer)r9r:r=r>s r"r+z#ZoeDepthFeatureFusionStage.__init__sx moo s634455 C CA K  9&AA B B B B C Cr!c|ddd}g}d}t||jD]4\}}| ||}n |||}||5|S)NrC)r/r.r2)r9rfused_hidden_statesfused_hidden_staterYlayers r"rZz"ZoeDepthFeatureFusionStage.forwards%ddd+  !#&}dk#B#B ; ; L%!)%*U<%8%8""%*U+=|%L%L"  & &'9 : : : :""r!)rrrrr+rZr^r_s@r"rsrssZC~CCCCCC #######r!rscFeZdZdZfdZdejdejfdZxZS)ZoeDepthPreActResidualLayerz ResidualConvUnit, pre-activate residual unit. Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. c6t|j|_|j|jn|j }t j|_t j|j |j ddd||_ t j|_ t j|j |j ddd||_ |jrLt j |j |j|_t j |j |j|_dSdS)Nrr)rdrfrgbias)eps)r*r+!use_batch_norm_in_fusion_residualuse_batch_normuse_bias_in_fusion_residualrReLU activation1rhfusion_hidden_size convolution1 activation2 convolution2 BatchNorm2dbatch_norm_eps batch_norm1 batch_norm2)r9r:rr>s r"r+z$ZoeDepthPreActResidualLayer.__init__s$ $F1=  . .(( $ 799I  %  %,    799I  %  %,       d!~f.GVMbcccD !~f.GVMbcccD    d dr!rYr?c(|}||}||}|jr||}||}||}|jr||}||zSro)rrrrrrrr9rYresiduals r"rZz#ZoeDepthPreActResidualLayer.forwards'' 55 ((66   :++L99L'' 55 ((66   :++L99Lh&&r!) rrrrr+rr]rZr^r_s@r"r}r}sn d d d d dD'EL'U\''''''''r!r}cheZdZdZd dedeffd Zd dejdejdzd ejfd Z xZ S) rwa8Feature fusion layer, merges feature maps from different stages. Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. align_corners (`bool`, *optional*, defaults to `True`): The align_corner setting for bilinear upsample. Tr: align_cornersct||_tj|j|jdd|_t||_t||_ dS)NrT)rdr) r*r+rrrhrrir}residual_layer1residual_layer2)r9r:rr>s r"r+z#ZoeDepthFeatureFusionLayer.__init__si *)F$=v?Xfgnrsss:6BB:6BBr!NrYrr?ct|c|j|jkr;tj||jd|jdfdd}|||z}||}tj|dd|j}||}|S)Nr)rbilinearFsizemoder scale_factorrr)rEr functional interpolaterrrrirs r"rZz"ZoeDepthFeatureFusionLayer.forwards  !X^33=44L$6q$9<;Ma;P#QXbrw5($*>*>x*H*HHL++L99 }00 qzI[1  |44 r!)Tro) rrrrrboolr+rr]rZr^r_s@r"rwrwsCC~CdCCCCCCELEL4s r"r+zZoeDepthNeck.__init__#s    ! -&2H2SW_2_2_$(D ! !$;F$C$CD !]__ / s sG J  bi1JXYcdkpqqq r r r r7v>>r!rr?c|t|ttfstdt |t jjkrtdj|||}fdt|D} |}||dfS)z Args: hidden_states (`list[torch.FloatTensor]`, each of shape `(batch_size, sequence_length, hidden_size)` or `(batch_size, hidden_size, height, width)`): List of hidden states from the backbone. z2hidden_states should be a tuple or list of tensorszOThe number of hidden states should be equal to the number of neck hidden sizes.NcBg|]\}}j||Sr )r).0ifeaturer9s r" z(ZoeDepthNeck.forward..Ds-VVVzq'MDJqM'**VVVr!rC) isinstancerr\ TypeErrorrvr:r0 ValueErrorrrMr)r9rrOrPfeaturesoutputs` r"rZzZoeDepthNeck.forward4s -%77 RPQQ Q }  T[%B!C!C C Cnoo o  , 11-{[[MVVVVY}=U=UVVV""8,,x|##r!) rrrrrr+r\rr]rZr^r_s@r"rrs  ?~??????"$T%,%7$W[\a\hWi$$$$$$$$r!rcReZdZdZfdZdeejdejfdZxZ S)#ZoeDepthRelativeDepthEstimationHeada Relative depth estimation head consisting of 3 convolutional layers. It progressively halves the feature dimension and upsamples the predictions to the input resolution after the first convolutional layer (details can be found in DPT's paper's supplementary material). ct|j|_d|_|jrt jddddd|_|j}t j||dzddd|_t j ddd |_ t j|dz|j ddd|_ t j|j dddd |_ dS) N)rr)rrrer)rrrTrr)r*r+ head_in_indexriadd_projectionrrhrconv1Upsampleupsamplenum_relative_featuresconv2conv3)r9r:rr>s r"r+z,ZoeDepthRelativeDepthEstimationHead.__init__Ss #1   e iSfV]cdddDO,YxQAaYZ[[[  SWXXX Yx1}f.JXYbcmnooo Yv;QAVWabccc r!rr?c||j}|j1||}tj|}||}||}||}tj|}|}||}tj|}|d}||fS)NrrA) rrirrrrrrsqueeze)r9rrrs r"rZz+ZoeDepthRelativeDepthEstimationHead.forwardbs%d&89 ? & OOM::M%BGIIm44M =11  m44  =11 ! -00   =11 ! -00 '//A/66((r!r[r_s@r"rrLsr d d d d d)T%,%7)EL))))))))r!rHz>c||z}||z}|tj|z|tj|zz ||z tj||z |zzz S)z%log(nCk) using stirling approximation)rlog)nkrs r" log_binomrxsY CA CA uy|| a%)A,,. .!a%59QUS[;Q;Q1Q QQr!c6eZdZdejffd ZddZxZS)LogBinomialSoftmaxrc xt||_||_|dt jd|ddddd|dt j|jdz gddddddS) a7Compute log binomial distribution for n_classes Args: n_classes (`int`, *optional*, defaults to 256): Number of output classes. act (`torch.nn.Module`, *optional*, defaults to `torch.softmax`): Activation function to apply to the output. k_idxrrrCF) persistent k_minus_1N) r*r+ractregister_bufferrarangeviewtensor)r9 n_classesrr>s r"r+zLogBinomialSoftmax.__init__s  Wel1i&@&@&E&EaQPQ&R&R_deee [%, |*D*D*I*I!RQRTU*V*Vchiiiiir!?-C6?c|jdkr|d}d|z |d}||d}t|j|j|jt j|zz|j|jz t j|zz}|||z dS)aCompute the log binomial distribution for probabilities. Args: probabilities (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Tensor containing probabilities of each class. temperature (`float` or `torch.Tensor` of shape `(batch_size, num_channels, height, width)`, *optional*, defaults to 1): Temperature of distribution. eps (`float`, *optional*, defaults to 1e-4): Small number for numerical stability. Returns: `torch.Tensor` of shape `(batch_size, num_channels, height, width)`: Log binomial distribution logbinomial(p;t). rrr)minmaxrA) ndimrKclamprrrrrr)r9 probabilities temperaturerone_minus_probabilitiesys r"rZzLogBinomialSoftmax.forwards   " ")33A66M#$}#4";";";"M"M%+++== dndj 1 1j59]333 4~ *ei8O.P.PP Q xxKQx///r!)rr)rrrrsoftmaxr+rZr^r_s@r"rrs^!$%- j j j j j j00000000r!rc*eZdZ dfd ZdZxZS)%ZoeDepthConditionalLogBinomialSoftmaxrr)c t||z|z}tjtj||z|dddtjtj|ddddtj|_d|_|j |_ |j |_ t|tj |_dS)aPer-pixel MLP followed by a Conditional Log Binomial softmax. Args: in_features (`int`): Number of input channels in the main feature. condition_dim (`int`): Number of input channels in the condition feature. n_classes (`int`, *optional*, defaults to 256): Number of classes. bottleneck_factor (`int`, *optional*, defaults to 2): Hidden dim factor. rrrer)rN)r*r+rr6rhGELUSoftplusmlpp_epsmax_tempmin_temprrrlog_binomial_transform)r9r: in_features condition_dimrbottleneck_factor bottleneckr>s r"r+z.ZoeDepthConditionalLogBinomialSoftmax.__init__s* !M16GG = IkM1:1UV`a b b b GII Ij%Qq! L L L KMM      &8 &V&V&V###r!c|tj||fd}|dddddf|dddddf}}||jz}|ddddf|ddddf|ddddfzz }||jz}|ddddf|ddddf|ddddfzz }|d}|j|jz |z|jz}|||S)az Args: main_feature (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Main feature. condition_feature (torch.Tensor of shape `(batch_size, num_channels, height, width)`): Condition feature. Returns: `torch.Tensor`: Output log binomial distribution rrANr).r)rrconcatrrKrrr)r9 main_featurecondition_featureprobabilities_and_temperaturerrs r"rZz-ZoeDepthConditionalLogBinomialSoftmax.forwards@)-|M^>_ef1g1g1g(h(h% )!!!RaR* 5 )!!!QRR* 5# & 2 %aaaCi0M!!!Q)4L}]^]^]^`acf]fOg4gh !DJ. !!!!Q), AAAq#I0FUVUVUVXY[^U^I_0_` !++A.. }t}4 CdmS **=+FFFr!)rr)rqr_s@r"rrsb  #W#W#W#W#W#WJGGGGGGGr!rc&eZdZdfd ZdZxZS)ZoeDepthSeedBinRegressorrMbP? ct|j|_|j|_||_||_tj|j|ddd|_ tj d|_ tj||ddd|_ |jdkrtj dntj |_dS)adBin center regressor network. Can be "normed" or "unnormed". If "normed", bin centers are bounded on the (min_depth, max_depth) interval. Args: config (`int`): Model configuration. n_bins (`int`, *optional*, defaults to 16): Number of bin centers. mlp_dim (`int`, *optional*, defaults to 256): Hidden dimension. min_depth (`float`, *optional*, defaults to 1e-3): Min depth value. max_depth (`float`, *optional*, defaults to 10): Max depth value. rrTinplacenormedN)r*r+bottleneck_featuresrbin_centers_type min_depth max_depthrrhrract1rract2)r9r:n_binsmlp_dimrrr>s r"r+z!ZoeDepthSeedBinRegressor.__init__s" !5 & 7""Yt/!QBB GD))) Yw1a88 -1-Bh-N-NBGD))))TVT_TaTa r!c||}||}||}||}|jdkr|dz}||ddz }|j|jz |z}tj |dd|j}tj |d }d |d d d d d f|d d dd d fzz}||fS||fS)z] Returns tensor of bin_width vectors (centers). One vector b for every pixel rrrTrBkeepdim)rrrrrrconstant)rvaluerAg?NrC.) rrrrrsumrrrrpadrcumsum)r9x bin_centersbin_widths_normed bin_widths bin_edgess r"rZz ZoeDepthSeedBinRegressor.forward s  JJqMM IIaLL JJqMMiill  H , ,%,K +koo!To.R.R R .4>9=NNJ**:7IPZbfbp*qqJ ZQ777I111crc3;!7)AAAqrr3J:O!OPK$k1 1 + +r!)rrrrrqr_s@r"rrsRbbbbbb:,,,,,,,r!r,r)alphagammac^|d|||zzS)a:Inverse attractor: dc = dx / (1 + alpha*dx^gamma), where dx = a - c, a = attractor point, c = bin center, dc = shift in bin center This is the default one according to the accompanying paper. Args: dx (`torch.Tensor`): The difference tensor dx = Ai - Cj, where Ai is the attractor point and Cj is the bin center. alpha (`float`, *optional*, defaults to 300): Proportional Attractor strength. Determines the absolute strength. Lower alpha = greater attraction. gamma (`int`, *optional*, defaults to 2): Exponential Attractor strength. Determines the "region of influence" and indirectly number of bin centers affected. Lower gamma = farther reach. Returns: torch.Tensor: Delta shifts - dc; New bin centers = Old bin centers + dc r)divpow)dxrrs r" inv_attractorr&s*" 66!ebffUmm++ , ,,r!c0eZdZ d fd Zd dZxZS) ZoeDepthAttractorLayerrrrFct|j|_|j|_|j|_||_||_ ||_ ||_ ||_ |j x}}tj||ddd|_tjd|_tj||dzddd|_tjd|_dS)zq Attractor layer for bin centers. Bin centers are bounded on the interval (min_depth, max_depth) rrTrr)N)r*r+attractor_alpharattractor_gammagemmaattractor_kindkind n_attractorsrrrmemory_efficientbin_embedding_dimrrhrrrrr r9r:rr#rrr$rrr>s r"r+zZoeDepthAttractorLayer.__init__;s + + ) ( "" 0!' 88 gY{GQ1== GD))) Yw q(8!QBB GD))) r!NTcn|7|r0tj||jdddd}||z}||}||}||}||}|dz}|j\}}}} |||j d|| }|dddddd f} tj||| fdd} |j shtj tj d |j} | t| d| d z d } nztj| | j } t'|j D]5}| t| dd|d fd | z z } 6|jdkr | |j z } | | z}|j|jz |z|jz} tj| d \} }tj| |j|j} || fS)ao The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape `(batch_size, prev_number_of_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers normed and scaled. NrTrrrr)r.meanr rrAdevicer+)rrrrErrrrrr#r$rr+r r"rrK zeros_liker-rurrsortclip)r9rprev_binprev_bin_embeddingr attractorsrQr=heightwidthattractors_normedrfuncdelta_crbin_new_centerss r"rZzZoeDepthAttractorLayer.forwardZsP&  ) %']%>%>& :UY&?&&"&&A JJqMM IIaLL JJqMMYYq\\ $& '1'7$ Avu__Z1BAvuUU 'qqq!!!Q|4m//65/PZjn/oo $ 6!Juy99$)DDd=):)D)DQ)G)G+J_J_`aJbJb)bccijkkkGG&{;;MNNNG4,-- b b=):111a9)E)O)OPQ)R)RU`)`aaayF""!D$55%/~6/IDNZ KQ777 QjdndnMM  ++r!)rrrFNTrqr_s@r"rr:s_ ******><,<,<,<,<,<,<,<,r!rc0eZdZ dfd Zd dZxZS) ZoeDepthAttractorLayerUnnormedrrrTct||_||_||_||_|j|_|j|_|j |_ ||_ |j x}}tj||ddd|_tjd|_tj||ddd|_tj|_dS)zL Attractor layer for bin centers. Bin centers are unbounded rrTrN)r*r+r#rrrrrrr!r"r$r%rrhrrrrrrr&s r"r+z'ZoeDepthAttractorLayerUnnormed.__init__s ( ""+ + ) 0 & 88 gY{GQ1== GD))) Yw aA>> KMM r!Ncr|7|r0tj||jdddd}||z}||}||}||}||}|jdd\}}tj|||fdd}|jshtj tj d|j } | t|d|dz d } nztj||j } t#|jD]5} | t|dd| d fd|z z } 6|j d kr | |jz } || z} | }| |fS) a The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape (batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape (batch_size, prev_num_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers unbounded. Two outputs just to keep the API consistent with the normed version. Nr(rTr)r*r)rrAr,.r+)rrrrErrrrr$rr+r r"rrKr.r-rur#) r9rr1r2rr3r4r5rr7r8rr9s r"rZz&ZoeDepthAttractorLayerUnnormed.forwards&  ) %']%>%>& :UY&?&&"&&A JJqMM IIaLL JJqMMYYq\\ "(- m//65/PZjn/oo $ 6!Juy99$)DDd=)=)=a)@)@;CXCXYZC[C[)[\\bcdddGG&{;;MNNNG4,-- [ [=AAAq#I)>)H)H)K)Kk)YZZZyF""!D$55%/%  ++r!)rrrTr:rqr_s@r"r<r<s_ """""":3,3,3,3,3,3,3,3,r!r<cDeZdZdfd ZdejdejfdZxZS)ZoeDepthProjectorcttj||ddd|_tjd|_tj||ddd|_dS)aProjector MLP. Args: in_features (`int`): Number of input channels. out_features (`int`): Number of output channels. mlp_dim (`int`, *optional*, defaults to 128): Hidden dimension. rrTrN)r*r+rrhrrrr)r9r out_featuresrr>s r"r+zZoeDepthProjector.__init__se Y{GQ1== 74(((Yw aA>> r!rYr?c||}||}||}|Sro)rrrrps r"rZzZoeDepthProjector.forwards;zz,// xx -- zz,// r!)rArrrr+rr]rZr^r_s@r"r@r@sc??????"ELU\r!r@ceZdZdZfdZ d dejdejdejdejdzd edzd e ejf d Z xZ S) ZoeDepthMultiheadAttentionzKEquivalent implementation of nn.MultiheadAttention with `batch_first=True`.c t||zdkrtd|d|d||_t ||z |_|j|jz|_tj||j|_ tj||j|_ tj||j|_ tj|||_ tj ||_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ())r*r+rnum_attention_headsrmattention_head_size all_head_sizerr7querykeyr out_projDropoutdropout)r9r<rJrQr>s r"r+z#ZoeDepthMultiheadAttention.__init__ s  , , 1 11K11-111  $7 #&{5H'H#I#I !58PPY{D,>?? 9[$*<==Y{D,>??  +{;; z'** r!NFquerieskeysvaluesattention_maskoutput_attentionsr?c|j\}}}|||d|j|jdd} |||d|j|jdd} |||d|j|jdd} tj | | dd} | tj |jz } || |z} tj | d} || } tj | | }|dddd}|dd|jfz}||}||}|r|| fn|f}|S)NrCrr)r(rArr)rErMrrJrK transposerNr rmatmulmathsqrtrrrrQrHrIrrLrO)r9rRrSrTrUrVrQ seq_lengthr= query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r"rZz"ZoeDepthMultiheadAttention.forward s%,M! J JJw   T*b$":DS"S%**+BCC  m44 6G]=/22mM]r!)NF) rrrrr+rr]rrrrZr^r_s@r"rGrGsUU+++++248). ...l. . )D0 .  $; . u| ........r!rGc@eZdZdfd Z ddejdzfdZxZS) ZoeDepthTransformerEncoderLayer皙?reluc&t|j}|j}|j}t ||||_tj|||_ tj ||_ tj|||_ tj ||_tj ||_tj ||_tj ||_t$||_dS)N)rQ)r*r+patch_transformer_hidden_size#patch_transformer_intermediate_size%patch_transformer_num_attention_headsrG self_attnrr7linear1rPrQlinear2 LayerNormnorm1norm2dropout1dropout2r activation)r9r:rQrur<intermediate_sizerJr>s r"r+z(ZoeDepthTransformerEncoderLayer.__init__Rs : "F$J3KAT^efffy.?@@ z'** y!2K@@ \+.. \+..  7++  7++  ,r!Nsrc_maskc |x}}|||||d}|||z}||}|||||}|||z}||}|S)N)rRrSrTrUr) rmrsrqrorQrurnrtrr)r9srcrwrRrSsrc2s r"rZz'ZoeDepthTransformerEncoderLayer.forwardfs $~~gDU]~^^_`aDMM$'''jjoo||DLLc9J9J)K)KLLMMDMM$'''jjoo r!)rgrhrorEr_s@r"rfrfQsl------.)-  ,%        r!rfc:eZdZfdZdejfdZdZxZS)ZoeDepthPatchTransformerEncoderc tj}tjfdt jD|_tj|j ddd|_ dS)zViT-like transformer block Args: config (`ZoeDepthConfig`): Model configuration class defining the model architecture. c.g|]}tSr )rf)rr=r:s r"rz.s" i i i ,V 4 4 i i ir!rrreN) r*r+rrr-runum_patch_transformer_layerstransformer_encoderrhrjembedding_convPxP)r9r:rbr>s ` r"r+z(ZoeDepthPatchTransformerEncoder.__init__vs 0 #%= i i i ieFDg>h>h i i i$ $  "$ =1UV`a" " " r!cpuc tjd|||d}tjd|d||d}tj|tjtjd| |z z}||z} tjtj| tj| gd} | d |dd} | S)zGenerate positional encodings Args: sequence_length (int): Sequence length embedding_dim (int): Embedding dimension Returns: torch.Tensor: Positional encodings. r)dtyper-rr)g@r,rA) rrrKexprrrFsincosrepeat) r9rQrT embedding_dimr-rpositionindexdiv_term pos_encodings r"positional_encoding_1dz6ZoeDepthPatchTransformerEncoder.positional_encoding_1ds<?%OOOYYZ[\\ Q qfMMMWWXYZZ9Uuygf1U1U1U'V'V&VYf&fghh(* y%)L"9"959\;R;R!SYZ[[[ #--!-44;;J1MM r!cx||d}tj|d}|ddd}|j\}}}||||||j|j z}tdD]}|j ||}|S)zForward pass Args: x (torch.Tensor - NCHW): Input feature tensor Returns: torch.Tensor - Transformer output embeddings of shape (batch_size, sequence_length, embedding_dim) r))rrrr)r-rr) rrJrrr rHrErr-rrur)r9r embeddingsrQrTrrs r"rZz'ZoeDepthPatchTransformerEncoder.forwards++A..66q99 ]&&z6:: ''1a00 5?5E2 O]$"="= z?PXbXh#># #  q A AA41!4Z@@JJr!) rrrr+rfloat32rrZr^r_s@r"r|r|ush     &Y^ejer$r!r|c&eZdZdfd ZdZxZS)ZoeDepthMLPClassifierr?Nct|}tj|||_tj|_tj|||_dSro)r*r+rr7rnrruro)r9rrChidden_featuresr>s r"r+zZoeDepthMLPClassifier.__init__sU %yo>> '))y,?? r!c||}||}||}|Sro)rnruro)r9rYrs r"rZzZoeDepthMLPClassifier.forwards;||L11 |44  \22 r!)r?Nrqr_s@r"rrsR@@@@@@r!rc(eZdZdZfdZdZxZS)*ZoeDepthMultipleMetricDepthEstimationHeadszn Multiple metric depth estimation heads. A MLP classifier is used to route between 2 different heads. c4tjjj|_j|_j}tj||ddd|_ t|_ tdd|_ |jdkrtn|jdkrttjfd jD|_t%|dz |_tjfd t+d D|_tjfd jD|_jtjfdjD|_dS)NrrrerAr)rrCrsoftplusc pi|]2}|dt|ddz|d|d3S)namerr)rr)rrrr)r)rconfr%r:s r" zGZoeDepthMultipleMetricDepthEstimationHeads.__init__..sc   V 6>-2";/";/    r!rrCrcDg|]}tjdzS)r)rr@rrr=r%r:s r"rzGZoeDepthMultipleMetricDepthEstimationHeads.__init__..sK     " & 9!2-2   r!rc i|]FdtjfdttDGS)rc Rg|]#}|dd$S)rrrrrr )rr Attractorr: configurationr#s r"rzRZoeDepthMultipleMetricDepthEstimationHeads.__init__...sV " "#/?&3K&@&3K&@ r!)rr-rurv)rrrr:r#s @r"rzGZoeDepthMultipleMetricDepthEstimationHeads.__init__..s   "f%r}"'s<'8'8!9!9 ( (   r!c Ri|]#}|dt|dd$S)rrr)r)r)rrr%r:last_ins r"rzGZoeDepthMultipleMetricDepthEstimationHeads.__init__..sV   "f%'L%!(+&' (((   r!)r*r+r%num_attractorsbin_configurationsrrrrhrr|patch_transformerrmlp_classifierrr< ModuleDictseed_bin_regressorsr@seed_projectorr-ru projectorsr3rconditional_log_binomial)r9r:rrr%rr#r>s ` @@@@r"r+z3ZoeDepthMultipleMetricDepthEstimationHeads.__init__s "4, "("; & 7%8Y24GUV_`jklll "A!H!H3RSTTT  H , ,.II  "j 0 06I$&=     #5    $ $  0+:KUfjkUk   -      q       -      &,%>     ".(*       &,%>    ) ) %%%r!c||}||dddddf}||}tj|ddd}d|jD} | tj|d  fd|jDd} n!#t$rtdd wxYw| d } | d } |j } | |\}}|j d vr || z | | z z }n|}||}|j}t!|j||D]'\}}}||}||||d \}}|}|}(|}t$j||jdddd}t$j||jdddd}|j}|||}tj||zdd}||fS)NrTrrCrAcg|] }|d Srr )rrs r"rzFZoeDepthMultipleMetricDepthEstimationHeads.forward..*sTTT=v&TTTr!c,g|]}|dk|Srr )rr:bin_configurations_names r"rzFZoeDepthMultipleMetricDepthEstimationHeads.forward...s(nnnvF6NVmDmDmFDmDmDmr!zbin_configurations_name z! not found in bin_configurationssrrrhybrid2rr(rr)r)rrrrrr rargmaxritem IndexErrorrrrrr3r/rrrrrEr)r9outconv_activationrfeature_blocksrelative_depthr embeddingr domain_votenamesrrrseed_bin_regressorr=seed_bin_centersr1r2r3 projector attractorr bin_embeddingbinrlastrrWrs @r"rZz2ZoeDepthMultipleMetricDepthEstimationHeads.forwards JJz " "**1--aaaAAAg6 ++I66 mM$5$5!T$5$J$JPRSSS UTDZoeDepthMetricDepthEstimationHead.__init__..rs:   "f.GVghhh   r!rc <g|]}|S))rr#rrr )rrrr:rrr#rs r"rz>ZoeDepthMetricDepthEstimationHead.__init__..xsO    !!-a''    r!)r)r*r+rr%rrrrrrrhrrr<rrr@rr-rurr3rrr) r9r:bin_configurationrrrrr%rrr#rr>s ` @@@@@@r"r+z*ZoeDepthMetricDepthEstimationHead.__init__Ss "5a8"8,%k2 %k2 "4, !2"" 0%8Y24GUV_`jklll   H , ,.II  "j 0 06I": 6Y)# # # 09dnt~>]^HH'H!0033.1$/Sa-b-b 7 7 )Iy'%Ig..M(yBTbfggg Cyy{{H!.!4!4!6!6  !!/ 8 8 ; ; " 9 9 ! 122ZW[!:! ! y$ 56A>>> 11-BCCWaqu1vv  ) )$ > >m// QWRSS\PZjn/oo iKQ===Dyr!rqr_s@r"rrRsH9 9 9 9 9 v"""""""r!rc:eZdZUeed<dZdZdZdZfdZ xZ S)ZoeDepthPreTrainedModelr:zoedepth pixel_values)imageTc t|t|trt j|jtjd|j  ddddt j|j tj |j dz g dddddSdS)NrrrC) r* _init_weightsrrinitcopy_rrrrrrr)r9moduler>s r"rz%ZoeDepthPreTrainedModel._init_weightss f%%% f0 1 1 Y Jv|U\!VX%>%>%C%CAr1a%P%P Q Q Q Jv'vx!|n)E)E)J)J1bRSUV)W)W X X X X X Y Yr!) rrrrrbase_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointingrr^r_s@r"rrsf "$O!&*#YYYYYYYYYr!rzU ZoeDepth model with one or multiple metric depth estimation head(s) on top. ceZdZfdZe d dejdejdzdedzdedzdedzde ej e zf d Z xZ S) ZoeDepthForDepthEstimationc@t|t||_t |jjdrGt |jjdr-|jjj|_|jjj|_ntdt||_ t||_ t|jdkrt!|nt#||_|dS)Nr< patch_sizezXZoeDepth assumes the backbone's config to have `hidden_size` and `patch_size` attributesr)r*r+rbackbonehasattrr:r<r5rrrneckr relative_headrvrrr metric_head post_init)r9r:r>s r"r+z#ZoeDepthForDepthEstimation.__init__s    %f-- 4=' 7 7 GDMDXZf*JF '"m2=DOOj !(( @HH6,--11 7v > > >26::  r!NrlabelsrVoutput_hidden_states return_dictr?c d}|td||n |jj}||n |jj}||n |jj}|j|||}|j} |j\} } } } |j } | | z}| | z}| | ||\} }|g| z}| | \}}|g|z}| |d|d|dd|\}}| d}|s*|||f|ddz}n|f|ddz}||f|zn|St||||j|j S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth depth estimation maps for computing the loss. Examples: ```python >>> from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation >>> import torch >>> import numpy as np >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti") >>> model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> # interpolate to original size >>> post_processed_output = image_processor.post_process_depth_estimation( ... outputs, ... source_sizes=[(image.height, image.width)], ... ) >>> # visualize the prediction >>> predicted_depth = post_processed_output[0]["predicted_depth"] >>> depth = predicted_depth * 255 / predicted_depth.max() >>> depth = depth.detach().cpu().numpy() >>> depth = Image.fromarray(depth.astype("uint8")) ```NzTraining is not implemented yet)rrVrrr))rrrrrA)rrrrr)NotImplementedErrorr:rrrVrforward_with_filtered_kwargs feature_mapsrErrrrrrrr)r9rrrVrrkwargsrrdrr=r4r5rrOrPrrWr metric_depthrrs r"rZz"ZoeDepthForDepthEstimation.forwards`  %&GHH H%0%rs !!!!!! &&&&&&!!!!!!++++++444444------9999999999222222  H % %   < < < < <; < <   <GGGGGbiGGGTbi0########2;';';';';'");';';'~""""""""J3$3$3$3$3$293$3$3$l))))))))))")))))))XRRRR)0)0)0)0)0)0)0)0X@G@G@G@G@GBI@G@G@GF5,5,5,5,5,ry5,5,5,p--U-----&\,\,\,\,\,RY\,\,\,~Q,Q,Q,Q,Q,RYQ,Q,Q,h 6FFFFFFFFR!!!!!bi!!!H<<<<<bi<<<~BI"J"J"J"J"J"J"J"J"Z^^^^^ ^^^F Y Y Y Y Yo Y Y Y y y y y y !8y y  y x ()B Cr!