tj1dZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z ddlmZddlZddlmZmZdd lmZdd lmZdd lmZmZd d lmZd dlmZd dlm Z m!Z!m"Z"d dl#m$Z$d dl%m&Z&m'Z'd dl(m)Z)d dl*m+Z+d dl,m-Z-m.Z.m/Z/ej0e1Z2e.de Gdde-Z3dNdZ4dOdZ5 dPdZ6Gdd ej7Z8e.Gd!d"e+Z9Gd#d$ej7Z:Gd%d&ej7Z;Gd'd(ej7Z<Gd)d*ej7Z=Gd+d,ej7Z>Gd-d.ej7Z?Gd/d0ej7Z@Gd1d2e)ZAGd3d4ej7ZBe>jCZDd5ZEGd6d7ej7eZFGd8d9eFZGGd:d;eFZHGd<d=eFZIGd>d?ej7ZJeGeHeId@ZKdAedBe eFfdCZLGdDdEe9ZMe.GdFdGe9ZNe.dHGdIdJe9e$ZOe.GdKdLe9ZPgdMZQdS)QzPyTorch UDOP model.N)ABCabstractmethod)Sequence)deepcopy) dataclass)Any)Tensornn)CrossEntropyLoss) UdopConfig)Seq2SeqLMOutputSeq2SeqModelOutput)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)PreTrainedModel) ModelOutputauto_docstringis_torchdynamo_compilingz Class for the model's outputs that may also contain a past key/values (to speed up sequential decoding). Includes an additional attention mask. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZe ejdzed<dS) BaseModelOutputWithAttentionMaska last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Attention mask used in the model's forward pass to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` and `config.add_cross_attention=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nlast_hidden_stateattention_maskpast_key_values hidden_states attentionscross_attentions)__name__ __module__ __qualname____doc__r torch FloatTensor__annotations__r!r"rr#tupler$r%d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/udop/modeling_udop.pyrr5s:37u(4/666/3NE%,333$(OUT\(((59M5*+d299926Je'(4/6668<eE-.5<<<<q&A1 E E #2#  % %&>q&A1 E E O OPQST U U !""  $ $%=a%@! D D !""  $ $%=a%@! D D N NqRS T T  *..r155 r/cHt|tjr|jd}n#t |}tj|}||z }|dkrCtj|g|z|}tj||gd}|d|S)Nrr7) isinstancer*r shapelentensorr;tocat)seq target_len pad_valuenmrets r0 pad_sequencerSys#u|$$ IaL HHl3QA1uuk9+/**--c22ic *** { { r/c B|} tjtjdddddfdddddfzdz | zd| dz } tjtjdddddfdddddfzdz | zd| dz | z} | | z} tjddkdd kz} tjd| d dd d}d|| <||z }tj dddddfd  tj tjt| dddf d| ddddddf| | dddddfgd }|dd}t#|\}}d ||f<fd t%tD}ht'||d dddjfdt%tDfdD}dkr dn| dz tjfd|D}tjfdD!tjfd|D}tj ||gd}tj gdtj |gd|fS)a Combine the image and text embeddings for the input to the encoder/decoder of UDOP. First, the image embeddings are created by checking for each visual patch if it is inside the bounding box of a token. If it is, the visual patch is combined with the token embedding. Then, the visual bounding boxes are combined with the text bounding boxes. Finally, the visual bounding boxes are combined with the text attention mask. Nrg@r5rr6r4Tr7Fc8g|]}||Sr.r.).0iimage_embeddings patch_indss r0 z1combine_image_text_embeddings..s(___1,Q/ 1 >___r/)r?r@c8g|]}||Sr.r.)rYrZr\ visual_bboxs r0r]z1combine_image_text_embeddings..s&QQQQ;q>*Q-0QQQr/cxg|]6}tj|djj7S)rdtypedevice)r*onessizerbrc)rYitemr!s r0r]z1combine_image_text_embeddings..sG! ! ! cgEJtyy||>+?H] ^ ^ ^! ! ! r/c bg|]+}t|tjd,S)rrrSr* zeros_like)rYrfr[max_lens r0r]z1combine_image_text_embeddings..s7pppSWdGU%56Ft6L%M%M N Npppr/c bg|]+}t|tjd,Srhri)rYrfbboxrks r0r]z1combine_image_text_embeddings..s6qqq]a|D'5;KDQUJ;W;WXXqqqr/c bg|]+}t|tjd,Srhri)rYrfr!rks r0r]z1combine_image_text_embeddings..s6 s s sUY\$)9.:N)O)O P P s s sr/)r*clipfloorlongrKfloat64meangather unsqueezer<re full_likeboolrLr:rIflattenziprangerErcr;)r[ inputs_embedsrmr_r! num_patchesrkr?r@sequence_length ocr_points_x ocr_points_y ocr_points target_segrepeated_vision_embedsindrowscolsinput_vision_patchesvisual_attention_maskinputs_vision_patchesr\s` ``` ` @r0combine_image_text_embeddingsrs&"O: T!!!QQQ']T!!!QQQ']2c9OKLLQQSSUVXgjkXkL  5;QQQ1W QQQ1W =DVWW\\^^`acruvcvww   ,J 775= ! !D))B--3&499R==C+?@J"\!Z11"55<2D/ L&% T_Q' ' '5DOA4F+F+FwVwwewwDO\]L^wwaeapqraswww YY|,, ''**44Q:: r/)r&r'r(r)rr __classcell__rs@r0rrsR&& d d d d dr/rcpeZdZUeed<dZdZdZdZdgZ e j fdZ dZ xZS) UdopPreTrainedModelr transformer)imagetextTFwoc> t||jj}t |t rt j|j|dzdSt |tj r@t j |jd||j t j |j dSdSt |trA|jj}|jj}t j|jjd||dzzdSt |t$r&t j|jjd|dzdSt |t(rFt+|dr2|jjs(t j|jjd|dzdSdSdSt |t0rt j|jjd||jjdzzt+|jdr*|jj t j |jj t j|jjd||jjdzzt+|jdr,|jj "t j |jj dSdSdSt |t8rVt j|jjd||jjdzzt+|jdr*|jj t j |jj t j|jjd||jjdzzt+|jdr*|jj t j |jj t j|jjd||jjdzzt+|jdr,|jj "t j |jj dSdSdSt |t>r|jj}|jj }|jj!}t j|j"jd|||zdzzt j|j#jd||dzzt j|j$jd||dzzt j|j%jd|||zdzz|j&r+t j|jjd||dzzdSdSdS)zInitialize the weightsr4rW)rsstdNlm_headbias)'r _init_weightsrinitializer_factorrG UdopLayerNorminit constant_weightr r trunc_normal_rzeros_RelativePositionBiasBased_modelnormal_relative_attention_bias UdopModelsharedUdopForConditionalGenerationhasattrtie_word_embeddingsrUdopDenseActDensewird_ffUdopDenseGatedActDensewi_0wi_1 UdopAttentiond_kv num_headsqkvohas_relative_attention_bias)rmodulefactorrkey_value_proj_dimn_headsrs r0rz!UdopPreTrainedModel._init_weightss f%%%/ fm , ,) p N6=&3, 7 7 7 7 7  * *' p  v}3F C C C C{& FK((((('&  8 9 9# p[3Fk)G L7>SfY`eiXiNj k k k k k k  * * p L-CVc\ J J J J J J  < = = pvy)) P$+2Q P V^2&3,OOOOOO P P P P  1 2 2 p L)DKDW\`C`9a b b b bvy&)) ,fin.H FIN+++ L)DKDTY]C]9^ _ _ _ _vy&)) ,fin.H FIN+++++ , ,.H.H  6 7 7 p L+#6dkFY^bEb;c d d d dv{F++ . 0@0L FK,--- L+#6dkFY^bEb;c d d d dv{F++ . 0@0L FK,--- L)DKDTY]C]9^ _ _ _ _vy&)) ,fin.H FIN+++++ , ,.H.H  . . pk)G!%!1 k+G Ls7M_C_dhBh8i j j j j Ls'4-8P Q Q Q Q Ls'4-8P Q Q Q Q Ls7M_C_dhBh8i j j j j1 p V;BRX]dim\mRnoooooo p p p pr/c|jj}|jj}| Jd||j}|dddf|dddf<||d<| Jd||dk|tj|dk s Jd |S) Nzself.model.config.decoder_start_token_id has to be defined. In Udop it is usually set to the pad_token_id. See Udop docs for more information.r6r5).rz1self.model.config.pad_token_id has to be defined.rz8Verify that `shifted_input_ids` has only positive values) rdecoder_start_token_id pad_token_id new_zerosrHclone masked_fill_r*allrf)r input_idsrrshifted_input_idss r0 _shift_rightz UdopPreTrainedModel._shift_right1s!%!C{/ %11 @211 &// @@%.sCRCx%8%>%>%@%@#qrr'"$:&!'')\'''&&'8D'@,OOOy*a/005577ss9sss7  r/)r&r'r(r r,base_model_prefixinput_modalitiessupports_gradient_checkpointing_can_compile_fullgraph_keep_in_fp32_modulesr*no_gradrrrrs@r0rrs %(&*#"!FU]__-p-p-p-p_-p`!!!!!!!r/rc&eZdZdfd ZdZxZS)rư>cttjt j||_||_dS)ze Construct a layernorm module in the Udop style. No bias and no subtraction of mean. N)rrr Parameterr*rdrvariance_epsilon)rrepsrs r0rzUdopLayerNorm.__init__JsD l5:k#:#:;; #r/ch|tjddd}|tj||jzz}|jjtj tj fvr||jj}|j|zS)NrVr6T)keepdim) rKr*float32powrsrsqrtrrrbfloat16bfloat16)rr#variances r0rzUdopLayerNorm.forwardRs !##EM2266q99>>r4>PP% Ht?T4T(U(UU  ;  ? ? ?),,T[->??M{]**r/)rr&r'r(rrrrs@r0rrIsL$$$$$$ + + + + + + +r/rc*eZdZdeffd ZdZxZS)rrcJttj|j|jd|_tj|j|jd|_tj|j |_ t|j |_ dSNFr)rrr LinearrrrrDropout dropout_ratedropoutr dense_act_fnactrrrs r0rzUdopDenseActDense.__init__dsx )FNFKeDDD)FKeDDDz&"566 &-.r/c||}||}||}t|jjt jr]|j|jjjkrC|jjjt j kr$| |jjj}||}|SN) rrrrGrrr*r rbint8rK)rr#s r0rzUdopDenseActDense.forwardks .. //  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r/r&r'r(r rrrrs@r0rrcsS/z//////       r/rc*eZdZdeffd ZdZxZS)rrcttj|j|jd|_tj|j|jd|_tj|j|jd|_tj |j |_ t|j |_dSr)rrr rrrrrrrrrrrrrs r0rzUdopDenseGatedActDense.__init__{s IfnfkFFF IfnfkFFF )FKeDDDz&"566 &-.r/c|||}||}||z}||}t |jjtjr]|j |jjj krC|jjj tj kr$| |jjj }||}|Sr) rrrrrGrrr*r rbrrK)rr# hidden_gelu hidden_linears r0rzUdopDenseGatedActDense.forwardshhtyy7788  -00 #m3  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r/rrs@r0rrzsS/z//////r/rc*eZdZdeffd ZdZxZS) UdopLayerFFrc$t|jrt||_nt ||_t |j|j|_ tj |j |_ dS)Nr)rr is_gated_actrDenseReluDenserrrlayer_norm_epsilon layer_normr rrrrs r0rzUdopLayerFF.__init__sx    <"8"@"@D  "3F";";D 'F+B,,,    4<eDDD4<eDDD4<eDDD4>4=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rrVr5) rKr*rqabsminrjlogfloatmathrvwhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r0_relative_position_bucketz'UdopAttention._relative_position_buckets>,  c A K !2Q!6 : :5: F F T T  % *; < <  !&+NO`>a>a!b!b b  1$ $y0&/ I'--//); < <h|i/00 1Y& ( "UZ.. &" &+Y &8RT_bcTc(d(d& & " EK2CE_```r/rc||jjj}tj|tj|dddf|z}tj|tj|dddf}||z }|||j |j|j }||} | gd d} | S)z%Compute binned relative position biasNrar(r)r*)rVrr5r) rrrcr*r:rqr/rrrpermuteru) r query_length key_lengthrcpast_seen_tokenscontext_positionmemory_positionr'relative_position_bucketvaluess r0 compute_biaszUdopAttention.compute_biass >18?F < EJvVVVWXWXWXZ^W^_brr,zFSSSTXZ[Z[Z[T[\+.>>#'#A#A #.;= $B$ $  --.FGG **44Q77 r/c |jdd}g|d|jR} |||jnd} t | t jr| n| } |du} || |  dd} d} t |tr1|j |j} | r|j}n |j}n|}| r|n|}| r3|1| r/|j|jj}|j|jj}ng|jddd|jR}|| | dd}|| | dd}|E||||j\}}| r$t |trd|j |j<t j| | dd}||jd }|jsLt jd| jd|d|f|j|j }|jr|jrd|_n$||d||j| }|$|ddddddd|jd f}||z}|}||z }t>j !|"d #|}t>j $||j$|j }t j||}| dd%}|j&g|dR}|'|}||f}|r||fz}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr6rr5rVFTrrcrb)rcr5r7)ptraining)(rHrget_seq_lengthrrGr*r rrr>r=r is_updatedgetcross_attention_cacheself_attention_cachelayerskeysr9rrupdatematmulrzerosrcrbrr? requires_gradr:r functionalsoftmaxr$type_asr contiguousreshaper)rr#maskkey_value_states position_biasr"output_attentionskwargs input_shape hidden_shaper5is_cross_attention query_statesrAcurr_past_key_valuescurrent_states key_states value_stateskv_shapescoresr4 causal_maskposition_bias_masked attn_weights attn_outputoutputss r0rzUdopAttention.forwards$)#2#. BBbB$*ABB M\Mh?99$.IIIno7ABRTYT`7a7aw+11333gw.T9vvm,,11,??II!QOO  o': ; ; 3(377GGJ! L'6'L$$'6'K$$#2 -?R))]  F/"=*"=-4T^DIJ/6t~FMLLP-crc2PBP8OPPH//44X>>HHANNJ66.1166x@@JJ1aPPL*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>lJ,@,@A,F,FGG  #)"-J3  %  *1-{1~zJSYS`hnht!!! .74=726M/ $ 1 1NJv}Wg!2!! "111aaa,Bj.>r.B,B#BC - ; ,&&},,V\\^^,DDLLVTT },,\T\TXTa,bb l<>> !++Aq11<<>> )k);;;;;; ff[))  .  0/Gr/FN)Trr)Nr)NNNNF) r&r'r(r intr staticmethodr/r:rrrs@r0rrs%* $ , , ,: , , , , , ,D- - - \- ^([[[[[[[[r/rc>eZdZddedzffd Z ddZxZS)UdopLayerSelfAttentionFNrctt||||_t |j|j|_tj |j |_ dS)Nrrr ) rrr SelfAttentionrrrrr rrrrs r0rzUdopLayerSelfAttention.__init__osl * 0KW`   (F$>tAAAqqq!G}$M$M!$($>$>tAAAqqq!G}$M$M!$($>$>tAAAqqq!G}$M$M! %' (' (( ( r/)rrrrs@r0rr sR[[[[[[r/r)g?g?ceZdZdZ dfd Ze dd edzd eee fdzd efd Z dd edzd eee fdzd efdZ dZ dd edzd eee fdzd efdZ xZS)ra Base class of relative biases. Args: num_heads (`int`): Number of attention heads in the model, it will create embeddings of size `num_heads`, which will be added to the scores of each token pair. relative_attention_num_buckets (`int`, *optional*, defaults to 32): Pair token metric (distance in the sequence, distance in pixels etc.) will be bucketed, parameter is defining number of such buckets. bidirectional (`bool`, *optional*, defaults to `True`): Whether the distance should be bidirectional for a pair of tokens. If `False`, then distance(tok1, tok2) == distance(tok2, tok1). scaling_factor (`int`, *optional*, defaults to 1): Defining factor which will be used to scale relative distance. max_distance (`int`, *optional*, defaults to 128): All distances above this value will end up in the one/same bucket. augmentation (`bool`, *optional*, defaults to `False`): Whether to multiply relative distances by a random scalar. expand (`bool`, *optional*, defaults to `False`): Whether to expand an existing pretrained model with subsequent additions of prefix_bucket. NrTr5rtokensFc .t||_||_||_||_||_||_||_| |_ ||_ |r |j sdnd} tj |j | z|j|_ dS)NrVr)rr prefix_bucket augmentationlevelr*scaling_factorr(rexpandrr rr) rrrr(rr*rrrr extra_headrs r0rz!RelativePositionBiasBase.__init__Bs *( (,*" .L+'B BQQ ')|D4WZd4dfjft'u'u$$$r/r!rmreturncdSrr.)rr!rms r0 prepare_inputz&RelativePositionBiasBase.prepare_input[s r/cv|||}t||j|j|j}|S)Nr1)rget_relative_position_bucketr(rr*)rr!rmr' rp_buckets r0 get_bucketz#RelativePositionBiasBase.get_bucketcsI ..~tDD8 ,;*    r/c|dddddf}|dddddf}||z }|jr|jr|tjtz}||jz}|tjSr) rr?randomuniformAUGMENTATION_RANGErrKr*rq)r positionsr6r7r's r0get_relative_positionz.RelativePositionBiasBase.get_relative_positionms$QQQ4Z0#AAAtQQQJ/+.>>   E E 1C!D D T00 ##EJ///r/c|jrr|jrktj|jdz|j}|jjj|jjd|j<d|jj|jd<||_d|_| ||}|jr| ddkrC| ddkr*| | ddd}|dddddfdk}| d}t|D]*\}}|j||d||df<|jdz|||dd|f<+||} | dkrt#d| gd } | S) NrVg?Frr5r6r9z Wrong dimension of values tensor)rrr5rV)rrr rrrrrdatarrer<sum enumeratecpunumpyr8rr2) rr!rmnew_biasr is_prefix num_prefixidxnum_prefix_rowr9s r0rz RelativePositionBiasBase.forwardws ; 4- |D$G!$KT^\\HJNJfJmJrHO !F4#F!F GJMHO !D!F!F G+3D (DKOOND99   k~~a  A%%.*=*=a*@*@1*D*D%,,^-@-@-C-CQJJ QQQ1W )I"r**J'01A1A1G1G1I1I'J'J k k#^CGCf #?@CGCfijCj #~?@@55i@@ ::<<1  ?@@ @ -- r/) NrTr5rrFFFNN)r&r'r(r)rrr dictstrrrrrrrrs@r0rr,sU.')vvvvvv2)-&*    38nt#      ^ $T#s(^VZEZfl000ftm$sCx.SWBWcir/rcReZdZd fd Zd dedzdeeefdzdefdZxZ S) RelativePositionBias1Dr5rc @tjd||d|dS)z Reimplementation of T5 relative position bias. Distance between given tokens is their distance in the sequence. Parameters are the same as in base class rr*Nr.rrrrr*rTrs r0rzRelativePositionBias1D.__init__0 \\\\U[\\\\\r/Nr!rmrc|jdkrtd|tj|dtj|jdddf}|S)Nr5zNo need to scale 1d featuresra)rrrr*r:rerqrc)rr!rmr's r0rz$RelativePositionBias1D.prepare_inputsv  ! # #;<< < 66 L,,Q//uz.J_ ` ` `aeghghghah i  ! r/)r5rr r&r'r(rr rrrrrrs@r0rrs]]]]]]!!FTM!SRUXY]H]!io!!!!!!!!r/rcReZdZdfd Zd dedzdeeefdzdefdZxZ S) RelativePositionBiasHorizontaldc @tjd||d|dS)z Represents in the bucket embeddings horizontal distance between two tokens. Parameters are the same as in base class rNr.rrs r0rz'RelativePositionBiasHorizontal.__init__rr/Nr!rmrc|jdkstd|td|ddddddgfd}||S)Nr4ENeed to scale the values of bboxes, as there are in small (0,1) rangez6Bbox is required for horizontal relative position biasrrVr6r7rrrsr)rr!rmhorizontal_positions r0rz,RelativePositionBiasHorizontal.prepare_inputsv"S((dee e <UVV V&*111aaa!Q<&8&=&="&=&E&E))*=>>>r/rrrrrs@r0rrs]]]]]]??FTM?SRUXY]H]?io????????r/rcReZdZdfd Zd dedzdeeefdzdefdZxZ S) RelativePositionBiasVerticalrc @tjd||d|dS)z Represents in the bucket embeddings vertical distance between two tokens. Parameters are the same as in base class rNr.rrs r0rz%RelativePositionBiasVertical.__init__rr/Nr!rmrc|jdkstd|td|ddddddgfd}||S)Nr4rz4Bbox is required for vertical relative position biasr5rr6r7r)rr!rmvertical_positions r0rz*RelativePositionBiasVertical.prepare_inputsv"S((dee e <STT T$(AAA1v$6$;$;$;$C$C))*;<<>$%[\\\?,; K(   \)4CC{CC D D D D r/ceZdZdZfdZededefdZdZ dZ d de e zfd Z xZS) UdopStackz This class is based on `T5Stack`, but modified to take into account the image modality as well as 2D position embeddings. c~ttjjj|_t|_j |_ j |_ tj fdt|j D|_ tjj|_tjj|_|j st)jj|_||_|dS)Nc Vg|]%}tt|dk|&S)rrj)rxrw)rYrZrs r0r]z&UdopStack.__init__.. s4 v v vZ[Yv4Q<t|}t|Sr)rr)rrelative_bias_lists r0rzUdopStack._get_relative_biass1&99-.@AAAr/c|jSrrrs r0get_output_embeddingszUdopStack.get_output_embeddingss   r/c||_dSrrrnew_embeddingss r0set_input_embeddingszUdopStack.set_input_embeddings!s*r/Nc | | n |jj} | | n |jj} ||n |jj}||n |jj}|#|!|jrdnd}t d|d|d|Jtj|dkr2| }| d|d}n3||tj|dkrtj d|jj |j |j }tjd|j |j }tjd |j |j }| }tj|ddddddf|j} t$d n@|| dd}n!|jrdnd}t d |d |d|+|jt d||}|||} | d|jj|jjz}t1| |||||d|jj|jj \}}}| dd}|js||||z }|\}}| dur|jsJd|d|jr]| rZ| X|jjr7t7t9|jt9|j} nt9|j} n |jsd} | | nd}|/t=s!||z}tj|||j }|jjrtA|j||| }nO|ddddddf}||j}d|z tj!|jj"z}|jr|tG|j|||}nd}|rdnd}| rdnd}| r |jrdnd}|jrd} n|$||} | |z} d}|}|%|}tM|j'D]g\}} |r||fz}| ||| |||| | |  }!|!d}|!d} |jr| |!| rdnd}| r||!dfz}|jr ||!dfz}h|(|}|%|}|r||fz}|stSd ||| |||fDStU||| |||!S)"Ndecoder_zYou cannot specify both z inputs and zinputs_embeds at the same timerr6)r9rr=)r9rr9z Empty batchzYou have to specify either z inputs or r{zz$UdopStack.forward..s4  =!===  r/)r r!r"r#r$r%)+rrmrSoutput_hidden_statesrrrr*numelrer>fullrrcrbrIrjrKrwarningrrr?r@rris_encoder_decoderrrr@rrdrrr"rrrrrrr-r)"rrr!rmrrr{rr_r[rRr"rmrSrrrTerr_msg_prefixrUr|r seq_lengthpast_key_values_lengthmask_seq_lengthr_encoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrr#rZ layer_module layer_outputss" r0rzUdopStack.forward$s&"+!6IIDK+/?BZZNt>ttnttt  "u{9'='='A'A#..**K!r;r?;;II  "y'<YAWAW[\A\A\ 9dk.FyO_gpgvwwwI"[9;KS\SbcccN;|I4DIO\\\D#..**K!,^AAAtT111rrUcrrrss s   ( !_``` --i88M  ##11,??   '+0DK4JJK2O  & & 3 3 /M4(,,..ss3K :4#3 T33D99 9M!, J   ? k k$kPT$k$k$k k k? ? # G_4;1G&9$DK888,dk:Z:Z:Z''OO'3$+&F&F&FO ##OETE`!?!?!A!A!Afg  !*B*D*D !4zAO"Z OML`aaaN ; ! U,{+- / KK)D$)9:K%..}/B.CCK, M[aaZ[0\-  V!/=3C2E!E?V+?=QRCSBU+U(--m<<  ]33   E 1]4D D    ""#%"(       0+)++%1     r/NNNNNNNNNNNNNNN)r&r'r(r)rrfr rrrrr-rrrrs@r0rrs ,B:B2PBBB\B!!!+++  "#!!D D $ 1 1%D D D D D D D D r/rc#0eZdZdddddZfdZdZdZe dd edzd edzd e e e fdzd edzd e e e fdzdedzdedzdedzdedzde dzdedzde dzde dzde dzde dzdeezf dZxZS)r shared.weightpatch_embed.proj.weightpatch_embed.proj.bias)encoder.embed_tokens.weightdecoder.embed_tokens.weight!encoder.embed_patches.proj.weightencoder.embed_patches.proj.biasct|tj|j|j|_t||_t|}d|_ d|_ t||_ t|}d|_ |j|_t||_|dS)NFT)rrr rrrrr patch_embedrrrmrencodernum_decoder_layersrdecoderrrrencoder_configdecoder_configrs r0rzUdopModel.__init__s    l6#4fnEE .v66!&))$)!#(  00 !&))$(!$*$=! 00  r/c|jSrrrs r0get_input_embeddingszUdopModel.get_input_embeddings {r/c|||_|j||j|dSrrr!rr#rs r0rzUdopModel.set_input_embeddings ;$  )).999 )).99999r/Nrr!rmrr_decoder_input_idsdecoder_attention_maskr{encoder_outputsr"decoder_inputs_embedsrmrSrrrc | | n |jj} ||n |jj}| |||||||| || } | d}|r| jn| d}|||| | ||| | || }|sQt dt|D}t dt| D} || zSt|j |j |j |j |j | j | j | j S) aA bbox (`torch.LongTensor` of shape `({0}, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. Note that `sequence_length = token_sequence_length + patch_sequence_length + 1` where `1` is for [CLS] token. See `pixel_values` for `patch_sequence_length`. visual_bbox (`torch.LongTensor` of shape `(batch_size, patch_sequence_length, 4)`, *optional*): Bounding boxes of each patch in the image. If not provided, bounding boxes are created in the model. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoProcessor, AutoModel >>> from datasets import load_dataset >>> import torch >>> # load model and processor >>> # in this case, we already have performed OCR ourselves >>> # so we initialize the processor with `apply_ocr=False` >>> processor = AutoProcessor.from_pretrained("microsoft/udop-large", apply_ocr=False) >>> model = AutoModel.from_pretrained("microsoft/udop-large") >>> # load an example image, along with the words and coordinates >>> # which were extracted using an OCR engine >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> inputs = processor(image, words, boxes=boxes, return_tensors="pt") >>> decoder_input_ids = torch.tensor([[model.config.decoder_start_token_id]]) >>> # forward pass >>> outputs = model(**inputs, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1, 1024] ```N) rr!rmrr_r{rSrrrr5 rr!r{r"rrrmrSrrc3,K|]\}}|dk |VdSr5Nr.rYrvalues r0rz$UdopModel.forward..|2#c#cjc5Z]abZbZbEZbZbZbZb#c#cr/c3,K|]\}}|dk |VdSr5r.r6s r0rz$UdopModel.forward..}r8r/)r r"decoder_hidden_statesdecoder_attentionsr%encoder_last_hidden_staterencoder_attentions)rrmrr!r!r#r-rrr r"r#r$r%)rrr!rmrr_r.r/r{r0r"r1rmrSrrrTr#rdecoder_outputss r0rzUdopModel.forwardshP"+!6IIDK>> from transformers import AutoProcessor, UdopForConditionalGeneration >>> from datasets import load_dataset >>> # load model and processor >>> # in this case, we already have performed OCR ourselves >>> # so we initialize the processor with `apply_ocr=False` >>> processor = AutoProcessor.from_pretrained("microsoft/udop-large", apply_ocr=False) >>> model = UdopForConditionalGeneration.from_pretrained("microsoft/udop-large") >>> # load an example image, along with the words and coordinates >>> # which were extracted using an OCR engine >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> # one can use the various task prefixes (prompts) used during pre-training >>> # e.g. the task prefix for DocVQA is "Question answering. " >>> question = "Question answering. What is the date on the form?" >>> encoding = processor(image, question, text_pair=words, boxes=boxes, return_tensors="pt") >>> # autoregressive generation >>> predicted_ids = model.generate(**encoding) >>> print(processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]) 9/30/92 ```N rrmr_rr!r{rSrrrr5r3rr) ignore_indexr6rV) losslogitsr"r:r;r%r<rr=)rrmrrr!r!r#rrrr r>rer r"r#r$r%r )rrr!rmrr_r.r/r{r0r"r1rmrSrrrFrTr#rr>sequence_output lm_logitsrJloss_fctrs r0rz$UdopForConditionalGeneration.forwards\"+!6IIDK>> from transformers import AutoProcessor, UdopEncoderModel >>> from huggingface_hub import hf_hub_download >>> from datasets import load_dataset >>> # load model and processor >>> # in this case, we already have performed OCR ourselves >>> # so we initialize the processor with `apply_ocr=False` >>> processor = AutoProcessor.from_pretrained("microsoft/udop-large", apply_ocr=False) >>> model = UdopEncoderModel.from_pretrained("microsoft/udop-large") >>> # load an example image, along with the words and coordinates >>> # which were extracted using an OCR engine >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = processor(image, words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NrH)rrSrrr!) rrrmr!rr_r{rSrrrTr0s r0rzUdopEncoderModel.forwardnsv2C1N--TXT_Tq$8$D $+Jj &1%rgs ########$$$$$$!!!!!! %%%%%%###### '&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999------  8 $ $  #=#=#=#=#={#=#=  #=L.    $ P/P/P/P/f"):N!N!N!N!N!/N!N!N!d+++++BI+++4 .RY<")&BIFRYDbi@Y Y Y Y Y *Y Y Y x8 -F!eeeeery#eeeP!!!!!5!!!$?????%=???&=====#;===&RY* !0, 9Q0R,k k k k k #k k k \] ] ] ] ] #] ] ] @w w w w w #6w w w tlllll*lll^ c b br/