tjdZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZdd lmZmZmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'er/ddl(Z(ddl)m*Z*ejj+j,e(j-j.j/_,ej0e1Z2Gddej+Z3Gddej+Z4Gddej+Z5Gddej+Z6Gddej+Z7Gddej+Z8Gdd eZ9d:d$Z:Gd%d&ej+Z;eGd'd(eZGd,d-ej+Z?eGd.d/e<Z@ed01Gd2d3e<ZAed41Gd5d6e<ZBeGd7d8e<ZCgd9ZDdS)STTTz&"<==   EL)GHHOOPWXXej      c^ ||dddddf}||dddddf}||dddddf}||dddddf}n"#t$r}td|d}~wwxYw||dddddf|dddddfz }||dddddf|dddddfz }t j||||||gd} | S)Nrrrz;The `bbox` coordinate values should be within 0-1000 range.r&dim)r4r5 IndexErrorr7r8rAcat) rEbboxleft_position_embeddingsupper_position_embeddingsright_position_embeddingslower_position_embeddingser7r8spatial_position_embeddingss rH!_calc_spatial_position_embeddingsz6LayoutLMv2Embeddings._calc_spatial_position_embeddingsJs c'+'A'A$qqq!!!Qw-'P'P $(,(B(B4111a=(Q(Q %(,(B(B4111a=(Q(Q %(,(B(B4111a=(Q(Q % % c c cZ[[ab b c!% : :4111a=4PQPQPQSTSTSTVWPW=;X Y Y $ : :4111a=4PQPQPQSTSTSTVWPW=;X Y Y&+i()))%%   ' ' ' #+*sBB B*B%%B*)__name__ __module__ __qualname____doc__r*rW __classcell__rGs@rHrr5sMQQ     $+++++++rIrcDeZdZfdZdZ ddeefdZxZS)LayoutLMv2SelfAttentionct|j|jzdkr0t |ds t d|jd|jd|j|_|j|_t|j|jz |_|j|jz|_ |j |_ |j |_ |jrtj |jd|j zd|_tjt!jd d |j |_tjt!jd d |j |_nltj |j|j |_tj |j|j |_tj |j|j |_tj|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rFbiasr)r)r*r-num_attention_headshasattr ValueErrorfast_qkvintattention_head_size all_head_sizehas_relative_attention_biashas_spatial_attention_biasrLinear qkv_linear ParameterrAzerosq_biasv_biasquerykeyvaluer=attention_probs_dropout_probr?rDs rHr*z LayoutLMv2SelfAttention.__init__es    : :a ? ?PVXhHiHi ?8F$688 4888  #)#= #&v'9FS"S* *,CD o--rINNN) rXrYrZr*rrrrr\r]s@rHr_r_dsGGGGG4( $.$. +, $.$.$.$.$.$.$.$.rIr_c>eZdZfdZ ddeefdZxZS)LayoutLMv2Attentionctt||_t ||_dSN)r)r*r_rELayoutLMv2SelfOutputoutputrDs rHr*zLayoutLMv2Attention.__init__s; +F33 *622 rINrc `|}|j||f||d|\}}|||}|SNrr)rEr) rEr}rrrrresidualattention_output_s rHrzLayoutLMv2Attention.forwardsa!'di   !      ! ;;'7BBrIr)rXrYrZr*rrrr\r]s@rHrrsl33333   +,         rIrc$eZdZfdZdZxZS)rcttj|j|j|_tj|j|j|_tj|j |_ dSNr") r)r*rrnr-denser;r<r=r>r?rDs rHr*zLayoutLMv2SelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== rIc||}||}|||z}|Srrr?r;rEr} input_tensors rHrzLayoutLMv2SelfOutput.forward@ =11  ]33 }|'CDD rIrXrYrZr*rr\r]s@rHrrsG>>>>> rIrcBeZdZfdZdejdejfdZxZS)LayoutLMv2Intermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r)r*rrnr-intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrDs rHr*zLayoutLMv2Intermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rIr}returncZ||}||}|Sr)rr)rEr}s rHrzLayoutLMv2Intermediate.forwards, =11 00?? rIrXrYrZr*rATensorrr\r]s@rHrrs^99999U\elrIrcPeZdZfdZdejdejdejfdZxZS)LayoutLMv2Outputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r)r*rrnrr-rr;r<r=r>r?rDs rHr*zLayoutLMv2Output.__init__sf Yv79KLL f&8f>STTTz&"<== rIr}rrc||}||}|||z}|Srrrs rHrzLayoutLMv2Output.forwardrrIrr]s@rHrrsi>>>>> U\RWR^rIrcFeZdZfdZ ddeefdZdZxZS)LayoutLMv2Layerct|j|_d|_t ||_t ||_t||_ dS)Nr) r)r*chunk_size_feed_forward seq_len_dimr attentionr intermediaterrrDs rHr*zLayoutLMv2Layer.__init__s^ '-'E$,V4426::&v.. rINFrc z|||||}t|j|j|j|}|Sr)rrfeed_forward_chunkrr) rEr}routput_attentionsrrrr layer_outputs rHrzLayoutLMv2Layer.forwardsU >>  ! *  1  #T%A4CSUe  rIc\||}|||}|Sr)rr)rErintermediate_outputrs rHrz"LayoutLMv2Layer.feed_forward_chunks2"//0@AA{{#68HII rI)NFNN) rXrYrZr*rrrrr\r]s@rHrrs~///// +,,rIrT c:d}|r8|dz}||dk|zz }tj|}n(tj| tj|}|dz}||k}|tj||z tj||z z ||z ztjz}tj |tj ||dz }|tj |||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on. Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rrKr) longrAabsmax zeros_likelogrrrr full_likewhere) relative_position bidirectional num_buckets max_distanceretn max_exactis_small val_if_larges rHrelative_position_bucketr#s* CO  !A%++-- ;; I' ( ( I((%*:;L*M*M N Nq I9}H !''))i'((48L94L+M+MMQ\_hQhibnnL9\5?<WX+Y+YZZL5;xL 1 11C JrIcJeZdZfdZdZdZ ddeefdZxZ S)LayoutLMv2Encodercdt|_tjfdt jD|_j|_j |_ |jr>j |_ j |_ tj |j j d|_|j rdj|_j|_tj |jj d|_tj |jj d|_d|_dS)Nc.g|]}tS)r).0rrFs rH z.LayoutLMv2Encoder.__init__..Ss!#e#e#eOF$;$;#e#e#erIFrc)r)r*rFr ModuleListrangenum_hidden_layerslayerrlrm rel_pos_bins max_rel_posrnre rel_pos_biasmax_rel_2d_posrel_2d_pos_binsrel_pos_x_biasrel_pos_y_biasgradient_checkpointingrDs `rHr*zLayoutLMv2Encoder.__init__Ps  ]#e#e#e#eU6KcEdEd#e#e#eff +1+M(*0*K'  + e & 3D %1D  " $*;V=W^c d d dD   * j"("7D #)#9D "$)D,@&B\ch"i"i"iD "$)D,@&B\ch"i"i"iD &+###rIc|d|dz }t||j|j}t j5|jj| dddd}dddn #1swxYwY| }|S)Nrr&rrrrrrK) unsqueezerrrrAno_gradrweighttrr)rEr$ rel_pos_matrs rH!_calculate_1d_position_embeddingsz3LayoutLMv2Encoder._calculate_1d_position_embeddingses",,R00<3I3I"3M3MM * ))   ]__ P P'.00227;CCAq!QOOG P P P P P P P P P P P P P P P$$&&s;B""B&)B&c|dddddf}|dddddf}|d|dz }|d|dz }t||j|j}t||j|j}t j5|jj| dddd}|j j| dddd}dddn #1swxYwY| }| }||z}|S)Nrrrr&rrrK) rrrrrArrrrrrr) rErPposition_coord_xposition_coord_yrel_pos_x_2d_matrel_pos_y_2d_mat rel_pos_x rel_pos_yrs rH!_calculate_2d_position_embeddingsz3LayoutLMv2Encoder._calculate_2d_position_embeddingsus111a=111a=+55b99>v}UUUM&*+=>> q..,  $' /   %x11&-@@ "X// << %+%8 "$*$6 !,1L%*U[UhUo,p,p,p ),,..XX e  '@ 'V'VWWWW rIc*eZdZfdZdZdZxZS)r"c t||_|jjj}t j||j}t|j tj j j sJ|j |_ t|jjjt|jjjksJt|jjj}|dt#j|jjj|ddd|dt#j|jjj|dddd|_t#jrt,dd}|j |jj}t5jt9jt9j|d |z |jd z t9jt9j|d|z |jdz f|_n&t5j |jdd |_t|jd krA|j!|j |jj"|j |jj"|jd ksJdS) Nr'rFr'r(p2z0using `AvgPool2d` instead of `AdaptiveAvgPool2d`)rPrrK)#r)r*get_detectron2_configr$r%META_ARCHITECTURErgetrbackboner-modelingFPNr#r&r)r@rArr|out_feature_key$are_deterministic_algorithms_enabledloggerwarning output_shapestrider AvgPool2drceilimage_feature_pool_shapepoolAdaptiveAvgPool2dappendchannels)rErF meta_archmodelr5 input_shapebackbone_striderGs rHr*z!LayoutLMv2VisualBackbone.__init__s //11HN4 1"&y11$(;;%.**=*F*JKKKKK 48>,--TX^5M1N1NNNNN48>455    L2 3 3 8 8q! L L     dhn&>??DD\STVWXXej     $  5 7 7 R NNM N N N$K"m88::4;OPWO Idi A(HIIFLklmLnnooIdi A(HIIFLklmLnnooDII,V-LRaR-PQQDI v. / /1 4 4  + 2 24=3M3M3O3OPTPd3e3n o o o}))++D,@AJfNmnoNppppppprIcBtj|r|n|j|jz |jz }||}||j}||d dd }|S)NrK) start_dimr) rA is_tensorrEr'r(rTrWr`flattenrr)rEimages images_inputfeaturess rHrz LayoutLMv2VisualBackbone.forwards#(?6#:#:M QUQ``dhdrr ==..D0199X&&...;;EEaKKVVXXrIctjr@tjr"tjdkst dtj}tjtj}|zdkst dfdt|zDfdt|zD}|z}t|j |||_ dS)Nr&z/Make sure torch.distributed is set up properly.rzGMake sure the number of processes can be divided by the number of nodesc \g|](}tt|z|dzz)Sry)listr)ri node_sizes rHrzCLayoutLMv2VisualBackbone.synchronize_batch_norm..s9uuuQRT%I A7J"K"KLLuuurIc\g|](}tj|)S))ranks)rA distributed new_group)rrrnode_global_rankss rHrzCLayoutLMv2VisualBackbone.synchronize_batch_norm..s@   HIE  ' '.?.B ' C C   rI)r=) rArv is_availableis_initializedget_rank RuntimeErrorcuda device_countget_world_sizerrIrT)rE self_rank world_sizesync_bn_groups node_rankrxrss @@rHsynchronize_batch_normz/LayoutLMv2VisualBackbone.synchronize_batch_normsF   * * , , R!0022 R!**,,r11PQQ Q%..00 J++-- &5577 Y&!++hii iuuuuV[\fjs\sVtVtuuu    MRS]ajSjMkMk   * 1$-~^gOhiii rI)rXrYrZr*rrr\r]s@rHr"r"sc!q!q!q!q!qFjjjjjjjrIr"c$eZdZfdZdZxZS)LayoutLMv2Poolercttj|j|j|_tj|_dSr)r)r*rrnr-rTanh activationrDs rHr*zLayoutLMv2Pooler.__init__ sC Yv163EFF '))rIcr|dddf}||}||}|S)Nr)rr)rEr}first_token_tensor pooled_outputs rHrzLayoutLMv2Pooler.forward%s@+111a40 #566  66 rIrr]s@rHrrsG$$$$$ rIrc>eZdZeedZfdZdZdZddZ dZ dZ dd Z e ee dd ejdzd ejdzd ejdzd ejdzdejdzdejdzdejdzdeedeezfdZxZS)r*)r} attentionsct|dt|||_|j|_t ||_t||_tj |j d|j |_ |jrvisual_dropoutrencoderrpooler post_initrDs rHr*zLayoutLMv2Model.__init__2s$ ---     ,2,O).v66.v66 9V%DR%H&J\]]  , h,.LaI[9\9\9cde9f,g,gD ) " V-?VEZ [ [ [ j)CDD(00 &v..  rIc|jjSrrr/rEs rHget_input_embeddingsz$LayoutLMv2Model.get_input_embeddingsFs ..rIc||j_dSrr)rErvs rHset_input_embeddingsz$LayoutLMv2Model.set_input_embeddingsIs*/'''rINc||}n|dd}|d}|Ntj|tj|j}|d|}|tj|}||j |}|j |}|j |} |j |} ||z| z| z} |j | } |j| } | S)Nr&rr>r)rrArBrr?r expand_asrrr/r1rWr:r;r?) rE input_idsrPr$token_type_ids inputs_embedsrf seq_lengthr1rVr:rs rH_calc_text_embeddingsz%LayoutLMv2Model._calc_text_embeddingsLs9  #..**KK',,..ss3K ^   < %*YM]^^^L'11!44>>yIIL  !"-i88N   O;;IFFM"oAA,OO&*o&W&WX\&]&]# $ E En U U"%88;VVYnn _..z:: _,,Z88 rIcD|||}|j|}|j|}||z|z}|jr ||jz }||}||}|Sr) rrrr1rWrrrr)rErrPr$visual_embeddingsr1rVrs rH_calc_img_embeddingsz$LayoutLMv2Model._calc_img_embeddingses ,,T[[-?-?@@"oAA,OO&*o&W&WX\&]&]#&)<> rIc||td||S||ddStd)NDYou cannot specify both input_ids and inputs_embeds at the same timer&5You have to specify either input_ids or inputs_embeds)rgr)rErrs rH_get_input_shapez LayoutLMv2Model._get_input_shapesb  ]%>cdd d  ">>## #  & %%'', ,TUU UrIrrPrrrr$rrrc T|||} ||jn|j} t| } |jjd|jjdz| d<t j| } t|||} | dxx| dz cc<t j| } ||jj|| | } t j|| gd}|t j | | }t j | |j | }t j||gd}|!t j | t j | }|5| d}|j jddd|f}|| }t jd| dt j | | dd}t j||gd}|?t j t%t| dgzt j | }||||||}||| | }t j||gd}|dd }||j }d |z t j|j jz}|j||f||d |}|j}||}t9||S)a bbox (`torch.LongTensor` of shape `((batch_size, sequence_length), 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. image (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)` or `detectron.structures.ImageList` whose `tensors` is of shape `(batch_size, num_channels, height, width)`): Batch of document images. Examples: ```python >>> from transformers import AutoProcessor, LayoutLMv2Model, set_seed >>> from PIL import Image >>> import torch >>> from datasets import load_dataset >>> set_seed(0) >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased") >>> model = LayoutLMv2Model.from_pretrained("microsoft/layoutlmv2-base-uncased") >>> dataset = load_dataset("hf-internal-testing/fixtures_docvqa") >>> image = dataset["test"][0]["image"] >>> encoding = processor(image, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> last_hidden_states.shape torch.Size([1, 342, 768]) ``` NrrrL)r?r>)rrPrr$rrrPr$rK)rr)rPr$)r pooler_output)rr?rqrFr_rASizerrOonesrrqrrr$rCrBrtuplerrrrrrrrrr )rErrPrrrr$rrrfr? visual_shaperr final_bboxvisual_attention_maskfinal_attention_maskrvisual_position_idsfinal_position_idstext_layout_emb visual_emb final_embextended_attention_maskencoder_outputssequence_outputrs rHrzLayoutLMv2Model.forwardsYb++I}EE %.%:!! @TK(( +>qADKDhijDkk Qz,// 400MJJKK A,q/)j-- ,,T[-QSWY_almm Yk2:::  !"Z FCCCN % <~?S\b c c c$y.:O)PVWXXX  !"[EJvVVVN  $QJ?7;J;GL'..{;;L#l1l1oUZX^___ff NA  #Y 6I'JPQRRR <;uT+%6%6!%<==UZX^___D44)%' 5  ..,/  I ;CCC "6"@"@"C"C"M"Ma"P"P"9"<"<4:"<"N"N#&)@#@EKPTPZD[D[D_"_&$,  # +      *; O44 )-'    rIr)NN)NNNNNNN)rXrYrZrr__can_record_outputsr*rrrrrrrrrrA LongTensor FloatTensorrrrr rr\r]s@rHr*r*.s,;Kbcc(///0002   ###JVVVV .2(,*.37260426s s #d*s %s  4' s )D0 s (4/ s &-s (4/s +,s  + +s s s ^_ s s s s s rIr*ax LayoutLMv2 Model with a sequence classification head on top (a linear layer on top of the concatenation of the final hidden state of the [CLS] token, average-pooled initial visual embeddings and average-pooled final visual embeddings, e.g. for document image classification tasks such as the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset. ) custom_introceZdZfdZdZee ddejdzdejdzdej dzdej dzdejdzd ejdzd ej dzd ejdzd e e d e e zfdZxZS)#LayoutLMv2ForSequenceClassificationc<t||j|_t||_t j|j|_t j |j dz|j|_ | dS)Nr r)r* num_labelsr*rrr=r>r?rnr- classifierrrDs rHr*z,LayoutLMv2ForSequenceClassification.__init__!s~     +)&11z&"<== )F$6$:F 1` a classification loss is computed (Cross-Entropy). Example: ```python >>> from transformers import AutoProcessor, LayoutLMv2ForSequenceClassification, set_seed >>> from PIL import Image >>> import torch >>> from datasets import load_dataset >>> set_seed(0) >>> dataset = load_dataset("aharley/rvl_cdip", split="train", streaming=True) >>> data = next(iter(dataset)) >>> image = data["image"].convert("RGB") >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased") >>> model = LayoutLMv2ForSequenceClassification.from_pretrained( ... "microsoft/layoutlmv2-base-uncased", num_labels=dataset.info.features["label"].num_classes ... ) >>> encoding = processor(image, return_tensors="pt") >>> sequence_label = torch.tensor([data["label"]]) >>> outputs = model(**encoding, labels=sequence_label) >>> loss, logits = outputs.loss, outputs.logits >>> predicted_idx = logits.argmax(dim=-1).item() >>> predicted_answer = dataset.info.features["label"].names[4] >>> predicted_idx, predicted_answer # results are not good without further fine-tuning (7, 'advertisement') ``` Nrr&rrrr>rrrPrrrr$rrL regressionsingle_label_classificationmulti_label_classificationlosslogitsr}rr) rg%warn_if_padding_and_no_attention_maskrr?rqrFr_rArrrrBrrrrrrOr?r problem_typerrrirsqueezerr|rrr}r)rErrPrrrr$rrrrfr?rrrrinitial_image_embeddingsoutputsrrfinal_image_embeddingscls_final_outputpooled_initial_image_embeddingspooled_final_image_embeddingsrrloss_fcts rHrz+LayoutLMv2ForSequenceClassification.forward.sE^  ]%>cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @TK(( +>qADKDhijDkk Qz,// ;'' A,q/)j-- o77 K 0$   $l1l1oUZX^___ff NA  $(?#G#G,$H$ $  />do / ))%' / /  / /   #..**KK',,..ss3K ^  %aaa*n 5  %aaan 50 +111a73+C*G*GA*G*N*N'(>(C(C(C(J(J%) >@] ^de   ,,7711  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     rINNNNNNNN)rXrYrZr*rrrrArrrrrrrr\r]s@rHrrsR:::.2(,*.37260426*.k k #d*k %k  4' k )D0 k (4/ k &-k (4/k  4'k +,k  ) )k k k ^k k k k k rIra LayoutLMv2 Model with a token classification head on top (a linear layer on top of the text part of the hidden states) e.g. for sequence labeling (information extraction) tasks such as [FUNSD](https://guillaumejaume.github.io/FUNSD/), [SROIE](https://rrc.cvc.uab.es/?ch=13), [CORD](https://github.com/clovaai/cord) and [Kleister-NDA](https://github.com/applicaai/kleister-nda). ceZdZfdZdZee ddejdzdejdzdej dzdej dzdejdzd ejdzd ej dzd ejdzd e e d e e zfdZxZS) LayoutLMv2ForTokenClassificationc6t||j|_t||_t j|j|_t j |j |j|_ | dSrrrDs rHr*z)LayoutLMv2ForTokenClassification.__init__sy     +)&11z&"<== )F$68IJJ rIc$|jjjSrrrs rHrz5LayoutLMv2ForTokenClassification.get_input_embeddingsrrINrrPrrrr$rrrrc |jd|||||||d| } ||} n|dd} | d} | jddd| f} || } || }d}|Ft }||d|j|d}t||| j | j S)af input_ids (`torch.LongTensor` of shape `batch_size, sequence_length`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. image (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)` or `detectron.structures.ImageList` whose `tensors` is of shape `(batch_size, num_channels, height, width)`): Batch of document images. token_type_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Example: ```python >>> from transformers import AutoProcessor, LayoutLMv2ForTokenClassification, set_seed >>> from PIL import Image >>> from datasets import load_dataset >>> set_seed(0) >>> datasets = load_dataset("nielsr/funsd", split="test") >>> labels = datasets.features["ner_tags"].feature.names >>> id2label = {v: k for v, k in enumerate(labels)} >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr") >>> model = LayoutLMv2ForTokenClassification.from_pretrained( ... "microsoft/layoutlmv2-base-uncased", num_labels=len(labels) ... ) >>> data = datasets[0] >>> image = Image.open(data["image_path"]).convert("RGB") >>> words = data["words"] >>> boxes = data["bboxes"] # make sure to normalize your bounding boxes >>> word_labels = data["ner_tags"] >>> encoding = processor( ... image, ... words, ... boxes=boxes, ... word_labels=word_labels, ... padding="max_length", ... truncation=True, ... return_tensors="pt", ... ) >>> outputs = model(**encoding) >>> logits, loss = outputs.logits, outputs.loss >>> predicted_token_class_ids = logits.argmax(-1) >>> predicted_tokens_classes = [id2label[t.item()] for t in predicted_token_class_ids[0]] >>> predicted_tokens_classes[:5] # results are not good without further fine-tuning ['I-HEADER', 'I-HEADER', 'I-QUESTION', 'I-HEADER', 'I-QUESTION'] ``` rNr&rrr) rrrr?rrr|rrr}r)rErrPrrrr$rrrrrfrrrrrs rHrz(LayoutLMv2ForTokenClassification.forwards"p/>do / ))%' / /  / /   #..**KK',,..ss3K ^ !3AAA{ {NC,,7711  '))H8FKKDO<>> from transformers import AutoProcessor, LayoutLMv2ForQuestionAnswering, set_seed >>> import torch >>> from PIL import Image >>> from datasets import load_dataset >>> set_seed(0) >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased") >>> model = LayoutLMv2ForQuestionAnswering.from_pretrained("microsoft/layoutlmv2-base-uncased") >>> dataset = load_dataset("hf-internal-testing/fixtures_docvqa") >>> image = dataset["test"][0]["image"] >>> question = "When is coffee break?" >>> encoding = processor(image, question, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_start_idx = outputs.start_logits.argmax(-1).item() >>> predicted_end_idx = outputs.end_logits.argmax(-1).item() >>> predicted_start_idx, predicted_end_idx (30, 191) >>> predicted_answer_tokens = encoding.input_ids.squeeze()[predicted_start_idx : predicted_end_idx + 1] >>> predicted_answer = processor.tokenizer.decode(predicted_answer_tokens) >>> predicted_answer # results are not good without further fine-tuning '44 a. m. to 12 : 25 p. m. 12 : 25 to 12 : 58 p. m. 12 : 58 to 4 : 00 p. m. 2 : 00 to 5 : 00 p. m. coffee break coffee will be served for men and women in the lobby adjacent to exhibit area. please move into exhibit area. ( exhibits open ) trrf general session ( part | ) presiding : lee a. waller trrf vice president " introductory remarks " lee a. waller, trrf vice presi - dent individual interviews with trrf public board members and sci - entific advisory council mem - bers conducted by trrf treasurer philip g. kuehn to get answers which the public refrigerated warehousing industry is looking for. plus questions from' ``` ```python >>> target_start_index = torch.tensor([7]) >>> target_end_index = torch.tensor([14]) >>> outputs = model(**encoding, start_positions=target_start_index, end_positions=target_end_index) >>> predicted_answer_span_start = outputs.start_logits.argmax(-1).item() >>> predicted_answer_span_end = outputs.end_logits.argmax(-1).item() >>> predicted_answer_span_start, predicted_answer_span_end (30, 191) ``` rNr&rrLr) ignore_indexrK)r start_logits end_logitsr}rr) rrrrsplitrrr#clamprr r}r)rErrPrrrr$rrrrrrfrrrrr total_loss ignored_indexr start_lossend_losss rHrz&LayoutLMv2ForQuestionAnswering.forwardsn/>do / ))%' / /  / /   #..**KK',,..ss3K ^ !3AAA{ {NC11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM']CCCH!,@@Jx M::H$x/14J+%!!/)     rI)T) NNNNNNNNN)rXrYrZr*rrrrArrrrrr rr\r]s@rHrrnsm      :::.2(,*.372604263715E E #d*E %E  4' E )D0 E (4/ E &-E (4/E )D0E '$.E +,E  - -E E E ^E E E E E rIr)rrrrr*r)Trrr)Er[rrArtorch.nnrrrrr activationsr modeling_layersr modeling_outputsr r r rrmodeling_utilsrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrrrutils.output_capturingrconfiguration_layoutlmv2rr-detectron2.modelingrModule_load_from_state_dictr. batch_normr/ get_loggerrXrYrr_rrrrrrrrrIr"rr*rrr__all__rrIrHrsS  AAAAAAAAAA&&&&&&!!!!!!999999.-----&&&&&&666666XXXXXXXXXXXX]]]]]]]]]]555555666666q666666LQ8?KpJ 2H  H % %,+,+,+,+,+29,+,+,+^P.P.P.P.P.biP.P.P.f     ")   4     29        RY        ry   """""0"""J))))XT@T@T@T@T@ T@T@T@nAAAAAAAA<0?j?j?j?j?jry?j?j?jD     ry   f f f f f /f f f R{ { { { { *C{ { { |E E E E E '@E E E PY Y Y Y Y %>Y Y Y x   rI