tj XdZddlZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZejeZGddej Z!Gddej Z"Gddej#Z$Gddej#Z%GddeZ&eGddeZ'Gdde'Z(edGd d!e'Z)ed"Gd#d$e'e Z*d$dgZ+dS)%z/PyTorch TrOCR decoder model (based on RoBERTa).N)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging) TrOCRConfigcbeZdZdZdedeffd Z d dejded ejdzffd ZxZ S) TrOCRLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcjd|_t||jz|dSN)offsetsuper__init__)selfrr __class__s f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/trocr/modeling_trocr.pyrz(TrOCRLearnedPositionalEmbedding.__init__*s3  $+5}EEEEErN input_idspast_key_values_length position_idsc0|V|jdd\}}tj|||ztj|jj|d}n|d}t ||j zS)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr(expand unsqueezerforwardr)rr#r$r%bszseq_lenr s r!r1z'TrOCRLearnedPositionalEmbedding.forward0s  $?2A2.LC <&(>(HPUPZcgcncufS"oo L(11!44Lww|dk9:::r")rN) __name__ __module__ __qualname____doc__intrr+Tensorr1 __classcell__r s@r!rr%sFsF3FFFFFFmq ; ; ;?B ;V[VbeiVi ; ; ; ; ; ; ; ; ; ;r"rc VeZdZdZd dededededzffd Zd ejffd Z xZ S) TrOCRScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?rr padding_idx embed_scaleNc\t|||||_dSN)rrr@)rrrr?r@r s r!rz!TrOCRScaledWordEmbedding.__init__Fs-  DDD&r"r#cVt||jzSrB)rr1r@)rr#r s r!r1z TrOCRScaledWordEmbedding.forwardJs!wwy))D,<<) r4r5r6r7r8floatrr+r9r1r:r;s@r!r=r=As''s'3'S'_dgk_k''''''===========r"r=c eZdZdZddedededzffd ZeddedededzfdZej dd ej d efd Z dd ej ded edzfd Z xZ S)"TrOCRSinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length.N num_positionsrr?ctd|_||_||_|||||_dSr)rrrrr? get_embeddingweights)rrGrr?r s r!rz+TrOCRSinusoidalPositionalEmbedding.__init__QsM  *&))- TT r"rc|dz}tjd|dz z }tjtj|tj| z}tj|tjd|dz}tjtj |tj |gd |d}|dzdkr+tj|tj |dgd}| d||ddf<| tjS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'r)r'rdimr)N)mathlogr+expr,int64rDr0catsincosviewzerostoget_default_dtype)rrr?half_dimembs r!rIz0TrOCRSinusoidalPositionalEmbedding.get_embeddingXs? !A%huooA.i XU[AAAGGIISDPQQl>===CCEEOOPQRRUXUbUbcdUeUeei338a@@@EEnVXYY 1  ! !)S%+na"@"@AqIIIC  ""#C QQQ vve-//000r"rr#r$c|\}}|||j||j}|jdz|z}|j||jdkr&|||j|j|_|jd| d ||d }|S)Nrrr)) size"create_position_ids_from_input_idsr?rWr(rJrIr index_selectrUdetach)rr#r$r2r3r%max_posxs r!r1z*TrOCRSinusoidalPositionalEmbedding.forwardks ~~'' W>>y$JZ\rssvv    "Q&0 < 7T\->->q-A-A#A#A--gt7I4K[\\DL L % %a):):2)>)> ? ? D DS'SU V V ] ] _ _r"c||}tj|d||z|z}||zS)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrL)ner8r+cumsumtype_asr-)rr#r?r$maskincremental_indicess r!r]zETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_ids}sg||K((,,..$|Da888@@FFI__cgg"''))K77r"rB)r)r4r5r6r7r8r staticmethodrIr+no_gradr9r1r]r:r;s@r!rFrFNs(NNUUcU#UCRVJUUUUUU11c1#1CRVJ111\1$U]__s_$_` 8 8 847 8QTW[Q[ 8 8 8 8 8 8 8 8r"rFc*eZdZdZ ddedededzd edzd edzd edzd edzd edzdedzffd Z ddej dej dzde dzdej dzdedzde ej ej dze ej dzff dZ xZ S)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper.NFT embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc <t||_||n||_||n||_||_||_||z|_|j|z|jkstd|jd|d|jdz|_ ||_ | |_ tj |j|||_tj |j|||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rs)rrrmrorprnrqhead_dim ValueErrorscalingrrrurLineark_projv_projq_projout_proj) rconfigrmrnrorprqrrrsrtrur s r!rzTrOCRAttention.__init__s- " ,DD)  ,DD) " !Y.  )T^;;"dn""""" }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r" hidden_stateskey_value_statespast_key_valuesattention_maskoutput_attentionsreturnc  |du}|\}} } |||jz} d} |Ht|tr1|j|j} |r|j} n |j } n|} |r|n|}|r3|1| r/| j |jj }| j |jj }n| |}||}||d|j|jdd}||d|j|jdd}|E| |||j\}}|r$t|trd|j|j<||jzd|jf}| || |j|jdd} | j|} |j|}|j|}|d}t+j| |dd}|||jz| |fkr2t/d||jz| |fd||||d| |fkr+t/d |d| |fd||||j| ||z}|||jz| |}t0j|d }|r=|||j| |}|||jz| |}nd}t0j||j|j }t+j||}|||jz| |jfkr5t/d ||j| |jfd||||j| |j}|dd}||| | }||}||fS) z#Input shape: Batch x Time x ChannelNFr)rrTz$Attention weights should be of size z , but is z!Attention mask should be of size rLptrainingz `attn_output` should be of size )r\r~rz isinstancer is_updatedgetrucross_attention_cacheself_attention_cachelayerskeysvaluesr|r}rUrnrx transposeupdatereshaper+bmmryr functionalsoftmaxrqrr)rrrrrrkwargsrtr2tgt_lenrm query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r!r1zTrOCRAttention.forwards.T9"/"4"4"6"6Wi{{=11DL@   &/+>?? 7,7;;DNKK %P+:+P((+:+O(('6$-?R))]  F/"=*"=-4T^DIJ/6t~FMLL^44J;;~66L#b$.$-PPZZ[\^_``J',,S"dndmTT^^_`bcddL*+?+F+FzS_aeao+p+p( L%F*_FY*Z*ZFAEO.t~>DN*B > #((gt~t}UU__`acdee +|+Z8 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4?"(";$&L$@$@!   H .. 8770#'# ! ! !D ,.<+G+GD (9T^V-CDD9V3T^DD " T^ < <r"FTrrencoder_hidden_statesencoder_attention_maskrr use_cachec |} |||||\}} tj||j|j}| |z}||}d} |e|} ||||||\}} tj||j|j}| |z}||}|} || |}tj||j |j}| |}tj||j|j}| |z}| |}|f} |r| | | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrrrN)rrrrr) rrrrqrrrrrrrrr) rrrrrrrrrresidualself_attn_weightscross_attn_weightsoutputss r!r1zTrOCRDecoderLayer.forward<s2!,0>>'+)/ ,:, , ( ( --mt|VZVc-dd  =0 11-@@ " ,$H040A0A+!65 /"3 1B11 -M-M11-4 >Gr"rB)NNNNFT) r4r5r6rrr+r9rrr1r:r;s@r!rrs"="={"="="="="="=N/3596:(,).!%GG|G t+G %|d2 G !& t 3 G  G $;G$;GGGGGGGGr"rc(eZdZUeed<dZdZdgZdS)TrOCRPreTrainedModelrmodelTrN)r4r5r6r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr"r!rrs3 &*#,-r"rcDeZdZdZdeffd Z ddZxZS) TrOCRDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rctj|_j|_j|_jrtj j nd}tj j |j||_ jr tjj |_n0t%j|jzdzj |j|_jrt)jj |_nd|_t)jfdt/jD|_d|_|dS)Nr>)r@rc2g|]}t|S))ru)r).0irs r! z)TrOCRDecoder.__init__..s($r$r$rPQ%6v%K%K%K$r$r$rr"F)rrrqdecoder_layerdrop layerdrop pad_token_idr?scale_embeddingrNsqrtrr= vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsrFlayernorm_embeddingrr ModuleListrangedecoder_layersrgradient_checkpointing post_init)rrr@r s ` r!rzTrOCRDecoder.__init__s\    ~ 1!.7=7MVdi 2333SV 4  v143CQ\     1 #B6Cacicu#v#vD #E.1AAAE" $$D  % ,')|F4F'G'GD $ $'+D $m$r$r$r$rUZ[a[pUqUq$r$r$rss &+# r"Nc ||n |jj}| | n |jj} ||n |jj}| | n |jj} ||t d|$|} |d| jd}n!||dddddf} nt d|jr%|j r|rt dd}|r[|Y| |jj r6tt|jt|jnt|j}||nd} |||}|jjr|| | }n||| }||j}||z}|j||}t,j||j|j }t3|j||| }||t5|j||| }| rd nd}|rd nd}|r|d nd}t7|jD]j\}}| r||fz }|j r t;jg}||jkr4||||||||}|d}|r||dfz }| ||dfz }k| r||fz }| stAd|||||fDStC|||||S)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer)zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...F)rr)r$r)r inputs_embedsrr)rrrrr)rrrrrrc3K|]}||V dSrBr)rvs r! z'TrOCRDecoder.forward..[s0= ===r")last_hidden_staterr attentionscross_attentions)"rroutput_hidden_statesr return_dictryrUr*rrlogger warning_onceis_encoder_decoderr rget_seq_lengthrrrrWr(rrrrqr r enumeraterr+randrrr)rr#rrrrrrrrrrinputr$ embed_posrall_hidden_statesall_self_attnsall_cross_attentionsidx decoder_layerdropout_probability layer_outputss r!r1zTrOCRDecoder.forwards@2C1N--TXT_Tq$8$D $+Jj "+!6IIDKstt t  "E!r5;r?;;II  &!!!!QQQ(+EEdee e  & "4= " "##t"  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  FUE`!?!?!A!A!Afg   --i88M ; 6 g,,UKa,bbII,,YOe,ffILL!566 % 1  # / 44]CCM --mt|VZVc-dd +;')+     ! ,1G1S%>{+5&; &&& "#7@BBD0:d&7h>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r#rrrrrrrrrrr)r)losslogitsrrrr)rrrrrrrrrUrrrrrr)rr#rrrrrrrrrrrrrrloss_fctoutputs r!r1zTrOCRForCausalLM.forwardsNP2C1N--TXT_Tq$8$D $+Jj &1%:>(,26*.!%)-,0#'m m #d*m  t+m  %047 m !& 04 7 m  m (4/m  4'm $;m  $;m #Tkm D[m  2 2m m m ^m m m m m r"r),r7rNr+rtorch.nnr activationsr cache_utilsrrr generationr masking_utilsr r modeling_layersr modeling_outputsrrmodeling_utilsrutilsrrconfiguration_trocrr get_loggerr4r Embeddingrr=ModulerFrkrrrrr__all__rr"r!r+s+65 %%%%%%!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999llllllll------,,,,,,,,,,,,,,  H % %;;;;;bl;;;8 = = = = =r| = = =9898989898989898xJ2J2J2J2J2RYJ2J2J2Zlllll2lll^.....?...X X X X X 'X X X v-----.--- H H H H H +_H H  H V 5 6r"