tjsdZddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZddlm Z ej!e"Z#edeGddeZ$Gddej%Z&Gddej%Z'Gddej%Z(Gddej%Z)Gddej%Z*Gddej%Z+Gd d!ej%Z,Gd"d#ej%Z-Gd$d%eZ.Gd&d'ej%Z/eGd(d)eZ0eGd*d+e0Z1Gd,d-ej%Z2ed.Gd/d0e0Z3Gd1d2ej%Z4Gd3d4ej%Z5ed5Gd6d7e0Z6ed8Gd9d:e0Z7ed;Gd<d=e0Z8eGd>d?e0Z9gd@Z:dS)AzPyTorch ViLT model.N) dataclass)nn)CrossEntropyLoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) ViltConfigzF Class for outputs of [`ViltForImagesAndTextClassification`]. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ e e ejdzed<dZ e e ejdzed<dS)(ViltForImagesAndTextClassificationOutputa7 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`list[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. Nlosslogits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrlisttuplerd/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/vilt/modeling_vilt.pyrr,s  &*D% d ")))'+FE  $+++;?M4e/01D8???8z/ViltEmbeddings.visual_embed..is   Aq !!M--#V'&* .  1fqj1   r'ij)indexingdeviceF)as_tuplec<g|]}dddf|kSNrr&)rPu valid_idxs r(rVz/ViltEmbeddings.visual_embed..s/NNNQ9QQQT?a#78NNNr'c<g|]}dddf|kSr_r&)rPr` non_valid_idxs r(rVz/ViltEmbeddings.visual_embed..s0ZZZ]=A+>!+CDZZZr'c8g|]}|dSrrErPvs r(rVz/ViltEmbeddings.visual_embed..s"777AaffQii777r'c8g|]}|dSrerErfs r(rVz/ViltEmbeddings.visual_embed..s"???!&&))???r'cg|]}|z Sr&r&)rPrgmax_image_lengths r(rVz/ViltEmbeddings.visual_embed..s===Q$q(===r'T) replacement)&r6 projectionweightshapefloatrrMrOlongsumr? image_size patch_sizer8 transposeviewr!catzipflattenstackmeshgridarangetor\expand isinstanceintmaxminnonzerounique enumerate multinomialonesappendr4r>)$r@ pixel_values pixel_maskrj_phpwxx_maskx_hx_w batch_size num_channels patch_dim pos_embed patch_indexeffective_resolution unique_rows valid_row_idxnon_valid_row_idx valid_numsnon_valid_numspad_numsselectirgnvp valid_choice pad_choice cls_tokensrSrcrTrarUs$ ` @@@@@r( visual_embedzViltEmbeddings.visual_embed\s,7>D 1b"  ! !, / /AAAtQQQM*0022**6QWQZ8P*QQVVXXQQQTl1%%aaad+QQQTl1%%aaad+23'/ L&%K*dk.DD .qqq!""aaax8BB1aHHMMaQ]_hjstt I       SMM      %%a((221a88 IIaLL " "1a ( (k N5< R(8995< UWHX;Y;Ydh i i ioq   "FM" " " "$aaaAAA"56 !((a&,q/2rSUVV !))!Q// "" a  #3#;:N^`cCdCd#; $'9 37799  #&9 "#7#;#;#=#=?OPP NNEN22 V,,e,<< 1o,,.. NNNN+NNN ZZZZkZZZ77777 ??->???====*===&s:~x'P'PQQ f fMAz2qAvv$0A1D1D1F1FHXYY  mA.|<====".uz"~~/C/C/E/EqVZ[[[  eiq)9;LQ;OPZ;[(\bcdddeeee6q))) fQQQTlF111a4L( ) . .z2| L Lqqq!t fQQQTl2388RHH!&A,qqq!t "<=BB:rSTUU fQQQTlF111a4L89>>z2|\\ ^**:r2>> Iz1o1 - - -I  %aaaAAAg .qqq$z : A A*bRT U UW` agh     M LLOOEJv|A::==fEEvNTUVVV&;888r'rc ||||} |'||||jj\}} } n|d} |d}| |t j|t j| j z} ||t j | |t j| j z}t j | |gd} t j || gd} | | fS)N) input_idstoken_type_ids inputs_embeds)rjrdtyper\rG) r0rr?rjrxr;r! zeros_likerpr\ full_likerv)r@rattention_maskrrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmaskss r(forwardzViltEmbeddings.forwards0**m+   595F5Fj4;;W6G66 2L+{{%,,Q//K '#$ !D$>$>  ^5:kFX Y Y Y% %  $d&@&@ OK)=UZXcXj k k k' '  Y \:BBB  >;7Q???5  r')rB)r)rrrr r.rr __classcell__rAs@r(r*r*Dst     V9V9V9V9B'!'!'!'!'!'!'!'!r'r*c*eZdZdZfdZddZxZS)r/zGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jdd|dt%j|jt$jddS) N) padding_idxeps position_idsrrXF) persistentrr)r-r.rr9 vocab_sizer3 pad_token_idword_embeddingsmax_position_embeddingsr8type_vocab_sizer; LayerNormlayer_norm_epsr<r=r>register_bufferr!r{r}r2rrFrpr@r?rAs r(r.zTextEmbeddings.__init__s3 !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg      r'Ncn||}n|dd}|d}||jddd|f}|mt|dr2|jddd|f}||d|}|}n+t j|t j|jj}|| |}| |} || z} | |} | | z } | | } | | } | S)NrXrrrr)rFrhasattrrr}r!r2rpr\rr;r8rr>) r@rrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr;rr8s r(rzTextEmbeddings.forwardsT  #..**KK',,..ss3K ^  ,QQQ  ^ J J"%:: "66|DD)) ^^J// \\*-- r')NNNNrrrr r.rrrs@r(r/r/sRQQ            r'r/c(eZdZdZfdZdZxZS)r5z# Image to Patch Embedding. ct|j|j}}|j|j}}t |tjj r|n||f}t |tjj r|n||f}|d|dz|d|dzz}||_||_||_||_ tj |||||_ dS)Nrr) kernel_sizestride)r-r.rrrsrr3r~ collectionsabcIterabler7rConv2drl)r@r?rrrsrr3r7rAs r(r.zViltPatchEmbeddings.__init__s !'!2F4EJ $*$79Kk #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq !!} 1 5*Q-:VW=:XY $$(&)L+:^hiiir'c|j\}}}}||jkrtd|jjj}|||}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rnr ValueErrorrlrmrr|)r@rrrrSrU target_dtypers r(rzViltPatchEmbeddings.forward(si2>2D/ L&% 4, , ,w -3 OOLOO,O?? @ @r'rrs@r(r5r5sV j j j j jr'r5c&eZdZfdZddZxZS)ViltSelfAttentionct|j|jzdkr0t |ds t d|jd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|j |_ tj |j|j|j |_ tj |j|j|j |_tj|j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)r-r.r3num_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer<attention_probs_dropout_probr>rs r(r.zViltSelfAttention.__init__4s.    : :a ? ?PVXhHiHi ?76#5773777  $*#= #&v'9Fpermute contiguousrFr)r@rroutput_attentionsr hidden_shape query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r(rzViltSelfAttention.forwardFs#)#2#. CCbC$*BCC jj//44\BBLLQPQRR HH]++00>>HHANN jj//44\BBLLQPQRR !< Y5H5HR5P5PQQ+di8P.Q.QQ  %/.@ -"*,,,-=>>,,77 _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S* *,CD 6G]=/22mM]r'NFrrrr.rrrs@r(rr3sQGGGGG$r'rcZeZdZdZdeffd ZdejdejdejfdZxZ S)ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r?cttj|j|j|_tj|j|_dSN) r-r.rrr3denser<r=r>rs r(r.zViltSelfOutput.__init__msJ Yv163EFF z&"<== r'r input_tensorreturncZ||}||}|Srrr>r@rrs r(rzViltSelfOutput.forwardrs* =11  ]33 r') rrrr rr.r!Tensorrrrs@r(rrgs >z>>>>>> U\RWR^r'rc&eZdZfdZddZxZS) ViltAttentionctt||_t ||_dSr)r-r.r attentionroutputrs r(r.zViltAttention.__init__ys; *622$V,, r'NFc||||}||d|}|f|ddz}|S)Nrr)r r )r@rrr self_outputsattention_outputrs r(rzViltAttention.forward~sK~~m^EVWW ;;|A FF#% QRR(88r'rrrs@r(r r xsL----- r'r cHeZdZdeffd ZdejdejfdZxZS)ViltIntermediater?cttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r-r.rrr3intermediate_sizerr~ hidden_actstrrintermediate_act_fnrs r(r.zViltIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r'rrcZ||}||}|Sr)rrr@rs r(rzViltIntermediate.forwards, =11 00?? r' rrrrr.r!rrrrs@r(rrsj9z999999U\elr'rcVeZdZdeffd ZdejdejdejfdZxZS) ViltOutputr?cttj|j|j|_tj|j|_ dSr) r-r.rrrr3rr<r=r>rs r(r.zViltOutput.__init__sJ Yv79KLL z&"<== r'rrrcd||}||}||z}|Srrrs r(rzViltOutput.forwards4 =11  ]33 % 4 r'rrs@r(rrsu>z>>>>>> U\RWR^r'rc*eZdZdZfdZddZxZS) ViltLayerz?This corresponds to the Block class in the timm implementation.czt|j|_d|_t ||_t ||_t||_ tj |j |j |_tj |j |j |_dS)Nrr)r-r.chunk_size_feed_forward seq_len_dimr r r intermediaterr rrr3rlayernorm_beforelayernorm_afterrs r(r.zViltLayer.__init__s '-'E$&v..,V44 (( " V-?VEZ [ [ [!|F,>FDYZZZr'NFcF|||||}|d}|dd}|||jz}||}||}|||}|f|z}|S)N)rrr)r r$r|r\r%r#r )r@rrrself_attention_outputsrr layer_outputs r(rzViltLayer.forwards!%  ! !- 0 0 /"0" "  2!4(,)=+;+;.s!#_#_#_!If$5$5#_#_#_r'F) r-r.r?r ModuleListrangenum_hidden_layerslayergradient_checkpointingrs `r(r.zViltEncoder.__init__s`  ]#_#_#_#_uVE]?^?^#_#_#_`` &+###r'NFTc|rdnd}|rdnd}t|jD]0\}} |r||fz}| |||} | d}|r || dfz}1|r||fz}|std|||fDSt|||S)Nr&rrc3K|]}||V dSrr&rfs r( z&ViltEncoder.forward..s(mmq_`_l_l_l_l_lmmr')last_hidden_staterr)rr0r%r ) r@rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_module layer_outputss r(rzViltEncoder.forwards#7@BBD$5?bb4(44 P POA|# I$58H$H!(LHYZZM)!,M  P&9]1=MeZdZUeed<dZdZdZddgZfdZ xZ S)ViltPreTrainedModelr?vilt)imagetextTr*rcHt|t|trjt j|jtj|jj d dt j |j dSdS)NrXr) r- _init_weightsr~r/initcopy_rr!r{rnr}zeros_r)r@modulerAs r(rBz!ViltPreTrainedModel._init_weightss f%%% fn - - / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h K- . . . . . / /r') rrrrr#base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modulesrBrrs@r(r=r=sd (&*#)+>?/////////r'r=c0eZdZdfd ZdZdZe ddejdzdej dzdejdzd ej dzd ejdzd ej dzd ej dzd e dzde dzde dzde dzde e ej zfdZxZS) ViltModelTcJt|||_t||_t ||_tj|j |j |_ |rt|nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r-r.r?r*rr*encoderrrr3r layernorm ViltPoolerpooler post_init)r@r?add_pooling_layerrAs r(r.zViltModel.__init__s     (00"6** f&8f>STTT,=Gj(((4  r'c$|jjjSrrr0rr@s r(get_input_embeddingszViltModel.get_input_embeddingss.>>r'c(||jj_dSrrU)r@rs r(set_input_embeddingszViltModel.set_input_embeddingss:?'777r'Nrrrrrrrrrr6r7rc | | n |jj} | | n |jj} | | n |jj} ||t d|+||||} n.||dd} nt d| \}}||jn|j}|tj ||f|}||t d||t d| |j dn |j d}||krt d |-tj ||jj |jj f|}| |||||||| \}}t|j|| }|||| | | }|d}||}|j||nd}| s||f|d dzSt#|||j|jS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerXz5You have to specify either input_ids or inputs_embedsr[zFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)r?rr)rrr6r7r)r5 pooler_outputrr)r?rr6r7r%warn_if_padding_and_no_attention_maskrFr\r!rrnrrrr rNrOrQr rr)r@rrrrrrrrrr6r7kwargsrtext_batch_sizerr\image_batch_sizeembedding_outputencoder_outputssequence_output pooled_outputs r(rzViltModel.forwardsX2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U&1#%.%:!! @T  !"Z/:)FPVWWWN  # (@eff f  !l&:VWW W4@4L<-a00R^RdefRg  . .`aa a  %5t{7Mt{Oe$fouvvvJ+/??       !5,; , , (.3;*)    ,, )/!5# '  *!,..998< 8O O444UY  J#]3oabb6II I)-')7&1     r')T NNNNNNNNNNN)rrrr.rWrYrr! LongTensorr"rboolr r%rrrs@r(rLrLs"???@@@.2372615.22615+/)-,0#'p p #d*p )D0p (4/ p '$. p $t+ p (4/p '$.p "Djp  $;p #Tkp D[p  $eE,=&> >p p p ^p p p p p r'rLc$eZdZfdZdZxZS)rPcttj|j|j|_tj|_dSr)r-r.rrr3rTanh activationrs r(r.zViltPooler.__init__sC Yv163EFF '))r'cr|dddf}||}||}|Sr_)rrj)r@rfirst_token_tensorrcs r(rzViltPooler.forwards@+111a40 #566  66 r'rrs@r(rPrPsG$$$$$ r'rPzU ViLT Model with a language modeling head on top as done during pretraining. c@eZdZddiZfdZdZdZe ddej dzdej dzd ej dzd ej dzd ej dzd ej dzd ej dzdej dzde dzde dzde dzde e ej zfdZxZS)ViltForMaskedLMzmlm_score.decoder.weightz6vilt.embeddings.text_embeddings.word_embeddings.weightct|t||_t ||_|dSr)r-r.rLr> ViltMLMHead mlm_scorerRrs r(r.zViltForMaskedLM.__init__sQ    f%% $V,, r'c|jjSr)rqdecoderrVs r(get_output_embeddingsz%ViltForMaskedLM.get_output_embeddingss ~%%r'c@||j_|j|j_dSr)rqrsr)r@new_embeddingss r(set_output_embeddingsz%ViltForMaskedLM.set_output_embeddingss!/,1r'Nrrrrrrrlabelsrr6r7rc T| | n |jj} ||||||||| | |  } | dd\}}| |jdn |jd}|ddd|f|dd|df}}||}d}|et }||j}||d|jj |d}| s|f| ddz}||f|zn|St||| j | j S)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrrrrrr6r7rDrrXrrrr) r?r7r>rnrqrr|r\rurr rr)r@rrrrrrrrxrr6r7r]rrbrc text_seq_len text_featuresr mlm_logitsmasked_lm_lossloss_fctr s r(rzViltForMaskedLM.forwardsV&1%STTTr'c||}||}||}|Sr)rrrrs r(rz#ViltPredictionHeadTransform.forward6s= =11 --m<< }55 r'rrs@r(rr,sLUUUUUr'rc$eZdZfdZdZxZS)rpct||_t||_t j|j|j|_ dSr) r-r.r?r transformrrr3rrsrs r(r.zViltMLMHead.__init__>sL  4V<<y!3V5FGG r'cZ||}||}|Sr)rrs)r@rs r(rzViltMLMHead.forwardDs' NN1   LLOOr'rrs@r(rprp=sLHHHHH r'rpz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. c,eZdZfdZe ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd edzd edzd edzde e ejzfdZ xZ S)ViltForQuestionAnsweringc t||j|_t||_t jt j|j|jdzt j |jdzt j t j|jdz|j|_ | dS)NrD) r-r. num_labelsrLr>r Sequentialrr3rGELU classifierrRrs r(r.z!ViltForQuestionAnswering.__init__Qs     +f%% - If(&*>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrzrrDr{)r?r7r>r[rr|r\rrM binary_cross_entropy_with_logitsrnrrr)r@rrrrrrrrxrr6r7r]rr[rrr s r(rz ViltForQuestionAnswering.forwardbsf&1%