tj  dZddlZddlZddlmZmZddlmZddlm Z ddl m Z ddl m Z mZddlZddlZddlmZdd lmZmZmZd d lmZd d lmZd d lmZd dlmZd dl m!Z!d dl"m#Z#d dl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ee+e,fZ-e.ej/ej/dzej/fZ0ede0fZ1ede fZ2e'j3e4Z5e&deGdde%Z6e&deGdde%Z7e&deGdde%Z8e&deGd d!e%Z9Gd"d#ej:Z;Gd$d%ej:Z<Gd&d'ej:Z=Gd(d)ej:Z>Gd*d+ej:Z?Gd,d-ej:Z@Gd.d/ej:ZAe&Gd0d1e!ZBe&d2Gd3d4eBZCe&d5Gd6d7eBZDe&d8Gd9d:eBZEe&d;Gd<d=eBZFe&d>Gd?d@eBZGe&dAGdBdCeBZHe&dDGdEdFeBZIe&dGGdHdIeBZJ ddKZKGdLdMej:ejLNZMGdOdPeMZNGdQdReMZOGdSdTeMZPGdUdVeMZQGdWdXeMZRdYe-dZej/d[ee+ej/ffd\ZSGd]d^eMZTdd_ej/d`e,dae,d[ej/fdbZUGdcddejVZWGdedfej:ZXddjZYddlZZGdmdnej:ejLNZ[Gdodpe[Z\dqZ]Gdrdse[Z^Gdtduej:Z_Gdvdwe_Z`Gdxdyej:ZaGdzd{ej:ZbGd|d}ej:ZcGd~dej:ZdGddej:ZeGdde_ZfGdde_ZgGdde_ZhGdde_ZigdZjdS)zPyTorch Perceiver model.N)CallableMapping) dataclass)reduce)__add__)AnyOptional)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)"BaseModelOutputWithCrossAttentions)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringlogging torch_int)PerceiverConfig.zw Base class for Perceiver base model's outputs, with potential hidden states, attentions and cross-attentions. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ e ejdzed<dZ e ejdzed<dZ e ejdzed<dS)PerceiverModelOutputz logits (`torch.FloatTensor` of shape `(batch_size, num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). Nlogitslast_hidden_state hidden_states attentionscross_attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rr tupler!r"n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/perceiver/modeling_perceiver.pyrr/s (,FE  $+++26u(4/66659M5*+d299926Je'(4/6668<eE-.5<<<<>,,77 _f== %--aAq99DDFF "/"4"4"6"6ss";wj"H* *,CD 6G]=/22mM]r,)FNNrNNNNNF) r#r$r%r&r;rnr'Tensorr(boolr*rJrLrMs@r-rOrOsjj !'G'G'G'G'G'GR%%%48+/04). ::|:)D0:!D( : &- :  $; : u| ::::::::r,rOcBeZdZfdZdejdejfdZxZS)PerceiverSelfOutputc|ttj|||_dSr9)r:r;r r\dense)rBrCinput_channelsoutput_channelsrDs r-r;zPerceiverSelfOutput.__init__s0 Y~?? r,r rsc0||}|Sr9)rrBr s r-rJzPerceiverSelfOutput.forwards =11 r,)r#r$r%r;r'rrJrLrMs@r-rrsc@@@@@U\elr,rceZdZdZ dfd Z ddejdejdzd ejdzd ejdzd edzd e ejf d Z xZ S)PerceiverAttentionz*Attention module, including a dense block.FNrTc dt|r6|4|jdkr|}n.|jdkr|}n td|jd||}||}t ||||||||_d} |r|} n| |} t ||jj| |_||_ dS)NqkvzUnknown value z) for cross_attention_shape_for_attention.)rcrTrUrRrdre)rr) r:r;#cross_attention_shape_for_attentionrSrOrBrrUoutputuse_query_residual) rBrCrcrTrUrRrdrerrrDs r-r;zPerceiverAttention.__init__s    )+"59S@@# ;tCC$  ;V%O;;; "# !( * 1#!      -#OO&",)&AUgvwww "4r,r rorprqrrrsc||||||}||d}|jr||z}|f|ddz}|SNrr)rBrr) rBr rorprqrr self_outputsattention_outputrs r-rJzPerceiverAttention.forward2sryy         ;;|A77  " @/-? #% QRR(88r,)FNNrNNTr) r#r$r%r&r;r'rr(rr*rJrLrMs@r-rrs44 !-5-5-5-5-5-5d48+/04). |)D0!D(  &-   $;  u| r,rcFeZdZdZfdZdejdejfdZxZS) PerceiverMLPz5A Transformer-style dense module to follow attention.c8ttj|||z|_t |jtrt|j|_ n |j|_ tj||z||_ dSr9) r:r;r r\dense1 isinstance hidden_actstrrintermediate_act_fndense2)rBrC input_sizewidening_factorrDs r-r;zPerceiverMLP.__init__Rs i Oj,HII f' - - 9'-f.?'@D $ $'-'8D $i* 48+/04). |)D0!D(  &-   $;  u| :r,rceZdZdZdfd Z ddejdejdzdejdzd ejdzd edzd edzd edzd e e zfdZ xZ S)PerceiverEncoderz=The Perceiver Encoder: a scalable, fully attentional encoder.Nc t||_|j|jzdkr t d|jd|jd|j|jzdkr t d|jd|jdt|d|j|j |j|j||j |j  |_ g}t|jD]L}t|d|j|j |j|j|j|j }||Mt#j||_dS) Nrznum_z_channels (z.) must be divisible by num_self_attend_heads (rQz/) must be divisible by num_cross_attend_heads (TrcrTrUrRrdrerrF)rcrTrUrRrdrer)r:r;rCr?num_self_attention_headsrSnum_cross_attention_headsrrTrUcross_attention_widening_factorrcross_attentionrangenum_self_attends_per_blockself_attention_widening_factorappendr ModuleList self_attends)rBrCreself_attention_layersrlayerrDs r-r;zPerceiverEncoder.__init__s    f= = B BO6#3OO+1+JOOO   f> >! C CQ6#3QQ,2,LQQQ  . #*(6""B%8 !#v899 0 0A"#(".!, 9&' & E   E " ( ( / / / /M*?@@r,FTr rorprqrroutput_hidden_states return_dictrsc|rdnd}|rdnd} |rdnd} ||||||} | d}|r | | dfz} t|jjD]P} t |jD]1\} }|r||fz}||||} | d}|r | | dfz} 2|r||fz}Q|st d||| | fDSt||| | S)Nr+rorprqrrrr)rorrc3K|]}||V dSr9r+).0vs r- z+PerceiverEncoder.forward..s0= ===r,)rr r!r")rrrC num_blocks enumeraterr*r)rBr rorprqrrrrall_hidden_statesall_self_attentionsall_cross_attentions layer_outputsri layer_modules r-rJzPerceiverEncoder.forwards#7@BBD$5?bb4%6@rrD,, )#/ -  &a(  N#7=;K:M#M t{-.. I IA#,T->#?#? T T<'M(9] Note that it's possible to fine-tune Perceiver on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ceZdZ ddeddedeffd ZdZdZe dd e j d e j dzd e e e jfdzd edzdedzdededzdeezfdZxZS)PerceiverModelNdecoderPerceiverAbstractDecoderinput_preprocessoroutput_postprocessorct|||_||_||_t ||_t|||jn|j |_ ||_ | dS)a decoder (`PerceiverDecoder`, *optional*): Decoder module that transforms latent representations into task predictions. input_preprocessor (`PreprocessorType`, *optional*): Preprocessor that encodes raw inputs into tensors for the model. output_postprocessor (`PostprocessorType`, *optional*): Postprocessor that transforms model outputs into final predictions. N)re) r:r;rCrrr7 embeddingsr num_channelsd_modelencoderr post_init)rBrCrrrrDs r-r;zPerceiverModel.__init__2s     "4$8!-f55' >P>\-::bhbp      r,c|jjSr9rr@rBs r-get_input_embeddingsz#PerceiverModel.get_input_embeddingsOs &&r,c||j_dSr9r)rBr_s r-set_input_embeddingsz#PerceiverModel.set_input_embeddingsRs"'r,Frprosubsampled_output_pointsrrrinterpolate_pos_encodingrrsc ||n |jj}||n |jj}||n |jj}|j|||\}} } ndd} d} |d|jjkr8td|dd|jjd|\} } } |j}|tj | | f|}| | }t|j|| }| |d||||| }|d }d}|jr|)|d jd |d jd dd}n| }|j|| | |}|||||}|j}|r(|j!|r|j|jz|_n ||jz}|jr|||}|s|||f|ddzS|f|ddzSt)|||j|j|jS)a inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. subsampled_output_points (`dict[str, torch.Tensor]`, *optional*): Dictionary of tensors used as queries for the decoder. The decoder maps these queries to the latent representation of the model. Used for subsampled decoding, e.g. when only decoding certain image patches. Examples: ```python >>> from transformers import PerceiverConfig, PerceiverTokenizer, PerceiverImageProcessor, PerceiverModel >>> from transformers.models.perceiver.modeling_perceiver import ( ... PerceiverTextPreprocessor, ... PerceiverImagePreprocessor, ... PerceiverClassificationDecoder, ... ) >>> import torch >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> # EXAMPLE 1: using the Perceiver to classify texts >>> # - we define a TextPreprocessor, which can be used to embed tokens >>> # - we define a ClassificationDecoder, which can be used to decode the >>> # final hidden states of the latents to classification logits >>> # using trainable position embeddings >>> config = PerceiverConfig() >>> preprocessor = PerceiverTextPreprocessor(config) >>> decoder = PerceiverClassificationDecoder( ... config, ... num_channels=config.d_latents, ... trainable_position_encoding_kwargs=dict(num_channels=config.d_latents, index_dims=1), ... use_query_residual=True, ... ) >>> model = PerceiverModel(config, input_preprocessor=preprocessor, decoder=decoder) >>> # you can then do a forward pass as follows: >>> tokenizer = PerceiverTokenizer() >>> text = "hello world" >>> inputs = tokenizer(text, return_tensors="pt").input_ids >>> with torch.no_grad(): ... outputs = model(inputs=inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 2] >>> # to train, one can train the model using standard cross-entropy: >>> criterion = torch.nn.CrossEntropyLoss() >>> labels = torch.tensor([1]) >>> loss = criterion(logits, labels) >>> # EXAMPLE 2: using the Perceiver to classify images >>> # - we define an ImagePreprocessor, which can be used to embed images >>> config = PerceiverConfig(image_size=224) >>> preprocessor = PerceiverImagePreprocessor( ... config, ... prep_type="conv1x1", ... spatial_downsample=1, ... out_channels=256, ... position_encoding_type="trainable", ... concat_or_add_pos="concat", ... project_pos_dim=256, ... trainable_position_encoding_kwargs=dict( ... num_channels=256, ... index_dims=config.image_size**2, ... ), ... ) >>> model = PerceiverModel( ... config, ... input_preprocessor=preprocessor, ... decoder=PerceiverClassificationDecoder( ... config, ... num_channels=config.d_latents, ... trainable_position_encoding_kwargs=dict(num_channels=config.d_latents, index_dims=1), ... use_query_residual=True, ... ), ... ) >>> # you can then do a forward pass as follows: >>> image_processor = PerceiverImageProcessor() >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = image_processor(image, return_tensors="pt").pixel_values >>> with torch.no_grad(): ... outputs = model(inputs=inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 2] >>> # to train, one can train the model using standard cross-entropy: >>> criterion = torch.nn.CrossEntropyLoss() >>> labels = torch.tensor([1]) >>> loss = criterion(logits, labels) ```N)rrHzLast dimension of the inputs: z' doesn't correspond to config.d_model: z0. Make sure to set config.d_model appropriately.device)rE)rC inputs_embedsro)rorprqrrrrraudiorrr rlabelsubsampled_points)z query_maskrr)modality_sizes)rrr r!r")rCrrrrrrhrrSr r'onesrrrrrz decoder_queryrr"rrr r!)rBrprorrrrrrkwargsrinputs_without_posrE seq_lengthrr embedding_outputencoder_outputssequence_outputroutput_modality_sizesrdecoder_outputss r-rJzPerceiverModel.forwardUs"`2C1N--TXT_Tq$8$D $+Jj &1%DQG5g>DQG))%% )7% L66(:Nf7M#ll!)"3 +O %+F! Y_%E%QY'8?;[[$44'68X&XO( a226J_2`` @!0?1223FFF')OABB,???#-)7&1,=     r,NNN)NNNNFN)r#r$r%r PreprocessorTypePostprocessorTyper;rrrr'r(dictrrrr*rrJrLrMs@r-rr$sU"9=/326 45-  0 :'''(((48CG)-,0).#'I I !I )D0I #'sEL'8"9D"@ I  $; I #Tk I #'I D[I  % %I I I ^I I I I I r,rz@ Example use of Perceiver for masked language modeling. ceZdZdeffd Ze d dejdzdejdzdedzdedzdejdzd edzd ejdzd e e zfd Z xZ S)PerceiverForMaskedLMrCcZt|t|}|j|jd}t ||t ||j|j|jd|jddd| |_t||_ | dS)Nr index_dimsF) output_num_channelsoutput_index_dimsrrTrUrRr final_project"trainable_position_encoding_kwargsrr) r:r;PerceiverTextPreprocessorrmax_position_embeddingsrPerceiverBasicDecoderr?rPerceiverEmbeddingDecoderembedding_decoderr)rBrCtext_preprocessor*trainable_position_encoding_kwargs_decoderrDs r-r;zPerceiverForMaskedLM.__init__(s    5f==.: 86 6 2 ( 0)$*$4"("@.;",9#(#3]       ";6!B!B r,Nrprorrrlabelsr input_idsrsc ||td|||}||n |jj}||||||} ||r| jn| d|jjj} d} |Kt} | | d|jj | d} |s| f| ddz} | | f| zn| St| | | j | j | jS) a inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, PerceiverForMaskedLM >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("deepmind/language-perceiver") >>> model = PerceiverForMaskedLM.from_pretrained("deepmind/language-perceiver") >>> # training >>> text = "This is an incomplete sentence where some words are missing." >>> inputs = tokenizer(text, padding="max_length", return_tensors="pt") >>> # mask " missing." >>> inputs["input_ids"][0, 52:61] = tokenizer.mask_token_id >>> labels = tokenizer(text, padding="max_length", return_tensors="pt").input_ids >>> outputs = model(**inputs, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 19.87 >>> logits = outputs.logits >>> list(logits.shape) [1, 2048, 262] >>> # inference >>> text = "This is an incomplete sentence where some words are missing." >>> encoding = tokenizer(text, padding="max_length", return_tensors="pt") >>> # mask bytes corresponding to " missing.". Note that the model performs much better if the masked span starts with a space. >>> encoding["input_ids"][0, 52:61] = tokenizer.mask_token_id >>> # forward pass >>> with torch.no_grad(): ... outputs = model(**encoding) >>> logits = outputs.logits >>> list(logits.shape) [1, 2048, 262] >>> masked_tokens_predictions = logits[0, 52:61].argmax(dim=-1).tolist() >>> tokenizer.decode(masked_tokens_predictions) ' missing.' ```N,You cannot use both `inputs` and `input_ids`rprorrrrr)embedding_layerrHrgr2rr r!r")rSrCrrr1rrrr ri vocab_sizer1r r!r")rBrprorrrr4rr5rrrmasked_lm_lossloss_fctrs r-rJzPerceiverForMaskedLM.forwardGsL~  )"7KLL L ^ 5F%0%KlKw(    '))H%Xfkk"dk6L&M&Mv{{[]__N ZY,F3A3M^%..SY Y&!/)$5     r,NNNNNNN) r#r$r%rr;rr'rrr*r1rJrLrMs@r-r"r""s  >'+.2)-,0&*#')-` `  t#`  t+`  $; ` #Tk `  t# ` D[` <$&`  ( (` ` ` ^` ` ` ` ` r,r"z; Example use of Perceiver for text classification. ceZdZfdZe d dejdzdejdzdedzdedzdejdzdedzd ejdzd ee zfd Z xZ S) "PerceiverForSequenceClassificationc t||jdd}|j|_t |t |t ||j|d|_|dS)Nrr$Trr+rr,) r:r;r? num_labelsrr-PerceiverClassificationDecoderrr)rBrCr3rDs r-r;z+PerceiverForSequenceClassification.__init__s    FLFVfg5h5h2 +' 8@@2#-3]#'      r,Nrprorrrr4rr5rsc ||td|||}||n |jj}||||||} |r| jn| d} d} |Z|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt} |jdkr1| | | } n| | |} n|jjdkrGt} | | d |j|d } n*|jjdkrt} | | |} |s| f| d dz} | | f| zn| St!| | | j| j| j S) a  inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, PerceiverForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("deepmind/language-perceiver") >>> model = PerceiverForSequenceClassification.from_pretrained("deepmind/language-perceiver") >>> text = "hello world" >>> inputs = tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(inputs=inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 2] ```Nr7r8rr regressionsingle_label_classificationmulti_label_classificationrHrgr:)rSrCrrr problem_typerCdtyper'longrKr squeezer rir r5r r!r")rBrprorrrr4rr5rrrr2r=rs r-rJz*PerceiverForSequenceClassification.forwards)F  )"7KLL L ^ 5F%0%GAJ  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x// FY,F)-)9TGf$$v E(!/)$5     r,r> r#r$r%r;rr'rrr*r5rJrLrMs@r-r@r@s ('+.2)-,0&*#')-T T  t#T  t+T  $; T #Tk T  t# T D[T <$&T  * *T T T ^T T T T T r,r@a Example use of Perceiver for image classification, for tasks such as ImageNet. This model uses learned position embeddings. In other words, this model is not given any privileged information about the structure of images. As shown in the paper, this model can achieve a top-1 accuracy of 72.7 on ImageNet. [`PerceiverForImageClassificationLearned`] uses [`~models.perceiver.modeling_perceiver.PerceiverImagePreprocessor`] (with `prep_type="conv1x1"`) to preprocess the input images, and [`~models.perceiver.modeling_perceiver.PerceiverClassificationDecoder`] to decode the latent representation of [`PerceiverModel`] into classification logits. ceZdZfdZe ddejdzdejdzdedzdedzdejdzd ed edzd ejdzd ee zfd Z xZ S)&PerceiverForImageClassificationLearnedc <t|d|jdzd}|jdd}|j|_t |t |dddddd|t||j|d  |_| dS) Nr&rgr$rconv1x1 trainableconcat) prep_typespatial_downsample out_channelsposition_encoding_typeconcat_or_add_posproject_pos_dimr+TrBr,) r:r; image_sizer?rCrPerceiverImagePreprocessorrDrr)rBrC/trainable_position_encoding_kwargs_preprocessorr3rDs r-r;z/PerceiverForImageClassificationLearned.__init__+s    KN^d^oqr^r:s:s7FLFVfg5h5h2 +' 9##$ '2"* #3b   3#-3]#'    * r,NFrprorrrr4rr pixel_valuesrsc  f||td|||}||n |jj}|||||||} |r| jn| d} d} |||| |j} |s| f| ddz} | | f| zn| St | | | j| j| j S)aQ inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, PerceiverForImageClassificationLearned >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("deepmind/vision-perceiver-learned") >>> model = PerceiverForImageClassificationLearned.from_pretrained("deepmind/vision-perceiver-learned") >>> inputs = image_processor(images=image, return_tensors="pt").pixel_values >>> outputs = model(inputs=inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 1000] >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: tabby, tabby cat ```N/You cannot use both `inputs` and `pixel_values`)rprorrrrrrrgr: rSrCrrr loss_functionr5r r!r")rBrprorrrr4rrr]rrrr2rs r-rJz.PerceiverForImageClassificationLearned.forwardIs^  ,":NOO O ^ 8!F%0%GAJ  %%ffdkBBD FY,F)-)9TGf$$v E(!/)$5     r,)NNNNNFNNrMrMs@r-rOrOs<'+.2)-,0&*).#',0M M  t#M  t+M  $; M #Tk M  t# M #'M D[M lT)M  * *M M M ^M M M M M r,rOa Example use of Perceiver for image classification, for tasks such as ImageNet. This model uses fixed 2D Fourier position embeddings. As shown in the paper, this model can achieve a top-1 accuracy of 79.0 on ImageNet, and 84.5 when pre-trained on a large-scale dataset (i.e. JFT). [`PerceiverForImageClassificationLearned`] uses [`~models.perceiver.modeling_perceiver.PerceiverImagePreprocessor`] (with `prep_type="pixels"`) to preprocess the input images, and [`~models.perceiver.modeling_perceiver.PerceiverClassificationDecoder`] to decode the latent representation of [`PerceiverModel`] into classification logits. ceZdZfdZe d dejdzdejdzdedzdedzdejdzdedzd ejdzd ee zfd Z xZ S) &PerceiverForImageClassificationFourierc (t|ddddd}|jdd}|j|_t |t |dd| t ||j|d  |_|dS) NTrf@F concat_posmax_resolution num_bands sine_onlyrr$pixels)rTrU fourier_position_encoding_kwargsrBr, r:r;r?rCrr[rDrrrBrC-fourier_position_encoding_kwargs_preprocessorr3rDs r-r;z/PerceiverForImageClassificationFourier.__init__s    ( 9 9 5 GMFVfg5h5h2 +' 9"#$1^     3#-3]#'    " r,Nrprorrrr4rr]rsc d||td|||}||n |jj}||||||} |r| jn| d} d} |||| |j} |s| f| ddz} | | f| zn| St | | | j| j| j S)aQ inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, PerceiverForImageClassificationFourier >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("deepmind/vision-perceiver-fourier") >>> model = PerceiverForImageClassificationFourier.from_pretrained("deepmind/vision-perceiver-fourier") >>> inputs = image_processor(images=image, return_tensors="pt").pixel_values >>> outputs = model(inputs=inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 1000] >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: tabby, tabby cat ```Nr_r8rrgr:r` rBrprorrrr4rr]rrrr2rs r-rJz.PerceiverForImageClassificationFourier.forward\  ,":NOO O ^ 8!F%0%GAJ  %%ffdkBBD FY,F)-)9TGf$$v E(!/)$5     r,r>rMrMs@r-rcrcs>'+.2)-,0&*#',0J J  t#J  t+J  $; J #Tk J  t# J D[J lT)J  * *J J J ^J J J J J r,rcaS Example use of Perceiver for image classification, for tasks such as ImageNet. This model uses a 2D conv+maxpool preprocessing network. As shown in the paper, this model can achieve a top-1 accuracy of 82.1 on ImageNet. [`PerceiverForImageClassificationLearned`] uses [`~models.perceiver.modeling_perceiver.PerceiverImagePreprocessor`] (with `prep_type="conv"`) to preprocess the input images, and [`~models.perceiver.modeling_perceiver.PerceiverClassificationDecoder`] to decode the latent representation of [`PerceiverModel`] into classification logits. ceZdZfdZe d dejdzdejdzdedzdedzdejdzdedzd ejdzd ee zfd Z xZ S) -PerceiverForImageClassificationConvProcessingc *t|ddddd}|jdd}|j|_t |t |ddd | t ||j|d  |_|dS) NT)8rxrgFrhrr$convfourier)rTrUrWrnrBr,rorps r-r;z6PerceiverForImageClassificationConvProcessing.__init__#s    & 9 9 5 GMFVfg5h5h2 +' 9 #$'01^    3#-3]#'    $ r,Nrprorrrr4rr]rsc d||td|||}||n |jj}||||||} |r| jn| d} d} |||| |j} |s| f| ddz} | | f| zn| St | | | j| j| j S)aY inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, PerceiverForImageClassificationConvProcessing >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("deepmind/vision-perceiver-conv") >>> model = PerceiverForImageClassificationConvProcessing.from_pretrained("deepmind/vision-perceiver-conv") >>> inputs = image_processor(images=image, return_tensors="pt").pixel_values >>> outputs = model(inputs=inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 1000] >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: tabby, tabby cat ```Nr_r8rrgr:r`rss r-rJz5PerceiverForImageClassificationConvProcessing.forwardCrtr,r>rMrMs@r-rvrvs@'+.2)-,0&*#',0J J  t#J  t+J  $; J #Tk J  t# J D[J lT)J  * *J J J ^J J J J J r,rva0 Example use of Perceiver for optical flow, for tasks such as Sintel and KITTI. [`PerceiverForOpticalFlow`] uses [`~models.perceiver.modeling_perceiver.PerceiverImagePreprocessor`] (with *prep_type="patches"*) to preprocess the input images, and [`~models.perceiver.modeling_perceiver.PerceiverOpticalFlowDecoder`] to decode the latent representation of [`PerceiverModel`]. As input, one concatenates 2 subsequent frames along the channel dimension and extract a 3 x 3 patch around each pixel (leading to 3 x 3 x 3 x 2 = 54 values for each pixel). Fixed Fourier position encodings are used to encode the position of each pixel in the patch. Next, one applies the Perceiver encoder. To decode, one queries the latent representation using the same encoding used for the input. ceZdZfdZe d dejdzdejdzdedzdedzdejdzdedzd ee zfd Z xZ S) PerceiverForOpticalFlowc<t|d|jddd}d|jddd}t|ddddd d | }t ||t ||j|jd dd d | |_|dS)NrgFTrkrjrlrirhpatchesr6rgrz)rTrUconv_after_patchingconv_after_patching_in_channelstemporal_downsamplerWrnY@)routput_image_shaperescale_factorrr(rWrnr,) r:r; train_sizer[rPerceiverOpticalFlowDecoderrrr)rBrCrq(fourier_position_encoding_kwargs_decoderimage_preprocessorrDs r-r;z PerceiverForOpticalFlow.__init__s    $/ 9 9 5$/ 4 4 08  $,. !#,-Z    ( 1//<#)#4$#($%(11Y      & r,Nrprorrrr4rrsc ||n |jj}d}|td||||||} |r| jn| d} |s| f| ddz} ||f| zn| St || | j| j| jS)a7 inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the optical flow loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import PerceiverForOpticalFlow >>> import torch >>> model = PerceiverForOpticalFlow.from_pretrained("deepmind/optical-flow-perceiver") >>> # in the Perceiver IO paper, the authors extract a 3 x 3 patch around each pixel, >>> # leading to 3 x 3 x 3 = 27 values for each pixel (as each pixel also has 3 color channels) >>> # patches have shape (batch_size, num_frames, num_channels, height, width) >>> # the authors train on resolutions of 368 x 496 >>> patches = torch.randn(1, 2, 27, 368, 496) >>> outputs = model(inputs=patches) >>> logits = outputs.logits >>> list(logits.shape) [1, 368, 496, 2] ```Nz*Optical flow training is not yet supportedr8rrgr: rCrNotImplementedErrorrrr5r r!r") rBrprorrrr4rrr2rrrs r-rJzPerceiverForOpticalFlow.forwardsF&1%GAJ FY,F)-)9TGf$$v E(!/)$5     r,)NNNNNNrMrMs@r-r}r}s/////b'+.2)-,0&*#'; ;  t#;  t+;  $; ; #Tk ;  t# ; D[;  * *; ; ; ^; ; ; ; ; r,r}a. Example use of Perceiver for multimodal (video) autoencoding, for tasks such as Kinetics-700. [`PerceiverForMultimodalAutoencoding`] uses [`~models.perceiver.modeling_perceiver.PerceiverMultimodalPreprocessor`] to preprocess the 3 modalities: images, audio and class labels. This preprocessor uses modality-specific preprocessors to preprocess every modality separately, after which they are concatenated. Trainable position embeddings are used to pad each modality to the same number of channels to make concatenation along the time dimension possible. Next, one applies the Perceiver encoder. [`~models.perceiver.modeling_perceiver.PerceiverMultimodalDecoder`] is used to decode the latent representation of [`PerceiverModel`]. This decoder uses each modality-specific decoder to construct queries. The decoder queries are created based on the inputs after preprocessing. However, autoencoding an entire video in a single forward pass is computationally infeasible, hence one only uses parts of the decoder queries to do cross-attention with the latent representation. This is determined by the subsampled indices for each modality, which can be provided as additional input to the forward pass of [`PerceiverForMultimodalAutoencoding`]. [`~models.perceiver.modeling_perceiver.PerceiverMultimodalDecoder`] also pads the decoder queries of the different modalities to the same number of channels, in order to concatenate them along the time dimension. Next, cross-attention is performed with the latent representation of [`PerceiverModel`]. Finally, [`~models.perceiver.modeling_perceiver.PerceiverMultiModalPostprocessor`] is used to turn this tensor into an actual video. It first splits up the output into the different modalities, and then applies the respective postprocessor for each modality. Note that, by masking the classification label during evaluation (i.e. simply providing a tensor of zeros for the "label" modality), this auto-encoding model becomes a Kinetics 700 video classifier. ceZdZdeffd Ze d dejdzdejdzdee ejfdzde dzde dzd ejdzd e dzd e e zfd Z xZS)"PerceiverForMultimodalAutoencodingrCcpt||j|jz}t dt |dd|fdddd|jt|dd |j|j|jfdddddd t|d d d dd}t|d|j |j dddd |j|j|jfddd}t|dt|d||jzf|j dddd|fddd|t|dddd|jd dd d|j d}t#t%||j t'|j dt)||j d }t+|||||_|dS)NrrzFTrr)rWrnrTsamples_per_patch r)rWrnrTrUrr r?)rr r )min_padding_size modalities mask_probs)concat_preprocessed_input output_shaper(rposition_encoding_onlyrWrn)rr)r(rrrWrnrRr$)rrrrWr+)rr num_outputsr(r) in_channelsr)rrV)r)rrr)r:r; num_framesaudio_samples_per_framePerceiverMultimodalPreprocessorPerceiverAudioPreprocessorrr[rZPerceiverOneHotPreprocessor&PerceiverBasicVideoAutoencodingDecoderrr(PerceiverMultimodalDecoderr/rD_label_trainable_num_channels PerceiverMultimodalPostprocessorPerceiverAudioPostprocessor PerceiverProjectionPostprocessor$PerceiverClassificationPostprocessorrrr)rBrCn_audio_samplesr image_decoderrrrDs r-r;z+PerceiverForMultimodalAutoencoding.__init__-s     +f.LL<3+4%(+:*<%*&* 66 (&,&>   4+4%'+1+OQWQb*c%*&* 66 ('(()   5V<<58"%sSAA=   B? &+, & :$#'#,#)#4f6GIZ"["" ..   "- &+/.3'6&:R'R&T(.(B',+/+4%(+:*<%*&* 66 '7.3',+/+6(.(L&'88   %> & :$O( ( ( T @4VIcddd9fF`opqqq=fRXRlmmm   ( 1!5     r,Nrprorrrrr4rrsc ||n |jj}d} |td|||||||} |r| jn| d} |s| f| ddz} | | f| zn| St | | | j| j| jS)a+ inputs (`torch.FloatTensor`): Inputs to the perceiver. Can be anything: images, text, audio, video, etc. subsampled_output_points (`dict[str, torch.Tensor]`, *optional*): Dictionary of tensors used as queries for the decoder. The decoder maps these queries to the latent representation of the model. Used for subsampled decoding, e.g. when only decoding certain image patches. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import PerceiverForMultimodalAutoencoding >>> import torch >>> import numpy as np >>> # create multimodal inputs >>> images = torch.randn((1, 16, 3, 224, 224)) >>> audio = torch.randn((1, 30720, 1)) >>> inputs = dict(image=images, audio=audio, label=torch.zeros((images.shape[0], 700))) >>> model = PerceiverForMultimodalAutoencoding.from_pretrained("deepmind/multimodal-perceiver") >>> # in the Perceiver IO paper, videos are auto-encoded in chunks >>> # each chunk subsamples different index dimensions of the image and audio modality decoder queries >>> nchunks = 128 >>> image_chunk_size = np.prod((16, 224, 224)) // nchunks >>> audio_chunk_size = audio.shape[1] // model.config.samples_per_patch // nchunks >>> # process the first chunk >>> chunk_idx = 0 >>> subsampling = { ... "image": torch.arange(image_chunk_size * chunk_idx, image_chunk_size * (chunk_idx + 1)), ... "audio": torch.arange(audio_chunk_size * chunk_idx, audio_chunk_size * (chunk_idx + 1)), ... "label": None, ... } >>> outputs = model(inputs=inputs, subsampled_output_points=subsampling) >>> logits = outputs.logits >>> list(logits["audio"].shape) [1, 240] >>> list(logits["image"].shape) [1, 6272, 3] >>> list(logits["label"].shape) [1, 700] ```Nz5Multimodal autoencoding training is not yet supported)rprorrrrrrrgr:r) rBrprorrrrr4rrr2rrrs r-rJz*PerceiverForMultimodalAutoencoding.forwardsz&1%GAJ FY,F)-)9TGf$$v E(!/)$5     r,r>)r#r$r%rr;rr'rr rrr*r5rJrLrMs@r-rrs`aa 9 , ,/ SQRR R9]]<\]]U