o sh"U@s`dZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZmZmZd d lmZeeZeGdddeZGddde jZ d&de jdej dej dej deej de!de!fddZ"Gddde jZ#Gddde jZ$Gd d!d!eZ%Gd"d#d#e jZ&Gd$d%d%e jZ'dS)'zTPyTorch IdeficsVision model: a copy of CLIPVisionModel using a simpler config objectN) dataclass)CallableOptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONS) ModelOutputcan_return_tuplelogging)IdeficsVisionConfigc@sjeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)IdeficsVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrtuplerrr`/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/transformers/models/idefics/vision.pyr(s rcs\eZdZdeffdd Zdejdededejfdd Zdd ej d e dejfd dZ Z S)IdeficsVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr position_ids)r) persistent)super__init__r" hidden_size embed_dim image_size patch_sizer Parameterrrandnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferarangeexpandselfr" __class__rr r-Gs"  "z IdeficsVisionEmbeddings.__init__ embeddingsheightwidthreturncCs|jdd}||j}|jdd}||kr||kr|S|dddf}|ddddf}|jd} ||jj} ||jj} | d| d} } t|} |dt| t| | }| dddd}|j t j k} | rvt d|t j}tjj|| | | | fd d d }| r|t j }t| |jd kst| |jdkrtd t| t| fd|jd |jdfd| dddddd| }t j|d|fddS)a# This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174 rNrr*g?rr(zUpcasting patch_pos_embed to fp32 for interpolation since `upsample_bicubic2d_out_frame` in nn.functional.interpolate is not implemented for 'torch.bfloat16' dtype. This will result in a slight overhead.bicubicF) scale_factormode align_cornerszNumber of patches for images (z/) don't match the shape of position embedding ()dim)shaper;r)r"r1mathsqrtreshapeintpermutedtyperbfloat16logger warning_oncetofloatr functional interpolate ValueErrorviewcat unsqueeze)r@rCrDrEr8 pos_embedr9class_pos_embedpatch_pos_embedr/ num_h_patches num_w_patchessqrt_num_positionsfp32_upcastingrrr interpolate_pos_encoding^sH        $z0IdeficsVisionEmbeddings.interpolate_pos_encodingF pixel_valuesrhc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model (z8). You should try to set `interpolate_pos_encoding=True`)rUr(rr*rM)rOr0r]r7weightrUrYflatten transposer4r>rr_rhr;r)) r@rirh batch_sizer6rDrE target_dtype patch_embeds class_embedsrCrrr forwards( zIdeficsVisionEmbeddings.forwardF) rrrrr-rTensorrSrhrboolrr __classcell__rrrAr r!Fs$1r!modulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr*rK)rNrU)ptrainingrr() rmatmulrmrr[softmaxfloat32rYrUr~r contiguous) rxryrzr{r|r}r~kwargs attn_weights attn_outputrrr eager_attention_forwards  rcsneZdZdZdeffdd Z   ddejdeejd eejd ee d e ejeejff d d Z Z S)IdeficsVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr"cst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)r,r-r"r.r/num_attention_heads num_headshead_dimr]scaleattention_dropoutr~ is_causalrLineark_projv_projq_projout_projr?rArr r-s$   zIdeficsVisionAttention.__init__NFrr|causal_attention_maskoutput_attentionsrFc CsL|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkrY|durR|durR||}n |durX|}n|du|_ t } |jj dkrz|jj dkrt|rtt dnt|jj } | ||| | ||j |j|jsdn|jd \} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x Channelrr(flash_attention_2Neagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rw)rr}r~)rOrrrr^rrrmr"_attn_implementationrrrWrXr rrr~rRrr)r@rr|rrrn seq_lengthr/querieskeysvaluesattention_interfacerrrrr rrsH          zIdeficsVisionAttention.forward)NNF) rrrrrr-rrtrrurrrrvrrrAr rs"rcs2eZdZfddZdejdejfddZZS)IdeficsVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r,r-r"r hidden_act activation_fnrrr.intermediate_sizefc1fc2r?rArr r-s  zIdeficsVisionMLP.__init__rrFcCs"||}||}||}|Sr)rrr)r@rrrr rrs   zIdeficsVisionMLP.forward)rrrr-rrtrrrvrrrAr rs rc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) IdeficsVisionEncoderLayerr"csRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)eps) r,r-r.r/r self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r?rArr r-"s   z"IdeficsVisionEncoderLayer.__init__Frr|rrrFcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rr|rr)rrrr)r@rr|rrresidualroutputsrrr rr*s"     z!IdeficsVisionEncoderLayer.forwardrs) rrrrr-rrtrrurrrrrvrrrAr r!s rcsxeZdZdZdeffdd Ze     ddeej deej dee d ee d ee d e e e ff d d ZZS)IdeficsVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`IdeficsVisionEncoderLayer`]. Args: config: IdeficsVisionConfig r"cs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr)r).0_r"rr `sz1IdeficsVisionEncoder.__init__..F) r,r-r"r ModuleListrangenum_hidden_layerslayersgradient_checkpointingr?rArr r-]s   zIdeficsVisionEncoder.__init__Nr|rroutput_hidden_states return_dictrFc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD] \} } |r<|| f}| | |||d} | d} |rQ|| df}q1|rY|| f}t| ||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr)rrr)rrr)r"rruse_return_dict enumeraterr ) r@ inputs_embedsr|rrrrencoder_statesall_attentionsridx encoder_layer layer_outputsrrr rrcs2'    zIdeficsVisionEncoder.forward)NNNNN)rrrrrr-rrrrtrurrr rrrvrrrAr rTs, rcsneZdZdeffdd Z     ddeejdeedeed eed eed e e e ff d d Z Z S)IdeficsVisionTransformerr"csRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r,r-r"r.r!rCrrr pre_layrnormrencoderpost_layernorm)r@r"r/rArr r-s   z!IdeficsVisionTransformer.__init__NFrirrrhrrFc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}||}|j||||d}|d}|dddddf} || } |s[|| f|ddSt || |j |j dS)z Returns: Nz You have to specify pixel_values)rh)rrrrrr)r pooler_outputrr) r"rrrr]rCrrrr rr) r@rirrrhrrencoder_outputsr pooled_outputrrr rrs2   z IdeficsVisionTransformer.forward)NNNFN)rrrrr-rrrrurrr rrrvrrrAr rs(  r)rw)(rrP dataclassesrtypingrrrrtorch.utils.checkpointr activationsrmodeling_layersr modeling_outputsr r modeling_utilsr utilsr rrconfiguration_ideficsr get_loggerrrWrModuler!rtrZrrrrrrrrrr sL       k P3X