h dZddlmZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZmZmZdd lmZmZdd lmZmZmZmZmZmZd dlm Z m!Z!m"Z"ejFe$Z%dejLdejLfdZ'dejLdejLfdZ(dejLdejLfdZ)eedGddeZ*eedGddeZ+eeGddeZ,Gdd e jZZ.Gd!d"e jZZ/ dJd#e jZd$ejLd%ejLd&ejLd'eejLd(e0d)e0d*e1fd+Z2Gd,d-e jZZ3Gd.d/e jZZ4Gd0d1eZ5eGd2d3eZ6Gd4d5e jZZ7Gd6d7e jZZ8ed8Gd9d:e6Z9Gd;dd?e6Z;eGd@dAe6ZedFGdGdHe6Z?gdIZ@y)KzPyTorch CLIP model.) dataclass)AnyCallableOptionalUnionN)nn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplefilter_out_non_signature_kwargslogging torch_int) CLIPConfigCLIPTextConfigCLIPVisionConfiglogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr!)rs e/var/www/html/aiagenthome/venv/lib/python3.12/site-packages/transformers/models/clip/modeling_clip.pycontrastive_lossr(%s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r(t)r* caption_loss image_losss r' clip_lossr/)s,#J/L!*,,.1J : % ,,r)tensorctj|d}tj|dd}tj|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 T)dimkeepdim?)r$powsum)r0 square_tensor sum_tensor normed_tensors r'_get_vector_normr</s< IIfa(M=b$?JIIj#.M r)z} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)CLIPVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r@rr$ FloatTensor__annotations__rArBtuplerCr)r'r?r?:sr 15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r)r?ze Base class for text model's outputs that also contains a pooling of the last hidden states. ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)CLIPTextModelOutputz text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. N text_embedsrA.rBrC) rDrErFrGrNrr$rHrIrArBrJrCrKr)r'rMrMLsr 04K%++,359x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r)rMceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd Zy) CLIPOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. Nlosslogits_per_imagelogits_per_textrNr@text_model_outputvision_model_outputrcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))rTrUN)getattrto_tuple).0kselfs r' z&CLIPOutput.to_tuple..}s=   LLDGRYZ^`aRbRkRkRm m s-0)rJkeysr\s`r'rYzCLIPOutput.to_tuple|s# YY[   r))rDrErFrGrQrr$rHrIrRrSrNr@rTrrUrJrrYrKr)r'rPrP^s&)-D(5$$ %,48hu001837OXe//07/3K%++,304L(5,,-448186:3: %* r)rPceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) CLIPVisionEmbeddingsconfigct|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_tj"|j |j|_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasr2r position_idsrr3 persistent)super__init__rb hidden_size embed_dim image_size patch_sizer Parameterr$randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr%expandr\rb __class__s r'rnzCLIPVisionEmbeddings.__init__s   ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr) embeddingsheightwidthrc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr3r6r r2bicubicF)sizemode align_cornersr4)shaper|weight unsqueezer$jit is_tracingrirrrreshapepermuterr" interpolateviewcat) r\rrrryr|rzclass_pos_embedpatch_pos_embedr4 new_height new_widthsqrt_num_positionss r'interpolate_pos_encodingz-CLIPVisionEmbeddings.interpolate_pos_encodingsv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr) pixel_valuesc `|j\}}}}|sJ||jk7s||jk7r,td|d|d|jd|jd |jjj }|j|j |}|jdjdd}|jj|dd} tj| |gd } |r| |j| ||z} | S| |j|jz} | S) NzInput image size (*z) doesn't match model ().)dtyper2rr3r)rrq ValueErrorrxrrtoflatten transposerur~r$rrr|ri) r\rr batch_size_rr target_dtype patch_embeds class_embedsrs r'forwardzCLIPVisionEmbeddings.forwards6'3'9'9$ Avu'Vt-F%SWSbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr)F) rDrErFrrnr$TensorintrrHr __classcell__rs@r'raraseq/q,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr)rac eZdZdeffd Z ddeejdeejdeejdejfdZ xZ S) CLIPTextEmbeddingsrbcNt||j}tj|j ||_tj|j||_|jdtj|jjddy)NrirjFrk) rmrnrorr{ vocab_sizetoken_embeddingmax_position_embeddingsr|r}r$r%r~r\rbrprs r'rnzCLIPTextEmbeddings.__init__s && !||F,=,=yI"$,,v/M/My"Y  ELL)G)GHOOPWXej  r) input_idsri inputs_embedsrc8||jdn|jd}|jjjd}||kDrtd|d|||jddd|f}||j |}|j|}||z}|S)Nr3rzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rr|rrrir)r\rrir seq_lengthmax_position_embeddingposition_embeddingsrs r'rzCLIPTextEmbeddings.forwards -6,AY__R(}GZGZ[]G^ !%!8!8!?!?!E!Ea!H . .d,<=S> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r)rBrcausal_attention_maskrrc j|j\}}}|j|}|j|} |j|} |j ||d|j j dd}| j ||d|j j dd} | j ||d|j j dd} |jjdk(r |du|_ n||||z}n||}t} |jjdk7rt|jj} | ||| | ||j|j|jsdn |j| \} } | j|||j!} |j#| } |sd} | | fS) z#Input shape: Batch x Time x Channelr3rr2flash_attention_2Neager)rrrr)rrrrrrrrb_attn_implementationrrrrrrrrr)r\rBrrrrrrpqueriesr^valuesattention_interfacerrs r'rzCLIPAttention.forward-s-:,?,?) J ++m,{{=)]+,,z:r4==ISSTUWXYyyZT]]CMMaQRSZRGQQRSUVW ;; + +/B B2$>DN).C.O!/2G!G&2!6(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,,/ % ! \"))*j)LWWY mmK0  LL((r))NNF)rDrErFrGrrrrnr$rrboolrJrrrs@r'rrsGBu%5~%EFB.268<,1 1)||1)!.1) ( 5 1) $D> 1) u||Xell33 4 1)r)rcVeZdZfdZdej dej fdZxZS)CLIPMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rmrnrbr hidden_act activation_fnrrrointermediate_sizefc1fc2rs r'rnzCLIPMLP.__init__bsd  #F$5$5699V//1I1IJ99V55v7I7IJr)rBrcl|j|}|j|}|j|}|Sr)rrr)r\rBs r'rzCLIPMLP.forwardis4/ **=9 / r))rDrErFrnr$rrrrs@r'rras$KU\\ellr)rc eZdZdeeefffd Z d dejdejdejde e de ejf dZ xZS) CLIPEncoderLayerrbcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yN)eps) rmrnrorpr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rs r'rnzCLIPEncoderLayer.__init__qsl ++&v.<<F & u  ! &r)rc6eZdZUeed<dZdZdZdZdZ dZ dZ y)CLIPPreTrainedModelrbclipTc |jj}t|trj|jj j jd|dz|jj j jd|dznMt|tr|jj}tjj|jd|jdz|ztjj|jj |jj|ztjj|jj |jj|znQt|t r-|jj}|jdzd|jj"zdzz|z}|jdz|z}tjj|j$j |tjj|j&j |tjj|j(j |tjj|j*j |nt|t,r|jj}|jj.dzd|jj"zdzz|z}d|jj.zdz|z}tjj|j0j |tjj|j2j |n)t|t4rtjj|j6j |j8dz|jjztjj|j:j |j<dz|jjznet|t>rdtjj|j:j |jj.dz|jjznt|t@rdtjj|j6j |jj.dz|jjzn}t|tBrmtjj|jDj |jjFj.dz|jjzt|tjHrI|jJj jM|j j jOdt|tjPr2|jJ%|jJj jMyyy) zInitialize the weightsrg{Gz?)meanstdr)rr2g?N))rbinitializer_factor isinstancerrrdatanormal_r|rarinitrurprxinitializer_rangernum_hidden_layersrrrrrrorr CLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimCLIPVisionModelWithProjectionCLIPTextModelWithProjectionCLIPForImageClassification classifier vision_configrrhzero_fill_r)r\rfactor in_proj_std out_proj_stdfc_stds r' _init_weightsz!CLIPPreTrainedModel._init_weightss// f0 1  " " ) ) . . 6 6CVd] 6 S  % % , , 1 1 9 9sQU 9 V  4 5[[33F GGOOF22&BRBRTXBX[aBaO b GGOOF2299v}}?^?^ag?gO h GGOOF55<<&--BaBadjBjO k  .[[33F!++T1q6==;Z;Z7Z_c6cdgmmK",,d2f GGOO((//KK++T1DKK4R4RR   ; < GGOO&&--KK++T1DKK4R4RR   : ; GGOO!!((KK--994?$++B`B``   fbll + KK   " " $ MM   $ $S ) fbii (V[[-D KK   " " $.E (r)N) rDrErFrrIbase_model_prefixsupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrrKr)r'rrs0 &*#N"&6%r)rc eZdZdZdeffd Z d deejdeejdee dee de f d Z xZ S) CLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig rbct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rmrnrbr ModuleListranger rlayersgradient_checkpointing)r\rbrrs r'rnzCLIPEncoder.__init__sU  mmuVMeMeGf$gGf!%5f%=Gf$gh &+#%hsA#rrroutput_hidden_statesrc6||n|jj}||n|jj}|rdnd}|rdnd}|}t|jD]*\} } |r||fz}| ||||} | d}|s"|| dfz},|r||fz}t |||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrK)rrr)rArBrC)rbrr, enumerater*r) r\rrrrr,encoder_statesall_attentionsrBidx encoder_layer layer_outputss r'rzCLIPEncoder.forwardsJ2C1N-TXT_T_TqTq$8$D $++JjJj  40d% "+DKK"8 C#!/=2B!B)%"3 M*!,M !/=3C2E!E#9 +}.>>N+(%  r)NNNN) rDrErFrGrrnrr$rrrrrrs@r'r&r&sy,z,268<,0/3 D !.D  ( 5 D $D> D 'tn D  D r)r&ceZdZdeffd Ze d deejdeejdeejdee dee de f d Z xZ S) CLIPTextTransformerrbct|||_|j}t ||_t ||_tj||j|_ |j|_ yr) rmrnrbrorrr&encoderrrrfinal_layer_norm eos_token_idrs r'rnzCLIPTextTransformer.__init__<sa  && ,V4"6* " YF F 'tn F  $F F r)r6zI The text model from CLIP without any head or projection on top. ceZdZUeed<ddgZdZdeffd ZdejfdZ dZ e e dd eej d eej d eej d eed eedef dZxZS) CLIPTextModelrbrrFcdt||t||_|j yr)rmrnr6 text_model post_initrs r'rnzCLIPTextModel.__init__s&  -f5 r)rcB|jjjSrrFrrr_s r'get_input_embeddingsz"CLIPTextModel.get_input_embeddings))999r)c:||jj_yrrIr\rs r'set_input_embeddingsz"CLIPTextModel.set_input_embeddings5:""2r)rrrirr,c.|j|||||S)a9 Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrirr,)rF)r\rrrirr,s r'rzCLIPTextModel.forwards)4)%/!5   r)rB)rDrErFrrI_no_split_modulesr"rnrModulerJrNrrrr$rrrrrrs@r'rDrDs -/AB ~ :bii:;-115/3,0/3  ELL) !. u||,  $D>  'tn   $  r)rDc eZdZdeffd Ze d deejdee dee dee de f dZ xZ S) CLIPVisionTransformerrbc t|||_|j}t ||_t j||j|_ t||_ t j||j|_ yr) rmrnrbrorarrrr pre_layrnormr&r8post_layernormrs r'rnzCLIPVisionTransformer.__init__sj  && .v6LL8M8MN"6*  ll9&:O:OPr)rrr,rrc||n|jj}||n|jj}| td|j ||}|j |}|j |||}|j}|dddddf}|j|}t|||j|jS)Nz You have to specify pixel_values)r)rrr,rr<) rbrr,rrrWr8rArXrrBrC) r\rrr,rrBr@rArAs r'rzCLIPVisionTransformer.forwards2C1N-TXT_T_TqTq$8$D $++JjJj   ?@ @ Ogh ))-8 +/<<'/!5,8,  ,==)!Q'2 ++M: )/')77&11   r)NNNF) rDrErFrrnrrr$rHrrrrrs@r'rUrUs{Q/Q59,0/338 ! u001! $D>! 'tn ! #+4. ! $ ! ! r)rUzK The vision model from CLIP without any head or projection on top. c eZdZUeed<dZdgZdeffd ZdejfdZ e e d de ejde ede ed edef d ZxZS) CLIPVisionModelrbrrcdt||t||_|j yr)rmrnrU vision_modelrGrs r'rnzCLIPVisionModel.__init__s'  1&9 r)rcB|jjjSrr^rrxr_s r'rJz$CLIPVisionModel.get_input_embeddings   ++;;;r)rr,rc,|j||||S)a Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrr,r)r^)r\rrr,rs r'rzCLIPVisionModel.forwards(<  %/!5%= !  r)rZ)rDrErFrrImain_input_namerRrnrrSrJrrrr$rHrrrrrs@r'r\r\s $O+,/ ! 'tn ! #' ! $ ! ! r)r\ceZdZUeed<gdZdZdeffd Zee dde jde e jde e jde jfd Zee dd e jd ede jfd Zee dde e j$d e e jde e jde e j$d e ede ede ed edefdZxZS)rrb)rrraFct||t|jts"t dt |jdt|jts"t dt |jd|j}|j}|j|_ |j|_ |j|_ tj|}|j|_t j|}|j"|_t%j&|j|jd|_t%j&|j|jd|_t%j,t/j0|j2j4|_|j9y)NzKconfig.text_config is expected to be of type CLIPTextConfig but is of type .zOconfig.vision_config is expected to be of type CLIPVisionConfig but is of type Frh)rmrnr text_configr TypeErrortyperrprojection_dimrorrrD _from_configrFr\r^rrrrrsr$r0rblogit_scale_init_value logit_scalerG)r\rbrirrFr^rs r'rnzCLIPModel.__init__:sx  &,,n=++,-Q0  &..0@A--./q2  (( ,, $33)55 - 9 9"// < $//&33MB (55!#4+@+@$BUBU\a!b!yy)<)Q>QX]^<< T[[5W5W(XY r)rrrirch|j|||}|j}|j|}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```)rrri)rFr=r)r\rrri text_outputsrA text_featuress r'get_text_featureszCLIPModel.get_text_features]sD648??)%4C4 %22 ,,]; r)rrcf|j||}|j}|j|}|S)ai Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```)rr)r^r=r)r\rrvision_outputsrAimage_featuress r'get_image_featureszCLIPModel.get_image_featuressC<6:5F5F%%=6G6 '44 // >r) return_lossrr,c ||n|jj}||n|jj}|j||||} |j |||||} | j } |j | } | j } |j| } | t| z } | t| z } tj| | jj| j} | |jjj| jz} | j}d}|r t!| }t#||| | | | | S)a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NrcrQ)rQrRrSrNr@rTrU)rbrr,r^rFr=rrr<r$rr,rr!roexpr/rP)r\rrrrirxrr,rrurqr@rNrSrRrQs r'rzCLIPModel.forwardsuP2C1N-TXT_T_TqTq$8$D $++JjJj 6:5F5F%/!5%= 6G6 48??)%/!5 4C4 &33 --l; "00 **;7 $&6|&DD !$4[$AA  ,,{LNN4D4G4G HZHZ4[\)D,<,<,@,@,B,E,EkFXFX,YY*,,. _-D-+#%* .  r))NNr)NNNNNNNF)rDrErFrrIrRr"rnrrr$rrrHrsrrwrrrPrrrs@r'rr4s Z !z!F%&26/3 !<<!!.!u||, !    !'!F%&*/#''##'#    #'#J15481537&*,0/3).V E,,-V u001V !. V u//0 V d^ V $D>V 'tnV #'V  V V r)rceZdZUeed<dZddgZdeffd ZdejfdZ dZ e e dd eej d eej d eej d eed eedef dZxZS)rrbFrrct||tj|}|j|_t j |j|jd|_ |jyNFrh) rmrnrDrmrFrrrorlrrG)r\rbrFrs r'rnz$CLIPTextModelWithProjection.__init__ s[  "//7 $//!yy););V=R=RY^_ r)rcB|jjjSrrIr_s r'rJz0CLIPTextModelWithProjection.get_input_embeddingsrKr)c:||jj_yrrIrMs r'rNz0CLIPTextModelWithProjection.set_input_embeddingsrOr)rrrirr,c|j|||||}|j}|j|}t||j|j |j S)a@ Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> text_embeds = outputs.text_embeds ```rQ)rNrArBrC)rFr=rrMrArBrC) r\rrrirr,rqrArNs r'rz#CLIPTextModelWithProjection.forwardso648??)%/!5 4C4 %22 **=9 "#*<<&44#..   r)rB)rDrErFrrIr"rRrnrrSrJrNrrrr$rrrMrrrs@r'rrs  -/AB ~ :bii:;-115/3,0/3 ( ELL)( !.( u||, ( $D> ( 'tn (  ( ( r)rc eZdZUeed<dZdeffd ZdejfdZ e e d de e jde ede ededef d ZxZS) rrbrct||tj|}|j|_t j |j|jd|_ |jyr}) rmrnr\rmr^rrrorlrrGr\rbr^rs r'rnz&CLIPVisionModelWithProjection.__init__Ns\  &33F; (55!#6+=+=v?T?T[`!a r)rcB|jjjSrr`r_s r'rJz2CLIPVisionModelWithProjection.get_input_embeddingsYrar)rr,rc|j||||}|j}|j|}t||j|j |j S)a Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection >>> from transformers.image_utils import load_image >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> image_embeds = outputs.image_embeds ```rc)r@rArBrC)r^r=rr?rArBrC)r\rrr,rrurAr@s r'rz%CLIPVisionModelWithProjection.forward\sn<6:5F5F%/!5%= 6G6  '44 --m< $%,>>(66%00   r)rZ)rDrErFrrIrdrnrrSrJrrrr$rHrr?rrrs@r'rrIs $O / * 'tn * #' *  * * r)rz CLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. c eZdZdZdeddffd Zee d dee jdee jdee dee de f d Z xZS) rrrbrNc~t|||j|_tj |j }|j |_|jdkDr4tj|j j|jntj|_ |jy)Nr) rmrn num_labelsr\rmrr^rrroIdentityrrGrs r'rnz#CLIPForImageClassification.__init__s   ++&33F4H4HI (55OUN_N_bcNcBIIf**668I8I Jikititiv  r)labelsrr,c||n|jj}||n|jj}|j|||}|j}t j |ddddddfd}|j|}d}||j|||j}t|||j|jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rr,rr)rQrrBrC) rbrr,r^rAr$rr loss_functionrrBrC) r\rrrr,rsequence_outputrrQs r'rz"CLIPForImageClassification.forwards2C1N-TXT_T_TqTq$8$D $++JjJj /3.?.? /!5/@/  "33 **_QAX%>AF1  %%ffdkkBD$!//))   r)r4)rDrErFrdrrnrrrr$rrrrrrs@r'rrs%O z d 04)-,0/3 ( u||,( &( $D> ( 'tn (  ( ( r)r)rrrDrr\rr)rT)ArG dataclassesrtypingrrrrr$r activationsr modeling_attn_mask_utilsr r modeling_layersr modeling_outputsrrrmodeling_utilsrrutilsrrrrrrconfiguration_cliprrr get_loggerrDloggerrr(r/r<r?rMrPrSrarfloatrrrrrrr&r6rDrUr\rrrr__all__rKr)r'rs!11 !d9bbFwwLL   H % `U\\`ell`-%,,-5<<- U\\ell  ?K ?  ?  ?+ ?  ?        FP299Pf%%^"% II% <<% % << % U\\* %  %%%0H)BIIH)V bii /1/d?%/?%?%DS "))S lS "))S l 2 '2  2 j- BII- ` 1 )1  1 hL #L L ^A "5A A H> $7> > B < !4<  < ~ r)