tjDN\dZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z ddlmZdd lmZdd lmZmZmZdd lmZdd lmZd dlmZmZmZmZd dlm Z m!Z!m"Z"d dl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-d dl.m/Z/m0Z0m1Z1m2Z2m3Z3edeGdde!Z4edeGdde"Z5edeGdde Z6Gdde(Z7Gdde/Z8GddeZ9Gd d!e3Z:Gd"d#eZ;Gd$d%e0Z<Gd&d'e1Z=Gd(d)eZ>Gd*d+e&Z?Gd,d-e2Z@Gd.d/e%ZAGd0d1e$ZBGd2d3e'ZCGd4d5e&ZDGd6d7e*ZEGd8d9e)ZFGd:d;e+ZGed<=Gd>d?eFZHGd@dAeFZIGdBdCeeFZJgdDZKdS)EzPyTorch AltCLIP model.N)strict)initialization)create_bidirectional_mask)BaseModelOutputWithPooling'BaseModelOutputWithPoolingAndProjection)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)merge_with_config_defaults)capture_outputs)ChineseCLIPModelChineseCLIPTextAttentionChineseCLIPTextLayerChineseCLIPTextSelfAttention) CLIPConfigCLIPTextConfigCLIPVisionConfig) CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayer CLIPOutputCLIPPreTrainedModelCLIPVisionEmbeddingsCLIPVisionModel_get_vector_normimage_text_contrastive_loss)RobertaEmbeddingsRobertaIntermediate RobertaOutput RobertaPoolerRobertaSelfOutputz BAAI/AltCLIP) checkpointc,eZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d Z e ed <dZ eezed<dZeezed<dZeed<dZeed<dZeed<dZedzed<dZedzed<dZedzed<dZeed<eZeZdS)AltCLIPTextConfigaR project_dim (`int`, *optional*, defaults to 768): The dimensions of the teacher model before the mapping layer. Examples: ```python >>> from transformers import AltCLIPTextModel, AltCLIPTextConfig >>> # Initializing a AltCLIPTextConfig with BAAI/AltCLIP style configuration >>> configuration = AltCLIPTextConfig() >>> # Initializing a AltCLIPTextModel (with random weights) from the BAAI/AltCLIP style configuration >>> model = AltCLIPTextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```i vocab_sizei hidden_sizenum_hidden_layersnum_attention_headsiintermediate_sizegelu hidden_actg?hidden_dropout_probrattention_probs_dropout_probimax_position_embeddingstype_vocab_sizeg{Gz?initializer_factorN pad_token_id bos_token_idr eos_token_id project_dim)__name__ __module__ __qualname____doc__r*int__annotations__r+r-r/r0r2strr3floatr4r5r7r8r9r:r;r=AttributeErrorprojection_dimattention_dropouti/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/altclip/modular_altclip.pyr)r):sD  &JKs!!!!!s!!!J'*u***01 #+111#&S&&&OS $$$$ L#*    L#*    L#*   K#^%%N&((rJr)ceZdZdZdS)AltCLIPVisionConfiga Example: ```python >>> from transformers import AltCLIPVisionConfig, AltCLIPVisionModel >>> # Initializing a AltCLIPVisionConfig with BAAI/AltCLIP style configuration >>> configuration = AltCLIPVisionConfig() >>> # Initializing a AltCLIPVisionModel (with random weights) from the BAAI/AltCLIP style configuration >>> model = AltCLIPVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```N)r>r?r@rArIrJrKrMrMcs    rJrMc"eZdZUdZdZeed<dS) AltCLIPConfiga Example: ```python >>> from transformers import AltCLIPConfig, AltCLIPModel >>> # Initializing a AltCLIPConfig with BAAI/AltCLIP style configuration >>> configuration = AltCLIPConfig() >>> # Initializing a AltCLIPModel (with random weights) from the BAAI/AltCLIP style configuration >>> model = AltCLIPModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> # We can also initialize a AltCLIPConfig from a AltCLIPTextConfig and a AltCLIPVisionConfig >>> # Initializing a AltCLIPText and AltCLIPVision configuration >>> config_text = AltCLIPTextConfig() >>> config_vision = AltCLIPVisionConfig() >>> config = AltCLIPConfig(text_config=config_text, vision_config=config_vision) ```r<rGN)r>r?r@rArGrBrCrIrJrKrOrOws.  0NCrJrOceZdZdS) AltCLIPOutputNr>r?r@rIrJrKrQrQDrJrQceZdZdS)AltRobertaEmbeddingsNrRrIrJrKrUrUrSrJrUceZdZfdZxZS)AltRobertaSelfAttentioncXt|d|_dS)NF)super__init__ is_causalselfconfig __class__s rKrZz AltRobertaSelfAttention.__init__s&    rJr>r?r@rZ __classcell__r_s@rKrWrWs8rJrWceZdZdS)AltRobertaSelfOutputNrRrIrJrKrdrdrSrJrdceZdZfdZxZS)AltRobertaAttentionctt||_t ||_dSN)rYrZrWr]rdoutputr\s rKrZzAltRobertaAttention.__init__s; +F33 *622 rJr`rbs@rKrfrfs8333333333rJrfceZdZdS)AltRobertaIntermediateNrRrIrJrKrkrkrSrJrkceZdZdS)AltRobertaOutputNrRrIrJrKrmrmrSrJrmceZdZfdZxZS)AltRobertaLayerctt||_t ||_t ||_dSrh)rYrZrf attentionrk intermediatermrir\s rKrZzAltRobertaLayer.__init__sK ,V4426::&v.. rJr`rbs@rKroros8/////////rJroc(eZdZdZdeffd ZxZS)AltRobertaEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`AltRobertaEncoderLayer`]. Args: config: AltCLIPTextConfig r^cttjfdt jD|_dS)Nc.g|]}tSrI)ro).0_r^s rK z.AltRobertaEncoder.__init__..s!$f$f$f_V%<%<$f$f$frJ)rYrZnn ModuleListranger-layersr\s `rKrZzAltRobertaEncoder.__init__sQ    m$f$f$f$feFLdFeFe$f$f$fgg rJ)r>r?r@rAr)rZrarbs@rKrtrtsZh0hhhhhhhhhhrJrtceZdZdS)AltRobertaPoolerNrRrIrJrKrrrSrJrceZdZdS)AltCLIPAttentionNrRrIrJrKrrrSrJrceZdZdS) AltCLIPMLPNrRrIrJrKrrrSrJrc$eZdZdeffd ZxZS)AltCLIPEncoderLayerr^cJt|dSrh)rYrZr\s rKrZzAltCLIPEncoderLayer.__init__s!      rJ)r>r?r@rMrZrarbs@rKrrsE!2!!!!!!!!!!rJrceZdZdS)AltCLIPEncoderNrRrIrJrKrrrSrJrceZdZdS)AltCLIPVisionEmbeddingsNrRrIrJrKrrrSrJrcPeZdZgdZeedZejdZ dS)AltCLIPPreTrainedModel)rUrorr hidden_states attentionsctj|||jj}t |t rt j|jd|j dz|zt j|j j |jj |zt j|j j |jj |zt j|jt!j|jddSt |t(r|j dzd|jjzdzz|z}|j dz|z}t j|jj |t j|jj |t j|jj |t j|jj |dSt |t4r||jjdzd|jjzdzz|z}d|jjzdz|z}t j|jj |t j|jj |dSt |t<rXt j|jj |j dz|zt j|j!j |j"dz|zdSt |tFrjt j|jt!j|jj$ddt j%|j&dSdS) zInitialize the weightsgg)meanstd)r)r6rrN)'r _init_weightsr^r8 isinstancerinitnormal_class_embedding embed_dimpatch_embeddingweightinitializer_rangeposition_embeddingcopy_ position_idstorcharange num_positionsexpandrr-q_projk_projv_projout_projrr+fc1fc2 AltCLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrUshapezeros_token_type_ids)r]modulefactor in_proj_std out_proj_stdfc_stds rKrz$AltCLIPPreTrainedModel._init_weightss %dF333/ f5 6 6 / L/cv?OQU?UX^?^ _ _ _ _ L/6FM<[^dr?r@_no_split_modulesrr_can_record_outputsrno_gradrrIrJrKrrsVuuu,& U]__ / /_ / / /rJrceZdZfdZxZS)AltCLIPVisionModelc 6tjdi|S)a Examples: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, AltCLIPVisionModel >>> model = AltCLIPVisionModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rI)rYforwardr] super_kwargsr_s rKrzAltCLIPVisionModel.forwards!.uww.....rJ)r>r?r@rrarbs@rKrrs8/////////rJraE The model behaves as an encoder following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 ) custom_introceZdZUeed<dZdZeedZ dfd Z e e e ddejdzd ejdzd ejdzd ejdzd ejdzd eedeezfdZxZS)AltRobertaModelr^textword_embeddingsrTct|t||_t ||_|rt |nd|_|dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rYrZrU embeddingsrtencoderrpooler post_init)r]r^add_pooling_layerr_s rKrZzAltRobertaModel.__init__>sk    .v66(00 2CM&v...  rJN input_idsattention_maskrr inputs_embedskwargsreturnc |du|duz rtd|||||}t|j||}|j|fd|i|}|d}|j||nd} t || S)aK Examples: ```python >>> from transformers import AutoTokenizer, AltRobertaModel >>> model = AltRobertaModel.from_pretrained("openai/alt_roberta-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/alt_roberta-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```Nz:You must specify exactly one of input_ids or inputs_embeds)rrrr)r^rrrr)last_hidden_state pooler_output) ValueErrorrrr^rrr) r]rrrrrrencoder_outputssequence_output pooled_outputs rKrzAltRobertaModel.forwardJs6  -t"; < [YZZ Z%)' (  3;')    '$,   )    *!,8< 8O O444UY )-'    rJ)TNNNNN)r>r?r@r)rCinput_modalities_input_embed_layerrorWrrZrrr rTensorr r tuplerrrarbs@rKrr+s*  *(-       *..2.2,0-1 3 3 <$&3  t+3  t+ 3 lT) 3 |d* 3 +,3  + +3 3 3 ^_ 3 3 3 3 3 rJrceZdZUeed<dZdZdZfdZe e dde j dzde j dzd e j dzd e j dzd e j dzd e ed eezfdZxZS)AltCLIPTextModelr^rrrobertac0t|t|d|_t j|j|j|_t j |j|j |_ | dS)NF)r)eps) rYrZrrrzLinearr+r=transformation LayerNormlayer_norm_epspre_LNrr\s rKrZzAltCLIPTextModel.__init__s{    &vGGG  i(:F>> from transformers import AutoProcessor, AltCLIPTextModel >>> model = AltCLIPTextModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> texts = ["it's a cat", "it's a dog"] >>> inputs = processor(text=texts, padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```)rrrrrrN)rrrrrI)rrrrrr) r]rrrrrroutputsrprojection_staters rKrzAltCLIPTextModel.forwards:$, ))%'      "!*++o66 ..??(A. 6.'!/)     rJr)r>r?r@r)rCrrbase_model_prefixrZr r rrr r rrrrarbs@rKrrs   *!*..2.2,0-1 3 3 <$&3  t+3  t+ 3 lT) 3 |d* 3 +,3  8 83 3 3 ^3 3 3 3 3 rJrceZdZUeed<deffd ZfdZfdZ ddej dzdej dzd ej dzd ej dzd ej dzd e dzd e de edeezfdZxZS)rr^ct||j}|j|_t |jj|_t|jj |_ dSrh) rYrZ text_configr=rr _from_configr^ text_modelr vision_config vision_model)r]r^rr_s rKrZzAltCLIPModel.__init__si    ( )5*77 8OPP.;;DK>> import torch >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```rI)rYget_text_featuresrs rKrzAltCLIPModel.get_text_featuress")uww(88<888rJc 6tjdi|S)ao Examples: ```python >>> import torch >>> from transformers import AutoProcessor, AltCLIPModel >>> from transformers.image_utils import load_image >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```rI)rYget_image_featuresrs rKrzAltCLIPModel.get_image_featuress"&*uww)99L999rJNFr pixel_valuesrrr return_lossinterpolate_pos_encodingrrc b|jd||d|} |jd||||d|} | d} || } | d} || } | t | z } | t | z } t j| | | j } | |j | j z} | }d}|rt| }t||| | | | | S)u return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, AltCLIPModel >>> from transformers.image_utils import load_image >>> model = AltCLIPModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16") >>> processor = AutoProcessor.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16") >>> url = "https://clip-cn-beijing.oss-cn-beijing.aliyuncs.com/pokemon.jpeg" >>> image = load_image(url) >>> inputs = processor(text=["杰尼龟", "妙蛙种子", "小火龙", "皮卡丘"], images=image, return_tensors="pt", padding=True) >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```)rr)rrrrr6N)losslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrI)rrrrr rmatmulttodevice logit_scaleexpr!rQ)r]rrrrrrrrvision_outputs text_outputsrrrrrs rKrzAltCLIPModel.forwards|F+* %%=     't ))%      &a( --l;; "1o **;77 $&6|&D&DD !$4[$A$AA  ,{LNN4D4D4G4G HZ4[4[\\)D,<,@,@,B,B,E,EkFX,Y,YY*,,..  @.??D-+#%* .    rJ)NNNNNNF)r>r?r@rOrCrZrrr LongTensor FloatTensorrboolr r rrQrrarbs@rKrrsZ W}WWWWWW99999":::::..215.2.204#').L L #d*L '$.L  t+ L  t+ L &- L D[L #'L +,L   L L L L L L L L rJr)rrrrr)rMrO)LrArtorch.nnrzhuggingface_hub.dataclassesrrr masking_utilsrmodeling_outputsrrmodeling_utilsr processing_utilsr utilsr r r utils.genericrutils.output_capturingr"chinese_clip.modeling_chinese_cliprrrrclip.configuration_cliprrrclip.modeling_cliprrrrrrrrr r!roberta.modeling_robertar"r#r$r%r&r)rMrOrQrUrWrdrfrkrmrortrrrrrrrrrrr__all__rIrJrKrs ......&&&&&&666666.-----&&&&&&IIIIIIIIII777777555555 SRRRRRRRRR                        >***$)$)$)$)$)$)$)+*$)N>***     *  +* $>***J+*8     J        ,   :      ,   333332333     0        }   /////*/// h h h h h h h h     }        }           !!!!!*!!!      [        2   (/(/(/(/(/0(/(/(/V////////6L L L L L ,L L L ^B B B B B -B B B J| | | | | #%;| | | ~   rJ