tj dZddlmZddlmZddlZddlmZddlmZddl m Z dd l m Z dd lmZdd lmZdd lmZmZmZdd lmZddlmZddlmZmZmZmZmZm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ej,e-Z.dej/dej/fdZ0dej/dej/fdZ1edeGddeZ2edeGddeZ3ed eGd!d"eZ4eeGd#d$eZ5Gd%d&ej6Z7Gd'd(ej6Z8Gd)d*ej6Z9Gd+d,ej6Z:Gd-d.eZ;eGd/d0eZ<Gd1d2ej6Z=Gd3d4e<Z>ed5Gd6d7e<Z?ed8Gd9d:eGd?d@e<ZBgdAZCdS)BzPyTorch BLIP model.) dataclass)AnyN)nn) normalize)initialization)ACT2FN)GenerationMixin)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)merge_with_config_defaults)capture_outputs) BlipConfigBlipTextConfigBlipVisionConfig)BlipTextLMHeadModel BlipTextModellogitsreturnctj|tjt ||jS)Ndevice)r functional cross_entropytorcharangelenr#)rs d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/blip/modeling_blip.pycontrastive_lossr*-s3 = & &vu|CKKPVP]/^/^/^ _ __ similaritycXt|}t|j}||zdz S)Ng@)r*T)r, caption_loss image_losss r)image_text_contrastive_lossr12s.#J//L!*,//J : % ,,r+z Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder. ) custom_introceZdZUdZdZeejdzed<dZ eejdzed<dZ ejdzed<dZ ejdzed<dZ eejdfdzed<dZ eejdfdzed <dS) 'BlipForConditionalGenerationModelOutputa+ loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the text decoder. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`, *optional*): Prediction scores of the language modeling head of the text decoder model. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*): The image embeddings obtained after applying the Vision Transformer model to the input image. Nlossr image_embedslast_hidden_state. hidden_states attentions)__name__ __module__ __qualname____doc__r5tupler& FloatTensor__annotations__rr6r7r8r9r+r)r4r48s-1D%! "T )000.2FE%# $t +222-1L%#d*11126u(4/666:>M5*C/047>>>7;Je',-4;;;;;r+r4ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdfdzed<dZ e ejdfdzed<dS) BlipTextVisionModelOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss from the text decoder. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. Nr5r6r7.r8r9) r:r;r<r=r5r&r?r@r6r7r8r>r9rAr+r)rCrCQs&*D% d ")))-1L%#d*11126u(4/666:>M5*C/047>>>7;Je',-4;;;;;r+rCz Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder as well as the image-text similarity scores. c0eZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ ejdzed<dZ e ejdfdzed<dZ ejdzed <dZe ejdfdzed <dZe ejdzed <dS) BlipImageTextMatchingModelOutputa itm_score (`torch.FloatTensor`): The image-text similarity scores. loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss from the text decoder. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. vision_pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, *optional*): Last layer hidden-state of the vision of the vision-only branch of the model. question_embeds (`torch.FloatTensor`): The question embeddings obtained by the text projection layer. N itm_scorer5r6r7.r8vision_pooler_outputr9question_embeds)r:r;r<r=rFr&r?r@r5r6r7r8r>rGr9rHrAr+r)rErEgs  +/Iu 4'...%)D% d ")))-1L%#d*11126u(4/666:>M5*C/047>>>59%+d29997;Je',-4;;;7;OU5,-4;;;;;r+rEceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ ejdzed<dZ ejdzed<dZ e ed<dZe ed <d eefd ZdS) BlipOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`BlipTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`BlipVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`BlipTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`BlipVisionModel`]. Nr5logits_per_imagelogits_per_text text_embedsr6text_model_outputvision_model_outputr c^tfdDS)Nc3tK|]2}|dvr|n!t|V3dS))rNrON)getattrto_tuple).0kselfs r) z&BlipOutput.to_tuple..sc   LLLDGGRYZ^`aRbRbRkRkRmRm      r+)r>keysrVs`r)rSzBlipOutput.to_tuplesC    YY[[      r+)r:r;r<r=r5r&r?r@rKrLrMr6rNr rOr>rrSrAr+r)rJrJs&&*D% d ")))15e'$.55504OU&-444,0K"T)000-1L%#d*1114818886:3::: %*      r+rJczeZdZdeffd ZdejdededejfdZd d ej d e dejfd Z xZ S) BlipVisionEmbeddingsconfigct||_|j|_|j|_|j|_tjtj dd|j|_ tj d|j|j|j|_ |j|jzdz|_|jdz|_tjtj d|j|j|_dS)Nrr) in_channels out_channels kernel_sizestride)super__init__r\ hidden_size embed_dim image_size patch_sizer Parameterr&randnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingrVr\ __class__s r)rdzBlipVisionEmbeddings.__init__s  + + +!|EK1dn,M,MNN!yDO\`\k   !Ot>1D!-1"$,u{1d>PRVR`/a/a"b"br+ embeddingsheightwidthr c|jddz }|jjddz }tjs||kr ||kr|jS|jddddf}|jddddf}|jd}||jz} ||jz} t |dz} |d| | |}|dddd}tj || | fdd }|dddd dd|}tj ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrrbbicubicF)sizemode align_cornersdim)shaperpr&jit is_tracingrhrreshapepermuterr$ interpolateviewcat) rVrsrtrurnroclass_pos_embedpatch_pos_embedr} new_height new_widthsqrt_num_positionss r)interpolate_pos_encodingz-BlipVisionEmbeddings.interpolate_pos_encodingsr!&q)A- /5a81< y##%% ++*F*F6UZ??* *1!!!RaR%81!!!QRR%8r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr+F pixel_valuesrc(|j\}}}}|jjj}|||}|ddd}|j|dd|} tj | |gd} |r| | ||} n|j } | | ddd| dddf|z} | S)Ndtyperbrrwr|)r~rmweightrtoflatten transposerkexpandr&rrrpry) rVrr batch_size_rtru target_dtype patch_embeds class_embedsrsrps r)forwardzBlipVisionEmbeddings.forwards'3'9$ Avu+28 ++LOO,O,O,OPP #++A..88A>> +22:q"EEHHVV Y l;CCC # 9!%!>!>z6SX!Y!Y  !%!8 "4QQQ8L*//!:L:L8Laaa5O"P"S"ST`"a"aa r+)F) r:r;r<rrdr&Tensorintrr?boolr __classcell__rrs@r)r[r[sc/cccccc$&D5<&D&DUX&D]b]i&D&D&D&DP  E$5 QU bgbn        r+r[c ~eZdZdeffd Z d dejdzdejdzdejdzdejfdZ xZ S) BlipTextEmbeddingsr\cVt|j}tj|j||_tj|j||_| dtj |j dddS)N position_idsrrwF) persistent) rcrdrer Embedding vocab_sizetoken_embeddingmax_position_embeddingsrpregister_bufferr&r'rrVr\rfrrs r)rdzBlipTextEmbeddings.__init__s & !|F,=yII"$,v/My"Y"Y  EL)GHHOOPWXXej      r+N input_idsr inputs_embedsr c.| |jdn |jd}|jjjd}||krtd|d|||jddd|f}|||}||}||z}|S)NrwrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )r~rpr ValueErrorrr)rVrrr seq_lengthmax_position_embeddingposition_embeddingsrss r)rzBlipTextEmbeddings.forwards -6,AY_R((}GZ[]G^ !%!8!?!Ea!H . . .VVV=SVV   ,QQQ  ^ )T^ ; ;'dn''N''' ]D( z&":;; 9T^Q-?@@)DNDNCCr+tensorseq_lenbszc||||j|jddS)Nrrb)rrrr contiguous)rVrrrs r)_shapezBlipAttention._shape4s<{{3GGQQRSUVWWbbdddr+r8kwargsr c |\}}}||||d|j||jzddddd}|d|d|d} }}t j||dd} | |jz} tj | d} | | } t j| | dddd} | d d|j fz} | | } || }|| fS) z#Input shape: Batch x Time x Channelrrbrrrwrr|N)ryrrrrr&matmulrrrr$softmaxrrfr)rVr8rrtgt_lenrf mixed_qkv query_states key_states value_statesattention_scoresattention_probs context_layernew_context_layer_shapeoutputs r)rzBlipAttention.forward7s^#0"4"4"6"6Wi HH] # # WS'1dni4>6Q R R WQ1a # #  2;1y|YWX\,j !< j6J6J2r6R6RSS+dj8-//0@b/II,,77 _lCCKKAqRSUVWW "/"4"4"6"6ss";t~>O"O%--.EFF //&&r+)r:r;r<r=rdr&rrrrrr>rrrs@r)rrsGGDDDDD$eU\eCeceeee#'|#'+,#' u|U\) * #'#'#'#'#'#'#'#'r+rcBeZdZfdZdejdejfdZxZS)BlipMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSN) rcrdr\r hidden_act activation_fnrrreintermediate_sizefc1fc2rqs r)rdzBlipMLP.__init___sf  #F$569V/1IJJ9V5v7IJJr+r8r c||}||}||}|Sr)rrr)rVr8s r)rzBlipMLP.forwardfs=// **=99 // r+)r:r;r<rdr&rrrrs@r)rr^scKKKKKU\elr+rcheZdZdeffd Zedejdee dej fdZ xZ S)BlipEncoderLayerr\cDt|j|_t ||_t j|j|j|_ t||_ t j|j|j|_ dSN)eps) rcrdrerfr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rqs r)rdzBlipEncoderLayer.__init__ns} +&v..<F  '    q&0  ((77 // %0 r+) r:r;r<rrdrr&rrrr?rrrs@r)rrmsSzSSSSSS|+,   ^r+rcneZdZUeed<dZdZdZddgZdgZ e j fdZ xZ S) BlipPreTrainedModelr\blip)imagetextTrrpast_key_valuesct||jj}t |t r`t |jdr|jjj}tj |j d|tj |j d|dSt |trQtj |jtj|jjdddSdS)zInitialize the weights vision_configg)meanstdrwrN)rc _init_weightsr\initializer_range isinstancer[hasattrrinit trunc_normal_rprkrcopy_rr&r'r~r)rVmodulerrrs r)rz!BlipPreTrainedModel._init_weightss f%%%k+ f2 3 3 it{O44 Bk/A  v8s L L L L  v5CS I I I I I I  2 3 3 i Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h h h i ir+)r:r;r<rr@base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placementr&no_gradrrrs@r)rrs (&*#+-AB#4"5U]__ i i i i_ i i i i ir+rcZeZdZdZdeffd Zedeede e zfdZ xZ S) BlipEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`BlipEncoderLayer`]. Args: config (`BlipConfig`): The corresponding vision configuration for the `BlipEncoder`. r\ct|_tjfdt jD|_d|_dS)Nc.g|]}tSrA)r)rTrr\s r) z(BlipEncoder.__init__..s"$g$g$g!%5f%=%=$g$g$gr+F) rcrdr\r ModuleListrangenum_hidden_layerslayersgradient_checkpointingrqs `r)rdzBlipEncoder.__init__s`  m$g$g$g$guVMeGfGf$g$g$ghh &+###r+rr c L|}|jD] }||fi|} t|S)N)r7)rr )rVrrr8 encoder_layers r)rzBlipEncoder.forwardsP & ![  M)MMM ????r+) r:r;r<r=rrdrrrr>r rrrs@r)rrs,z,,,,,,  @+, @  @ @ @^ @ @ @ @ @r+rc eZdZUdZdZeed<eedZ deffd Z e e de ddejdzd ed eed eezfd Zd ZxZS)BlipVisionModelr)rr\)r8r9ct|||_|j}t ||_t ||_tj ||j |_ | dSr) rcrdr\rer[rsrencoderrrrpost_layernorm post_initrs r)rdzBlipVisionModel.__init__sx     & .v66"6**  l9&:OPPP r+F)tie_last_hidden_statesNrrr c |td|||}|jdd|i|}|j}||}|dddddf}||}t ||S)Nz You have to specify pixel_values)rrr)r7 pooler_outputrA)rrsrr7rr )rVrrrr8encoder_outputsr7 pooled_outputs r)rzBlipVisionModel.forwards  ?@@ @ Oghh +74<, , ', , ,  ,= //0ABB)!!!Q'2 ++M:: )/'    r+c|jSr)rsrYs r)get_input_embeddingsz$BlipVisionModel.get_input_embeddingss r+NF)r:r;r<main_input_namerrr@rr_can_record_outputsrdrrrr&r?rrrr>r rrrrs@r)rrs $O! )# /       _E22226).  '$. #' +,  + +    ^32  6r+rz This model is going to be deprecated in future versions. Please use `BlipForConditionalGeneration`, `BlipForQuestionAnswering` or `BlipForImageTextRetrieval` depending on your usecase. c@eZdZUeed<deffd ZdZdZee dde j dzde j dzde j dzd e e d eezf d Zee dd e jdzded e e d eezfdZe dde jdzd e jdzde j dzded e jf dZee dde jdzd e jdzde j dzde jdzdedzded e e d eezfdZxZS) BlipModelr\cXt|t|jts%t dt |jdt|jts%t dt |jd|j}|j}|j |_ |j |_ |j |_ t||_t||_t#j|j |j d|_t#j|j |j d|_t#jt-j|jj|_t6d|dS)NzKconfig.text_config is expected to be of type BlipTextConfig but is of type .zOconfig.vision_config is expected to be of type BlipVisionConfig but is of type F)biasz`BlipModel` is going to be deprecated in future release, please use `BlipForConditionalGeneration`, `BlipForQuestionAnswering` or `BlipForImageTextRetrieval` depending on your usecase.)rcrdr text_configr TypeErrortyperrprojection_dimretext_embed_dimvision_embed_dimr text_modelr vision_modelrrvisual_projectiontext_projectionrir&rr\logit_scale_init_value logit_scaleloggerwarningr)rVr\r(rrrs r)rdzBlipModel.__init__s    &,n== 0+,,000  &.0@AA 2-..222  ( , $3)5 - 9' 44+M::!#4+@$BU\a!b!b!b!y)QX]^^^< T[5W(X(XYY G   r+c4|jSr)r.rrYs r)rzBlipModel.get_input_embeddings&s33555r+c:|j|dSr)r.set_input_embeddingsrVvalues r)r8zBlipModel.set_input_embeddings)s ,,U33333r+Nrattention_maskrrr c l|jd|||dd|}|j}|||_|S)a Examples: ```python >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```T)rr;r return_dictrA)r.rr1)rVrr;rr text_outputsrs r)get_text_featureszBlipModel.get_text_features,sd*FUT_F )% F F  F F %2 %)%9%9-%H%H "r+Frrc j|jd||dd|}|j}|||_|S)a Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```T)rrr=rA)r/rr0)rVrrrvision_outputsrs r)get_image_featureszBlipModel.get_image_featuresMs\:6GT5F6 %%=6 6  6 6  '4 '+'='=m'L'L$r+c$|||}|d}tj|ddtj}|||||}|d} || } | S)a Returns: multimodal_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The multimodal embeddings obtained by applying the image embeddings to the text encoder using the cross-attention mechanism. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> texts = ["a photo of a cat", "a photo of a dog"] >>> inputs = processor(images=image, text=texts, padding=True, return_tensors="pt") >>> multimodal_features = model.get_multimodal_features(**inputs) ```rrrNrwrrr;encoder_hidden_statesencoder_attention_maskr)r/r&onesrylongr.r1) rVrrr;rrAr6 image_attsr>rmultimodal_featuress r)get_multimodal_featuresz!BlipModel.get_multimodal_featuresus>**%%=+   &a( Z 1 1 3 3CRC 8 KKK )".#- '  %Q "22=AA""r+ return_lossc |jd ||d|}|jd |||d|} |j} || } | j} || } | | dddz } | | dddz } |j| j } | | j | j } tj | | | z} | }d }|rt| }t||| | | | | S) a@ return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```rD)rr;rrbrwT)pr}keepdimr")r#rN)r5rKrLrMr6rNrOrA)r/r.rr0r1normr3exprr#rr&rtr1rJ)rVrrr;rrMrrrAr>r6rMr3rLrKr5s r)rzBlipModel.forwardsN+* %%=     't )%     &3 --l;; "0 **;77 $l&7&7!T&7&R&RR !K$4$4qb$$4$O$OO &**,,//{7I/JJ #k.@ HYZZ ,{LNN4D4DEE S*,,..  @.??D-+#%* .    r+rr )NNNF)NNNNNF)r:r;r<rr@rdrr8rrr&rrrr>r r?r?rrBrrLrJrrrs@r)r$r$s "z""""""H666444*..2,0 <$& t+lT)  +,  + + ^>26).$$'$.$#'$+, $ + + $$$^$L.215.2). 0#0##d*0#'$.0# t+ 0# #' 0#   0#0#0#^0#d.215.204#').N N #d*N '$.N  t+ N &- N D[ N #'N +,N   N N N ^N N N N N r+r$a BLIP Model for image captioning. The model consists of a vision encoder and a text decoder. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the text decoder continue the prompt. Otherwise, the decoder starts generating text from the [BOS] (beginning-of-sequence) token. will start generating the caption from the text input. If no text input is provided, the decoder will start with the [BOS] token only. c|eZdZUeed<dZdddZdeffd ZdZdZ e e dde j d e jd zd e jd zde jd zdedee jzdeedeezfdZe j dde j d e jd zd e jd zdede jf dZxZS)BlipForConditionalGenerationr\r!text_decoder.cls.predictions.bias3text_decoder.bert.embeddings.word_embeddings.weightz)text_decoder.cls.predictions.decoder.biasz+text_decoder.cls.predictions.decoder.weightct|t|j|_t |j|_|jj|_ |jj |_ | dSr) rcrdrrr/rr( text_decoder bos_token_iddecoder_input_ids pad_token_iddecoder_pad_token_idrrqs r)rdz%BlipForConditionalGeneration.__init__ su    +F,@AA/0BCC!'!3!@$*$6$C! r+c4|jSr)rZrrYs r)rz1BlipForConditionalGeneration.get_input_embeddings 55777r+c:|j|dSr)rZr8r9s r)r8z1BlipForConditionalGeneration.set_input_embeddings ..u55555r+NFrrr;labelsrlogits_to_keeprr c |jd||d|}|j} |jd||| |d|d|} t| j| j| |j|j|jS)a Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "A picture of" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) ```rDr)rr;rFrc reductionrd)r5rr6r7r8r9rA)r/r7rZr4r5rr8r9) rVrrr;rcrrdrrAr6outputss r)rz$BlipForConditionalGeneration.forwardsD+* %%=     &7 #$# )".)     7>%,>(6%0     r+c |jd}|||}|d}tj|ddtj|j} t|trtj |}nY|Wtj |j |j j j gg|d|j}|j j j|dddf<||ddddfnd}|jjd|ddddf|j j j|j j j||| d|} | S) a Overrides *generate* function to be able to use the model as a conditional generator Parameters: pixel_values (*torch.FloatTensor* of shape *(batch_size, num_channels, image_height, image_width)*: Input image to be processed input_ids (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): The sequence used as a prompt for the generation. attention_mask (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipForConditionalGeneration >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) two cats sleeping on a couch ``` rrDNrwrr#r)r eos_token_idr]r;rFrGrA)r~r/r&rHryrIr#rlistrr\r\r(rjrepeatrr[rZgenerate sep_token_idr]) rVrrr;rgenerate_kwargsrrAr6image_attention_maskrgs r)rmz%BlipForConditionalGeneration.generate[sV"'* **%%=+   &a( $z,*;*;*=*=crc*B%*]i]pqqq i & & (33II   4#94;;R;_"`!abb A&&L'((  +1> !!!Q$3A3M3B3//SW,$#, 3B3'0=0=)".#7     r+)NNNFr)NNF)r:r;r<rr@r!_tied_weights_keysrdrr8rrr&r?rrrrrrr>r4rrrmrrs@r)rUrUs $O5X7l z      888666.226*.).-.9 9 '9 #d*9 (4/ 9  4' 9 #' 9 el*9 +,9  8 89 9 9 ^9 vU]__.226). JJ'J#d*J(4/ J #' J  JJJ_JJJJJr+rUaS BLIP Model for visual question answering. The model consists of a vision encoder, a text encoder as well as a text decoder. The vision encoder will encode the input image, the text encoder will encode the input question together with the encoding of the image, and the text decoder will output the answer to the question. c~eZdZUeed<dddZdeffd ZdZdZe e dd e j d e j d e j dzd e j dzde j dzde j dzdedeedeezfdZe j dd e j d e j de j dzdede j f dZxZS)BlipForQuestionAnsweringr\rVrWrXcPt|t|j|_t |jd|_t|j|_ |jj |_ |jj |_ |dS)NFadd_pooling_layer)rcrdrrr/rr( text_encoderrrZr]r^r[decoder_start_token_idrrqs r)rdz!BlipForQuestionAnswering.__init__s    +F,@AA)&*>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipForQuestionAnswering >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> # training >>> text = "How many cats are in the picture?" >>> label = "2" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> labels = processor(text=label, return_tensors="pt").input_ids >>> inputs["labels"] = labels >>> outputs = model(**inputs) >>> loss = outputs.loss >>> loss.backward() >>> # inference >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ```NaEither `decoder_input_ids` or `labels` should be passed when calling `forward` with `BlipForQuestionAnswering`. if you are training the model make sure that `labels` is passed, if you are using the model for inference make sure that `decoder_input_ids` is passed or call `generate`rDrwrrErr)rr;rFrGrcrf)r5r6r7r8r9rA)rr/r7r&rHryrIrwrZr5rrCr8r9)rVrrr\r}r;rcrrrAr6rprH answer_output decoder_losss r)rz BlipForQuestionAnswering.forwardsl^ >/7u  +* %%=     &7 $z,*;*;*=*=crc*B%*UUU+$+ )".#7        "3"; & )!,)) '1"1#1       (-2244LLL(%,>(6%0     r+c |||}|d}tj|ddtj|j}t |trtj|}| ||||d} | d} tj| ddtj| j} tj | ddf|j | j } |j j d | |jjj|jjj| | d |} | S) a Overrides *generate* function to be able to use the model as a conditional generator Parameters: input_ids (*torch.LongTensor* of shape *(batch_size, sequence_length)*): The sequence used as a prompt for the generation. pixel_values (*torch.FloatTensor* of shape *(batch_size, num_channels, image_height, image_width)*: Input image to be processed attention_mask (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`. `1` for tokens that are NOT MASKED, `0` for MASKED tokens. **generate_kwargs: Additional arguments passed to the *generate* function of the decoder Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipForQuestionAnswering >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ``` rDrNrwriF)rr;rFrGr=r) fill_valuer#)rrjr]rFrGrA)r/r&rHryrIr#rrkrrwfullrxrZrmr\r(rnr])rVrrr;rrorAr6rpquestion_outputsrHquestion_attention_maskbos_idsrgs r)rmz!BlipForQuestionAnswering.generate0s\**%%=+   &a( $z,*;*;*=*=crc*B%*]i]pqqq i & & 4(33I,,)".#7 -  +1-"'*  " "3B3 'uz/BX# # # *  ! !! $ $a (T5PYhYo   -$#, 0=0="1#:      r+)NNNNFr )r:r;r<rr@rqrdr8rrrr&rr?rrrr>rCrrrmrrs@r)rsrss 5X7l z      666888 6::>26*.).` ` #` '` !+d2 ` !& 04 7 ` (4/ `  4'` #'` +,`  * *` ` ` ^` DU]__ 37). TT#T'T(4/ T #' T  TTT_TTTTTr+rsa BLIP Model with a vision and text projector, and a classification head on top. The model is used in the context of image-text retrieval. Given an image and a text, the model returns the probability of the text being relevant to the image. ceZdZUeed<deffd ZdZdZee dde j d e j d e dzd e j dzd e d eedeezfdZxZS)BlipForImageTextRetrievalr\cft|t|j|_t |jd|_tj |jj |j |_ tj |jj |j |_ tj |jj d|_t|ds |jjn|j|_t|ds |jjn|j|_|dS)NFrurbr^rx)rcrdrrr/rr(rwrrreimage_text_hidden_size vision_proj text_projitm_headrr]r^r[rxrrqs r)rdz"BlipForImageTextRetrieval.__init__s    +F,@AA)&*>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, BlipForImageTextRetrieval >>> model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "an image of a cat" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) ``` rDNrwrrEr)rr;r|)rFr7r8r9rHrA)r/r7r&rHryrIrwrrrrrSrEr8r9)rVrrrr;rrrAr6rJrHr image_feat text_feats r)rz!BlipForImageTextRetrieval.forwardsF+* %%=     &7 Z 1 1 3 3CRC 8 KKK  0/d/#-&2'1   O.?O]]?111a7#;<(6%0+     r+)TNF)r:r;r<rr@rdrr8rrr&rr?rrrr>rCrrrs@r)rrs z<888666 %)26). H H #H 'H Tk H (4/ H #' H +,H  * *H H H ^H H H H H r+r)r$rrUrsrrr)Dr= dataclassesrtypingrr&rtorch.nn.functionalrrr activationsr generationr modeling_layersr modeling_outputsr r rmodeling_utilsrprocessing_utilsrutilsrrrrrr utils.genericrutils.output_capturingrconfiguration_bliprrrmodeling_blip_textrr get_loggerr:r4rr*r1r4rCrErJModuler[rrrrrrrr$rUrsr__all__rAr+r)rsC!!!!!! ))))))&&&&&&!!!!!!))))))999999 .-----&&&&&&jjjjjjjjjjjjjjjj777777555555LLLLLLLLLLBBBBBBBB  H % %`U\`el```` -EL-U\----   <<<<