tj dZddlmZddlmZddlmZddlZddlmZddl m Z dd l m Z dd lmZdd lmZdd lmZdd lmZddlmZmZmZmZmZmZddlmZmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7e)j8e9Z:e'eGddeZ;e'deGdde%Z<Gdd ej=Z> dOd"ej=d#ej?d$ej?d%ej?d&ej?dzd'e@d(e@fd)ZAGd*d+ej=ZBGd,d-ej=ZCGd.d/eZDe'Gd0d1eZEGd2d3ej=ZFGd4d5eEZGGd6d7ej=ZHGd8d9ej=ZIGd:d;ej=ZJGd<d=ej=ZKGd>d?ej=ZLGd@dAeZMGdBdCej=ZNGdDdEej=ZOGdFdGeEZPe'dHGdIdJeEZQe'dKGdLdMeEeZRgdNZSdS)PzPyTorch InstructBLIP model.)Callable) dataclass)AnyN)nn)initialization)ACT2FN)GenerationMixin)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentionsCausalLMOutputWithPastSeq2SeqLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)merge_with_config_defaults)OutputRecordercapture_outputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigc<eZdZUdZdZedzed<dZedzed<dS)'BaseModelOutputWithVisionQformerOutputsz vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). Nvision_outputsqformer_outputs) __name__ __module__ __qualname____doc__r+r__annotations__r,rt/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/instructblip/modeling_instructblip.pyr*r*2sJ9=N.5<<zKInstructBlipForConditionalGenerationModelOutput.to_tuple..\sf  WWW GGq!!**,,      r3)tuplekeysrCs`r4r@z8InstructBlipForConditionalGenerationModelOutput.to_tuple[sE    YY[[       r3)r-r.r/r0r8rEtorch FloatTensorr1r9r+rr,rr:rrrr@r2r3r4r7r7@s  -1D%! "T )000.2FE%# $t +22281D!-1"$,u{1d>PRVR`/a/a"b"br3 embeddingsheightwidthr;c|jddz }|jjddz }tjs||kr ||kr|jS|jddddf}|jddddf}|jd}||jz} ||jz} t |dz} |d| | |}|dddd}tj || | fdd }|dddd dd|}tj ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r%Ng?rrr!bicubicF)sizemode align_cornersdim)shaper_rHjit is_tracingrWrreshapepermuter functional interpolateviewcat) rCrbrcrdr]r^class_pos_embedpatch_pos_embedrl new_height new_widthsqrt_num_positionss r4interpolate_pos_encodingz5InstructBlipVisionEmbeddings.interpolate_pos_encodingxsr!&q)A- /5a81< y##%% ++*F*F6UZ??* *1!!!RaR%81!!!QRR%8r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr3F pixel_valuesr{c(|j\}}}}|jjj}|||}|ddd}|j|dd|} tj | |gd} |r| | ||} n|j } | | ddd| dddf|z} | S)N)dtyper!r%rfrk)rmr\weightr~toflatten transposerZexpandrHrur{r_rh) rCr|r{ batch_size_rcrd target_dtype patch_embeds class_embedsrbr_s r4forwardz$InstructBlipVisionEmbeddings.forwards'3'9$ Avu+28 ++LOO,O,O,OPP #++A..88A>> +22:q"EEHHVV Y l;CCC # 9!%!>!>z6SX!Y!Y  !%!8 "4QQQ8L*//!:L:L8Laaa5O"P"S"ST`"a"aa r3F) r-r.r/r(rSrHTensorintr{rIboolr __classcell__ras@r4rKrKesc7cccccc$&D5<&D&DUX&D]b]i&D&D&D&DP  E$5 QU bgbn        r3rKmodulequerykeyvalueattention_maskscalingdropoutc ztj||dd|z}|||z}tj|d}tj|||j}tj||} | dd} | |fS)Nrfrk)ptrainingr%r!) rHmatmulrrrrsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs r4eager_attention_forwardrs<s}}R'<'<==GL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r3ceZdZdZfdZdejdedefdZdejde ejejd ze ejd zffd Z xZ S) InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' papercVt||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ d|_ |j |_ tj |jd|jzd|_|jrWtjt#j|j}tjt#j|j}nd}d}|It#j|t#j|d|f}tj||j_tj |j|j|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).Fr)bias) requires_grad)rRrSrLrTrUnum_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasrXrHzerosru zeros_liker projection)rCrLq_biasv_biasrras r4rSzInstructBlipAttention.__init__s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( !'!99T^Q-?eLLL ? \%+dn"="=>>F\%+dn"="=>>FFFF  y&%*:6QV*W*W*WY_!`aaHL22DHM)DNDNCCr3tensorseq_lenbszc||||j|jddS)Nr%r!)rtrrrr)rCrrrs r4_shapezInstructBlipAttention._shapes<{{3GGQQRSUVWWbbdddr3 hidden_statesr;Nc |\}}}||}|||d|j||jzddddd}|d|d|d} }}t j|jjt} | |||| fd|j sdn|j |j d|\} } | ||d  } || } | | fS) z#Input shape: Batch x Time x Channelrr!rr%Nr)rrrrf)rhrrprrqr get_interfacerL_attn_implementationrrrrrr) rCrrrtgt_lenrU mixed_qkv query_states key_states value_statesattention_interfacerrs r4rzInstructBlipAttention.forwards=#0"4"4"6"6WiHH]++ %%c7At~yTXTbGbcckk q!Q  2;1y|YWX\,j (?(M K ,.E) ) %8$7     %  #}HCC$2HJ % %  % % ! \"))#w;;FFHH ook22 L((r3) r-r.r/r0rSrHrrrrErrrs@r4rrsGGDDDDD>eU\eCeceeee")|") u|U\D0% 2E2LL M ")")")")")")")")r3rcBeZdZfdZdejdejfdZxZS)InstructBlipMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSN) rRrSrLr hidden_act activation_fnrrrTintermediate_sizefc1fc2r`s r4rSzInstructBlipMLP.__init__sf  #F$569V/1IJJ9V5v7IJJr3rr;c||}||}||}|Sr)rrrrCrs r4rzInstructBlipMLP.forwards=// **=99 // r3r-r.r/rSrHrrrrs@r4rrscKKKKKU\elr3rcheZdZdeffd Zedejdee dej fdZ xZ S)InstructBlipEncoderLayerrLcDt|j|_t ||_t j|j|j|_ t||_ t j|j|j|_ dSNeps) rRrSrTrUr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r`s r4rSz!InstructBlipEncoderLayer.__init__$s +.v66<F  '    q&0  ((77 // %0 r3) r-r.r/r&rSrrHrrrrIrrrs@r4rr#sS1SSSSSS|+,   ^r3rc|eZdZUeed<dZdZdZdZdZ dZ dZ dZ gdZ ejfdZxZS)InstructBlipPreTrainedModelrLblip)imagetextT)InstructBlipQFormerEmbeddingsrrInstructBlipQFormerLayer%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputc:t||jj}t |t r:t j|jd|t j|j d|dSt |ttfrt j |j dSt |trQt j|jt#j|jjdddSdS)zInitialize the weightsr)meanstdrfr%rfN)rR _init_weightsrLinitializer_range isinstancerKinit trunc_normal_r_rZ$InstructBlipForConditionalGenerationInstructBlipModelzeros_ query_tokensrcopy_ position_idsrHarangermr)rCrfactorras r4rz)InstructBlipPreTrainedModel._init_weightsYs  f%%%. f: ; ; i  v8s O O O O  v5CV L L L L L L !EGX Y Z Z i K+ , , , , ,  = > > i Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h h h i ir3)r-r.r/r&r1base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrHno_gradrrrs@r4rrCs (&*#"&N!U]__ i i i i_ i i i i ir3rcZeZdZdZdeffd Zedeede e zfdZ xZ S)InstructBlipEncodera Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rLct|_tjfdt jD|_d|_dS)Nc.g|]}tSr2)r)rArrLs r4 z0InstructBlipEncoder.__init__..us"$o$o$o!%=f%E%E$o$o$or3F) rRrSrLr ModuleListrangenum_hidden_layerslayersgradient_checkpointingr`s `r4rSzInstructBlipEncoder.__init__rsa  m$o$o$o$ouU[UmOnOn$o$o$opp &+###r3rr;c L|}|jD] }||fi|} t|S)Nlast_hidden_state)rr)rC inputs_embedsrr encoder_layers r4rzInstructBlipEncoder.forwardxsP & ![  M)MMM ????r3) r-r.r/r0r&rSrrrrErrrrs@r4rrhs,1,,,,,,  @+, @  @ @ @^ @ @ @ @ @r3rc eZdZUdZdZeed<eedZ deffd Z e e de ddejdzd ed eed eezfd Zd ZxZS)InstructBlipVisionModelr|)rrL)r attentionsct|||_|j}t ||_t ||_tj ||j |_ | dSr) rRrSrLrTrKrbrencoderrrrpost_layernorm post_init)rCrLrUras r4rSz InstructBlipVisionModel.__init__sx     & 6v>>*622  l9&:OPPP r3F)tie_last_hidden_statesNr{rr;c |td|||}|jdd|i|}|j}||}|dddddf}||}t ||S)Nz You have to specify pixel_values)r{rrr pooler_outputr2)rrbrrrr)rCr|r{rrencoder_outputsr pooled_outputs r4rzInstructBlipVisionModel.forwards  ?@@ @ Oghh +74<, , ', , ,  ,= //0ABB)!!!Q'2 ++M:: )/'    r3c|jSr)rbrGs r4get_input_embeddingsz,InstructBlipVisionModel.get_input_embeddingss r3NF)r-r.r/main_input_namerr(r1rr_can_record_outputsrSrr rrHrIrrrrErrr#rrs@r4rrs $O! $$$$1+ 7       _E22226).  '$. #' +,  + +    ^32  6r3rc@eZdZdfd Z ddeefdZxZS)rFct||_|j|jzdkr.t |dst d|j|jfz|j|_t|j|jz |_|j|jz|_ |jdz|_ d|_ |j |_ tj|j|j |_|rJtj|j|j |_tj|j|j |_dStj|j|j |_tj|j|j |_dS)Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rF)rRrSrLrTrhasattrrrattention_head_size all_head_sizerrattention_probs_dropout_probrrrrencoder_hidden_sizerrrCrLis_cross_attentionras r4rSz.InstructBlipQFormerMultiHeadAttention.__init__sZ    : :a ? ?PVXhHiHi ?^%v'ABC  $*#= #&v'9F#)#2#. CCbC$*BCC  +2N3NN*NM^)#2#.MMD4LMMjj//44\BBLLQPQRR HH^,,11(;;EEaKK jj0055h??II!QOO (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k)E;+STTTz&"<== r3r input_tensorr;c||}||}|||z}|Srr>rrrCrrAs r4rz%InstructBlipQFormerSelfOutput.forward@ =11  ]33 }|'CDD r3rrs@r4rri>>>>> U\RWR^r3rc eZdZd fd Z d dejdejdzdejdzdejdzdeed ejf d Z xZ S) InstructBlipQFormerAttentionFctt|||_t ||_dSr)rRrSr attentionroutputr/s r4rSz%InstructBlipQFormerAttention.__init__s> >vGYZZ3F;; r3Nrrr2r3rr;c \|jd||||d|\}}|||}|S)N)rrr2r3r2)rJrK) rCrrr2r3rrrattention_outputs r4rz$InstructBlipQFormerAttention.forwardsW( ')"7#9       Q ;;{MBBr3rr;) r-r.r/rSrHrrIrrrrrs@r4rHrHs<<<<<<48:>;?   | )D0  %047  !& 1D 8  +,           r3rHcBeZdZfdZdejdejfdZxZS)InstructBlipQFormerIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) rRrSrrrTrr>rrstrr intermediate_act_fnr`s r4rSz(InstructBlipQFormerIntermediate.__init__1sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r3rr;cZ||}||}|Sr)r>rRrs r4rz'InstructBlipQFormerIntermediate.forward9s, =11 00?? r3rrs@r4rOrO0s^99999U\elr3rOcPeZdZfdZdejdejdejfdZxZS)InstructBlipQFormerOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) rRrSrrrrTr>rrr?r@rr`s r4rSz"InstructBlipQFormerOutput.__init__Asf Yv79KLL f&8f>STTTz&"<== r3rrAr;c||}||}|||z}|SrrCrDs r4rz!InstructBlipQFormerOutput.forwardGrEr3rrs@r4rUrU@rFr3rUcLeZdZfdZ ddeefdZdZdZxZ S) rct|j|_d|_t ||_||_||jzdkrt |d|_d|_ nd|_ t||_ t||_ t||_t||_dS)Nr%rT)r0F)rRrSchunk_size_feed_forward seq_len_dimrHrJ layer_idxcross_attention_frequencycrossattentionhas_cross_attentionrO intermediaterUrKintermediate_query output_query)rCrLr\ras r4rSz!InstructBlipQFormerLayer.__init__Os '-'E$5f==" v7 71 < <">vZ^"_"_"_D '+D $ $',D $;FCC/77 "A&"I"I5f==r3Nrrc |j|fd|i|}|dkr|ddd|ddf}|jr#|td|j|f|||d|}t |j|j|j|} |jd|kr`t |j |j|j|dd|dddf | j } tj | | gd} n!t |j |j|j|} | S)Nrrz>encoder_hidden_states must be given for cross-attention layers)rr2r3r%rk)rJr_rr^rfeed_forward_chunk_queryrZr[rmfeed_forward_chunkrdevicerHru) rCrrr2r3 query_lengthrrMquery_attention_output layer_outputlayer_output_texts r4rz InstructBlipQFormerLayer.forwardcs*4>   )    !  %5aaa,6I%J "' (0$%efff)<)<**#1*?+A **  **&5-, & L %a(<77$=+0$$QQQ qqq%89 %% "\()) " %y,8I)JPQRRR 4',  L r3c\||}|||}|Sr)r`rKrCrMintermediate_outputris r4rez+InstructBlipQFormerLayer.feed_forward_chunks2"//0@AA{{#68HII r3c\||}|||}|Sr)rarbrls r4rdz1InstructBlipQFormerLayer.feed_forward_chunk_querys4"556FGG(()<>NOO r3NNNr) r-r.r/rSrrrrerdrrs@r4rrNs>>>>>."# 33+,3333j r3rcPeZdZfdZe ddeefdZxZS)InstructBlipQFormerEncoderct|_tjfdt jD|_d|_dS)Nc0g|]}t|Sr2)r)rAr\rLs r4r z7InstructBlipQFormerEncoder.__init__..s$ j j jY %fi 8 8 j j jr3F) rRrSrLrr r r layerrr`s `r4rSz#InstructBlipQFormerEncoder.__init__sg  ] j j j j%PVPhJiJi j j j   ',###r3Nrrc t|jjD]}|j|}||||f||d|}t |S)N)r3rgr)r rLr rtr) rCrrr2r3rgri layer_modules r4rz"InstructBlipQFormerEncoder.forwardst{455  A:a=L(L%(>)   MM9+    r3ro) r-r.r/rSrrrrrrs@r4rqrqs|,,,,,"#   +,        r3rqc2eZdZdZfdZ ddZxZS)rz;Construct the embeddings from word and position embeddings.cttj|j|j|j|_tj|j|j|_ tj |j|j |_ tj |j|_|dt#j|jdd||_dS)N) padding_idxrrrF) persistent)rRrSr Embedding vocab_sizerT pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsrr layernormr?r@rregister_bufferrHrrrLr`s r4rSz&InstructBlipQFormerEmbeddings.__init__s !|F,=v?Q_e_rsss#%<0NPVPb#c#c f&8f>STTTz&"<==   EL)GHHOOPWXXej     r3Nrc||d}nd}|(|jdd|||zf}|b||}|||j}||z}|tj||fd}n|}||j j j }| |}| |}|S)Nr%rrk) rhrclonerrrrfrHrurrr~r)rC input_idsr query_embedspast_key_values_length seq_lengthrbrs r4rz%InstructBlipQFormerEmbeddings.forwards  "))!,JJJ  ,QQQ0FVlIl0l-lmssuuL  --i88J"&":":#8#>?? ^^J// \\*-- r3ro)r-r.r/r0rSrrrs@r4rrs`EE     " r3rcjeZdZdZdZdZdZdZee e ddge e ddgdZ de ffd Z d Zd Zeee dd ejdejd zdejd zdejd zdejd zdejd zdeedeejezfdZxZS)InstructBlipQFormerModelz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. Tr%z .attention)index layer_namez.crossattention)rrcross_attentionsrLct|||_t||_t ||_|dSr)rRrSrLrrbrqrrr`s r4rSz!InstructBlipQFormerModel.__init__sV     7??1&99  r3c|jjSrrbrrGs r4r#z-InstructBlipQFormerModel.get_input_embeddingss ..r3c||j_dSrr)rCrs r4set_input_embeddingsz-InstructBlipQFormerModel.set_input_embeddingss*/'''r3Nrrrrr2r3rr;c ||td| |jdnd}||||} t|j| |}|1|dkr| ddd|ddfn| } t|j| ||}|j| f||||d|} | j} | dddddf} t| | S) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner%r)rrr)rLrr)rLrrr2)rr2r3rgr)rrmrbr rLrrr)rCrrrrr2r3rrgembedding_outputquery_embedding_outputr sequence_outputr!s r4rz InstructBlipQFormerModel.forwardsN$  !5VWW W0<0H|)!,,a ??%%+   3;*)    " -O[]^N^N^%5aaa,6I%J%Jdt "%>{45&; &&& ",84< , )"7#9% , ,  , , *;'1aaa0 ;-'    r3)NNNNN)r-r.r/r0rrrrrrrr&r'rSr#rrr rrH LongTensorrIrrrrErrrrs@r4rrs #'N2 N@Vb c c c  N@Vg h h h 8///000 4804,0:>;?9 9 #9 )D09 &- 9 lT) 9 %047 9 !& 1D 89 +,9  u !$P P9 9 9 ^_ 9 9 9 9 9 r3rz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. cBeZdZdZdgZdeffd ZdZdej dej fdZ e e ddej d ej d ej d zdej d zd ej d zdej d zdej d zdejd zdedeedeezfdZxZS)rr|rrLct|t|j|_t jtjd|j |j j |_ t|j |_t j|j j |jj |_t%j|j|_|dSNr%)rRrSr vision_config vision_modelrrXrHrnum_query_tokensqformer_configrTrrqformerr text_configlanguage_projectionr" from_configlanguage_modelrr`s r4rSzInstructBlipModel.__init__gs    3F4HIILQ8OQWQfQr)s)stt/0EFF #%9V-B-NPVPbPn#o#o '3F4FGG r3c |j}t|dkr@d|vrD  ( 7 7 7 ? ?r3rrcN|e||tj|jjtj|jk}|d}n||jjk}|d |j}|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. Nr~rfrf r#rHrrLimage_token_idlongrfall unsqueezerrCrrspecial_image_masks r4get_placeholder_maskz&InstructBlipModel.get_placeholder_mask  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H /99"==@@AUVV!!r3NFqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskr{rr;c D|jd || d| } | d} tj| ddtj| j} |j| jddd}tj|ddtj| j}|tj |}| |j}tj ||gd}|j d |||| | d| }|dddd|dddf}|8|j |}|tj |}||}| |j|j}||| }|||}|jjr|j d ||d | }n|j d ||||d | }t+| || S)a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. )r|r{rNrfrr%rk)rrrr2r3rrr)rrrrr>r2)rrHonesrhrrfrrrm ones_likerrurrr#rr~rmasked_scatterrLuse_decoder_only_language_modelr7)rCr|rrrrrrrr{rr+ image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputss r4rzInstructBlipModel.forwardsP+* %%=     &a(  %z,*;*;*=*=crc*B%*]i]pqqq(// 0B10Er2NN $z,*;*;*=*=crc*B%*]i]pqqq ! )%*_5F%G%G "!7!:!:;O;V!W!W!&,@BX+Y_`!a!a!a$  '1%".#7      %Q'+A\->->q-A-A+A111(DE  FD/DDFFyQQM%!&!;!;!% 8 8 F F 5 8 89M}Ob c c!66yP]6^^%445GI^__ ; 6 )d)+-GG *d)+-"3'=   G?))#*    r3)NNNNNNF)r-r.r/r%_keep_in_fp32_modulesr&rSrrHrrIrrrrrrr rEr7rrrs@r4rr^s %O+, 1      ???( "e.> "uO` " " " " ;?.22659:>-1)._ _ '_ !,_ !& 04 7 _ $t+ _ (4/ _ !+d2_ !& 04 7_ |d*_ #'_ -._  @ @_ _ _ ^_ _ _ _ _ r3ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. ceZdZUeed<dZdZdgZdeffd ZdZ de j fdZ dfd Z d Zd Zee ddejdejdejd zded zdeedeezf dZdejdejfdZee ddejdejdejd zdejd zdejd zdejd zdejd zdejd zdejd zdedeedeezfdZej ddejdejd zdejd zdejd zdejd zdejd zdedejfdZxZ S) rrLr|TrcXt|t|j|_t jtj d|j |j j |_ t|j |_t j|j j |jj |_|jrt)j|j}nt-j|j}||_|dSr)rRrSr _from_configrrrrXrHrrrrTrrrrrrrr#rr$rr)rCrLrras r4rSz-InstructBlipForConditionalGeneration.__init__ s    3@@AUVVLQ8OQWQfQr)s)stt/<PQQNN2>v?QRRN, r3c:|j|dSr)rset_output_embeddings)rCnew_embeddingss r4rz:InstructBlipForConditionalGeneration.set_output_embeddingss 11.AAAAAr3r;c4|jSr)rget_output_embeddingsrGs r4rz:InstructBlipForConditionalGeneration.get_output_embeddings"s"88:::r3Nc~||jSt|S)N)modality)r get_encoderrR)rCrras r4rz0InstructBlipForConditionalGeneration.get_encoder%s9  &2244 477&&&99 9r3c4|jSr)r get_decoderrGs r4rz0InstructBlipForConditionalGeneration.get_decoder+s"..000r3c |j}t|dkr@d|vr6GT5F6 %%=6 6  6 6  Aqq>qqbpqqq%a(  %z,*;*;*=*=crc*B%*]i]pqqq(// 0B10Er2NN $z,*;*;*=*=crc*B%*]i]pqqq ! )%*_5F%G%G "!7!:!:;O;V!W!W!&,@BX+Y_`!a!a!a&$, '1%".#7     *9&&q)!!!-C|/@/@/C/C-CQQQ*FG 11,??'5$r3rrcN|e||tj|jjtj|jk}|d}n||jjk}|d |j}|Srrrs r4rz9InstructBlipForConditionalGeneration.get_placeholder_maskrr3rrrlabelsc j||||| d} | j} | j}| j}|||}|t j|}| |j|j } | ||}| || }|j j r=|jd ||d| }|d}d}| |jd || |j jjd| }n&d| d<|jd ||||| d | }|j}|j}t)||||| S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```Trrr{rNrrr)r9rr}r)rrrrr)r8r9r+r,r:r2)rrr,r+r#rHrrrfr~rrrLrr loss_functionrr}r8r9r7)rCr|rrrrrrrrr{rrrr,r+rrr9r8s r4rz,InstructBlipForConditionalGeneration.forwards^CGBYBY /#9%= CZC C !/ <(8'6  7D5577 BBM  !"_Y77N 5 8 89M}Ob c c!66yP]6^^%445GI^__ ; 6 $)d)+-G QZFD!)t)!&T[=T=_ci %)F= !)d)+-"3'=   G)+#*     r3c t|dr||jd} |||||d} | j} ||l|jjg|jjz} | |jjj gz} tj | gtj |j }|| d}||}|tj|}| |j |j} |||}||| }||d }|jjjs||d <|jjd i||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rrTrNrr%rrrr2)r*rrmrrrLimage_token_indexrr bos_token_idrHrrrfrepeatr#rrr~rrris_encoder_decodergenerate)rCr|rrrrrr{generate_kwargsrrr image_tokens start_tokensrinputsrs r4rz-InstructBlipForConditionalGeneration.generatesD 4 ) ) *  ' ' ) ) )!'* BFBYBY /#9%= CZC C !/ <   $ =>A]] +t{/F/S.TT !L,uzR^Refff %,,Z;; 7D5577 BBM  !"_Y77N 5 8 89M}Ob c c!66yP]6^^%445GI^__ #0NSS")< ,"+F; .$%.KKK?KKr3rr$)NNNNNNNF)NNNNNF)!r-r.r/r&r1r%rrrSrrModulerrrrrrrHrIrrrrrEr*rrr7rrrrrs@r4rrs $O!+,1(BBB;ry;;;;:::::: 111???( ;?05 @@'@!+@!& 04 7 @ #'+ @ +, @ 8 8@@@^@D "e.> "uO` " " " " ;?.22659:>26*.).B B 'B !,B !& 04 7 B $t+ B (4/ B !+d2B !& 04 7B (4/B  4'B #'B +,B  @ @B B B ^B HU]__6::>-12626).DD'D!+d2D!& 04 7 D #d* D (4/ D(4/D#'D  DDD_DDDDDr3r)rrrrr)r)Tr0collections.abcr dataclassesrtypingrrHrrr activationsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrrr utils.genericrutils.output_capturingrr autor"r#r$configuration_instructblipr&r'r( get_loggerr-rr*r7rrKrfloatrrrrrrrrrrHrOrUrrqrrrr__all__r2r3r4r s"!$$$$$$!!!!!! &&&&&&!!!!!!))))))666666BBBBBB999999GFFFFFFF&&&&&&666666jjjjjjjjjjjjjjjj777777EEEEEEEEIIIIIIIIIIoooooooooo  H % %  P P P P P.H P P  P       k