h&dZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZddlmZmZm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2e&jfe4Z5ee$dGdde"Z6Gdde jnZ8 dHde jnde jrde jrde jrdee jrd e:d!e:fd"Z;Gd#d$e jnZ<Gd%d&e jnZ=Gd'd(eZ>e$Gd)d*eZ?Gd+d,e jnZ@Gd-d.e?ZAGd/d0e jnZBGd1d2e jnZCGd3d4e jnZDGd5d6e jnZEGd7d8e jnZFGd9d:eZGGd;de jnZIGd?d@e?ZJe$dAGdBdCe?ZKe$dDGdEdFe?eZLgdGZMy)IzPyTorch InstructBLIP model.N) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)OutputRecordercheck_model_inputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigzQ Class defining the outputs of [`InstructBlipForConditionalGeneration`]. ) custom_introceZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<deefd Zy) /InstructBlipForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsreturncHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)r-r.r/N)getattrto_tuple).0kselfs u/var/www/html/aiagenthome/venv/lib/python3.12/site-packages/transformers/models/instructblip/modeling_instructblip.py zKInstructBlipForConditionalGenerationModelOutput.to_tuple..JsD !WW Gq!**, -!s-0)tuplekeysr8s`r9r5z8InstructBlipForConditionalGenerationModelOutput.to_tupleIs% YY[    )__name__ __module__ __qualname____doc__r+rr;torch FloatTensor__annotations__r,r-r.r/rr5r>r9r*r*.s 04D(5**+ ,315FHU5,,- .526NHU../6:>OXeE$5$567>AEHU5+<+<%=>E %* r>r*ceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) InstructBlipVisionEmbeddingsconfigcpt|||_|j|_|j |_|j |_tjtjdd|j|_ tjd|j|j |j |_ |j |j zdz|_|jdz|_tjtjd|j|j|_y)Nr$r ) in_channels out_channels kernel_sizestrider )super__init__rI hidden_size embed_dim image_size patch_sizer ParameterrCrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr8rI __class__s r9rPz%InstructBlipVisionEmbeddings.__init__Ts  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"br> embeddingsheightwidthr0c|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r$Ng?rr r bicubicF)sizemode align_cornersdim)shaper\rCjit is_tracingrTrreshapepermuter functional interpolateviewcat) r8r_r`rarZr[class_pos_embedpatch_pos_embedri new_height new_widthsqrt_num_positionss r9interpolate_pos_encodingz5InstructBlipVisionEmbeddings.interpolate_pos_encodingfs`!&&q)A- //55a81< yy##%+*F6UZ?** *11!RaR%811!QR%8r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr> pixel_valuesrxc|j\}}}}|jjj}|j|j |}|j dj dd}|jj|ddj |} tj| |gd} |r|j| ||} n |j} | | ddd| jdddfj |z} | S)Ndtyper r$rcrh)rjrYweightr|toflatten transposerWexpandrCrrrxr\re) r8ryrx batch_size_r`ra target_dtype patch_embeds class_embedsr_r\s r9forwardz$InstructBlipVisionEmbeddings.forwards'3'9'9$ Avu++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY l;C #!%!>!>z6SX!Y !%!8!8 "4Q8L*//!:L8La5O"P"S"ST`"aa r>F) r?r@rAr'rPrCTensorintrxrDboolr __classcell__r^s@r9rHrHSsmc7c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn r>rHmodulequerykeyvalueattention_maskscalingdropoutc ptj||jdd|z}|||z}tjj |d}tjj |||j}tj||} | jddj} | |fS)Nrcrh)ptrainingr$r ) rCmatmulrrrosoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs r9eager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r>c eZdZdZfdZdej dedefdZ d dej de ej d e ej e ej e e ej ffd Z xZ S) InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ d|_ |j|_ tj|jd|jzd|_|jretj t#j$|j}tj t#j$|j}nd}d}|Qt#j&|t#j(|d|f}tj ||j_tj|j|j|_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )bias) requires_grad)rOrPrIrQrRnum_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasrUrCzerosrr zeros_liker projection)r8rIq_biasv_biasrr^s r9rPzInstructBlipAttention.__init__su  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCr>tensorseq_lenbszc|j|||j|jjddj S)Nr$r )rqrrrr)r8rrrs r9_shapezInstructBlipAttention._shapes7{{3GQQRSUVWbbddr> hidden_states head_maskr0c 6|j\}}}|j|}|j||d|j||jzj ddddd}|d|d|d} } }t } |j jdk7rt|j j} | ||| | fd|jsdn |j|jd |\} } | j||d j} |j| } | | fS) z#Input shape: Batch x Time x Channelr r rr$eagerN)rrrrc)rerrmrrnrrI_attn_implementationrrrrrr)r8rrrrtgt_lenrR mixed_qkv query_states key_states value_statesattention_interfacerrs r9rzInstructBlipAttention.forwards-#0"4"4"6WiHH]+ %%c7At~~yTXTbTbGbckk q!Q 2;1y|YWX\,j (? ;; + +w 6"9$++:Z:Z"[ $7     %  #}}C$2H2HJJ %  % ! \"))#w;FFH ook2 L((r>N) r?r@rArBrPrCrrrrr;rrrs@r9rrsGD>eU\\eCece -1$)||$)ELL)$) u||Xell3XeELL>Q5RR S $)r>rcVeZdZfdZdej dej fdZxZS)InstructBlipMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) rOrPrIr hidden_act activation_fnrrrQintermediate_sizefc1fc2r]s r9rPzInstructBlipMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr>rr0cl|j|}|j|}|j|}|Sr)rrrr8rs r9rzInstructBlipMLP.forward s4/ **=9 / r>r?r@rArPrCrrrrs@r9rrs$KU\\ellr>rc eZdZdeffd Zedejdejdee dejfdZ xZ S)InstructBlipEncoderLayerrIcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) rOrPrQrRr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r]s r9rPz!InstructBlipEncoderLayer.__init__sm ++.v6<<Frrrr0c |}|j|}|jd||d|\}}||z}|}|j|}|j|}||z}|S)N)rrrF)rrrr)r8rrrresidualrs r9rz InstructBlipEncoderLayer.forwards!((7 )4>> '$   q &0  ((7 / %0 r>) r?r@rAr%rPrrCrrrrDrrrs@r9rrs`S1S|| +,     r>rcBeZdZUeed<dZdZdZdZdZ dZ dZ gdZ dZ y)InstructBlipPreTrainedModelrIblipT)InstructBlipQFormerEmbeddingsr%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputc|jj}t|tjtj frY|j jjd||j%|jjjyyt|tjr(|j jjd|yt|tjrJ|jjj|j jjdyt|trYtjj!|j"d|tjj!|j$d|yt|t&t(fr%|j*jjyy)zInitialize the weightsr)meanstdN?)rIinitializer_range isinstancerrrXr}datanormal_rzero_ Embeddingrfill_rHinit trunc_normal_r\rW$InstructBlipForConditionalGenerationInstructBlipModel query_tokens)r8rfactors r9 _init_weightsz)InstructBlipPreTrainedModel._init_weightsHsI.. fryy"))4 5 MM   & &CV & <{{&   &&('  - MM   & &CV & <  - KK   " " $ MM   $ $S )  < = GG ! !&";";#6 ! R GG ! !&"8"8s ! O !EGX Y Z    $ $ * * ,[r>N)r?r@rAr%rEbase_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrrFr>r9rr5s> &*#"&N!-r>rc teZdZdZdeffd Ze ddeejde e de e effdZxZS) InstructBlipEncodera Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rIct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wNF) rOrPrIr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r8rIrr^s r9rPzInstructBlipEncoder.__init__gsV  mmuU[UmUmOn$oOn!%=f%EOn$op &+#%psA#rrr0c V|}|jD] }||fd|i|}t|S)Nrlast_hidden_state)rr)r8 inputs_embedsrrr encoder_layers r9rzInstructBlipEncoder.forwardmsC& ![[M)-M)??r>r)r?r@rArBr%rPrrrCrrrrr;rrrrs@r9rr]sj,1, 26@!.@+, @ uo% & @@r>rc eZdZUdZeed<eedZdeffd Z e e d de e jdedeedeeeffdZd ZxZS) InstructBlipVisionModelryrI)r attentionsct||||_|j}t ||_t ||_tj||j|_ |jyr) rOrPrIrQrHr_rencoderrrrpost_layernorm post_init)r8rIrRr^s r9rPz InstructBlipVisionModel.__init__s]   && 6v>*62  ll9&:O:OP r>rxrr0c | td|j||}|jdd|i|}|j}|j |}|dddddf}|j |}t ||S)Nz You have to specify pixel_values)rxr rr  pooler_outputrF)rr_rr rr)r8ryrxrrencoder_outputsr  pooled_outputs r9rzInstructBlipVisionModel.forwards  ?@ @ Ogh +74<<, ', ,  ,== //0AB)!Q'2 ++M: )/'  r>c|jSr)r_r=s r9get_input_embeddingsz,InstructBlipVisionModel.get_input_embeddingss r>r)r?r@rAmain_input_namer'rErr_can_record_outputsrPrrrrCrDrrrrr;rrrrrs@r9rrs$O $$1+ 7 59). u001 #' +,  u00 1   6r>rcZeZdZd fd ZdZdZdZdZdZ d de e fdZ xZ S) rct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj |j"|_t'|dd|_|j(dk(s|j(dk(rF|j*|_tj,d|j*zd z |j|_d |_y) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryr r$F)rOrPrIrQrhasattrrrattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrr4r!max_position_embeddingsrdistance_embeddingsave_attentionr8rIis_cross_attentionr^s r9rPz.InstructBlipQFormerMultiHeadAttention.__init__s     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD ##r>c||_yrattn_gradients)r8r2s r9save_attn_gradientsz9InstructBlipQFormerMultiHeadAttention.save_attn_gradientss ,r>c|jSrr1r=s r9get_attn_gradientsz8InstructBlipQFormerMultiHeadAttention.get_attn_gradientss"""r>c||_yr attention_map)r8r8s r9save_attention_mapz8InstructBlipQFormerMultiHeadAttention.save_attention_maps *r>c|jSrr7r=s r9get_attention_mapz7InstructBlipQFormerMultiHeadAttention.get_attention_maps!!!r>c|jdd|j|jfz}|j|}|j ddddS)Nrcrr r$r )rerr&rqrn)r8x new_x_shapes r9transpose_for_scoresz:InstructBlipQFormerMultiHeadAttention.transpose_for_scoressNffhsmt'?'?AYAY&ZZ AFFK yyAq!$$r>rc |du}|rC|j|j|}|j|j|} |}n@|j|j|}|j|j|} |j|} |j| } t j | |j dd} |jdk(s|jdk(rF|jd} t j| tj|jjdd}t j| tj|jjdd}||z }|j||jzdz }|j| j }|jdk(rt j"d| |}| |z} nE|jdk(r6t j"d| |}t j"d ||}| |z|z} | t%j&|j(z } | j }|| |z} t+j,d | j|}|r8|j.r,|j1||j3|j4|j7|}|||z}t j || }|j9d d dd j;}|jdd|j<fz}|j|}||fS)Nrcrr#r$r$r|devicer{zbhld,lrd->bhlrzbhrd,lrd->bhlrrhrr r )r?rrrrCrrr!rearangelongrBrqr,r+r~r|einsummathsqrtr&rSoftmaxr-r9 register_hookr3rrnrr')r8rrrencoder_hidden_statesencoder_attention_maskrr/ key_layer value_layermixed_query_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtypeattention_probsattention_probs_dropped context_layernew_context_layer_shapes r9rz-InstructBlipQFormerMultiHeadAttention.forwards!3$> 11$((;P2QRI33DJJ?T4UVK3N11$((=2IJI33DJJ}4MNK JJ}5//0AB !<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||AABXY $"5"5  # #O 4  ) )$*B*B C#',,"?  &= &I # %S"S* **,CD o--r>rNNNN) r?r@rArPr3r5r9r;r?rrrrrs@r9rrsF$8-#+"%"# I.+,I.r>rcneZdZfdZdej dej dej fdZxZS)rc(t|tj|j|j|_tj |j|j|_tj|j|_ yr) rOrPrrrQdenserrr)hidden_dropout_probrr]s r9rPz&InstructBlipQFormerSelfOutput.__init__0s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r>r input_tensorr0cr|j|}|j|}|j||z}|Srrarrr8rrcs r9rz%InstructBlipQFormerSelfOutput.forward67 =1  ]3 }|'CD r>rrs@r9rr/1> U\\RWR^R^r>rceZdZd fd ZdZ d dej deejdeejdeejdeejde e d ej fd Z xZ S) InstructBlipQFormerAttentionct|t|||_t ||_t |_yr)rOrPr attentionroutputset pruned_headsr.s r9rPz%InstructBlipQFormerAttention.__init__?s3 >vGYZ3F; Er>c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr$rh)lenrrlrr&rorrrrrmrar'union)r8headsindexs r9 prune_headsz(InstructBlipQFormerAttention.prune_headsEs u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r>rrrrJrKrr0c `|jd|||||d|\}}|j||} | S)N)rrrrJrKrF)rlrm) r8rrrrJrKrrrattention_outputs r9rz$InstructBlipQFormerAttention.forwardWsO( ')"7#9     Q ;;{MBr>rr^) r?r@rArPrurCrrrDrrrrrs@r9rjrj>s" ;*7;15=A>B  || !!2!23 E--.  ((9(9:  !)):): ;  +,   r>rjcVeZdZfdZdej dej fdZxZS)InstructBlipQFormerIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rOrPrrrQrrarrstrr intermediate_act_fnr]s r9rPz(InstructBlipQFormerIntermediate.__init__ns] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r>rr0cJ|j|}|j|}|Sr)rar|rs r9rz'InstructBlipQFormerIntermediate.forwardvs& =1 00? r>rrs@r9ryryms#9U\\ellr>rycneZdZfdZdej dej dej fdZxZS)InstructBlipQFormerOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rOrPrrrrQrarrr)rbrr]s r9rPz"InstructBlipQFormerOutput.__init__~s` YYv779K9KL f&8&8f>S>STzz&"<"<= r>rrcr0cr|j|}|j|}|j||z}|Srrerfs r9rz!InstructBlipQFormerOutput.forwardrgr>rrs@r9rr}rhr>rcHeZdZfdZ ddeefdZdZdZxZ S)InstructBlipQFormerLayercft||j|_d|_t ||_||_||jzdk(rt |d|_d|_ nd|_ t||_ t||_ t||_t||_y)Nr$rT)r/F)rOrPchunk_size_feed_forward seq_len_dimrjrl layer_idxcross_attention_frequencycrossattentionhas_cross_attentionry intermediaterrmintermediate_query output_queryr8rIrr^s r9rPz!InstructBlipQFormerLayer.__init__s '-'E'E$5f=" v77 71 <">vZ^"_D '+D $',D $;FC/7 "A&"I5f=r>rc f|j|f||d|}|dkDr|ddd|ddf} |jr%| td|j| f||||d|} t |j |j |j| } |jd|kDrjt |j|j |j|dd|dddfj| j} tj| | gd} | St |j|j |j|} | S)N)rrrz>encoder_hidden_states must be given for cross-attention layers)rrrJrKr$rh)rlrrrrfeed_forward_chunk_queryrrrjfeed_forward_chunkr~rBrCrr) r8rrrrJrK query_lengthrrwquery_attention_output layer_outputlayer_output_texts r9rz InstructBlipQFormerLayer.forwardsv*4>>  )    ! %5a,6I%J "''(0$%eff)<)<)<**#1'*?+A *  *&5--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,8I)JPQR  5'',,   L r>cL|j|}|j||}|Sr)rrmr8rwintermediate_outputrs r9rz+InstructBlipQFormerLayer.feed_forward_chunks,"//0@A{{#68HI r>cL|j|}|j||}|Sr)rrrs r9rz1InstructBlipQFormerLayer.feed_forward_chunk_querys."556FG(()<>NO r>NNNNr) r?r@rArPrrrrrrrs@r9rrs9>."#6+,6p r>rcFeZdZfdZe ddeefdZxZS)InstructBlipQFormerEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wr) rOrPrIrrrrrlayerrrs r9rPz#InstructBlipQFormerEncoder.__init__s]  ]]JOPVPhPhJi jJiY %fi 8Ji j  ',# ksA$rc t|jjD])}|j|} |||nd} | ||| |f||d|}+t |S)N)rKrr )rrIrrr) r8rrrrJrKrri layer_modulelayer_head_masks r9rz"InstructBlipQFormerEncoder.forwardsyt{{445A::a=L.7.CilO(%  (>) M 69+  r>r) r?r@rArPrrrrrrs@r9rrs=,"# +,  r>rc2eZdZdZfdZ ddZxZS)rz;Construct the embeddings from word and position embeddings.cPt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|j!dt#j$|jj'ddt)|dd|_||_y) N) padding_idxr position_ids)r$rcF) persistentr!r")rOrPrr vocab_sizerQ pad_token_idword_embeddingsr+position_embeddingsrr layernormr)rbrregister_bufferrCrCrr4r!rIr]s r9rPz&InstructBlipQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ r>c||jd}nd}|&|jdd|||zfj}|k|j|}|jdk(r/|j |j |j}||z}|tj||fd}n|}|j |jjj}|j|}|j|}|S)Nr$rr"rh)rerclonerr!rr~rBrCrrrr}r|r)r8 input_idsr query_embedspast_key_values_lengthrQr_rs r9rz%InstructBlipQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J++z9&*&>&>|zO`O`?a&b#'*== '"YY j'AqI %J]]4>>#8#8#>#>? ^^J/ \\*- r>)NNNr)r?r@rArBrPrrrs@r9rr sE$ r>rceZdZdZdZdZdZdZee e ddge e ddgdZ de ffd Z d Zd Zd Z dd ej$deedej*dedej$f dZee ddej4d eej8deej4deej$deej8deej8deej8deedeeej8e ffdZ!xZ"S)InstructBlipQFormerModelz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr$z .attention)rt layer_namez.crossattention)rrcross_attentionsrIct||||_t||_t ||_|jyr)rOrPrIrr_rrrr]s r9rPz!InstructBlipQFormerModel.__init__Ss9   7?1&9  r>c.|jjSrr_rr=s r9rz-InstructBlipQFormerModel.get_input_embeddings]s...r>c&||j_yrrr8rs r9set_input_embeddingsz-InstructBlipQFormerModel.set_input_embeddings`s*/'r>c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrlru)r8heads_to_prunerrss r9 _prune_headsz%InstructBlipQFormerModel._prune_headscs> +002LE5 LL  u % / / ; ;E B3r>r input_shaperB has_queryr0c|jdk(r|dddddddf}n=|jdk(r|ddddddf}ntd|d|jd|j|j}d|z d z}|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nr z!Wrong shape for input_ids (shape z) or attention_mask (shape )r{rg)rirrjr~r|)r8rrrBrextended_attention_masks r9get_extended_attention_maskz4InstructBlipQFormerModel.get_extended_attention_maskks.    1 $&4Qa]&C #    !Q &'5QdA5E&F #3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&r>rrrrrJrKrc D| | td||jdnd} |j|||} | jdd} | \} } | j}|t j | | f|}|j|| |}|t|tr|dj\}}}n|j\}}}||f}t|tr|Dcgc]}|j|}}n?|)t j ||}|j|}n|j|}nd}|j||jj}|j| f||||| d|}|j}|dddddf}t!|| Scc}w) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner$r)rrrrc)rB)rrrJrKrr)rrjr_rerBrConesrrlistinvert_attention_mask get_head_maskrIrrr r)r8rrrrrrJrKrrembedding_outputrrrQrBrencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapemaskencoder_extended_attention_maskrsequence_outputrs r9rz InstructBlipQFormerModel.forwards$  !5VW W0<0H|))!,a ??%%+  '++-cr2 !, J!((  !"ZZ*j)A6RN#'"B"B>S^`f"g ! ,/6AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2w`vX\43M3Md3S`v/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y$++2O2OP +74<< , 2"7#B% , , *;;'1a0 ;-'  93xsFr)NNNNNN)#r?r@rArBrrrrrrrrr&rPrrrrCrr;rrBrrrr LongTensorrrDrrrrrrrs@r9rr>s #( N2 @Vb c  @Vg h 8/0C )' )'3Z)' )'  )'  )'V7;37/315=A>BO ##O !!2!23O u//0 O u||, O E--. O  ((9(9:O !)):): ;O +,O  uU&&')UU VO O r>rz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. ceZdZdZdgZdeffd ZdZdZdZ dZ d e jd e jfd Zee dde jd e jd ee jd ee jdee jdee jdee jd ee j$dedeedeeeffdZxZS)rryrrIc(t||t|j|_t j tjd|j|jj|_ t|j|_t j|jj|j j|_t%j&|j |_|j(j*/|j*j-|j(j*|j(j./|j.j-|j(j.|j1yNr$)rOrPr vision_config vision_modelrrUrCrnum_query_tokensqformer_configrQrrqformerr text_configlanguage_projectionr! from_configlanguage_modelrextend_keep_in_fp32_modulesrr]s r9rPzInstructBlipModel.__init__s  3F4H4HILLQ8O8OQWQfQfQrQr)st/0E0EF #%99V-B-B-N-NPVPbPbPnPn#o '33F4F4FG    0 0 <  " " ) )$*=*=*O*O P    4 4 @  & & - -d.A.A.W.W X r>c6|jjSrrrr=s r9rz&InstructBlipModel.get_input_embeddings""7799r>c:|jj|yrrrrs r9rz&InstructBlipModel.set_input_embeddings  007r>c|jjs_|jj|jj_|jj|jj _yyrrIuse_decoder_only_language_modelrsharedr embed_tokensdecoderr=s r9 _tie_weightszInstructBlipModel._tie_weights T{{::7;7J7J7Q7QD   ' ' 47;7J7J7Q7QD   ' ' 4;r>c|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r$raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maprqrCcuda device_countloggerwarningr%rrio_same_devicer8rs r9_preprocess_acceleratez(InstructBlipModel._preprocess_acceleratey ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3:>D   ( ( 7 4r>rr c|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j|j }|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. rArc rrCrrIimage_token_idrDrBall unsqueeze expand_asr~r8rr special_image_masks r9get_placeholder_maskz&InstructBlipModel.get_placeholder_mask%  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H /99"=GG VYYZgZnZno!!r>qformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskrxrr0c *|jd || d| } | d} tj| jddtj| j } |j j| jddd}tj|jddtj| j }|tj|}tj||gd}|jd |||| | d| }|dddd|jdddf}|7|jj|}|tj|}|j|}|j|j |j }|j#|| }|j%||}|j&j(r|jd ||d | }n|jd ||||d | }t+| || S)a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. )ryrxrNrcrAr$rh)rrrrJrKr r r)r rr r r3rF)rrCrrerDrBrrrj ones_likerrrrrrr~r|rmasked_scatterrIrr*)r8ryrr rrr r r rxrr- image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputss r9rzInstructBlipModel.forward4sPP+** %%=   &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!&,@BX+Y_`!a$  '1%".#7    %Q'+A\->->q-A+A1(DE  FD//DDFyQM%!&!;!% 8 8 F 5 8 89M9M}ObOb c!66yP]6^%445GI^_ ;; 6 6)d))+-G *d))+-"3'=   G?))#*  r>)NNNNNNF)r?r@rArrr%rPrrrrrCrrDrrrrrrrr rr;r*rrrs@r9rrs^ %O+,1&:8R ?( "e.>.> "uO`O` " >B15598<=A04).^ ''^ !,,^ !))9)9 : ^ E--. ^ !!1!12 ^ $E$4$45^ !))9)9 :^   -^ #'^ -.^  uEE F^ ^ r>ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. ceZdZUeed<dZdZdgZdeffd ZdZ dZ dZ d e jfd Zd Zd Zd ZdZ ddej(dej*deej*deedeef dZdej*dej(fdZee ddej(dej(deej*deej(deej*deej*deej*deej(deej*dedeed eee ffdZ!ejD d dej(deej*deej*deej*deej*deej(ded ej*fdZ#xZ$S)!rrIryTrc\t||tj|j|_t jtjd|j|jj|_ tj|j|_t j |jj|j"j|_|j&r t)j*|j"}nt-j*|j"}|j.%|j.j1|j.|j2%|j2j1|j2||_|j7yr)rOrPr _from_configrrrrUrCrrrrQrrrrrrrr"rr#rrrrr)r8rIrr^s r9rPz-InstructBlipForConditionalGeneration.__init__s3  3@@AUAUVLLQ8O8OQWQfQfQrQr)st/<P>PQN2>>v?Q?QRN  + + 7  " " ) ).*J*J K  / / ;  & & - -n.R.R S, r>c6|jjSrrr=s r9rz9InstructBlipForConditionalGeneration.get_input_embeddingsrr>c:|jj|yrrrs r9rz9InstructBlipForConditionalGeneration.set_input_embeddingsrr>c:|jj|yr)rset_output_embeddings)r8new_embeddingss r9rz:InstructBlipForConditionalGeneration.set_output_embeddingss 11.Ar>r0c6|jjSr)rget_output_embeddingsr=s r9r!z:InstructBlipForConditionalGeneration.get_output_embeddingss""88::r>c6|jjSr)r get_encoderr=s r9r#z0InstructBlipForConditionalGeneration.get_encoder""..00r>c6|jjSr)r get_decoderr=s r9r&z0InstructBlipForConditionalGeneration.get_decoderr$r>c|jjs_|jj|jj_|jj|jj _yyrrr=s r9rz1InstructBlipForConditionalGeneration._tie_weightsrr>c|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyrrrs r9rz;InstructBlipForConditionalGeneration._preprocess_acceleraterr>rr rx return_dictc|j||d}|d}tj|jddtj|j }|j j|jddd} tj| jddtj|j } |tj|}tj| |gd}|j||| ||d } | dddd| jdddf} |j| } |r| || fS| S) a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)ryrxr)rNrcrAr$rh)rrrrJrKr)) rrCrrerDrBrrrjrrrrr)r8ryrr rxr)r-rrrrrrrs r9get_image_featuresz7InstructBlipForConditionalGeneration.get_image_featuressY"**%%=+  &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!&,@BX+Y_`!a '1%".#7 % %Q'+A\->->q-A+A1(DE !% 8 8 F (.-G G$$r>rr c|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j|j }|Srrrs r9rz9InstructBlipForConditionalGeneration.get_placeholder_maskrr>rr r labelsrc |j|||| d\} } }||j|}|tj|}| j |j |j } |j||}|j|| }|jjrS|jd ||d| }|d}d}| j|jd || |jjjd| }n5d| d<|jd ||||| d | }|j}|j }t#||| || S) aD qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import requests >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```Trr rxr)Nr rr)r,r-rr))r rr r r-)r+r,r-r.r/rF)r+rrCrr~rBr|rrrIrr loss_functionrrr+r,r*)r8ryrr rrr r r r-rxrrr-rrrr,r+s r9rz,InstructBlipForConditionalGeneration.forward.sZ@D?V?V /#9%= @W@ <~}  7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ ;; 6 6)d))+-G QZFD!)t))!&T[[=T=T=_=_ci %)F= !)d))+-"3'=   G<))#*   r>c Bt|dr|j|jd} |j||||d\} } } |||jj g|jj z} | |jjjgz}tj|gtj|j}|j| d}|j|}|tj|}| j!|j|j"} |j%||}|j'|| }||d}|j(jj*s||d <|j(j,d i||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rrTr/rAr$r rrrF)r%rrjr+rIimage_token_indexrr bos_token_idrCrrDrBrepeatrrr~r|rrris_encoder_decodergenerate)r8ryrr rrr rxgenerate_kwargsrrr-r image_tokens start_tokensrinputsrs r9r6z-InstructBlipForConditionalGeneration.generatesD 4 )  ' ' )!''* ?C?V?V /#9%= @W@ <~}   $ = =>A]A]] +t{{/F/F/S/S.TT !LL,uzzR^ReRef %,,Z; 7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ #0NS""))<<"+F; .$%%..KK?Kr>)NFF)NNNNNNNF)NNNNNF)%r?r@rAr%rErrrrPrrrrModuler!r#r&rrrCrDrrrr+rrrrrrr;r*rno_gradr6rrs@r9rrs $O!+,14:8B;ryy;11R ?0>B38&+ /%''/%!++/%!))9)9 : /% #+4. /% d^ /%b "e.>.> "uO`O` " >B15598<=A59-1).} ''} !,,} !))9)9 : } E--. } !!1!12 } $E$4$45} !))9)9 :}   1 12} ))*} #'} +,}  uEE F} } ~U]]_9==A045959).C''C$E$4$45C!))9)9 : C E,,- C !!1!12 C  1 12C#'C   CCr>r)rrrrr)r)NrBrF dataclassesrtypingrrrrrCr activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrrutilsrrrrrr utils.genericrrautor!r"r#configuration_instructblipr%r&r' get_loggerr?rr*r;rHrfloatrrrrrrrrrrjryrrrrrrr__all__rFr>r9rNsp" !11 !)B9 G&lljj?IIoo   H %   k