PhYddlZddlmZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZmZddl m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1e%jde3Z4Gdde jjZ6e#GddeZ7 dGde jjdejpdejpdejpdeejpde9de9fdZ:Gd d!e jjZ;Gd"d#e jjZ<Gd$d%eZ=Gd&d'e jjZ>Gd(d)e7Z?Gd*d+e jjZ@Gd,d-e jjZAGd.d/e jjZBGd0d1e jjZCGd2d3e jjZDGd4d5eZEGd6d7e jjZFGd8d9e jjZGGd:d;e7ZHee#d<=Gd>d?e!ZIe#d@=GdAdBe7ZJe#dC=GdDdEe7e ZKgdFZLy)HN) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)OutputRecordercheck_model_inputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipVideoConfigInstructBlipVideoQFormerConfigInstructBlipVideoVisionConfigceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) !InstructBlipVideoVisionEmbeddingsconfigcpt|||_|j|_|j |_|j |_tjtjdd|j|_ tjd|j|j |j |_ |j |j zdz|_|jdz|_tjtjd|j|j|_y)Nr$r ) in_channels out_channels kernel_sizestrider )super__init__r* hidden_size embed_dim image_size patch_sizer Parametertorchrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingselfr* __class__s z/var/www/html/saasai/venv/lib/python3.12/site-packages/transformers/models/instructblipvideo/modeling_instructblipvideo.pyr1z*InstructBlipVideoVisionEmbeddings.__init__8s  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"b embeddingsheightwidthreturnc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r$Ng?rr r bicubicF)sizemode align_cornersdim)shaper>r7jit is_tracingr5rreshapepermuter functional interpolateviewcat) r@rDrErFr<r=class_pos_embedpatch_pos_embedrO new_height new_widthsqrt_num_positionss rBinterpolate_pos_encodingz:InstructBlipVideoVisionEmbeddings.interpolate_pos_encodingJs`!&&q)A- //55a81< yy##%+*F6UZ?** *11!RaR%811!QR%8r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCrC pixel_valuesr^c|j\}}}}|jjj}|j|j |}|j dj dd}|jj|ddj |} tj| |gd} |r|j| ||} n |j} | | ddd| jdddfj |z} | S)Ndtyper r$rIrN)rPr;weightrbtoflatten transposer9expandr7rXr^r>rK) r@r_r^ batch_size_rErF target_dtype patch_embeds class_embedsrDr>s rBforwardz)InstructBlipVideoVisionEmbeddings.forwardrs'3'9'9$ Avu++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY l;C #!%!>!>z6SX!Y !%!8!8 "4Q8L*//!:L8La5O"P"S"ST`"aa rCF) __name__ __module__ __qualname__r'r1r7Tensorintr^ FloatTensorboolrm __classcell__rAs@rBr)r)7smc<c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn rCr)cBeZdZUeed<dZdZdZdZdZ dZ dZ gdZ dZ y) InstructBlipVideoPreTrainedModelr*blipT)"InstructBlipVideoQFormerEmbeddingsInstructBlipVideoAttention*InstructBlipVideoQFormerMultiHeadAttention"InstructBlipVideoQFormerSelfOutputc|jj}t|tjtj frY|j jjd||j%|jjjyyt|tjr(|j jjd|yt|tjrJ|jjj|j jjdyt|trYtjj!|j"d|tjj!|j$d|yt|t&t(fr%|j*jjyy)zInitialize the weights)meanstdN?)r*initializer_range isinstancerLinearr:rcdatanormal_biaszero_ Embedding LayerNormfill_r)init trunc_normal_r>r9)InstructBlipVideoForConditionalGenerationInstructBlipVideoModel query_tokens)r@modulefactors rB _init_weightsz.InstructBlipVideoPreTrainedModel._init_weightssI.. fryy"))4 5 MM   & &CV & <{{&   &&('  - MM   & &CV & <  - KK   " " $ MM   $ $S )  A B GG ! !&";";#6 ! R GG ! !&"8"8s ! O !JLb c d    $ $ * * ,erCN)rorprqr%__annotations__base_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrrCrBryrys> ##&*#"&N!-rCryrquerykeyvalueattention_maskscalingdropoutc ptj||jdd|z}|||z}tjj |d}tjj |||j}tj||} | jddj} | |fS)NrIrN)ptrainingr$r ) r7matmulrfrrUsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs rBeager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rCc eZdZdZfdZdej dedefdZ d dej de ej d e ej e ej e e ej ffd Z xZ S) r|z=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ d|_ |j|_ tj|jd|jzd|_|jretj t#j$|j}tj t#j$|j}nd}d}|Qt#j&|t#j(|d|f}tj ||j_tj|j|j|_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )r) requires_grad)r0r1r*r2r3num_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrrqkvqkv_biasr6r7zerosrX zeros_liker projection)r@r*q_biasv_biasrrAs rBr1z#InstructBlipVideoAttention.__init__su  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCrCtensorseq_lenbszc|j|||j|jjddj S)Nr$r )rWrrrfr)r@rrrs rB_shapez!InstructBlipVideoAttention._shapes7{{3GQQRSUVWbbddrC hidden_states head_maskrGc 6|j\}}}|j|}|j||d|j||jzj ddddd}|d|d|d} } }t } |j jdk7rt|j j} | ||| | fd|jsdn |j|jd |\} } | j||d j} |j| } | | fS) z#Input shape: Batch x Time x Channelr r rr$eagerNr)rrrrI)rKrrSrrTrr*_attn_implementationrrrrrr)r@rrrrtgt_lenr3 mixed_qkv query_states key_states value_statesattention_interfacerrs rBrmz"InstructBlipVideoAttention.forwards-#0"4"4"6WiHH]+ %%c7At~~yTXTbTbGbckk q!Q 2;1y|YWX\,j (? ;; + +w 6"9$++:Z:Z"[ $7     %  #}}C$2H2HJJ %  % ! \"))#w;FFH ook2 L((rCN) rorprq__doc__r1r7rrrsrrtuplermrvrws@rBr|r|sGD>eU\\eCece -1$)||$)ELL)$) u||Xell3XeELL>Q5RR S $)rCr|cVeZdZfdZdej dej fdZxZS)InstructBlipVideoMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r0r1r*r hidden_act activation_fnrrr2intermediate_sizefc1fc2r?s rBr1zInstructBlipVideoMLP.__init__ sd  #F$5$5699V//1I1IJ99V55v7I7IJrCrrGcl|j|}|j|}|j|}|Sr)rrrr@rs rBrmzInstructBlipVideoMLP.forwards4/ **=9 / rCrorprqr1r7rrrmrvrws@rBrr s$KU\\ellrCrc eZdZdeffd Zedejdejdee dejfdZ xZ S)InstructBlipVideoEncoderLayerr*cDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) r0r1r2r3r| self_attnrrlayer_norm_eps layer_norm1rmlp layer_norm2r?s rBr1z&InstructBlipVideoEncoderLayer.__init__sm ++3F;<<F> '$   q &0  ((7 / %0 rC) rorprqr%r1rr7rrrrrtrmrvrws@rBrrs`S6S|| +,     rCrc teZdZdZdeffd Ze ddeejde e de e effdZxZS) InstructBlipVideoEncodera" Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipVideoEncoderLayer`]. Args: config (`InstructBlipVideoConfig`): The corresponding vision configuration for the `InstructBlipVideoEncoder`. r*ct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wNF) r0r1r*r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r@r*rirAs rBr1z!InstructBlipVideoEncoder.__init__GsW  mmTYZ`ZrZrTs$tTsq%B6%JTs$tu &+#%usA#rrrGc V|}|jD] }||fd|i|}t|S)Nrlast_hidden_state)rr)r@ inputs_embedsrrr encoder_layers rBrmz InstructBlipVideoEncoder.forwardMsC& ![[M)-M)??rCr)rorprqrr%r1rrr7rrrrrrrrmrvrws@rBrr=sj,6, 26@!.@+, @ uo% & @@rCrc eZdZUdZeed<eedZdeffd Z e e d de e jdedeedeeeffdZd ZxZS) InstructBlipVideoVisionModelr_r*)r attentionsct||||_|j}t ||_t ||_tj||j|_ |jyr) r0r1r*r2r)rDrencoderrrrpost_layernorm post_init)r@r*r3rAs rBr1z%InstructBlipVideoVisionModel.__init__gs]   && ;FC/7  ll9&:O:OP rCr^rrGc | td|j||}|jdd|i|}|j}|j |}|dddddf}|j |}t ||S)Nz You have to specify pixel_values)r^rrr pooler_outputr)rrDrrrr)r@r_r^rrencoder_outputsr pooled_outputs rBrmz$InstructBlipVideoVisionModel.forwardrs  ?@ @ Ogh +74<<, ', ,  ,== //0AB)!Q'2 ++M: )/'  rCc|jSr)rDr@s rBget_input_embeddingsz1InstructBlipVideoVisionModel.get_input_embeddingss rCr)rorprqmain_input_namer'rrr|_can_record_outputsr1rrrr7rtrurrrrrrmr rvrws@rBrr_s$O ))60 < 59). u001 #' +,  u00 1   6rCrcZeZdZd fd ZdZdZdZdZdZ d de e fdZ xZ S) r}ct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj |j"|_t'|dd|_|j(dk(s|j(dk(rF|j*|_tj,d|j*zd z |j|_d |_y) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryr r$F)r0r1r*r2rhasattrrrsattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrgetattrrmax_position_embeddingsrdistance_embeddingsave_attentionr@r*is_cross_attentionrAs rBr1z3InstructBlipVideoQFormerMultiHeadAttention.__init__s     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD ##rCc||_yrattn_gradients)r@r"s rBsave_attn_gradientsz>InstructBlipVideoQFormerMultiHeadAttention.save_attn_gradientss ,rCc|jSrr!r s rBget_attn_gradientsz=InstructBlipVideoQFormerMultiHeadAttention.get_attn_gradientss"""rCc||_yr attention_map)r@r(s rBsave_attention_mapz=InstructBlipVideoQFormerMultiHeadAttention.save_attention_maps *rCc|jSrr'r s rBget_attention_mapzbhlrzbhrd,lrd->bhlrrNrr r )r/rrrr7rrfrrKarangelongr2rWrrrdrbeinsummathsqrtrrSoftmaxrr) register_hookr#rrTrr)r@rrrencoder_hidden_statesencoder_attention_maskrr key_layer value_layermixed_query_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtypeattention_probsattention_probs_dropped context_layernew_context_layer_shapes rBrmz2InstructBlipVideoQFormerMultiHeadAttention.forwards!3$> 11$((;P2QRI33DJJ?T4UVK3N11$((=2IJI33DJJ}4MNK JJ}5//0AB !<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||AABXY $"5"5  # #O 4  ) )$*B*B C#',,"?  &= &I # %S"S* **,CD o--rCrnNNNN) rorprqr1r#r%r)r+r/rrrmrvrws@rBr}r}sF$8-#+"%"# I.+,I.rCr}cneZdZfdZdej dej dej fdZxZS)r~c(t|tj|j|j|_tj |j|j|_tj|j|_ yr) r0r1rrr2denserrrhidden_dropout_probrr?s rBr1z+InstructBlipVideoQFormerSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rCr input_tensorrGcr|j|}|j|}|j||z}|SrrQrrr@rrSs rBrmz*InstructBlipVideoQFormerSelfOutput.forward7 =1  ]3 }|'CD rCrrws@rBr~r~ 1> U\\RWR^R^rCr~ceZdZd fd ZdZ d dej deejdeejdeejdeejde e d ej fd Z xZ S) !InstructBlipVideoQFormerAttentionct|t|||_t ||_t |_yr)r0r1r} attentionr~outputset pruned_headsrs rBr1z*InstructBlipVideoQFormerAttention.__init__s3 CFL^_8@ ErCc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr$rN)lenrr\rrr_rrrrr]rQrunion)r@headsindexs rB prune_headsz-InstructBlipVideoQFormerAttention.prune_heads"s u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:rCrrrr:r;rrGc `|jd|||||d|\}}|j||} | S)N)rrrr:r;r)r\r]) r@rrrr:r;rrriattention_outputs rBrmz)InstructBlipVideoQFormerAttention.forward4sO( ')"7#9     Q ;;{MBrCrnrN) rorprqr1rer7rrrrtrrrmrvrws@rBrZrZs" ;*7;15=A>B  || !!2!23 E--.  ((9(9:  !)):): ;  +,   rCrZcVeZdZfdZdej dej fdZxZS)$InstructBlipVideoQFormerIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r0r1rrr2rrQrrstrr intermediate_act_fnr?s rBr1z-InstructBlipVideoQFormerIntermediate.__init__Js] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rCrrGcJ|j|}|j|}|Sr)rQrlrs rBrmz,InstructBlipVideoQFormerIntermediate.forwardRs& =1 00? rCrrws@rBririIs#9U\\ellrCricneZdZfdZdej dej dej fdZxZS)InstructBlipVideoQFormerOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r0r1rrrr2rQrrrrRrr?s rBr1z'InstructBlipVideoQFormerOutput.__init__Ys` YYv779K9KL f&8&8f>S>STzz&"<"<= rCrrSrGcr|j|}|j|}|j||z}|SrrUrVs rBrmz&InstructBlipVideoQFormerOutput.forward_rWrCrrws@rBroroXrXrCrocHeZdZfdZ ddeefdZdZdZxZ S)InstructBlipVideoQFormerLayercft||j|_d|_t ||_||_||jzdk(rt |d|_d|_ nd|_ t||_ t||_ t||_t||_y)Nr$rT)rF)r0r1chunk_size_feed_forward seq_len_dimrZr\ layer_idxcross_attention_frequencycrossattentionhas_cross_attentionri intermediateror]intermediate_query output_queryr@r*rwrAs rBr1z&InstructBlipVideoQFormerLayer.__init__gs '-'E'E$:6B" v77 71 <"CF_c"dD '+D $',D $@H4V< "Fv"N:6BrCrc f|j|f||d|}|dkDr|ddd|ddf} |jr%| td|j| f||||d|} t |j |j |j| } |jd|kDrjt |j|j |j|dd|dddfj| j} tj| | gd} | St |j|j |j|} | S)N)rrrz>encoder_hidden_states must be given for cross-attention layers)rrr:r;r$rN)r\rzrryrfeed_forward_chunk_queryrurvrPfeed_forward_chunkrdr2r7rX) r@rrrr:r; query_lengthrrgquery_attention_output layer_outputlayer_output_texts rBrmz%InstructBlipVideoQFormerLayer.forward{sv*4>>  )    ! %5a,6I%J "''(0$%eff)<)<)<**#1'*?+A *  *&5--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,8I)JPQR  5'',,   L rCcL|j|}|j||}|Sr)r{r]r@rgintermediate_outputrs rBrz0InstructBlipVideoQFormerLayer.feed_forward_chunks,"//0@A{{#68HI rCcL|j|}|j||}|Sr)r|r}rs rBrz6InstructBlipVideoQFormerLayer.feed_forward_chunk_querys."556FG(()<>NO rCNNNNr) rorprqr1rrrmrrrvrws@rBrsrsfs:C."#6+,6p rCrscFeZdZfdZe ddeefdZxZS)InstructBlipVideoQFormerEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wr) r0r1r*rrrrrslayerrr~s rBr1z(InstructBlipVideoQFormerEncoder.__init__s]  ]]OTU[UmUmOn oOn) *69 =On o  ',# psA$rc t|jjD])}|j|} |||nd} | ||| |f||d|}+t |S)N)r;rr)rr*rrr) r@rrrr:r;rri layer_modulelayer_head_masks rBrmz'InstructBlipVideoQFormerEncoder.forwardsyt{{445A::a=L.7.CilO(%  (>) M 69+  rCr) rorprqr1rrrrmrvrws@rBrrs=,"# +,  rCrc2eZdZdZfdZ ddZxZS)r{z;Construct the embeddings from word and position embeddings.cPt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|j!dt#j$|jj'ddt)|dd|_||_y) N) padding_idxr position_ids)r$rIF) persistentrr)r0r1rr vocab_sizer2 pad_token_idword_embeddingsrposition_embeddingsrr layernormrrRrregister_bufferr7r3rgrrr*r?s rBr1z+InstructBlipVideoQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ rCc||jd}nd}|&|jdd|||zfj}|k|j|}|jdk(r/|j |j |j}||z}|tj||fd}n|}|j |jjj}|j|}|j|}|S)Nr$rrrN)rKrclonerrrrdr2r7rXrrcrbr)r@ input_idsr query_embedspast_key_values_lengthrArDrs rBrmz*InstructBlipVideoQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J++z9&*&>&>|zO`O`?a&b#'*== '"YY j'AqI %J]]4>>#8#8#>#>? ^^J/ \\*- rC)NNNr)rorprqrr1rmrvrws@rBr{r{sE$ rCr{ceZdZdZdZdZdZdZee e ddge e ddgdZ de ffd Z d Zd Zd Z dd ej$deedej*dedej$f dZee ddej4d eej8deej4deej$deej8deej8deej8deedeeej8e ffdZ!xZ"S)InstructBlipVideoQFormerModelz Querying Transformer (Q-Former), used in InstructBlipVideo. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr$z .attention)rd layer_namez.crossattention)rrcross_attentionsr*ct||||_t||_t ||_|jyr)r0r1r*r{rDrrrr?s rBr1z&InstructBlipVideoQFormerModel.__init__-s9     rCc.|jjSrrDrr s rBr z2InstructBlipVideoQFormerModel.get_input_embeddings7s...rCc&||j_yrrr@rs rBset_input_embeddingsz2InstructBlipVideoQFormerModel.set_input_embeddings:s*/'rCc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr\re)r@heads_to_prunerrcs rB _prune_headsz*InstructBlipVideoQFormerModel._prune_heads=s> +002LE5 LL  u % / / ; ;E B3rCr input_shaper2 has_queryrGc|jdk(r|dddddddf}n=|jdk(r|ddddddf}ntd|d|jd|j|j}d|z d z}|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nr z!Wrong shape for input_ids (shape z) or attention_mask (shape )rarg)rOrrPrdrb)r@rrr2rextended_attention_masks rBget_extended_attention_maskz9InstructBlipVideoQFormerModel.get_extended_attention_maskEs.    1 $&4Qa]&C #    !Q &'5QdA5E&F #3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&rCrrrrr:r;rc D| | td||jdnd} |j|||} | jdd} | \} } | j}|t j | | f|}|j|| |}|t|tr|dj\}}}n|j\}}}||f}t|tr|Dcgc]}|j|}}n?|)t j ||}|j|}n|j|}nd}|j||jj}|j| f||||| d|}|j}|dddddf}t!|| Scc}w) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner$r)rrrrI)r2)rrr:r;rr)rrPrDrKr2r7onesrrlistinvert_attention_mask get_head_maskr*rrrr)r@rrrrrr:r;rrembedding_outputrrhrAr2rencoder_batch_sizeencoder_sequence_lengthriencoder_hidden_shapemaskencoder_extended_attention_maskrsequence_outputrs rBrmz%InstructBlipVideoQFormerModel.forwardps$  !5VW W0<0H|))!,a ??%%+  '++-cr2 !, J!((  !"ZZ*j)A6RN#'"B"B>S^`f"g ! ,/6AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2w`vX\43M3Md3S`v/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y$++2O2OP +74<< , 2"7#B% , , *;;'1a0 ;-'  93xsFrn)NNNNNN)#rorprqrrrrrrsrr}r r&r1r rrr7rrrrsr2rurrr LongTensorrrtrrrrrmrvrws@rBrrs #( N7 EQ[g h  EQ[l m =/0C )' )'3Z)' )'  )'  )'V7;37/315=A>BO ##O !!2!23O u//0 O u||, O E--. O  ((9(9:O !)):): ;O +,O  uU&&')UU VO O rCrzV Class defining the outputs of [`InstructBlipVideoForConditionalGeneration`]. ) custom_introceZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<deefd Zy) 4InstructBlipVideoForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsrGcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)rrrN)rto_tuple).0kr@s rB zPInstructBlipVideoForConditionalGenerationModelOutput.to_tuple..sD !WW Gq!**, -!s-0)rkeysr s`rBrz=InstructBlipVideoForConditionalGenerationModelOutput.to_tuples% YY[    rC)rorprqrrrrr7rtrrrrrrrrrCrBrrs 04D(5**+ ,315FHU5,,- .526NHU../6:>OXeE$5$567>AEHU5+<+<%=>E %* rCrz` InstructBlipVideo base Model consisting of language model, qformer and vision encoder. c#eZdZdZdgZdeffd ZdZdZdZ dZ d e jd e jfd Zee dde jd e jd ee jd ee jdee jdee jdee jd ee j$deedeedeededeedeedeeeffdZxZS)rr_rr*c(t||t|j|_t j tjd|j|jj|_ t|j|_t j|jj|j j|_t%j&|j |_|j(j*/|j*j-|j(j*|j(j./|j.j-|j(j.|j1yNr$)r0r1r vision_config vision_modelrr6r7rnum_query_tokensqformer_configr2rrqformerr text_configlanguage_projectionr! from_configlanguage_modelrextend_keep_in_fp32_modulesrr?s rBr1zInstructBlipVideoModel.__init__s  89M9MNLLQ8O8OQWQfQfQrQr)st4V5J5JK #%99V-B-B-N-NPVPbPbPnPn#o '33F4F4FG    0 0 <  " " ) )$*=*=*O*O P    4 4 @  & & - -d.A.A.W.W X rCc6|jjSrrr r s rBr z+InstructBlipVideoModel.get_input_embeddings""7799rCc:|jj|yrrrrs rBrz+InstructBlipVideoModel.set_input_embeddings 007rCc|jjs_|jj|jj_|jj|jj _yyrr*use_decoder_only_language_modelrsharedr embed_tokensdecoderr s rB _tie_weightsz#InstructBlipVideoModel._tie_weights T{{::7;7J7J7Q7QD   ' ' 47;7J7J7Q7QD   ' ' 4;rCc|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r$raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maprar7cuda device_countloggerwarningrrrio_same_devicer@rs rB_preprocess_acceleratez-InstructBlipVideoModel._preprocess_acceleratey ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3:>D   ( ( 7 4rCrrc|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j|j }|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. r1rI) r r7rr*image_token_idr4r2all unsqueeze expand_asrdr@rrspecial_image_masks rBget_placeholder_maskz+InstructBlipVideoModel.get_placeholder_mask"  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H /99"=GG VYYZgZnZno!!rCqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskoutput_attentionsoutput_hidden_states return_dictr^ use_cacherrGc | | n|jj} |j\}}}}}|j||z|||}|j || | | | }|d}t j |jddt j|j}|jj|jddd}t j |jddt j|j}|t j|}|j|d}|j|d}t j||gd}|j|||||| | | }|dddd|jdddf}|j!|}|j||jj"|zd}|Q|j$j'|}||jj(k(}|t j|}nl||j't j*|jj(t j|jk(}|j-d}|j/dj1|j3|j}|j3|j|j4}|j7||}|jj8r|j$d ||| | | | d |}n|j$d ||||| | | | d |}t;||| S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)r_r r r r^rrIr1rNr$)rrrr:r;r r r rrr r r r)rrr r r r r rrr)r*use_return_dictrPrSrr7rrKr4r2rrg ones_likerepeat_interleaverXrrrrr video_token_idrrrrrdrbmasked_scatterrr)r@r_rrrrr r rr r r r^rrrhframeschannelrErFr image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputss rBrmzInstructBlipVideoModel.forward1sR&1%->q-A+A1(DE !% 8 8 F!6 = =j$++JfJfioJoqs t  FD//DDFyQM!*dkk.H.H!H %!&!;!.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; /99"=GG VYYZgZnZno 5 8 89M9M}ObOb c%445GI^_ ;; 6 6)d))+-"3%9'# G*d)) +-"3'="3%9'#  GD))#*  rC) NNNNNNNNNFN)rorprqr rr%r1r rrrr7rrtrrrrrrrurr rrrrmrvrws@rBrrs %O+,6&:8R ?( "e.>.> "uO`O` " >B15598<=A04,0/3&*).$( '' !,, !))9)9 :  E--.  !!1!12  $E$4$45 !))9)9 :   - $D> 'tn d^ #' D> -. uJJ K!  rCra InstructBlipVideo Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c%(eZdZUeed<dZdZdgZdeffd ZdZ dZ dZ d e jfd Zd Zd Zd ZdZ d"dej(dej*deej*deedeef dZdej*dej(fdZee d#dej(dej(deej*deej(deej*deej*deej*deej(deedeedeej*deededeedeed eee ff dZ!ejD d$dej(deej*deej*deej*deej*deej(ded ej*fd Z# d"dej(dej*deej*deedeef d!Z$xZ%S)%rr*r_Trc\t||tj|j|_t jtjd|j|jj|_ tj|j|_t j |jj|j"j|_|j&r t)j*|j"}nt-j*|j"}|j.%|j.j1|j.|j2%|j2j1|j2||_|j7yr)r0r1r _from_configrrrr6r7rrrr2rrrrrrrr"rr#rrrrr)r@r*rrAs rBr1z2InstructBlipVideoForConditionalGeneration.__init__s3  8EEfFZFZ[LLQ8O8OQWQfQfQrQr)st4AA&BWBWX #%99V-B-B-N-NPVPbPbPnPn#o  1 11==f>P>PQN2>>v?Q?QRN  + + 7  " " ) ).*J*J K  / / ;  & & - -n.R.R S, rCc6|jjSrrr s rBr z>InstructBlipVideoForConditionalGeneration.get_input_embeddingsrrCc:|jj|yrrrs rBrz>InstructBlipVideoForConditionalGeneration.set_input_embeddingsrrCc:|jj|yr)rset_output_embeddings)r@new_embeddingss rBr%z?InstructBlipVideoForConditionalGeneration.set_output_embeddingss 11.ArCrGc6|jjSr)rget_output_embeddingsr s rBr(z?InstructBlipVideoForConditionalGeneration.get_output_embeddingss""88::rCc6|jjSr)r get_encoderr s rBr*z5InstructBlipVideoForConditionalGeneration.get_encoder""..00rCc6|jjSr)r get_decoderr s rBr-z5InstructBlipVideoForConditionalGeneration.get_decoderr+rCc|jjs_|jj|jj_|jj|jj _yyrrr s rBrz6InstructBlipVideoForConditionalGeneration._tie_weightsrrCc|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyrrrs rBrz@InstructBlipVideoForConditionalGeneration._preprocess_acceleraterrCrrr^r cy)$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. Nr)r@r_rrr^r s rBget_image_featureszInstructBlipVideoForConditionalGeneration.get_placeholder_maskrrCrr r r r labelsrrc H| | n|jj} |j|||| d\}}}| s|jn|}| s|jn|}||j |}|t j |}|j|j|j}|j||}|j||}|jjre|jd ||| | | |d|}| r |jn|d}d}| w|jd || |jj j"d|}nB|jd ||||| | | | |d |}| r |j$n|d}| r |jn|d }t'||||| S) a qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length)): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. Examples: ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTrrr^r rrr)rr4r) rrr r r r r r4rr$)rrrrrr)r*rget_video_featuresrr r7rrdr2rbrrrrr loss_functionrrrr)r@r_rrrrr r rr r r4r r^rrrrrrrrrs rBrmz1InstructBlipVideoForConditionalGeneration.forward*sf&1%8C ..0  7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ ;; 6 6)d))+-"3%9'# G(3W^^ FD!)t))!&T[[=T=T=_=_ci *d)) +-"3'="3%9'#  G$/7<A]A]]`aa +t{{/F/F/S/S.TT !LL,uzzR^ReRef %,,Z; 7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ #0NS""))<<"+F; .$%%..KK?KrCc|j\}}}} } |j||z|| | }|j||d} | d} tj| j ddtj | j} |jj| jddd}tj|j ddtj | j}|tj|}|j|d}|j|d}tj||gd}|j|||| | d }|dddd|j dddf}|j|}|j||jj |zd}|r|| |fS|S) r1T)r_r^r rNrIr1rNr$)rrrr:r;r )rPrSrr7rrKr4r2rrgrrrXrrr*r)r@r_rrr^r rhrrrErFrrrrrrrrs rBr8z->q-A+A1(DE !% 8 8 F!6 = =j$++JfJfioJoqs t (.-G G$$rC)NFF) NNNNNNNNNNFN)NNNNNF)&rorprqr%rr rrr1r rr%rModuler(r*r-rrr7rtrrrur2rrrrrrrrrmno_gradr?r8rvrws@rBrrsz $#$O!+,64:8B;ryy;11R ?0>B38&+  '' !++ !))9)9 :  #+4.  d^  " "e.>.> "uO`O` " >B15598<=A59,0/3-1&*).$(N ''N !,,N !))9)9 : N E--. N !!1!12 N $E$4$45N !))9)9 :N   1 12N $D>N 'tnN ))*N d^N #'N D>N +,!N " uJJ K#N N `U]]_9==A045959).C''C$E$4$45C!))9)9 : C E,,- C !!1!12 C  1 12C#'C   CCR>B38&+ 9%''9%!++9%!))9)9 : 9% #+4. 9% d^ 9%rCr)rryrrr)r)Mr6 dataclassesrtypingrrrrr7r activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrrutilsrrrrrr utils.genericrrautor!r"r#configuration_instructblipvideor%r&r' get_loggerrorrEr)ryrrfloatrr|rrrrr}r~rZrirorsrr{rrrr__all__rrCrBrXsr, !11 !)B9 G&lljj?II   H %G GT#-#-#-\% II% <<% % << % U\\* %  %%.I)I)X 299 $>D@ryy@D1#C1hw.w.t  +  + \ 299  RYY U$>Up$ bii$ N00fi $Di X   ;   : F =F  F R}%0PRa}%}%@  rC