tjh.pddlZddlmZddlmZmZmZmZmZddlm Z ddl m Z ddl m Z mZddlmZdd lmZmZmZdd lmZd d lmZejeZGd deZGddeZGddejZGddeZ GddeZ!GddeZ"gdZ#dS)N)nn)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)ACT2FN)Cache)BaseModelOutputWithPastBaseModelOutputWithPooling)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)VipLlavaConfigceZdZdS)VipLlavaModelOutputWithPastN__name__ __module__ __qualname__k/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/vipllava/modular_vipllava.pyrr&DrrceZdZdS)VipLlavaCausalLMOutputWithPastNrrrrr r *rrr c*eZdZdeffd ZdZxZS)VipLlavaMultiModalProjectorconfigctt|jtrdnt |j}t j||jj z|j |_ t j ||jj z|j j d|_t|j|_t j |j j |j j d|_dS)Nr)epsT)bias)super__init__ isinstancevision_feature_layersintlenr LayerNorm vision_config hidden_sizeprojector_layernorm_epsprojector_layernormLinear text_configlinear_1r projector_hidden_actactlinear_2)selfr#num_feature_layers __class__s rr(z$VipLlavaMultiModalProjector.__init__/s ",V-I3"O"OvQQUXY_YuUvUv#%< !5!A AvGe$ $ $   !5!A A   *    &56 &"4"@&BTB`gklll rc||}||}||}||}|SN)r1r4r6r7)r8 hidden_statess rforwardz#VipLlavaMultiModalProjector.forward>sN00??  m44 //  m44 r)rrrrr(r> __classcell__)r:s@rr"r".sZ m~ m m m m m mrr"ceZdZdS)VipLlavaPreTrainedModelNrrrrrArAFrrrAczeZdZeed ddejdeeezdzde e de e zfdZ ee dd ejdzdejdzd ejdzd ejdzd edzd ejdzdeeezdzdedzde e de ezfdZdS) VipLlavaModelzWObtains image last hidden states from the vision tower and apply multimodal projection.) custom_introN pixel_valuesr*kwargsreturnc ,||n |jj}d|d<|j|fi|t|trj|ddddf}n$fd|D}t j|d}||}|_ S)\ pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. NToutput_hidden_statesrcBg|]}j|ddddfS)Nr)r=).0index image_outputss r z4VipLlavaModel.get_image_features..ls2kkkEm9%@ABBGkkkr)dim) r#r* vision_towerr)r+r=torchcatmulti_modal_projector pooler_output)r8rEr*rFimage_featuresrNs @rget_image_featuresz VipLlavaModel.get_image_featuresKs$&;%F ! !DKLm *.%&))      +S 1 1 ?*89NOPQPQPQSTSUSUPUVNNlkkkUjkkkN"Y~2>>>N33NCC&4 #r input_idsattention_mask position_idspast_key_values inputs_embeds use_cache lm_kwargsc ||n |jj}|du|duz rtd|||}|j|||j} | |j|j} | ||| } | | | }|j d|||||d| } t| j | j| j| j|| nd} | S)z vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedsrEr*)r]rW)rZr[r\r]r^)last_hidden_stater\r= attentionsimage_hidden_statesr)r#r* ValueErrorget_input_embeddingsrXrVtodevicedtypeget_placeholder_maskmasked_scatterlanguage_modelrrbr\r=rc)r8rYrErZr[r\r]r*r^r_rWspecial_image_maskoutputsoutputs rr>zVipLlavaModel.forwardssb(&;%F ! !DKLm   -t"; < [YZZ Z  7D5577 BBM  #!44)AV5 ,..}/C]EXYYN!%!:!:~";"" *889K^\\M+>4+>, )%+' , ,  , , -%7#3!/)2>2JPT     rr<)NNNNNNNN)rrrrrrS FloatTensorr+listrrtupler rX LongTensorTensorr boolrr>rrrrCrCJs^n 9=""'" #T#Y5"+, " + + """"H.215.204(,268>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```N)rYrErZr[r\r]r^r*)logitsrz vocab_size)lossr}r\r=rcrdr)r#r*ryrbr)r+slicelm_head loss_functionr3r~r r\r=rcrd)r8rYrErZr[r\r]r*rzr^r{r_rnr= slice_indicesr}rs rr>z(VipLlavaForConditionalGeneration.forwardsj&;%F ! !DKLm 0:tz 0 %)%+'"7 0 0  0 0  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %%VFt{OfOq%rrD-#3!/) ' ;     rr<) NNNNNNNNNr)rrrrrSrpr+rqrrrrr rXrrsrtr rur r>rrrrwrws9=  '  #T#Y5 +,  + +    ^ ".215.204(,268<*.!%-.R R #d*R '$.R  t+ R &- R  R (4/R  #T#Y5R  4'R $;R el*R ./R  / /R R R ^R R R rrw)rCrwrA)$rSr(transformers.models.llava.modeling_llavarrrrr activationsr cache_utilsr modeling_outputsr r processing_utilsrutilsrrr utils.genericrconfiguration_vipllavar get_loggerrloggerrr Moduler"rArCrw__all__rrrrs4 "!!!!! SSSSSSSS&&&&&&@@@@@@@@@@------222222  H % %     ":        %@   ")0     2   `````J```Fg g g g g 'Dg g g T [ Z Zr