tjEVdZddlmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z mZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddlmZddlmZeje Z!edeGdde Z"edeGddeZ#Gddej$Z%eGddeZ&edGdde&Z'ed Gd!d"e&e Z(gd#Z)dS)$zPyTorch Llava model.) dataclassN)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringloggingtorch_compilable_check)can_return_tuplemerge_with_config_defaults) AutoModel) LlavaConfigzJ Base class for Llava outputs, with hidden states and attentions.  custom_introc2eZdZUdZdZejdzed<dS)LlavaModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/llava/modeling_llava.pyrr$s7  59*T188888r%rzQ Base class for Llava causal language model (or autoregressive) outputs. ceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ejdzed<dZ e ejdzed<dZejdzed<dS) LlavaCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrr r)r!r"r#r*r+rr,tupler-rr$r%r&r(r(9s  &*D% d ")))'+FE  $+++$(OUT\(((59M5*+d299926Je'(4/66648*T188888r%r(c*eZdZdeffd ZdZxZS)LlavaMultiModalProjectorconfigctt|jtrdnt |j}t j|jj |z|j j |j |_ t|j|_t j|j j |j j |j |_dS)Nrbias)super__init__ isinstancevision_feature_layerintlenrLinear vision_config hidden_size text_configmultimodal_projector_biaslinear_1rprojector_hidden_actactlinear_2)selfr1num_feature_layers __class__s r&r6z!LlavaMultiModalProjector.__init__Xs ",V-H#"N"NtQQTWX^XsTtTt  ,/A A   *1    &56   *F,>,JQWQq    r%c||}||}||}|SN)r@rBrC)rDimage_featuresr,s r&forwardz LlavaMultiModalProjector.forwardfs; n55 //  m44 r%)rrrrr6rJ __classcell__rFs@r&r0r0WsS  {       r%r0c@eZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ dS)LlavaPreTrainedModelr1model)imagetextTr+N)rrrrr#base_model_prefixinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_flex_attn_supports_attention_backendr$r%r&rNrNmsV (&*##4"5N!"&r%rNzu The Llava model which consists of a vision backbone and a language model, without a language modeling head. c$eZdZdeffd Zeeed ddej de e e ze e zdzde dzd e dzd eed eezf d Zd ejdej dej fdZee dd ejdzdej dzdejdzdejdzdedzdej dzde e e ze e zdzde dzdejdzd eed eezfdZxZS) LlavaModelr1ct|tj|j|_t ||_tj|j|_ | dSrH) r5r6r from_configr< vision_towerr0multi_modal_projectorr>language_model post_initrDr1rFs r&r6zLlavaModel.__init__sm    %1&2FGG%=f%E%E"'3F4FGG r%zWObtains image last hidden states from the vision tower and apply multimodal projection.rN pixel_valuesr8vision_feature_select_strategyoutput_hidden_stateskwargsreturnc  d|D}|j|fddd| t|tr" j|}|dkr|ddddf}n6 fd|D}|dkr d|D}t j|d }||}|d }t j |d |j |jj z d  } t j|d | }nt!|}| _ S) Nci|] \}}||| SrHr$).0kvs r& z1LlavaModel.get_image_features..sCCC41aQ]!Q]]]r%T)rf return_dictdefaultrc*g|]}j|Sr$)r,)rk layer_idx image_outputss r& z1LlavaModel.get_image_features..s!ddd)}29=dddr%c*g|]}|ddddfS)Nrr$)rkhss r&rtz1LlavaModel.get_image_features..s(7772aaae9777r%dim image_sizes)devicer)itemsr_r7r9r,r!catr`get as_tensorr{ patch_sizeprodtolistsplitsqueezelist pooler_output) rDrdr8rerfrgselected_image_featurehs_poolrI split_sizesrss @r&get_image_featureszLlavaModel.get_image_featuressDC6<<>>CCC))  !%      *C 0 0 @%2%@AU%V "-::)?122)F&ddddOcdddG-::77w777%*YwB%?%?%? "334JKK ::m $ $ 0 !6~?TUUUY]YjYuu"  #[)?)?)B)BKPPNN!.11N&4 #r% input_ids inputs_embedsrIc |e||tj|jjtj|jk}|d}n||jjk}|}|j d|j dz}| d |j}t||j dz| kd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)dtyper{rwrrz6Image features and image tokens do not match, tokens: z , features: )get_input_embeddingsr!tensorr1image_token_idlongr{allsumshape unsqueezetornumel)rDrrrIspecial_image_maskn_image_tokensn_image_featuress r&get_placeholder_maskzLlavaModel.get_placeholder_masks  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11)/2^5I!5LL/99"==@@AUVV ]04 48L8L8N8N N s^ s saq s s   "!r%attention_mask position_idsr+rzc  |du|duz rtd|||}|||||| dj} t j| d|j|j} | ||| } | | | }|j d ||||d| } t| j | j| j| j|| ndS) Nz:You must specify exactly one of input_ids or inputs_embedsT)rdr8rerzrorrx)rrI)rrr+r)last_hidden_stater+r,r-rr$) ValueErrorrrrr!r}rr{rrmasked_scatterrarrr+r,r-)rDrrdrrr+rr8rerzrgrIroutputss r&rJzLlavaModel.forwards\  -t"; < [YZZ Z  7D5577 BBM  #!44)%9/M' 5  #Y~1===@@AUWdWjkkN!%!:!:~";"" *889K^\\M%$% )%+'      (%7#3!/)2>2JPT     r%)NNN) NNNNNNNNN)rrrrr6rrrr!r"r9rstrboolr rr.r r LongTensorrTensorrrrJrKrLs@r&r\r\}sQ { ^n DH59,0 --'-"DIoS 9D@-),d - #Tk - +, - + +---  -^")":?:K"]b]n""""0.215.204(,26CG59+// / #d*/ '$./  t+ / &- /  / (4// "DIoS 9D@/ ),d / \D(/ +,/  ) )/ / / ^/ / / / / r%r\zS The LLAVA model which consists of a vision backbone and a language model. ceZdZddiZdeffd ZdejfdZe dde j d e e e ze e zdzd edzd eedeezf d Zee dde jdzde j dzde jdzde jdzdedzde j dzd e e e ze e zdzd edzde jdzde e jzde jdzd eedeezfdZ dfd ZxZS)LlavaForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightr1ct|t||_t j|jj|jjd|_ | dS)NFr3) r5r6r\rOrr;r>r= vocab_sizelm_headrbrcs r&r6z&LlavaForConditionalGeneration.__init__se    '' y!3!?ASA^ejkkk  r%rhc|jSrH)r)rDs r&get_output_embeddingsz3LlavaForConditionalGeneration.get_output_embeddingss |r%Nrdr8rergc .|jjd|||d|S)N)rdr8rer$)rOr)rDrdr8rergs r&rz0LlavaForConditionalGeneration.get_image_featuress:-tz, %!5+I      r%rrrrr+rlabelslogits_to_keeprzc n|jd||||||||| d | } | d}t| trt| dn| }||dd|ddf}d}| |jd|| |jjjd| }t||| j | j | j | j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "USER: \nWhat's the content of the image? ASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed" ```) rrdrrr+rr8rerzrN)r*rr)r)r*r+r,r-rr$)rOr7r9slicer loss_functionr1r>rr(r+r,r-r)rDrrdrrr+rr8rerrrzrgrr, slice_indicesr*r)s r&rJz%LlavaForConditionalGeneration.forward+s\$*  %)%+'!5+I#        8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD+#3!/) ' ;     r%Fc tj|f|||||d|} |s|dds|| d<| S)N)r+rrris_first_iteration use_cacheTrd)r5prepare_inputs_for_generationr~) rDrr+rrdrrrrg model_inputsrFs r&rz;LlavaForConditionalGeneration.prepare_inputs_for_generationzst=uww<  +'))1       8VZZ T%B%B 8 ,8L (r%)NN) NNNNNNNNNrN)NNNNNF)rrr_tied_weights_keysrr6rModulerrr!r"r9rrr rr.r rrrrrr(rJrrKrLs@r&rr sv +,VW{ ryDH59   '  "DIoS 9D@  ),d  +,  + +    ^  .215.204(,26CG59*.-.+/K K #d*K '$.K  t+ K &- K  K (4/K "DIoS 9D@K ),d K  4'K el*K \D(K +,K  , ,K K K ^K ` r%r)rrNr\)*r dataclassesrr!r activationsr cache_utilsr generationrmodeling_outputsr r r modeling_utilsr processing_utilsr utilsrrrr utils.genericrrautorconfiguration_llavar get_loggerrloggerrr(rr0rNr\r__all__r$r%r&rs!!!!!! !!!!!! ))))))``````````------&&&&&&XXXXXXXXXXXXIIIIIIII,,,,,,  H % %   9 9 9 9 96 9 9   9  99999+99   90ry, ' ' ' ' '? ' ' ' F F F F F %F F  F R HHHHH$8/HH  HV R Q Qr%