tj?=.ddlmZddlZddlmZddlmZddlmZ ddl m Z ddl m Z mZddlmZmZdd lmZdd lmZmZmZd d lmZmZmZmZmZmZd d lm Z d dl!m"Z"m#Z#ej$e%Z&edeGdde Z'GddeZ(Gdde"Z)GddeZ*GddeZ+GddeZ,GddeZ-eGdd eZ.Gd!d"eZ/gd#Z0dS)$)CallableN)strict)initialization)PreTrainedConfig)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModel)eager_attention_forward)VisionRotaryEmbeddingapply_rotary_pos_emb_visionz&DeepGlint-AI/mlcd-vit-bigG-patch14-336) checkpointc(eZdZUdZdZdZdZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ eed<dZeeezeeefzed<dZeeezeeefzed<dZeed<dZeed<dZeezed<dZeed<dZeed<dS)MLCDVisionConfigav num_key_value_groups (`int`, *optional*, defaults to 1): Number of key-value groups used in Attention. Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configi hidden_sizei intermediate_size0num_hidden_layersnum_attention_headsnum_key_value_groupsr num_channelsiP image_size patch_sizegelu hidden_actgh㈵>layer_norm_epsattention_dropoutg{Gz?initializer_range?initializer_factorN)__name__ __module__ __qualname____doc__ model_typebase_config_keyrint__annotations__r r"r$r&r'r(listtupler*r,strr-floatr/r0r2c/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mlcd/modular_mlcd.pyrr)s-  &%J%OK!s!!!s!!!! !#!!!L#47Jd3i%S/177746Jd3i%S/1666J NE   %(us{(((#u### ######r@rceZdZdS)MLCDMLPNr3r4r5r?r@rArCrCQDr@rCceZdZdS)MLCDRotaryEmbeddingNrDr?r@rArGrGUrEr@rGcHeZdZdeffd ZdejdejfdZxZ S)MLCDVisionEmbeddingsconfigcNt||`dSN)super__init__position_embeddingselfrJ __class__s rArNzMLCDVisionEmbeddings.__init__Zs'     # # #r@ pixel_valuesreturncN|jd}|jjj}|||}|ddd}|j|dd}tj ||gd}|S)Nrdtyperr%dim) shapepatch_embeddingweightrWtoflatten transposeclass_embeddingexpandtorchcat)rQrS batch_size target_dtype patch_embeds class_embeds embeddingss rAforwardzMLCDVisionEmbeddings.forward^s!'* +28 ++LOO,O,O,OPP #++A..88A>> +22:q"EE Y l;CCC r@) r3r4r5rrNrc FloatTensorTensorrj __classcell__rRs@rArIrIYsl$/$$$$$$ E$5 %,        r@rIceZdZdZdeffd Z d dejdeejejfdejdzde e d eejejdzff d Z xZ S) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 rJcpt||j|_d|_dS)NF)rMrNr& is_causalrPs rArNzMLCDAttention.__init__ss1    $*$?!r@N hidden_statesposition_embeddingsattention_maskkwargsrTc |jdd\}}|||||j|jf}|||||j|jf}|||||j|jf} |dd} |dd} t||| | \}}| dddd }| dddd }| dddd } tj |jjt } | |||| |f|jsdn|j|j|jd|\} }| dddd } | ||d} || } | ddd } | |fS)NrXrr%rrr.)dropoutscalingrr)r[q_projreshape num_headshead_dimk_projv_proj unsqueezer>rpermute contiguousr get_interfacerJ_attn_implementationrtrainingrxscalerrviewout_proj)rQrsrtrurvre seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss rArjzMLCDAttention.forwardxst"/!4SbS!9 J{{=1199:zSWSacgcp:qrr [[//77ZQUQ_aean8opp {{=1199:zSWSacgcp:qrr "!$..q117799!!$..q117799#>|ZY\^a#b#b j$++Aq!Q77BBDD ''1a33>>@@ #++Aq!Q77BBDD (?(M K ,.E) ) %8$7      % $}>CC$,Jn % %  % % ! \"))!Q155@@BB !&&z:rBB mmK00 !))!Q22==?? L((r@rL) r3r4r5r6rrNrcrlr<r r rjrmrns@rArprpks//3 ,),)|,)#5<#=>,) t+ ,) +, ,) u|U\D00 1 ,),),),),),),),)r@rpc eZdZdeffd Z d dejdeejejfdejdzdee deej f d Z xZ S) MLCDEncoderLayerrJcrt|t||_dSrL)rMrNrp self_attnrPs rArNzMLCDEncoderLayer.__init__s.    &v..r@NrsrtrurvrTc |}||}|jd|||d|\}}||z}|}||}||}||z}|S)a Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. )rsrtrur?) layer_norm1r layer_norm2mlp)rQrsrtrurvresidual_s rArjzMLCDEncoderLayer.forwards$!((77 )4> ' 3)      q !=0  ((77 //  =0 r@rL) r3r4r5rrNrcrlr<r r rkrjrmrns@rArrs/////////3 ""|"#5<#=>" t+ " +, " u ! """"""""r@rc eZdZdZdeffd Z d dejdeej ej fdej dzde e d ee zf d Z xZS) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rJcJt|dS)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)rMrNrPs rArNzMLCDEncoder.__init__s!      r@N inputs_embedsrtrurvrTc P|}|jD] }||||fi|}t|S)a= Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)layersr)rQrrtrurvrs encoder_layers rArjzMLCDEncoder.forwardsa,& ![  M)M# MM+    r@rL)r3r4r5r6rrNrcrkr<rlr r rrjrmrns@rArrs!/!!!!!!/3 ! ! (! #5<#=>!  t+ ! +, !  ! ! ! ! ! ! ! ! r@rceZdZUeed<dZdgZdZdZdZ dZ dZ dZ e edZejfdZxZS)MLCDPreTrainedModelrJ vision_modelrTF)rs attentionsc`t||jj}t |t r|jj}t j|jd|j dz|zt j|j j |jj |zt j |jtj|jjddd St |t&r|jj}|j dzd|jjzdzz|z}|j dz|z}t j|jj |t j|jj |t j|jj |t j|jj |d St |t2r|jj}|jjdzd|jjzdzz|z}d|jjzdz|z}t j|jj |t j|jj |d St |t:rL|jj}|jj|jjzdzdz|z}t j|jd|d St |t@rVd|j!tjd |j"dtj# |j"z zz }t j |j$|d Sd S) zInitialize the weightsr.g)meanstd)rrX)r%rXrr1rrVN)%rM _init_weightsrJr2 isinstancerIinitnormal_ra embed_dimr\r]r0copy_ position_idsrcaranger[rbrpr"rzr~rrrCrfc1fc2MLCDVisionModelr$ class_pos_embrGthetarZr>inv_freq) rQmodulefactor in_proj_std out_proj_stdfc_std pos_emb_stdrrRs rArz!MLCDPreTrainedModel._init_weightss f%%%/ f2 3 3 2[3F L/cv?OQU?UX^?^ _ _ _ _ L/6FM<[^d>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```Nz You have to specify pixel_valuesrX)devicer%rrY)rrt)r pooler_outputr?) ValueErrorr[rJr*rcrrrrbstackr_rrdrrrri pre_layrnormencoderpost_layernormr )rQrSrvnum_patches_heightnum_patches_widthhpos_idswpos_idspos_idsrotary_pos_embembrtrsencoder_outputsr pooled_outputs rArjzMLCDVisionModel.forward9s:  ?@@ @)/3t{7MM(.r2dk6LL L+L4G H H H R RST U U \ \]_ar s s  L*<3F G G G Q QRS T T [ [\npr s s +x//1183C3C3E3EFBOOO55g>>D$6#GQOOOi8bAAA"wwyy#''))4 55 ))-88 &$, ' 3     ,A.)!!!Q'2 ++M:: )/'    r@rL) r3r4r5rrNrcrkr r r<r rjrmrns@rArr3sq/qqqqqq26> > '$.> +,>  + + > > > > > > > > r@r)rrr)1collections.abcrrctorch.nnrhuggingface_hub.dataclassesrrrconfiguration_utilsrmodeling_outputsrr modeling_utilsr r processing_utilsr utilsr rrclip.modeling_cliprrrrrrllama.modeling_llamarqwen2_vl.modeling_qwen2_vlrr get_loggerr3loggerrrCrGrIrprrrr__all__r?r@rArs-%$$$$$ ......&&&&&&333333KKKKKKKKFFFFFFFF&&&&&&@@@@@@@@@@;:::::[[[[[[[[  H % %CDDD#$#$#$#$#$'#$#$ED#$L     g        /   /$9)9)9)9)9)M9)9)9)x'''''''''T. . . . . +. . . b-2-2-2-2-2/-2-2-2`D D D D D oD D D N   r@