tjoddlZddlmZddlmZddlZddlmZddlm Z ddlm Z m Z m Z ddl mZddlmZdd lmZmZmZmZdd lmZdd lmZdd lmZmZmZdd lm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ej*e+Z,ede GddeZ-eGddeZ.eGddeZ/Gddej0Z1eGdde)Z2edGd d!e(Z3Gd"d#e&Z4eGd$d%e2Z5ed&Gd'd(e2Z6ed)Gd*d+e2Z7gd,Z8dS)-N) dataclass)Literal)strict)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)PreTrainedConfig)BaseModelOutputMaskedLMOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)CONFIG_MAPPING AutoConfig AutoModel)ModernBertPredictionHead) SmolVLMModelSmolVLMPreTrainedModelModernVBERT/modernvbert) checkpointceZdZUdZdZeedZdZee zdze d<dZ ee zdze d<dZ e e d<d Ze e d <d Zee d <d Zee d<dZede d<dZee ze d<dZee d<dZee d<fdZxZS)ModernVBertConfiga? pixel_shuffle_factor (`int | None`, *optional*, defaults to 4): Scale factor used by any pixel-shuffle / upsampling operations in the vision head. initializer_cutoff_factor (`float | None`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. classifier_pooling (`Literal["cls", "mean"]`, *optional*, defaults to `"cls"`): The pooling strategy to use for classification tasks. classifier_bias (`bool | None`, *optional*, defaults to `False`): Whether to add a bias term to the classification head Example: ```python >>> from transformers import ModernVBertConfig >>> # Initializing configuration >>> configuration = ModernVBertConfig() >>> # Initializing a model from the configuration (model class is implemented in >>> # `modernvbert.modeling_modernvbert`) >>> from transformers import ModernVBertModel >>> model = ModernVBertModel(configuration) >>> # Accessing the model configuration >>> cfg = model.config ``` modernvbert) text_config vision_configNr"r#iimage_token_idpixel_shuffle_factorg{Gz?initializer_range@initializer_cutoff_factorcls)r*meanclassifier_poolingclassifier_dropoutFclassifier_biastie_word_embeddingsc |jtd|_n6t|jtrtddi|j|_|jtd|_n6t|jtrtddi|j|_t jdi|dS)N modernbertsiglip_vision_model)r"r isinstancedictr#super __post_init__)selfkwargs __class__s q/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/modernvbert/modular_modernvbert.pyr8zModernVBertConfig.__post_init__Xs   #-l;==D   ($ / / P-l;OOd>NOOD    %!/0E!F!H!HD   *D 1 1 ]!/0E!F!\!\I[!\!\D ''''''')__name__ __module__ __qualname____doc__ model_typer sub_configsr"r r6__annotations__r#r$intr&r'floatr)r,rr.r/boolr0r8 __classcell__r;s@r<r r ,s  6J",zJJK26K!D(4/66648M#d*T1888NC !#!!!#u###'*u***16 .666&) )))!OT!!! %%%% ( ( ( ( ( ( ( ( (r=r ceZdZUdZdZejed<dZe ejdzed<dZ e ejdzed<dZ e ejdzed<dS)ModernVBertBaseModelOutputaY Base class for ModernVBERT model's outputs. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlast_hidden_state hidden_states attentionsimage_hidden_states) r>r?r@rArLtorch FloatTensorrDrMtuplerNrOr4r=r<rKrKfs,,0u(///59M5*+d299926Je'(4/666;?u01D8?????r=rKceZdZUdZdZejdzed<dZejed<dZ e ejdfdzed<dZ e ejdfdzed<dZ ejdzed<dS) ModernVBertMaskedLMOutputaG Base class for ModernVBERT model's outputs with masked language modeling loss. Args: loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided): Masked language modeling (MLM) loss. logits (`torch.FloatTensor`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlosslogits.rMrNrO) r>r?r@rArUrPrQrDrVrMrRrNrOr4r=r<rTrTs,&*D% d "))) $FE $$$:>M5*C/047>>>7;Je',-4;;;48*T188888r=rTc.eZdZdZfdZdZdZxZS)ModernVBertConnectorz Connector module for ModernVBERT. It performs a pixel shuffle operation followed by a linear projection to match the text model's hidden size. Based on https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html ct|j|_tj|jj|jdzz|jjd|_dS)NrFbias) r7__init__r&nnLinearr# hidden_sizer"modality_projectionr9configr;s r<r\zModernVBertConnector.__init__se $*$?!#%9  ,0KQ0N O   *$ $ $    r=c|\}}}t|dzx}}|||||}|||t||z ||z}|dddd}||t||z t||z ||dzz}|dddd}||t||dzz ||dzzS)Ng?rrr )sizerEviewpermutereshape)r9rOr& batch_size seq_length embed_dimheightwidths r< pixel_shufflez"ModernVBertConnector.pixel_shuffles=,?,D,D,F,F) J Z_---166z65R[\\166 E,@$@ A A9OcCc  299!Q1EE199  ,, - - -- . . -q0 1    299!Q1EE"** J*>*ABCCYRfhiRiEj   r=cb|||j}||SN)rnr&r`)r9rOs r<forwardzModernVBertConnector.forwards1"001DdF_``''(;<<r?r@rAr\rnrqrHrIs@r<rXrXs`         &=======r=rXcFeZdZeZgZejdZdS)ModernVBertPreTrainedModelctj|dtjdtffd }t |t rFjjtj djj j zz }||j |dSt |trFjjtj djj j zz }||j|dSt |t t"frCjjtj jj jz }||j|dSdS)Nmodulestdctjdd}tj|jd|| |z||zt |t jt jfr"|j tj |j dSdSdS)Nr)r(r-)r+rvab) getattrrbinit trunc_normal_weightr5r]r^Conv2dr[zeros_)rurv cutoff_factorr9s r< init_weightz=ModernVBertPreTrainedModel._init_weights..init_weights#DK1LcRRM    .3&#%     &29bi"899 -;*K ,,,,, - -**r=r()r _init_weightsr]ModulerFr5rXrbr'mathsqrtr"num_hidden_layersr`ModernVBertForMaskedLMlm_head$ModernVBertForSequenceClassification!ModernVBertForTokenClassificationr_ classifier)r9rurout_std final_out_stds` r<rz(ModernVBertPreTrainedModel._init_weightssU%dF333 - - - - - - - - f2 3 3 :k3didkF]Fo@o6p6ppG K2G < < < < <  6 7 7 :k3didkF]Fo@o6p6ppG K 0 0 0 0 0  41    :!K9DIdkF]Fir?r@r config_class_no_split_modulesrPno_gradrr4r=r<rsrss@$LU]__::_:::r=rsaG ModernVBertModel is a model that combines a vision encoder (SigLIP) and a text encoder (ModernBert). ModernVBert is the base model of the visual retriever ColModernVBert, and was introduced in the following paper: [*ModernVBERT: Towards Smaller Visual Document Retrievers*](https://arxiv.org/abs/2510.01149). ) custom_introceZdZdeffd Zeedd ddejdej dzd ejdzd ej dzd ej dzd ej dzd ej dzde e deezfdZxZS)ModernVBertModelrbct|t||_t j|j|_t j|j|_ t|jj |jj zdz|j dzz |_|dS)Nr)r7r\rX connectorr from_configr" text_modelr# vision_modelrE image_size patch_sizer& image_seq_len post_initras r<r\zModernVBertModel.__init__s    .f55#/0BCC%1&2FGG "-1E1PPUV V*A- /   r= Inputs fed to the model can have an arbitrary number of images. To account for this, pixel_values fed to the model have image padding -> (batch_size, max_num_images, 3, max_heights, max_widths) where max_num_images is the maximum number of images among the batch_size samples in the batch. Padding images are not needed beyond padding the pixel_values at the entrance of the model. For efficiency, we only pass through the vision_model's forward the real images by discarding the padding images i.e. pixel_values of size (image_batch_size, 3, height, width) where image_batch_size would be 7 when num_images_per_sample=[1, 3, 1, 2] and max_num_images would be 3. rrrN input_idsattention_mask position_ids inputs_embeds pixel_valuespixel_attention_maskrOr:returnc |:|j||j}||||j}|9||j|j}||||}|jd|||d|} t| j | j | j |S)a| pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. N)rr)dtypedevice)rrrO)rrr)rLrMrNrOr4) rget_input_embeddingstorget_image_features pooler_outputr inputs_mergerrKrLrMrN) r9rrrrrrrOr:outputss r<rqzModernVBertModel.forward s>  BDO@@BB9MMPPQZQabbM  #"&"9"9)@T#:##  *"5"8"8}?R[h[o"8"p"p  ..#=Vi/M "$/ ')%     *%7!/) 3     r=)NNNNNNN)r>r?r@r r\rrrP LongTensorTensorrQ BoolTensorrrrRrKrqrHrIs@r<rrs@0 ^ -   '+.20426158<8</ / #/  t+/ &- / (4/ / '$. / $.5/ #.5/ +,/  + +/ / /   / / / / / r=rceZdZdS)ModernVBertPredictionHeadN)r>r?r@r4r=r<rrJsDr=rc<eZdZddiZdeffd ZdZdZee dd  dd e j d e j d zd e j d zde j d zde j d zde jd zde j d zde j d zdeedeezfdZxZS)rzlm_head.weightz1model.text_model.embeddings.tok_embeddings.weightrbcXt||jj|_t ||_t |j|_tj |jj |j|jj |_ | dS)NrZ)r7r\r" vocab_sizermodelrprojection_headr]r^r_ decoder_biasrrras r<r\zModernVBertForMaskedLM.__init__Rs     ,7%f-- 89KLLy!3!?W]WiWvwww  r=c|jSrpr)r9s r<get_output_embeddingsz,ModernVBertForMaskedLM.get_output_embeddings^s |r=c||_dSrpr)r9new_embeddingss r<set_output_embeddingsz,ModernVBertForMaskedLM.set_output_embeddingsas % r=rrrNrrrrrrrOlabelsr:rc f|jd|||||||d| } | d} ||| } d} |Ft}|| d|j|d} t | | | j| j| j S) pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., text_config.]` or `model.image_token_id`. Tokens with indices set to `model.image_token_id` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.]`. rrrrrrrOrN)rUrVrMrNrOr4) rrrrrfrrTrMrNrO)r9rrrrrrrOrr:rrMrVrU criterions r<rqzModernVBertForMaskedLM.forwarddsH$*  )%'%!5 3        d22=AABB  (**I9V[[T_==v{{2OOD(!/) ' ;     r=NNNNNNNN)r>r?r@_tied_weights_keysr r\rrrrrPrrrQrrrrRrTrqrHrIs@r<rrNs*,_` 0      &&&^ -   '+.20426158<8<*.0 0 #0  t+0 &- 0 (4/ 0 '$. 0 $.50 #.50  4'0 +,0  * *0 0 0   0 0 0 0 0 r=rza The ModernVBert Model with a sequence classification head on top that performs pooling. c(eZdZdeffd Zeedd ddejdej dzd ejdzd ej dzd ej dzd ej dzd ej dzdejdzde e deezfdZxZS)rrbct||j|_||_t ||_t |j|_tj |j |_ tj |jj|j|_|dSrp)r7r\ num_labelsrbrrrr"headr]Dropoutr.dropr^r_rrras r<r\z-ModernVBertForSequenceClassification.__init__s     + %f-- -f.@AA Jv899 )F$6$BFDUVV r=rrrNrrrrrrrOrr:rc L|jd|||||||d| } | d} |jjdkr | dddf} n|jjdkr||jdd\} } n|jdd\} } ||jn|j}|#t j| | f|t j}| |dz d | d d z } | | }| |}| |}d}|Z|jj f|jd kr d |j_ nN|jd kr7|jt jks|jt jkr d|j_ n d|j_ |jj d krWt%}|jd kr1|||}n|||}n|jj dkrGt)}||d|j|d}n*|jj dkrt-}|||}t/||| j| jS)rrrr*Nr+r)rrrrd)dimT)rkeepdim regressionsingle_label_classificationmulti_label_classificationrUrVrMrNr4)rrbr,shaperrPonesrG unsqueezesumrrr problem_typerrlongrErsqueezerrfrrrMrN)r9rrrrrrrOrr:rrLriseq_lenr pooled_outputrVrUloss_fcts r<rqz,ModernVBertForSequenceClassification.forwardsF$*  )%'%!5 3      $AJ ; )U 2 2 1!!!Q$ 7   [ +v 5 5(&3&9"1"&=# GG&/obqb&9# G)2)>Y%%MDXF%!&Z,A&X]Xb!c!c!c!2^5M5Mb5Q5Q!Q V V[\ V ] ]`n`r`rtasaa!  "344  -00 //  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     r=r)r>r?r@r r\rrrPrrrQrrrrRrrqrHrIs@r<rrsh 0      ^ -   '+.20426158<8<*.Q Q #Q  t+Q &- Q (4/ Q '$. Q $.5Q #.5Q  4'Q +,Q  ) )Q Q Q   Q Q Q Q Q r=rzw The ModernVBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. c(eZdZdeffd Zeedd ddejdej dzd ejdzd ej dzd ej dzd ej dzd ej dzdejdzde e deezfdZxZS)rrbcrt||j|_t||_t |j|_tj |j |_ tj |jj |j|_|dSrp)r7r\rrrrr"rr]rr.rr^r_rrras r<r\z*ModernVBertForTokenClassification.__init__s     +%f-- -f.@AA Jv899 )F$6$BFDUVV r=rrrNrrrrrrrOrr:rc |jd|||||||d| } | d} || } || } || } d} |Ft }|| d|j|d} t| | | j| j S)rrrNrrr4) rrrrrrfrrrMrN)r9rrrrrrrOrr:rrLrVrUrs r<rqz)ModernVBertForTokenClassification.forward*sH$*  )%'%!5 3      $AJ II&788 II&788!233  '))H8FKKDO<r?r@r r\rrrPrrrQrrrrRrrqrHrIs@r<rrsU 0      ^ -   '+.20426158<8<*.1 1 #1  t+1 &- 1 (4/ 1 '$. 1 $.51 #.51  4'1 +,1  & &1 1 1   1 1 1 1 1 r=r)r rsrrrr)9r dataclassesrtypingrrPtorch.nnr]huggingface_hub.dataclassesrrrrr r{configuration_utilsr modeling_outputsr r rrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrautorrrmodernbert.modeling_modernbertrsmolvlm.modeling_smolvlmrr get_loggerr>loggerr rKrTrrXrsrrrrr__all__r4r=r<rsE !!!!!! ......AAAAAAAAAA&&&&&&333333 .-----&&&&&&@@@@@@@@@@------8888888888EEEEEEKKKKKKKK  H % %45555(5(5(5(5((5(5(655(p @@@@@@@ @: 9999999 9<$=$=$=$=$=29$=$=$=N$:$:$:$:$:!7$:$:$:NM M M M M |M M M `     8   R R R R R 7R R R j l l l l l +El l  l ^ K K K K K (BK K  K \   r=