tj<ddlZddlmZddlZddlZddlmZddlmcm Z ddl m Z ddlm Z ddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z ddl!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5ddl6m7Z7m8Z8ddl9m:Z:m;Z;mm?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHddlImJZJddlKmLZLmMZMe,jNeOZPe*de GddeZQe*de Gdd eZRe*de Gd!d"eZSGd#d$e;ZTGd%d&eCZUGd'd(e?ZVGd)d*e@ZWGd+d,ejXZYGd-d.ejXZZGd/d0eGZ[Gd1d2eHZ\Gd3d4e<Z]d5Z^dQd7Z_Gd8d9ejXZ`Gd:d;e:ZaGd<d=eZbGd>d?eDZcGd@dAeEZdGdBdCedZeGdDdEeFZfGdFdGeJZgGdHdIeAZhGdJdKeBZiGdLdMeMZjGdNdOeLZkgdPZldS)RN)Callable)strict) LayerNorm)initialization)ACT2FN)Cache DynamicCache)PreTrainedConfig)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPooling)RopeParameters)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)deprecate_kwarg)accepts_precomputed_kwargsmaybe_autocastmerge_with_config_defaults)capture_outputs)get_vision_cu_seqlensget_vision_position_ids)Glm4MLP Glm4RMSNormGlm4RotaryEmbeddingeager_attention_forward) Qwen2_5_VisionPatchEmbedQwen2_5_VisionRotaryEmbedding Qwen2_5_VLCausalLMOutputWithPast"Qwen2_5_VLForConditionalGeneration Qwen2_5_VLMLPQwen2_5_VLModelOutputWithPastQwen2_5_VLPreTrainedModelQwen2_5_VLTextModelQwen2_5_VLVisionAttentionQwen2_5_VLVisionBlock) Qwen2VLModel)Qwen2VLProcessorQwen2VLProcessorKwargszzai-org/GLM-4.1V-9B-Thinking) checkpointcleZdZUdZdZdZdZeed<dZ eed<dZ e ed <d Z e ed <d Zeezed <dZeed<dZeed<dZeeezeeefzed<dZeeezeeefzed<dZeed<dZeed<dZeeezeeefzed<dZeed<dZeed<dZeed <d!S)"Glm4vVisionConfiga, out_hidden_size (`int`, *optional*, defaults to 4096): The output hidden size of the vision model. Example: ```python >>> from transformers import Glm4vVisionConfig, Glm4vVisionModel >>> # Initializing a Glm4vVisionConfig GLM-4.1V-9B style configuration >>> configuration = Glm4vVisionConfig() >>> # Initializing a model (with random weights) from the GLM-4.1V-9B configuration >>> model = Glm4vVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` glm4v_vision vision_configdepthi hidden_sizesilu hidden_actFattention_biasattention_dropout num_headsr in_channelsiP image_size patch_sizeh㈵> rms_norm_epsr!spatial_merge_sizetemporal_patch_sizeout_hidden_size5intermediate_size{Gz?initializer_rangeN)__name__ __module__ __qualname____doc__ model_typebase_config_keyr9int__annotations__r:r<strr=boolr?floatrArBrClisttuplerErGrHrIrKrMrOe/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/glm4v/modular_glm4v.pyr5r5Es]  & J%OE3OOOKJ ND   %(us{(((IsK47Jd3i%S/177746Jd3i%S/1666L%=>tCy5c?:>>>OS"s"""#u#####r^r5cReZdZUdZdZdZdgZddddddd Zd gd gfd d gd gfd gd gfdZdhZ dZ e e d<dZ e e d<dZe e d<dZe e d<dZe e d<dZe dze d<dZee d<dZe e d <d!Zee d"<d#Zee d$<d%Zee d&<d'Zee ze d(<dZeezdze d)<dZe dze d*<fd+ZxZ S),Glm4vTextConfiga Example: ```python >>> from transformers import Glm4vTextModel, Glm4vConfig >>> # Initializing a GLM-4.1V style configuration >>> configuration = Glm4vConfig() >>> # Initializing a model from the GLM-4.1V style configuration >>> model = Glm4vTextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` glm4v_text text_configpast_key_valuescolwiserowwisecolwise_gather_outputrowwise_split_input)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnorm mrope_sectioniP vocab_sizerJr:rLrM(num_hidden_layers num_attention_headsr!Nnum_key_value_headsr;r<imax_position_embeddingsrNrOrFrGT use_cacher>r?rope_parameters pad_token_idc `|j |j|_tjdi|dS)Nr])rvrusuper __post_init__selfkwargs __class__s r_r}zGlm4vTextConfig.__post_init__s:  # +'+'?D $'''''''r^)!rPrQrRrSrTrUkeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planignore_keys_at_rope_validationrqrVrWr:rMrsrurvr<rXrwrOrZrGrxrYr?ryrdictrzr} __classcell__rs@r_raraos  J#O#4"5&/%.%.%.%<"7 &(9:#%568IJ!"_$56 '6%6"JK"s"""s!!!!&'t'''J#(S(((#u###L%It%(us{(((48O^d*T1888#L#*###(((((((((r^raceZdZUdZdZeedZdgZdZ e e zdze d<dZ e e zdze d<dZee d <d Zee d <d Zee d <dZee d<dZee d<dZee d<dZee d<fdZxZS) Glm4vConfiga image_start_token_id (`int`, *optional*, defaults to 151339): The image start token index to encode the start of image. image_end_token_id (`int`, *optional*, defaults to 151340): The image end token index to encode the end of image. video_start_token_id (`int`, *optional*, defaults to 151341): The video start token index to encode the start of video. video_end_token_id (`int`, *optional*, defaults to 151342): The video end token index to encode the end of video. ```python >>> from transformers import Glm4vForConditionalGeneration, Glm4vConfig >>> # Initializing a GLM-4.1V style configuration >>> configuration = Glm4vConfig() >>> # Initializing a model from the GLM-4.1V style configuration >>> model = Glm4vForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```glm4v)r7rcrdNrcr7i/Oimage_token_idi0Ovideo_token_idi+Oimage_start_token_idi,Oimage_end_token_idi-Ovideo_start_token_idi.Ovideo_end_token_idFtie_word_embeddingsc t|jtr|jddi|j|_n|j|jddi||_t|jtr|jddi|j|_n|j|jddi||_t jdi|dS)Nr7rcr]) isinstancer7r sub_configsrcr|r}r~s r_r}zGlm4vConfig.__post_init__s d($ / / M!B!1/!B!X!XTEW!X!XD     '!B!1/!B!L!LV!L!LD  d& - - I>t/ >RRAQRRD     %>t/ >HHHHD '''''''r^)rPrQrRrSrTr5rarrrcrr rWr7rrVrrrrrrrYr}rrs@r_rrs   .J$5oVVK#4"526K((4/66648M4**T1888 NC    NC   &#&&&$$$$ &#&&&$$$$ %%%% ( ( ( ( ( ( ( ( (r^rceZdZdS) Glm4vRMSNormNrPrQrRr]r^r_rrDr^rc&eZdZddeffd ZxZS) Glm4VisionMlpFbiascdt|||j|_dSN)r|__init__rKrM)rconfigrrs r_rzGlm4VisionMlp.__init__s. &&&!'!7r^F)rPrQrRrYrrrs@r_rrsI88T8888888888r^rceZdZdeddfdZdS)Glm4vVisionPatchEmbedrreturnNctj||j|_|j|_|j|_|j|_|j|j|jg}tj|j|j|||_ dS)N) kernel_sizestride) nnModulerrErIrBr: embed_dimConv3dproj)rrrs r_rzGlm4vVisionPatchEmbed.__init__sy 4    +#)#= !-+/$/R Id.K`klll r^)rPrQrRr5rr]r^r_rrs?m0mTmmmmmmr^rceZdZdS)Glm4vVisionRotaryEmbeddingNrr]r^r_rrrr^rc ZeZdZd dededededdf fd Zd ejdejfd Z xZ S) Glm4vVisionPatchMergerFdim context_dimr<rrNcttj||||_t ||_tj||||_tj||||_tj||||_ tj |_ t||_ dS)Nr)r|rrLinearrrpost_projection_norm gate_projup_proj down_projGELUact1ract_fn)rrrr<rrs r_rzGlm4vVisionPatchMerger.__init__s Ic3T222 $-cNN!3 $???yk=== ;$???GII Z( r^ hidden_statec||}|||}||||||zSr)rrrrrrr)rrs r_forwardzGlm4vVisionPatchMerger.forwardsnyy.. yy!:!:1D!-"$,t/A4>"R"R#,   r^rc|jj}|jd}|j}t |t r!t j||t j}|jd} t| dz} | | | | ddd d |t j} |jd} t j| |j} | d|d dkd}||df t j}||df t j}|dz|z dzdz }|dz|z dzdz }t j||fd  d d}t'j| ||jd d }|dd dd}| |j |j}||z}|S) a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. )devicedtyperg?r!rrrFborder)mode align_corners padding_mode)rweightshaperrr[rtensorlongrVviewpermute unsqueezetofloat32arangecumsumsumstackF grid_samplersqueezer)r embeddingslengths image_shapesh_coordsw_coordspos_embed_weightr:r orig_size_sq orig_size pos_embed_2d num_tokenstoken_positionsseq_idstarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32adapted_pos_embeds r_rzGlm4vVisionEmbeddings.forwardsi 29&,Q/ !( gt $ $ Ml76LLLG(-a0  c)**  ! !)Y D D WQ1   Yq\\ RvU]R 3 3  %a( ,z*:KLLL",,Q//7>>!3D3D3N3Nq3Q3QQVVWXYY +..U].CC +..U].CCc>X-2Q6c>X-2Q6{FF+444>>qAAKKANN#$- $T%=Uai# # #  "9!@!@!C!C!K!KB!O!O!W!WXY[\!]!]2556F6LMMPPQ[Qbcc "33 r^) rPrQrRr5rrrrrrs@r_rr sd -0 - - - - - -:PUP\::::::::r^rc(eZdZdeddffd ZxZS)Glm4vVisionAttentionrrNc t||j|_tj|j|jdz|j|_tj|j|jd|_dS)NrrF) r|rr?rrr:r=qkvrrs r_rzGlm4vVisionAttention.__init__Xsn    !'!99V/1Ca1GfNcdddIf0&2D5QQQ r^)rPrQrRr5rrrs@r_rrWsXR0RTRRRRRRRRRRr^rc eZdZdfd ZxZS)Glm4vVisionBlockrNct|t|j|j|_t|j|j|_t||_t|d|_ dS)NepsFr) r|rrr:rGnorm1norm2rattnrmlprs r_rzGlm4vVisionBlock.__init__`sv    !&"4&:MNNN !&"4&:MNNN (00  e444r^rN)rPrQrRrrrs@r_rr_s=5555555555r^rc2eZdZddeffd ZdZdZxZS)Glm4vTextRotaryEmbeddingNrct|jdgd|_dS)Nrp)r@r@)r|rrygetrp)rrrrs r_rz!Glm4vTextRotaryEmbedding.__init__is> #377UUr^c^|jdddddfd|jddd}|dddddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}| ||j }tj ||fd }||jz}||jz} dddn #1swxYwY||j | |j fS) NrrrmpscpuF) device_typeenabledr!rr)inv_freqrZexpandrrrtyperXr transpose apply_mroperprcatcosattention_scalingsinrr) rx position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r_rz Glm4vTextRotaryEmbedding.forwardms!M$aaa*=>DDFFMMaQ]QcdeQfhjlmnn ,QQQ4] ; A A C C'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E$$UD,>??E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5vvAGv$$cff17f&;&;;;s9B)E..E25E2c|}||d}tjdt|Dd}|S)Nrrc*g|]\}}||dzS)rr]).0ichunks r_ z8Glm4vTextRotaryEmbedding.apply_mrope..s$KKKXQE!a%LKKKr^)splitrr enumerate)rr!rpsectionchunksresults r_rz$Glm4vTextRotaryEmbedding.apply_mrope}sMW"--KK69J9JKKKQSTTT r^r)rPrQrRrarrrrrs@r_r r hsoVVVVVVVV<<< r^r c|ddddf}|ddddf}tj| |fddS) z*Rotates half the hidden dims of the input..rNr!rrr)rrflatten)rx1x2s r_rotate_half_llmr3sQ 319B 319B ;Ryb ) ) ) 1 1" 5 55r^rcT||}||}|dd|jddzfdd}|dd|jddzfdd}|jd}|dd|f|d|df}}|dd|f|d|df} }||zt||zz} ||zt||zz} t j| |gd} t j| | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nrr!r)rrrepeat_interleaver3rr) qkrr unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds r_apply_rotary_pos_embr@s\$ -- & &C -- & &C c'SYr]a''' ( : :1" : E EC c'SYr]a''' ( : :1" : E EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{u55;@PW\]]] r^rkposition_embeddingsrlrdrrc d|\}}}||} ||} ||} | ||d|jdd} | ||d|jdd} | ||d|jdd} |\} } t| | | | \} } ||| | |j \} } tj |j j t}||| | | |f|jsdn|j|jd|\}}|||d}||}||fS)Nrrr!r>)dropoutrH)sizerIrJrKrrErr@updaterCr get_interfacer_attn_implementationr%trainingr?rHreshape contiguousrL)rrkrNrlrdrbszq_len_ query_states key_states value_statesrrattention_interface attn_output attn_weightss r_rzGlm4vTextAttention.forwards&**,, UA{{=11 [[// {{=11 #((eRGGQQRSUVWW __S%T]CCMMaQRSS #((eRGGQQRSUVWW &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \"))#ub99DDFF kk+.. L((r^r)NNN)rPrQrRrSrarVrrrr\r rr rrrs@r_rBrBs ^^^3:^^^^^^.IM.2(, ))))|))#5<#=>E)) t+ ))  )) -. )) u|U\D0% 2E2LL M))))))))))))))))r^rBceZdZdS) Glm4vTextMLPNrr]r^r_rbrbrr^rbc eZdZdedeffd Ze ddejde ejejfdzdejdzd ej dzd e dzd e dzd e ej e ej ej fdzffd ZxZS)Glm4vTextDecoderLayerrrCct|j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ dS)Nr) r|rr:rB self_attnrbrrrGinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormrMs r_rzGlm4vTextDecoderLayer.__init__s !-+FI>>''+F,>FDWXXX(4V5GVM`(a(a(a%(4V5GVM`(a(a(a%".v/AvGZ"["["[r^NFrkrNrlrrdrxrc "|}||}|jd||||||d|\}} ||}||z}|}||}||}||}||z}|S)N)rkrNrlrrdrxr])rgrfrirhrrj) rrkrNrlrrdrxrresidualrZs r_rzGlm4vTextDecoderLayer.forwards!,,];; *4> ' 3)%+      q55mDD  =0 !55mDD // // >>  =0 r^)NNNNF)rPrQrRrarVrrrrr\ LongTensorr rY FloatTensorrrrs@r_rdrds\\3\\\\\\IM.204(,!&##|##5<#=>E# t+ # &- #  #$;# u %(95;L(L"MPT"TT U###^#####r^rdceZdZdS)Glm4vModelOutputWithPastNrr]r^r_rprp/rr^rpceZdZddgZdZdS)Glm4vPreTrainedModelrdrctj||t|trVd|jt jd|jdt j|jz zz }tj |j |dSdS)Ng?rr!r) r _init_weightsrrthetarrrrZinitcopy_r)rmodulers r_rtz"Glm4vPreTrainedModel._init_weights6s%dF333 f8 9 9 2flu|Avz1TYT_/`/`/`cicm/mnoH Jv 1 1 1 1 1 2 2r^N)rPrQrR_no_split_modulesrtr]r^r_rrrr3s/02DE22222r^rrc eZdZUeed<dZdgZeedZ d fd Z dZ e e ed ejd ejd eedeezfd ZxZS)Glm4vVisionModelr)imagevideorrk attentionsrNctj|_j|_t |_t |_jj z}t|dz|_ tj fdtjD|_t#jjj|_t-jj|_tjjjjj|_t-jj|_d|_|dS)Nr!c.g|]}tSr])r)r%rZrs r_r(z-Glm4vVisionModel.__init__..Qs"$[$[$[!%5f%=%=$[$[$[r^)rrr<r)rB out_channelsrrF)r|rrHrErrr patch_embedr:rArrotary_pos_embr ModuleListranger9blocksrrKrMr<mergerrrGpost_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)rrrErs ` r_rzGlm4vVisionModel.__init__FsS    "("; +/77088%)998QGGm$[$[$[$[uV\GZGZ$[$[$[\\ ,&F4LY_Yj    $00BH[#\#\#\ )*/1,     +6+=6CVWWW&+# r^ctjd|jjdtdt ||j}||}||fS)N`z.rot_pos_emb` is deprecated and will be removed in v5.11. Use `get_vision_position_ids` from `transformers.vision_utils` and apply the rotary embedding module.r!) stacklevel)warningswarnrrP FutureWarningr rHr)rgrid_thwrrs r_ rot_pos_embzGlm4vVisionModel.rot_pos_embbsr  I' I I I     /x9PQQ ,,\::|++r^rkrrc t||j|}t||}||}||}||}t j||fd}|| f}|dd|ddz } | || ||dddf |j |dddf |j }|j D]} | |f||d|}||}|d|j|j|jd}|dddd }||d|jj}||} t-|| S) a\ hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. )rrrrNr) cu_seqlensrNrr!)last_hidden_state pooler_output)r rHrrrrrrrrrrrrrrrrrrrKrr) rrkrrrr rotary_embr"rNseqlensblkmerged_hidden_statess r_rzGlm4vVisionModel.forwardls/x9PY_``` *8FCCC ((77 00?? ((66 iZ0b999"wwyy#''))4QRR.:crc?2    A  ! !-"6 7 7 A  ! !-"6 7 7   ;  CC%$7 MM++M:: %** ')@-BUVXBY  &--aAq99  66;;B @[\\ #{{=99)+.    r^r )rPrQrRr5rWinput_modalitiesryrr_can_record_outputsrrrrrrrrrr\rrrrs@r_r{r{=s )+,)* 8,,, 3 "\3 5:\3 MSTfMg3 + +3 3 3 ^_ 3 3 3 3 3 r^r{ceZdZeedZdeffd Zee e dde j dzde j dzde j dzdedzd e jdzd edzd eed eezfd ZxZS)Glm4vTextModelr~rc,ttjfdt jD|_tjj |_ t|_ |` |`dS)Nc0g|]}t|Sr])rd)r%rCrs r_r(z+Glm4vTextModel.__init__..s$ g g g) "69 5 5 g g gr^rr)r|rrrrrsrnrr:rGror rrThas_sliding_layersrs `r_rzGlm4vTextModel.__init__s    m g g g guVMeGfGf g g g   !!39LMMM 2&AAA  %  # # #r^Nrirlrrdrjrxrrc p|du|duz rtd|r5|3tjst |j}|||}|y||nd}tj|j d|j |z}| ddd d|j dd}n3|j dkr(|d  d|j dd}|j dkr$|j dd kr|d} |dd}nd} |j|||| d } tdi| } |} || | } |jD]}|| f| | || d |}|} || } t%| |S)N:You must specify exactly one of input_ids or inputs_embedsrrrrrrr!)N.)rrjrlrdr)r)rlrrdrN)rrdr]) ValueErrorrjit is_tracingr rrmget_seq_lengthrrrrrndimr rrnror)rrirlrrdrjrxrpast_seen_tokenstext_position_ids mask_kwargs causal_maskrkrN decoder_layer layer_outputss r_rzGlm4vTextModel.forwards:  -t"; < [YZZ Z  ?09M9M9O9O0*$+>>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL',,Q266==aATUVAWY[\\LL  ! # #' 299!\=OPQ=RTVWWL   ! !l&8&;q&@&@ ,Q '+LL!% k*,.-   )77;77 % "oom,oWW![ * *M)M*. /$7   M*MM -00 &++    r^)NNNNNN)rPrQrRrdrBrrarrrrrrmrr rnrYrr r\rrrrs@r_rrs7.( $$$$$$$.2.204(,26!%J J #d*J  t+J &- J  J (4/ J $;J -.J  ( (J J J _ ^J J J J J r^rceZdZddgZfdZedee ddej dej dzd e e d e ezfd Z dd ej d ej dej dzdej dzfdZ ddej dzd e ejejfffd Zeddee dd ej dzdejdzdej dzdedzd ej dzdejdzdej dzdej dzdej dzdejdzd e e d e ezfdZxZS) Glm4vModelrdrct|t|j|_dSr)r|rr{ _from_configr7visualrs r_rzGlm4vModel.__init__s7    &33F4HII r^r})modalityNpixel_values_videosvideo_grid_thwrrc ||jj}|dddf}|ddddf}tj||d}||jdd}tj||gd}|j|f|dd|} |d|jj dzz } tj | j | } | | _ | S) a[ pixel_values_videos (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input videos. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. NrrrT)r return_dictrr!) rrrrr5new_onesrrprodrHtolistr)r) rrrrthw flattened_hw prefix_onesflattened_video_grid_thwvision_outputs split_sizes video_embedss r_get_video_featureszGlm4vModel.get_video_features s266t{7HII 111a4  AAAqrrE ".r1!<<< $--l.@.CQGG #(9k<-Ha#P#P#P $  *BPT  X^  &**2..$+2PRS2SS[[]] {>#?MM '3$r^rirjimage_featuresvideo_featuresc|||tj|jjtj|jk}|d}||tj|jjtj|jk}|d}n ||jjk}||jjk}| }| d |j}|Et||j dz|kd|d|j d| }| d |j}|Et||j dz|kd|d|j d||fS)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. Nrrrz6Image features and image tokens do not match, tokens: z , features: rz6Video features and video tokens do not match, tokens: )get_input_embeddingsrrrrrrallrrrrrrnumel) rrirjrrspecial_image_maskspecial_video_maskn_image_tokensn_video_tokenss r_get_placeholder_maskzGlm4vModel.get_placeholder_mask,s   !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!; !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  "+dk.H!H !*dk.H!H +//11/99"==@@AUVV  % "!4R!88N  %"4^^TUTUTUWXTXEY_`aaaN#$N111a4 %uww%TT^T|TTTr^ rope_deltaszv5.10)versionrlrrd pixel_valuesimage_grid_thwmm_token_type_idsc |du|duz rtd|||}|{|j||fddi| j} t j| d|j|j} | ||| \} }| | | }|{|j || fddi| j}t j|d|j|j}| |||\}}| ||}|| ||| |||| }|j d d||||d | }td i|d |jiS) aU image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. NrrTrr)r)r)rirrrjrlrdr)rirrlrdrjrr])rrget_image_featuresrrrrrrrmasked_scatterrcompute_3d_position_idslanguage_modelrpr)rrirlrrdrjrrrrrr image_embeds image_maskrZr video_maskoutputss r_rzGlm4vModel.forward{s&.  -t"; < [YZZ Z  7D5577 BBM  #242n:>BH !9\q999<<]=QS`SfggL 55i_k5llMJ)88\RRM  *242#^AEIO !9\q999<<]=QS`SfggL 55i_k5llMAz)88\RRM  77#--+- /"38L&$% %)+'      (    (    r^r)NN) NNNNNNNNNN)rPrQrRryrrrrrrnrmrrr\rrrrrrr IntTensorrprrrs@r_rrs02DEJJJJJ 11137".(4/+,  + + ^21B4837 (6(6#(6((6)D0 (6 )D0 (6(6(6(6X37#U#U(4/#U u|U\) * #U#U#U#U#U#UJ_]G444.2.204(,26,08<262648A A #d*A  t+A &- A  A (4/ A lT)A #.5A (4/A (4/A !?T1A +,A  ) )A A A ^54A A A A A r^rceZdZdS)Glm4vCausalLMOutputWithPastNrr]r^r_rrrr^rceZdZ ddejdzdejdzdejdzdedzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzde ejzde e de e zfdZ dfd Z ddejdzdejdzde ejejffdZxZS)Glm4vForConditionalGenerationNrrirlrrdrjlabelsrrrrrlogits_to_keeprrc  ~|jd|||| | | ||||d | }|d}t| trt| dn| }||dd|ddf}d}|'||||jjj}t|||j |j |j |j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) rirrrrrrrlrdrjrN)logitsrrq)lossrrdrkrrr])modelrrVslicelm_head loss_functionrrcrqrrdrkrr)rrirlrrdrjrrrrrrrrrrk slice_indicesrrs r_rz%Glm4vForConditionalGeneration.forwardst$*  % 3))/%)+'        9C>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %%VFt{OfOq%rrD*#3!/)+     r^TFc  ntj|f||||||| | || d | } | s |r d| d<d| d<| S)N) rdrlrjrrrrrrxis_first_iterationrr)r|prepare_inputs_for_generation)rrirdrlrjrrxrrrrrr model_inputsrs r_rz;Glm4vForConditionalGeneration.prepare_inputs_for_generation"s~"=uww<   +)'%% 3))1      " 7i 7+/L (26L. /r^cZ|||tj|jjtj|jkd}||tj|jjtj|jkd}||tj|jjtj|jkd}n0||jjk}||jjk}||jjk}tj | | z d}|dk}||z}| d} | d} | | fS)aa Get the number of images and videos for each sample to calculate the separation length of the sample tensor. These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Returns: image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`) video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`) Nr).rrrr) rrrrrrrrrrrVr) rrirjis_imageis_video_start is_video_end video_level inside_videostandalone_images image_counts video_countss r__get_image_nums_and_video_numsz#5#5#7#7,:J:J:L:L#LRSTTT "Q % 6),,,33 %))a)00 \))r^) NNNNNNNNNNNr) NNNNTNNNNFr)rPrQrRrrmrr rnrrVrrr\rrrrrrs@r_rrs.2.204(,26*.,08<262648-.Y Y #d*Y  t+Y &- Y  Y (4/ Y  4'Y lT)Y #.5Y (4/Y (4/Y !?T1Y el*Y +,Y  , ,Y Y Y Y |  $$$$$$R.26*6*#d*6*|d*6* u|U\) * 6*6*6*6*6*6*6*6*r^rc$eZdZddddddidZdS)Glm4vProcessorKwargsFT)paddingreturn_token_type_idsreturn_mm_token_type_idsreturn_metadata) text_kwargs videos_kwargsN)rPrQrR _defaultsr]r^r_rrs9%*(,  ,T2 IIIr^rcjeZdZd fd ZdededefdZdedeeefdZ d d efd Z xZ S)Glm4vProcessorNc 2t||||t|dsdn|j|_t|dsdn|j|_|d|_|d|_dS)N) chat_template image_tokenz <|image|> video_tokenz <|video|>z<|begin_of_video|>z<|end_of_video|>)r|rhasattrrrconvert_tokens_to_idsvideo_start_id video_end_id)rimage_processor tokenizervideo_processorrrrs r_rzGlm4vProcessor.__init__s )_Tabbb.5i.O.Oj;;U^Uj.5i.O.Oj;;U^Uj'==>RSS%;;qA '(89)DIIKK|[_ii 01)< <    e    &\1rrx| (1- C OO,, 6 6C  $ $Z_ 5 5 5 5/  <%&&33  & &BU'\':2'>'>[\ ] ] ]%&&33z** / /I/ :M"99-Zj9kkO  .OOr^ricg}|D]}tj|}tj|}tj||jkd}tj||jkd}||k}d|||jk|z<d|||jk|z<|||S)Nr)axisr!r) nparray zeros_likerrrrr'r) rririnput array_idsmm_token_typesstartsendsis_video_modalitys r_create_mm_token_type_idsz'Glm4vProcessor.create_mm_token_type_idss > >EI]511N YyD,??aHHHF9Y$*;;!DDDD & UVNI)<<@QQ RXYNI)<z<|end_of_image|>)rrV)rr1r s r_r(z%Glm4vProcessor.replace_frame_token_ids+mD$47G$GmmY\]jYkYkmmmr^)NNNNr) rPrQrRrrrVrXr3r[r?r(rrs@r_rrsPPPPPP>!$!4S ?!!!!*nncnnnnnnnnr^r) rrar5rrrrrrr{rA)mrcollections.abcrnumpyr6rtorch.nnrtorch.nn.functional functionalrhuggingface_hub.dataclassesrrrrrv activationsr cache_utilsr r configuration_utilsr masking_utilsr modeling_flash_attention_utilsr modeling_layersrmodeling_outputsrrmodeling_rope_utilsrmodeling_utilsrrprocessing_utilsrutilsrrrrrutils.deprecationr utils.genericrrrutils.output_capturingr vision_utilsrr glm4.modeling_glm4r"r#r$r%qwen2_5_vl.modeling_qwen2_5_vlr&r'r(r)r*r+r,r-r.r/qwen2_vl.modeling_qwen2_vlr0qwen2_vl.processing_qwen2_vlr1r2 get_loggerrPr#r5rarrrrrrrrrrr r3r@rBrbrdrprrr{rrrrrr__all__r]r^r_r]s$$$$$$ ......&&&&&&!!!!!!........333333//////BBBBBB999999SSSSSSSS111111FFFFFFFF&&&&&&100000cccccccccc555555JJJJJJJJcccccccccccc                        655555  H % %9:::%$%$%$%$%$(%$%$;:%$P9:::7(7(7(7(7(&7(7(;:7(t9:::1(1(1(1(1("1(1(;:1(j     ;   88888M888 m m m m m4 m m m     !>   fffffRYfff"GGGGGBIGGGTRRRRR4RRR55555,55528666%%%%PC)C)C)C)C)C)C)C)L     7   /////6///d     <   222224222e e e e e +e e e P] ] ] ] ] (] ] ] @z z z z z z z z z     "B   x*x*x*x*x*$Fx*x*x*v1=n=n=n=n=n%=n=n=n@   r^