tj"ddlZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl m cm Zddl mZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4ddl5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;ddlZ>ddl?m@Z@mAZAmBZBedGdde jCZDGdde jCZEGdd e jCZFGd!d"e jCZGGd#d$e jCZHGd%d&e jCZId'ZJd(e jKd)e jKd*e jKd+e jKd,eLe jKe jKff d-ZMd.e jKd/eNd,e jKfd0ZO dXd2e jCd3e jKd4e jKd5e jKd6e jKdzd7ePd8ePd9e-e/fd:ZQGd;de!ZSGd?d@e jCZTdAZUdYdBZVGdCdDe jCZWGdEdFe jCZXGdGdHe!ZYe0eGdIdJe#ZZe0GdKdLe+Z[GdMdNe[Z\e0GdOdPe[Z]e0GdQdRe[Z^e0eGdSdTe%Z_GdUdVe[eZ`gdWZadS)ZN)Callable) dataclass)AnyOptional) LayerNorm)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)deprecate_kwarg)accepts_precomputed_kwargsis_flash_attention_requestedmaybe_autocastmerge_with_config_defaults)capture_outputs)get_vision_cu_seqlensget_vision_position_ids) Glm4vConfigGlm4vTextConfigGlm4vVisionConfigRMSNormcTeZdZd deddffd ZdejdejfdZdZxZ S) Glm4vRMSNormư>epsreturnNcttjt j||_||_dS)z; Glm4vRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizer. __class__s f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/glm4v/modeling_glm4v.pyr2zGlm4vRMSNorm.__init__:sD l5:k#:#:;; # hidden_statesc|j}|tj}|ddd}|tj||jzz}|j||zS)NT)keepdim) dtypetor5float32powmeanrsqrtr8r7)r9r> input_dtypevariances r<forwardzGlm4vRMSNorm.forwardBs|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r=cHt|jjd|jS)Nz, eps=)tupler7shaper8)r9s r< extra_reprzGlm4vRMSNorm.extra_reprIs& )**II$2GIIIr=)r-) __name__ __module__ __qualname__floatr2r5TensorrKrO __classcell__r;s@r<r,r,8s$$$$$$$$$$;U\;el;;;;JJJJJJJr=r,c,eZdZddeffd ZdZxZS) Glm4VisionMlpFbiasct|j|_|j|_t j|j|j||_t j|j|j||_t j|j|j||_ t|j |_ dSNrY) r1r2r:out_hidden_sizeintermediate_sizer3Linear gate_projup_proj down_projr hidden_actact_fn)r9configrYr;s r<r2zGlm4VisionMlp.__init__Ns !-!'!74#3T5KRVWWWy!143IPTUUU 4#94;KRVWWWV./ r=c||||||zSN)rbrdr`rar9 hidden_states r<rKzGlm4VisionMlp.forwardWs>~~dkk$..*F*FGG$,,WcJdJddeeer=F)rPrQrRboolr2rKrUrVs@r<rXrXMs_00T000000fffffffr=rXcLeZdZdeddffd ZdejdejfdZxZS)Glm4vVisionPatchEmbedrer/Nct|j|_|j|_|j|_|j|_|j|j|jg}tj|j|j|||_ dS)N) kernel_sizestride) r1r2 patch_sizetemporal_patch_size in_channelsr: embed_dimr3Conv3dproj)r9reror;s r<r2zGlm4vVisionPatchEmbed.__init__\sz  +#)#= !-+/$/R Id.K`klll r=r>c|jjj}|d|j|j|j|j}|||d|j}|S)NrArC) rvr7rCviewrsrrrqrDrt)r9r> target_dtypes r<rKzGlm4vVisionPatchEmbed.forwardfswy'- %**  $":DOT_   -"2"2"2"F"FGGLLRQUQ_`` r= rPrQrRr)r2r5rTrKrUrVs@r<rmrm[szm0mTmmmmmmU\elr=rmcheZdZUejed<d dededdffd Zdejdejfd Z xZ S) Glm4vVisionRotaryEmbeddinginv_freq@dimthetar/Nct||_||_d|t jd|dtj|z zz }|d|ddS)N?rr@rxr~F persistent)r1r2rrr5arangerSregister_buffer)r9rrr~r;s r<r2z#Glm4vVisionRotaryEmbedding.__init__rsr  %ELC%+$N$N$NQT$TUV ZeDDDDDr= position_idscb|d|jzdS)NrAr&) unsqueezer~flatten)r9rs r<rKz"Glm4vVisionRotaryEmbedding.forwardys+&&r**T]:CCAFFFr=)r) rPrQrRr5rT__annotations__intrSr2rKrUrVs@r<r}r}oslEECEEDEEEEEEGELGU\GGGGGGGGr=r}c ZeZdZd dededededdf fd Zd ejdejfd Z xZ S) Glm4vVisionPatchMergerFr context_dimrcrYr/Ncttj||||_t ||_tj||||_tj||||_tj||||_ tj |_ t||_ dSr[)r1r2r3r_rvrpost_projection_normr`rarbGELUact1r rd)r9rrrcrYr;s r<r2zGlm4vVisionPatchMerger.__init__~s Ic3T222 $-cNN!3 $???yk=== ;$???GII Z( r=ric||}|||}||||||zSrg)rvrrrbrdr`rarhs r<rKzGlm4vVisionPatchMerger.forwardsnyy.. yy!:!:1D!-"$,t/A4>"R"R#,   r=r/c|jj}|jd}|j}t |t r!t j||t j}|jd} t| dz} | | | | ddd d |t j} |jd} t j| |j} | d|d dkd}||df t j}||df t j}|dz|z dzdz }|dz|z dzdz }t j||fd  d d}t'j| ||jd d }|dd dd}| |j |j}||z}|S) a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. r&devicerCrg?r@rrxrArFborder)mode align_corners padding_mode)rr7rNr isinstancelistr5tensorlongrrypermuterrDrErcumsumsumstackF grid_samplersqueezerC)r9 embeddingslengths image_shapesh_coordsw_coordspos_embed_weightr:r orig_size_sq orig_size pos_embed_2d num_tokenstoken_positionsseq_idstarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32adapted_pos_embeds r<rKzGlm4vVisionEmbeddings.forwardsi 29&,Q/ !( gt $ $ Ml76LLLG(-a0  c)**  ! !)Y D D WQ1   Yq\\ RvU]R 3 3  %a( ,z*:KLLL",,Q//7>>!3D3D3N3Nq3Q3QQVVWXYY +..U].CC +..U].CCc>X-2Q6c>X-2Q6{FF+444>>qAAKKANN#$- $T%=Uai# # #  "9!@!@!C!C!K!KB!O!O!W!WXY[\!]!]2556F6LMMPPQ[Qbcc "33 r=r{rVs@r<rrsd -0 - - - - - -:PUP\::::::::r=rc|dd|jddzf}|d|jddzdf}tj| |fdS)*Rotates half the hidden dims of the input..NrAr@r)rNr5catxx1x2s r< rotate_halfrs] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r=qkcossinr/c|j}|j}||}}|d|d}}||zt||zz}||zt||zz}||}||}||fS)N)rCrSrrrD)rrrr orig_q_dtype orig_k_dtypeq_embedk_embeds r<apply_rotary_pos_emb_visionrs7L7L 7799aggiiqA}}R  &&((#--*;*;*A*A*C*CC3w;q>>C/0G3w;q>>C/0Gjj&&Gjj&&G G r=r>n_repc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r&N)rNexpandreshape)r>rbatchnum_key_value_headsslenhead_dims r< repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr=modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nr@rrA)rrC)ptrainingr&)rnum_key_value_groupsr5matmul transposer3 functionalsoftmaxrErDrCrr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r<eager_attention_forwardrs3 ;< cu_seqlensposition_embeddingsc B|jd}||djdddddd\}}}|\} } t ||| | \}}|ddd}|ddd}|ddd}tj j j ttj rS|dd|ddz } |||fdjjsdnj||| | dd\} } nS|dd|ddz fd |||fD}fd t'|D}t)j|d } | |d} | } | S) NrrrAr&r@rF)rrr cu_seq_lens_q cu_seq_lens_k max_length_q max_length_krcbg|]+}tj|d,S)r@r)r5splittolist).0rrs r< z0Glm4vVisionAttention.forward..LsAAG FGNN$4$4!<<<r=c lg|]0\}}}|||fdjjsdnjddd1S)NrF)rrrrr)rrr)rrrvattention_interfacerr9s r<rz0Glm4vVisionAttention.forward..Ps   Aq!$#  $( L'+}PCC$:P#        r=r)rNrrrrunbindrrrr get_interfacere_attn_implementationrr maxrrrzipr5rrrv)r9r>r r r seq_length query_statesrrrr max_seqlenr_splits attn_outputsrrs` ` @@r<rKzGlm4vVisionAttention.forward s#(+ HH] # # + +J4>2 N N V VWXZ[]^`a b b i ijk l l / j,'S#>|ZY\^a#b#b j#--a33==a@@ ))!Q//99!<< #--a33==a@@ (?(M K ,.E) )  ( 4 4' 9$QRR.:crc?:??AAJ00  $ #'=Ld6L((''NK"!nz#2#6GLXZdfrKsF       #F|   L )La888K!))*b99DDFF ii ,, r=rg) rPrQrRr)r2rr5rTrMrKrUrVs@r<rrs 0 T      _%w777 IM AA|ALA#5<#=>E A  AAA87AAAAAr=rc eZdZd fd Zedde d dejdejd eejejfdzdejfd Z xZ S) Glm4vVisionBlockr/Nctt|j|j|_t|j|j|_t||_t|d|_ dS)Nr.Fr\) r1r2r,r: rms_norm_epsnorm1norm2rattnrXmlprs r<r2zGlm4vVisionBlock.__init__fst !&"4&:MNNN !&"4&:MNNN (00  e444r=rrrr>r r c ||j||f||d|z}||||z}|S)z cu_seqlens (`torch.Tensor`): Cumulative sequence lengths used for packed variable-length attention in Flash Attention kernels. r r )r+r)r,r*)r9r>r r rs r<rKzGlm4vVisionBlock.forwardmsq&  JJ} % %) ! 3) )  ) )  &M1J1J(K(KK r=r/Nrg) rPrQrRr2rrr5rTrMrKrUrVs@r<r%r%es555555_%w777 IM |L#5<#=>E   ^87r=r%c eZdZUejed<d deffd Ze ddedzde dde dzde d e ffd Z ejed Zd ZxZS)Glm4vTextRotaryEmbeddingr~Nrect|j|_|j|_||_|jjd|_|j}|jdkrt|j}||j|\}|_ | d|d| d| d|j dgd|_dS) N rope_typedefaultr~Froriginal_inv_freq mrope_section) r8)r1r2max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrerope_parametersr3compute_default_rope_parametersrattention_scalingrclonegetr6)r9rer rope_init_fnr~r;s r<r2z!Glm4vTextRotaryEmbedding.__init__s "("@$*$B! 4[A!%!E >Y & &.t~>L+7< V+L+L($( ZeDDD 0(..2B2BuUUU#377UUr=rz torch.deviceseq_lenr/z torch.TensorcV|jd}|jdd}t|ddp|j|jz}t ||z}d}d|t jd|dt j |t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorrrNrr@rxr) r<r@getattrr:num_attention_headsrr5rint64rDrS) rerrBbaserErrattention_factorr~s r<r=z8Glm4vTextRotaryEmbedding.compute_default_rope_parameterss&%l3 & 6 : :;RTW X X6:t44h8JfNh8h(2233 U\!S!5;???BB&X]XcBddgjj k )))r=c^|jdddddfd|jddd}|dddddddf}t |jjtr|jjdkr |jjnd}t|d5||z dd}| ||j }tj ||fd }||jz}||jz} dddn #1swxYwY||j | |j fS) Nrr&rAmpscpuF) device_typeenabledr@rrx)r~rSrrNrrtyperr!r apply_mroper6r5rrr>rrDrC) r9rrinv_freq_expandedposition_ids_expandedrNfreqsembrrs r<rKz Glm4vTextRotaryEmbedding.forwards !M$aaa*=>DDFFMMaQ]QcdeQfhjlmnn ,QQQ4] ; A A C C'1!(-'E'Ek!(-[`J`J`ahmmfk  U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E$$UD,>??E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5vvAGv$$cff17f&;&;;;s9B)E..E25E2c|}||d}tjdt|Dd}|S)NrArc*g|]\}}||dzS)r)richunks r<rz8Glm4vTextRotaryEmbedding.apply_mrope..s$KKKXQE!a%LKKKr=)rr5r enumerate)r9rTr6sectionchunksresults r<rQz$Glm4vTextRotaryEmbedding.apply_mropesMW"--KK69J9JKKKQSTTT r=rgNNN)rPrQrRr5rTrr(r2 staticmethodrrrMrSr=no_gradrrKrQrUrVs@r<r1r1slVVVVVVVV")-+/"**$&*(*t* ~u$ % ***\*>U]__<<_< r=r1c|ddddf}|ddddf}tj| |fddS) r.rNr@r&rArr)r5rrrs r<rotate_half_llmrcsQ 319B 319B ;Ryb ) ) ) 1 1" 5 55r=cT||}||}|dd|jddzfdd}|dd|jddzfdd}|jd}|dd|f|d|df}}|dd|f|d|df} }||zt||zz} ||zt||zz} t j| |gd} t j| | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .NrAr@r)rrNrepeat_interleavercr5r) rrrr unsqueeze_dim rotary_dimq_rotq_passk_rotk_passrrs r<apply_rotary_pos_embrls\$ -- & &C -- & &C c'SYr]a''' ( : :1" : E EC c'SYr]a''' ( : :1" : E EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{u55;@PW\]]] r=r>r rpast_key_valuesrr/c d|\}}}||} ||} ||} | ||d|jdd} | ||d|jdd} | ||d|jdd} |\} } t| | | | \} } ||| | |j \} } tj |j j t}||| | | |f|jsdn|j|jd|\}}|||d}||}||fS)NrAr&r@r)rr)sizerqrrrsryrrrlupdaterorrrerrrrrrrrt)r9r>r rrvrbszq_lenr!rrrrrrrrs r<rKzGlm4vTextAttention.forwards&**,, UA{{=11 [[// {{=11 #((eRGGQQRSUVWW __S%T]CCMMaQRSS #((eRGGQQRSUVWW &S#7 jRUWZ#[#[ j  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \"))#ub99DDFF kk+.. L((r=rgr_)rPrQrR__doc__r(rr2r5rTrMr rrrKrUrVs@r<rnrns ^^^3:^^^^^^.IM.2(, ))))|))#5<#=>E)) t+ ))  )) -. )) u|U\D0% 2E2LL M))))))))))))))))r=rncBeZdZfdZdejdejfdZxZS) Glm4vTextMLPc"t||_tj|jd|jzd|_tj|j|jd|_t|j |_ dS)Nr@Fr\) r1r2rer3r_r:r^ gate_up_projrbr rc activation_fnrs r<r2zGlm4vTextMLP.__init__Gsz  If&8!f>V:V]bccc6#;V=OV[\\\#F$56r=r>r/c||}|dd\}}|||z}||S)Nr@rAr)rrZrrb)r9r> up_statesgates r<rKzGlm4vTextMLP.forwardOsX%%m44 #//!/44i 2 24 8 88 ~~i(((r=)rPrQrRr2r5 FloatTensorrKrUrVs@r<r~r~Fs`77777)U%6)5;L))))))))r=r~c eZdZdedeffd Ze ddejde ejejfdzdejdzd ej dzd e dzd e dzd e ej e ej ej fdzffd ZxZS)Glm4vTextDecoderLayerreroct|j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ dS)Nr') r1r2r:rn self_attnr~r,r,r(input_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormrus r<r2zGlm4vTextDecoderLayer.__init__Ys !-+FI>>''+F,>FDWXXX(4V5GVM`(a(a(a%(4V5GVM`(a(a(a%".v/AvGZ"["["[r=NFr>r rrrv use_cacher/c "|}||}|jd||||||d|\}} ||}||z}|}||}||}||}||z}|S)N)r>r rrrvrrX)rrrrr,r) r9r>r rrrvrrresidualr!s r<rKzGlm4vTextDecoderLayer.forwardcs!,,];; *4> ' 3)%+      q55mDD  =0 !55mDD // // >>  =0 r=)NNNNF)rPrQrRr(rr2rr5rTrM LongTensorr rkrrKrUrVs@r<rrXs\\3\\\\\\IM.204(,!&##|##5<#=>E# t+ # &- #  #$;# u %(95;L(L"MPT"TT U###^#####r=rc2eZdZUdZdZejdzed<dS)Glm4vModelOutputWithPast rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. The attribute is deprecated and will be removed in v5.20, use `model.base_model.rope_deltas` instead. N rope_deltasrPrQrRr|rr5rrrXr=r<rr6 ,0K!D(/////r=rcTeZdZUeed<dZdZdZddgZdgZ dZ dZ dZ dZ fdZxZS) Glm4vPreTrainedModelremodel)imagevideotextTrr%rvc t|t|trVd|jt jd|jdt j|jz zz }tj |j |dSdS)Nrrr@rx) r1 _init_weightsrr}rr5rrrSinitcopy_r~)r9rr~r;s r<rz"Glm4vPreTrainedModel._init_weightss f%%% f8 9 9 2flu|Avz1TYT_/`/`/`cicm/mnoH Jv 1 1 1 1 1 2 2r=)rPrQrRr'rbase_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendrrUrVs@r<rrs 1&*#02DE#4"5N!"&222222222r=rc eZdZUeed<dZdgZeedZ d fd Z dZ e e ed ejd ejd eedeezfd ZxZS)Glm4vVisionModelre)rrr%r> attentionsr/Nctj|_j|_t |_t |_jj z}t|dz|_ tj fdtjD|_t#jjj|_t-jj|_tjjjjj|_t-jj|_d|_|dS)Nr@c.g|]}tSrX)r%)rr!res r<rz-Glm4vVisionModel.__init__..s"$[$[$[!%5f%=%=$[$[$[r=)rrrcr')rs out_channelsrorpF)r1r2spatial_merge_sizerqrrrm patch_embedr:rr}rr3 ModuleListrangedepthblocksrr]r^rcmergerr,r(post_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)r9rerr;s ` r<r2zGlm4vVisionModel.__init__sS    "("; +/77088%)998QGGm$[$[$[$[uV\GZGZ$[$[$[\\ ,&F4LY_Yj    $00BH[#\#\#\ )*/1,     +6+=6CVWWW&+# r=ctjd|jjdtdt ||j}||}||fS)N`z.rot_pos_emb` is deprecated and will be removed in v5.11. Use `get_vision_position_ids` from `transformers.vision_utils` and apply the rotary embedding module.r@) stacklevel)warningswarnr;rP FutureWarningr%rr)r9grid_thwrrs r< rot_pos_embzGlm4vVisionModel.rot_pos_embsr  I' I I I     /x9PQQ ,,\::|++r=r>rrc t||j|}t||}||}||}||}t j||fd}|| f}|dd|ddz } | || ||dddf |j |dddf |j }|j D]} | |f||d|}||}|d|j|j|jd}|dddd }||d|jj}||} t-|| S) a\ hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. )rrArr&Nrr.rr@)last_hidden_state pooler_output)r%rr$rrrr5rrrrrDrrrryrNrrrer]rr) r9r>rrrr  rotary_embrUr seqlensblkmerged_hidden_statess r<rKzGlm4vVisionModel.forwards/x9PY_``` *8FCCC ((77 00?? ((66 iZ0b999"wwyy#''))4QRR.:crc?2    A  ! !-"6 7 7 A  ! !-"6 7 7   ;  CC%$7 MM++M:: %** ')@-BUVXBY  &--aAq99  66;;B @[\\ #{{=99)+.    r=r/)rPrQrRr)rrrr%r_can_record_outputsr2rr"r#rr5rTrrrMrrKrUrVs@r<rrs )+,)* 8,,, 3 "\3 5:\3 MSTfMg3 + +3 3 3 ^_ 3 3 3 3 3 r=rceZdZUeed<dZeedZdeffd Z e e e dde jdzde jdzde jdzd edzd e jdzd edzd eed eezfdZxZS)Glm4vTextModelre)rrctj|_j|_t jjj|j|_t j fdtj D|_ tjj|_t!|_d|_|dS)Nc0g|]}t|SrX)r)rrores r<rz+Glm4vTextModel.__init__..#s$ g g g) "69 5 5 g g gr=r'reF)r1r2 pad_token_id padding_idx vocab_sizer3rr: embed_tokensrrnum_hidden_layerslayersr,r(normr1rrrrs `r<r2zGlm4vTextModel.__init__s    !. +L):Fr  decoder_layer layer_outputss r<rKzGlm4vTextModel.forward,s:  -t"; < [YZZ Z  ?09M9M9O9O0*$+>>>O   --i88M  CRC^==???de  < (;A(>}G[\\\_ooL',,Q266==aATUVAWY[\\LL  ! # #' 299!\=OPQ=RTVWWL   ! !l&8&;q&@&@ ,Q '+LL!% k*,.-   )77;77 % "oom,oWW![ * *M)M*. /$7   M*MM -00 &++    r=)NNNNNN)rPrQrRr(rrrrnrr2rr"r#r5rrTr rrkrrrMrrKrUrVs@r<rrsK  .(  .2.204(,26!%J J #d*J  t+J &- J  J (4/ J $;J -.J  ( (J J J _ ^J J J J J r=rczeZdZdZdZddgZfdZ d)ded eeeefe j zd ed ed ed e e j zdzf dZ d*de jde jde jdzde jdzde j dzdee j e j ff dZedee d+de jde jdzdeedeezfdZedee d+de jde jdzdeedeezfdZ d,de jde jde jdzd e jdzfd!Z d-de j dzde j dzde j dzde j dzde j dzd"e j dzde jdzde j dzfd#Zed$d%&ee d.de jdzde j dzd'e jdzd"edzde jdzde j dzde jdzde jdzde jdzde jdzdeedeezfd(Z xZ!S)/ Glm4vModelrFrr%ct|t|j|_t |j|_d|_ | dSrg) r1r2r _from_config vision_configvisualr text_configlanguage_modelrrrs r<r2zGlm4vModel.__init__sl    &33F4HII ,99&:LMM r=r&Nstart_positionrtemp_merge_sizer time_intervalrc|d|z|d|z|d|z} }}tj|||z} tj|||z} tj| ||z} tj| | | d\} }}tj| ||gddd }|dxx|z cc<|S) a Compute 3D positional indices for vision tokens derived from a single image or video input. The positions are generated from the input grid defined by temporal (T), height (H), and width (W) dimensions. Temporal and spatial dimensions can be downscaled according to the merge sizes used in the vision backbone. The resulting positions are offset by `start_position`. Args: start_position (`int`): Offset added to all computed positional indices. grid_thw (`Sequence[int]` or `torch.Tensor` of shape `(3,)`): The (T, H, W) grid representing the feature layout of the current image or video after patch embedding. temp_merge_size (`int`, *optional*): Factor by which the temporal dimension is reduced in the backbone. The temporal grid size is divided by this value. Defaults to 1. spatial_merge_size (`int`, *optional*): Factor by which the spatial dimensions (H and W) are reduced in the backbone. Both H and W are divided by this value. Defaults to 1. time_interval (`int`, *optional*): Spacing factor applied between consecutive temporal position indices.Defaults to 1. device (`str` or `torch.device`, *optional*): Device on which the resulting tensor is allocated. If `None`, uses the current default device. Returns: torch.LongTensor of shape (3, sequence_length): Positional indices for temporal, height, and width dimensions, flattened into sequence form and offset by `start_position`. rr&r@rij)indexingrrrA)itemr5rmeshgridrr)r9rrrrrr llm_grid_t llm_grid_h llm_grid_wposition_temporalposition_heightposition_widthT_gridH_gridW_gridvision_position_idss r<r%z"Glm4vModel.get_vision_position_idssL QK    / 1 QK    "4 4 QK    "4 4!+J "LFCCCmS,z&AAANRj@@@>Q!&0A?Tbmq!r!r!r#k666*BJJJRRSTVXYYA.0""r=rmm_token_type_idsimage_grid_thwvideo_grid_thwrr/c |*tj||dddfd}d|dddf<|jjj}g}tjd|jd|jd|j|j} |t|nd|t|ndd} t|D]w\} } || } |@| ||  } | ||  } g}tj t| dD]K\}}t|}|dd}|d ddz}||||fLd}g}|D]\}}}|dkrd||z }|tj||j dd dd |z||z }pt)| |}|||d||j }|||t-|d|d |zz }tj|ddd }|;|| j| dd| ||  f<n!|| j| dd| f<||dzt5| z ytj||j d}| |fS) a Difference from Qwen2VL/Qwen2.5VL's get_rope_index: - GLM4V uses timestamps to separate each video frame, so the video_grid_thw should also be split too. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. mm_token_type_ids (`torch.IntTensor` of shape `(batch_size, sequence_length)`): Token type ids matching each modality to a different value in the input sequence, i.e. text (0), image (1), video (2). image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) Nrrr&rrCr)r&r@c|dS)Nr&rX)rs r<z+Glm4vModel.get_rope_index..s `abc`dr=rArr@)r5rererrzerosrNrCriterr[rk itertoolsgroupbyrrappendrryrnextr%rrrrDlenrr)r9rrrrrrrmrope_position_deltasr grid_iters batch_idxcurrent_input_idsinput_token_typeinput_type_grouprgroup start_index end_index current_posllm_pos_ids_list modality_type start_idxend_idxtext_lenrr llm_positionss r<get_rope_indexzGlm4vModel.get_rope_indexsF  %"4^^TUTUTUWXTXEY_`aaaN#$N111a4 ![6I "{ OA  OA /#    (6'AtN###t'5'AtN###t  -6i,@,@$ [$ [ (I(0; )$5nY6O6T6T6V6V$W!#3N94M4R4R4T4T#U ! '/ :J:Q:Q:S:S0T0TVdVdee G G UU #Ahqk !"IaL1,  ''k9(EFFFFK! 5E W W1 y' A%%&2H$++ Xi6FGGGLLQPRSSZZ[\^`aadoo 8+KK $J}$=>>H*.*F*F#Xq2DYM]+G++'%++,?@@@3x{HQK#@#@DV#VVKK!I&6A>>>FFq"MMM)O\O_O_`l`sOtOt QQQ >)+D+I+I+K+KKLL-:-=-=l>Q-R-R QQQ \* ! ( ():):)<)#?MM '3$r=r pixel_valuesc ||jj}|j|fd|i|}|d|jjdzz}t j|j|}||_|S)T pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. rrAr@) rPrrCr+rrr5rr)r9r5rrr1r2 image_embedss r<get_image_featureszGlm4vModel.get_image_features=s$(():;; $\UUNUfUU%**2..$+2PRS2SS[[]] {>#?MM '3$r=rimage_featuresvideo_featuresc|||tj|jjtj|jk}|d}||tj|jjtj|jk}|d}n ||jjk}||jjk}| }| d |j}|Et||j dz|kd|d|j d| }| d |j}|Et||j dz|kd|d|j d||fS)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. Nr rAz6Image features and image tokens do not match, tokens: z , features: rz6Video features and video tokens do not match, tokens: )get_input_embeddingsr5rreimage_token_idrrallvideo_token_idrrrDrrNnumel) r9rrr:r;special_image_maskspecial_video_maskn_image_tokensn_video_tokenss r<get_placeholder_maskzGlm4vModel.get_placeholder_maskTs   !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!; !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  "+dk.H!H !*dk.H!H +//11/99"==@@AUVV  % "!4R!88NTXY>Y>Y(,(;(;---"3 )<)) %L+ +D     )/E/I/IYM^(5(; %J A)-2244;;B??!C +77!8KQOO +00JCCJJ1aQRSSVVWdWkll $|,BDZ]gDghh +00Ar::AA!ZQSTTWWXeXlmm $66zTEUE[\]E^7^de6ffE'%((-:N(*O*OOLL Lr=rrrrc |du|duz rtd|||}|{|j||fddi| j} t j| d|j|j} | ||| \} }| | | }|{|j || fddi| j}t j|d|j|j}| |||\}}| ||}|| ||| |||| }|j d d||||d | }td i|d |jiS) aU image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Nrr)Trr)r:)r;)rrrrrrvr)rrrrvrrrX)rr=r9rr5rrDrrCrFmasked_scatterr4rOrrr)r9rrrrvrr5r&rrrrr8 image_maskr!r3 video_maskoutputss r<rKzGlm4vModel.forwards&.  -t"; < [YZZ Z  7D5577 BBM  #242n:>BH !9\q999<<]=QS`SfggL 55i_k5llMJ)88\RRM  *242#^AEIO !9\q999<<]=QS`SfggL 55i_k5llMAz)88\RRM  77#--+- /"38L&$% %)+'      (    (    r=)r&r&r&Nr_rg)NN)NNNNN) NNNNNNNNNN)"rPrQrRraccepts_loss_kwargsrr2rrr5rTrrr%r IntTensorrMr$rrrrrrrr4r9rFrOrr rrKrUrVs@r<rr|s02DE !"#,02#2#2#sC}% 42# 2#  2#  2#el"T)2#2#2#2#p3726.2 [3[3#[3!?[3(4/ [3 (4/ [3  t+ [3 u|U\) *[3[3[3[3z 11137".(4/+,  + + ^21: 11137'(4/+,  + + ^2104837 (6(6#(6((6)D0 (6 )D0 (6(6(6(6\/3.2.2/348//<$&/|d*/ t+ /  t+ /  t+ /,/!?T1/  ////b_]G444.2.204(,26,08<262648A A #d*A  t+A &- A  A (4/ A lT)A #.5A (4/A (4/A !?T1A +,A  ) )A A A ^54A A A A A r=rc2eZdZUdZdZejdzed<dS)Glm4vCausalLMOutputWithPastrNrrrXr=r<rXrXrr=rXc!"eZdZddiZdZfdZe d#dejdej dzde e d e e zfd Ze d#d ejd ej dzde e d e e zfd Zeddee d$dej dzdejdzdej dzdedzdejdzdej dzd ejdzdejdzd ej dzdej dzdejdzdeejzde e d e ezfdZ d%fd ZfdZ d#dej dzdejdzd e ejejffdZ d&d ed!edej dzd e ej eeefffd"ZxZ S)'Glm4vForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightFct|t||_t j|jj|jjd|_ | dS)NFr\) r1r2rrr3r_rr:rlm_headrrs r<r2z&Glm4vForConditionalGeneration.__init__se    '' y!3!?ASA^ejkkk  r=Nr&rrr/c *|jj||fi|S)r()rr4)r9r&rrs r<r4z0Glm4vForConditionalGeneration.get_video_featuress%-tz,-@.[[TZ[[[r=r5rc *|jj||fi|S)r7)rr9)r9r5rrs r<r9z0Glm4vForConditionalGeneration.get_image_featuress#-tz,\>TTVTTTr=rrrrrrrrvrlabelsrlogits_to_keepc  ~|jd|||| | | ||||d | }|d}t| trt| dn| }||dd|ddf}d}|'||||jjj}t|||j |j |j |j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) rr5r&rrrrrrvrrN)logitsr_r)lossrbrvr>rrrX)rrrslicer\ loss_functionrerrrXrvr>rr)r9rrrrvrr_r5r&rrrr`rrTr> slice_indicesrbrcs r<rKz%Glm4vForConditionalGeneration.forward,sz$*  % 3))/%)+'        9C>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %%VFt{OfOq%rrD*#3!/)+     r=Tc  ntj|f||||||| | || d | } | s |r d| d<d| d<| S)N) rvrrrr5r&rrris_first_iterationr5r&)r1prepare_inputs_for_generation)r9rrvrrrrr5r&rrrhr model_inputsr;s r<riz;Glm4vForConditionalGeneration.prepare_inputs_for_generations~"=uww<   +)'%% 3))1      " 7i 7+/L (26L. /r=ct||}d}|dx}|}|dkr#|jj|d|jjz}|Sd|vr|djddkr|d}t|jdko|jtj tj fv}|r|dk|d|d Ad | D}|jj |fi|\}} | |j_nf|dd d d }tj|jddtj |j |j_|d}tj||gd}|S)Nrrvrrr&r@rrrc&i|]\}}|dk ||S)rrX)rrrs r< zVGlm4vForConditionalGeneration._prepare_position_ids_for_generation..s(VVVTQQ+EUEUAqEUEUEUr=rrAr r)r1$_prepare_position_ids_for_generationr@rrrrNrrCr5rritemsr$rrr rr) r9 inputs_tensor model_kwargstext_positions past_lengthcacher is_input_idsvision_positionsrr;s r<rnzBGlm4vForConditionalGeneration._prepare_position_ids_for_generationsEEmUabb !%%&788 8E E..00K !   6 B))4tz7MML  , & &< +D+J1+MPQ+Q+Q(5M=.//14g9LQVQZ\a\fPg9g    !455A!!"233?#5#5#7#7,:J:J:L:L#LRSTTT "Q % 6),,,33 %))a)00 \))r=r& expand_sizeis_encoder_decoderc dkrfSgdfd}fd}|d||r8dtd|dd<fS) Nr&)r5rr&rsecond_per_grid_tsc dd} dd}  dd\}}d}|D]}|dkrFtj|t |}d|D}|||| ||<O|dkr't |}|||| ||<||d krFtj|t |}d |D}|||| ||<|dkr't |}|||| ||<|d kr$|||t | ||<!|S) Nrrr)rctj||}|gdg|dz zztjfd|Dd}|S)Nr&c$g|] }|j SrX)rI)rsample repeat_argss r<rzGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samples..)s"#V#V#VFMFM;$?#V#V#Vr=rr)r5rrr)rr repeat_timessamplesr^rs @r<_repeat_interleave_sampleszGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samples&sb+a11+nsaeeggk/BB #V#V#V#Vg#V#V#V\]^^^ r=r5c^g|]*}tj|d+Sr&rr5r+rrrs r<rz{Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual..13UUU6uz&a888<<>>UUUr=)rrr&c^g|]*}tj|d+Srrrs r<rz{Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual..=rr=r)r@rr5rr) dict_to_expandrr image_nums video_numsrrrrrrrqr9s r<"_expand_dict_for_generation_visualzgGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visuals)--.>EEN)--.>EEN%)%H%H)9)9/4)P)P&I&& "J     &  .((#k.$z:J:JKKGUUWUUUG*D*D&s+W;+++N3'',,,":..G*D*D&s+W;+++N3''111#k.$z:J:JKKGUUWUUUG*D*D&s+W;+++N3'',,,":..G*D*D&s+W;+++N3''000*D*D&s+T*5E5ET_+++N3'" !r=c|D]}|dkr2||jdkr!||d||<:||Dt||tjr$|vr ||d||<|S)Nrrr&rr)rrerr5rT)rrr visual_keyss r<_expand_dict_for_generationz`Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generationLs% d d.((^C-@-E-J-J*8*=*O*OP[ab*O*c*cN3''"3'3">##6 EE4;..*8*=*O*OP[ab*O*c*cN3'! !r=rrencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rer@r)r9rrrrqrrrs`` `` @r<_expand_inputs_for_generationz;Glm4vForConditionalGeneration._expand_inputs_for_generations !  l* *www + "+ "+ "+ "+ "+ "+ "+ "Z " " " " " ":9,GG  !33KQ3GGI22<@@  k 122: !pqqq.I.I,WhJi.j.jL* +,&&r=rg) NNNNNNNNNNNr) NNNNTNNNNF)r&FN)!rPrQrR_tied_weights_keysrUr2rr5rrrrrMrr4r9rrrTr rVrrXrKrirnrrkdictrrrrUrVs@r<rZrZs*,VW37 \ \". \(4/ \+, \ + + \ \ \^ \37 U U' U(4/ U+, U + + U U U^ U_]G444.2.204(,26*.,08<262648-.Y Y #d*Y  t+Y &- Y  Y (4/ Y  4'Y lT)Y #.5Y (4/Y (4/Y !?T1Y el*Y +,Y  , ,Y Y Y ^54Y |  $$$$$$L$$$$$R.26*6*#d*6*|d*6* u|U\) * 6*6*6*6*t#(-1 V'V'V'!V'#d* V' uc3h/ 0 V'V'V'V'V'V'V'V'r=rZ)rZrrrr)r)r&)brrcollections.abcr dataclassesrtypingrrr5torch.nnr3torch.nn.functionalrrrr r activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrutils.deprecationr utils.genericrr r!r"utils.output_capturingr# vision_utilsr$r%configuration_glm4vr'r(r)Moduler,rXrmr}rrrrTrMrrrrSrrr%r1rcrlrnr~rrrrrrrXrZ__all__rXr=r<rs($$$$$$!!!!!! &&&&&&!!!!!!........))))))777777//////BBBBBB999999kkkkkkkkkkKKKKKKKKFFFFFFFF&&&&&&aaaaaaaaaaaa000000 655555JJJJJJJJPPPPPPPPPPY''JJJJJ29JJ('J( f f f f fBI f f fBI( G G G G G G G GfffffRYfff"GGGGGBIGGGT(((  |   +0< >Cl  5< %&     UU\ U# U%, U U U U&%% I% <% % < % L4' %  %%'(%%%%2PPPPP29PPPf1>JJJJJryJJJZ666%%%%PC)C)C)C)C)C)C)C)L)))))29)))$/////6///d 00000600 022222?222(e e e e e +e e e Pe e e e e )e e e Pv v v v v %v v v r  00000"800 0b'b'b'b'b'$8/b'b'b'J x w wr=