"S3S4\ Rh5r?"S5S6\ Rh5r@"S7S8\ Rh5rA"S9S:\ Rh5rB"S;S<\ Rh5rC"S=S>\ Rh5rD"S?S@\ R5rF"SASB\ Rh5rG"SCSD\ Rh5rH"SESF\ Rh5rI"SGSH\ Rh5rJ"SISJ\ Rh5rK\'"SKSL9"SMSN\"55rL"SOSP5rM\'"SQSR\"55rN"SSST\ Rh5rO\'"SUSV\N55rP\'"SWSX\N\55rQ"SYSZ\N5rR"S[S\\N\5rS/S]QrTg)`N)cached_property)CallableOptionalUnion)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)check_model_inputs) Emu3ConfigEmu3TextConfigEmu3VQVAEConfigcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s ^/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/emu3/modeling_emu3.py rotate_halfr*.sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer*)qkcossin position_ids unsqueeze_dimq_embedk_embeds r)apply_rotary_pos_embr65sS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r+ hidden_statesn_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r#expandreshape)r7r8batchnum_key_value_headsslenhead_dims r) repeat_kvrAPs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr+modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nr rr)r"dtype)ptrainingr)rAnum_key_value_groupsr$matmul transposer#nn functionalsoftmaxfloat32torLrHrN contiguous) rBrCrDrErFrGrHrI key_states value_states attn_weights causal_mask attn_outputs r)eager_attention_forwardr]\s3 ; ;[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-URS9Ul[R"UR URUR-URS9Ul[R"UR URUR-URS9Ul[R"UR UR-UR URS9Ulg)Nr@Tbias)super__init__rbrcgetattr hidden_sizenum_attention_headsr@r>rOrGattention_dropout is_causalrRLinearattention_biasq_projk_projv_projo_projselfrbrc __class__s r)riEmu3Attention.__init__ysI  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r+r7position_embeddingsrFpast_key_valuecache_positionrIr9c 4URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO URUR S.UD6unnUR""/UQSP76R%5nUR'U5nUU4$)Nrrr )r1r0r{eager)rHrG)r#r@rqviewrQrrrsr6updatercr]rb_attn_implementationrrNrmrGr<rWrt)rvr7ryrFrzr{rI input_shape hidden_shape query_statesrXrYr0r1 cache_kwargsattention_interfacer\rZs r)forwardEmu3Attention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<'))!. )) ! )) !!1!12 ))+,)) u||U\\) *))))r+r_RMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$) Emu3RMSNormc>[TU]5 [R"[R "U55UlX lg)z* Emu3RMSNorm is equivalent to T5LayerNorm N)rhrirR Parameterr$onesweightvariance_epsilon)rvrkepsrws r)riEmu3RMSNorm.__init__s/ ll5::k#:; #r+cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nr rT)keepdim) rLrVr$rUpowmeanrsqrtrr)rvr7 input_dtypevariances r)rEmu3RMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r+c^[URR5SUR3$)Nz, eps=)rrr#rrvs r) extra_reprEmu3RMSNorm.extra_reprs* ))*+6$2G2G1HIIr+)rr)ư>) rrrrrirrrrrs@r)rrs$;JJr+rc.^\rSrSrU4SjrSrSrU=r$)Emu3MLPc>[TU]5 XlURUlURUl[ R "URURURS9Ul[ R "URURURS9Ul [ R "URURURS9Ul [URUl g)Nrf)rhrirbrkintermediate_sizerRromlp_bias gate_projup_proj down_projr hidden_actact_fnrvrbrws r)riEmu3MLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r+cURURURU55URU5-5nU$N)rrrr)rvr&rs r)rEmu3MLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r+)rrbrrrkrrrrrrrirrrrs@r)rrs0r+rct^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \RS \ \ \R\R4S \\S \ \R \ \ \R \R 444SjjrSrU=r$)Emu3DecoderLayerrbrccV>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul [R"UR5Ulg)N)rbrcr)rhrirkr_ self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormrRDropoutrmrHrus r)riEmu3DecoderLayer.__init__s !--&fJ6?*6+=+=6CVCVW(3F4F4FFL_L_(`%zz&":":; r+r7rFr2rz use_cacher{ryrIr9c Un URU5nUR"SUUUUUUUS.UD6upXRU5-nUn URU5nUR U5nXRU5-nU$)N)r7rFr2rzrr{ry)rrrHrr) rvr7rFr2rzrr{ryrIresidual_s r)rEmu3DecoderLayer.forwards!,,]; >>  ')%)) 3     !<< #>>  55mD /  << #>> r+)rHrkrrrr)NNNFNN)rrrrrrrir$rrrr boolrrr FloatTensorrrrrs@r)rrs  !!1!12&eELL%,,,F&GH+, u  (51B1BEDUDU1U+V"WW Xr+rcV^\rSrSrSrS\4U4SjjrS\R4Sjr Sr U=r $)Emu3VQVAEVectorQuantizeria A module for vector quantization using learned embedding vectors. This module implements the quantization process similar to te one described in the VQ-VAE (Vector Quantized Variational AutoEncoder) paper. It quantizes continuous input vectors into discrete codebook vectors, which are learned during training. Current implementation improves over previous ones by avoiding costly matrix multiplications and allowing for post-hoc remapping of indices. rbc>[TU]5 [R"URUR 5UlUR RRRSUR- SUR- 5 g)Ng?) rhrirR Embedding codebook_size embed_dim embeddingrdatauniform_rs r)ri!Emu3VQVAEVectorQuantizer.__init__sb f&:&:FAtT $.."7"7":B  %;^^=R=R=\=\]^`a=bcc $4y@ $||I1=388v]##r+)r) rrrrrrrir$rrrrrs@r)rrs+ee $ELL$$r+rc.^\rSrSrU4SjrSrSrU=r$)Emu3VQVAEEncoderConvDownsamplei0cZ>[TU]5 [R"XSSSS9Ulg)Nrr r kernel_sizestridepaddingrhrirRConv2dconvrv in_channelsrws r)ri'Emu3VQVAEEncoderConvDownsample.__init__1% IIkAaYZ[ r+cV[R"USSSS9nURU5nU$)N)rrrrconstantr)padmoderE)Frrrvr7s r)r&Emu3VQVAEEncoderConvDownsample.forward5s+mJVWX  -0 r+rrrs@r)rr0s\r+rc.^\rSrSrU4SjrSrSrU=r$)Emu3VQVAEEncoderConvUpsamplei<cZ>[TU]5 [R"XSSSS9Ulg)Nrrrrrs r)ri%Emu3VQVAEEncoderConvUpsample.__init__=rr+cT[R"USSS9nURU5nU$)N@nearest scale_factorr)r interpolaterrs r)r$Emu3VQVAEEncoderConvUpsample.forwardAs( m#IV  -0 r+rrrs@r)rr<s\r+rc j^\rSrSrS\S\S\\S\\4U4SjjrS\R4Sjr S r U=r $) Emu3VQVAEConv3diG in_channel out_channelrrcP>[T U]5 [USSUSS5VVs/sH upVXV- PM nnnSUlUSSS2H&nU=RUS-US--US-4- slM( U=RS- sl[R "UUUUS9Ulgs snnf)Nrrrr )r r)r)rhriziprrRConv3dr) rvrrrr one_kernel one_stride padding_sizespad_sizerws r)riEmu3VQVAEConv3d.__init__Hs ORS^_`_aSbdjklkmdnOopOo5KZ0Oo p %dd+H LLX]X\98q=I IL,  II      qsB"r7ch[R"XR5nURU5nU$r)rrrrrs r)rEmu3VQVAEConv3d.forward^s(m\\:  -0 r+)rr) rrrrrrrir$rrrrrs@r)rrGsK   3Z  c  ,U\\r+rcn^\rSrSrS\S\4U4SjjrS\RS\R4SjrSr U=r $) Emu3VQVAESpatialNormidr out_channelsc>[TU]5 [R"USSSS9Ul[R "UUSSSS9Ul[R "UUSSSS9Ulg)N rT num_channels num_groupsraffinerrr)rhrirR GroupNorm norm_layerrconv_yconv_brvrr rws r)riEmu3VQVAESpatialNorm.__init__esn ,,%  ii     ii     r+r7 quant_statesc[R"X!RSSSS9nURU5nXR U5-UR U5-nU$)NrKr )sizer)rrr#r(r)r*)rvr7r-s r)rEmu3VQVAESpatialNorm.forwardsT}}\8K8KBC8PW`a  6 % L(AADKKP\D]] r+)r*r)r( rrrrrrir$rrrrrs@r)rrds:   8U\\r+rcV^\rSrSrS\S\4U4SjjrS\R4SjrSr U=r $)Emu3VQVAETemporalUpsampleirrcD>[TU]5 [UUSSS9Ulg)Nrrrrrrrrrhrirrrvrrrws r)ri"Emu3VQVAETemporalUpsample.__init__( #  !   r+r7cDURup#pEnURSSSSS5R5RUSU5n[R "USSS 9nURX#XVS5RSSSSS5R5nUR U5nU$) Nrrrrr rr r r )r#rrWrrrr)rvr7rrrrrs r)r!Emu3VQVAETemporalUpsample.forwards8E8K8K5 h%--aAq!<GGINNz[]_gh  m#IV %**:PRS[[\]_`bcefhijuuw  -0 r+rr1rs@r)r3r3s/      U\\r+r3cV^\rSrSrS\S\4U4SjjrS\R4SjrSr U=r $)Emu3VQVAETemporalDownsampleirrcD>[TU]5 [UUSSS9Ulg)N)rrr)r rrr7r8r9s r)ri$Emu3VQVAETemporalDownsample.__init__r;r+r7c(URU5nU$rrrs r)r#Emu3VQVAETemporalDownsample.forwards -0 r+rr1rs@r)r?r?s/      U\\r+r?c4^\rSrSrSU4SjjrSrSrU=r$)Emu3VQVAETemporalResnetBlockicf>[TU]5 XlUcUOUUl[R "U5Ul[UUSSS9Ul[R "U5Ul [UUSSS9Ul URUR:wa[R"UUSSSS9Ul gg)Nr5r6r7rrr) rhrirr rR BatchNorm3dnorm1rconv1norm2conv2r nin_shortcutr+s r)ri%Emu3VQVAETemporalResnetBlock.__init__s &+7+?K\^^K0 $  !   ^^L1 $  !     t00 0 "  !D  1r+cPUnURU5nU[R"U5-nURU5nUR U5nU[R"U5-nUR U5nUR UR:waURU5nX!-$r) rHr$sigmoidrIrJrKrr rL)rvr7rs r)r$Emu3VQVAETemporalResnetBlock.forwards  =1 }55  =1  =1 }55  =1   t00 0((2H''r+)rIrKrrLrHrJr rrrs@r)rErEs@ ( (r+rEc^\rSrSrS S\S\\S\\4U4SjjjrS S\RS\\R4Sjjr Sr U=r $) Emu3VQVAEResnetBlockirr quant_channelsc>[TU]5 XlUcUOUnX lX0lUc9[ R "USSSS9Ul[ R "USSSS9UlO [X15Ul[X25Ul[ R"UUSSSS9Ul [ R"UUSSSS9Ul URUR:wa[ R"UUSSSS9Ul gg) Nr"rTr#rrrr)rhrirr rSrRr'rHrJrrrIrKrL)rvrr rSrws r)riEmu3VQVAEResnetBlock.__init__s &&2&:{ (,  !;2SW`deDJ[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul URUl SUl [R"URUR5Ul[R"URUR5Ul[R"URUR5Ul[R"URUR5UlSUlg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).reFr)rhrirbrkrrl num_headsr@ ValueErrorscalermrHrnrRrorrrsrqout_projrOrs r)ri Emu3VQVAEAttentionBlock.__init__%s"  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A %&!r+r7rFr9c 2URupEnURU5nURU5nURU5n UR XEUR UR 5RSS5nUR XEUR UR 5RSS5nU R XEUR UR 5RSS5n [n URRS:wa[URRn U "UUUU UURURUR(dSO URS9upU R!XEU5R#5n UR%U 5n X4$)z#Input shape: Batch x Time x Channelrr r}r~)rnrGrH)r#rqrrrsrr]r@rQr]rbrrrnr_rNrHr<rWr`) rvr7rFrIr seq_lengthrquerieskeysvaluesrr\rZs r)rEmu3VQVAEAttentionBlock.forward<sS-:,?,?) ++m,{{=)]+,,zt~~t}}U__`acdeyyOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,, % ! "))*)LWWY mmK0 ((r+) rbrHrr@rnrrr]rOr`rqr_rsr)rrrrrrrir$rrrrrrrs@r)r[r["saG&&426$)||$)!.$) u||Xell33 4 $)$)r+r[c6^\rSrSrSrU4SjrSSjrSrU=r$)Emu3VQVAEGroupNormicz Same as the torch GroupNorm with the only difference that this ones accepts an optional kwarg `quant_states` which is not used. This class makes it easier to use SpatialNorm or GroupNorm without conditionals c &>[TU]"S0UD6 grW)rhri)rvrIrws r)riEmu3VQVAEGroupNorm.__init__js "6"r+c[R"XRURURUR 5$r)r group_normr%rrgr)rvinputr-s r)rEmu3VQVAEGroupNorm.forwardms'||E??DKKDHHUUr+rr) rrrrrrirrrrs@r)rirics #VVr+ricp^\rSrSrSU4SjjrSS\R S\\R 4SjjrSr U=r $)Emu3VQVAEMiddleBlockiqc>[TU]5 [UUUS9Ul[ U5UlUc[ USSSS9UlO[X25Ul[UUUS9Ul g)Nrr rSr"rTr#) rhrirRblock_1r[attn_1ri attn_normrblock_2)rvrbrrSrws r)riEmu3VQVAEMiddleBlock.__init__rsm +#$)  .f5  !/[UW]ajnoDN1.NDN+#$)  r+r7r-cNURX5nUnURX5nURupEpgURXEXg-5R SS5nUR U5SnUR XFXu5RSSSS5nX1-nURX5nU$)Nrr rr) rtrvr#rrQrur<rrw)rvr7r-rrrrrs r)rEmu3VQVAEMiddleBlock.forwards ]A  }C .;.A.A+ f%**:PZZ[\^_`  M215 %--j%RZZ[\^_abdef  0  ]A r+)rurvrtrwr) rrrrrir$rrrrrrs@r)rqrqqs1 ( U%6%6 huO`O`Fa  r+rqcJ^\rSrSrU4SjrS\R 4SjrSrU=r $)Emu3VQVAEDownBlockic >[T U]5 [UR5UlUR UlUR nURnS[U5-nX@l[R"5Ul [UR5GHLn[R"5n[R"5n[R"5nX$U-n X#U-n [UR 5H~n UR[U U S95 U n URcM-XQR;dM>UR[!U55 UR[R""U SSSS95 M [R$"5n XllX|lXlXPRS- :wa[-U 5U lURRU 5 GMO g)N)rrr r"rTr#r)rhrilenchannel_multipliernum_resolutionsnum_res_blocks base_channelsrin_channel_multiplierrR ModuleListdownrangeappendrRattn_resolutionsr[r'Moduleblockattn attn_normsr downsample)rvrbrrri_levelrrrblock_in block_outi_blockrrws r)riEmu3VQVAEDownBlock.__init__s "6#<#<=$33,, #66 $u-?'@ @%:"MMO T112GMMOE==?DJ$W'EEH%7(CCI !4!45 ($,%. %**67F]F];]KK 7 ?@%%bllUW]ajn&op699;DJI(O..22"@"J II  T "13r+r7c<[UR5GHup#[UR5HnURU"U5n[ UR 5S:dM3UnURU"U5nURupgpURXgX-5RSS5nUR U"U5SnURXhX5RSSSS5nXQ-nM X RS- :wdMURU5nGM U$)Nrrr r) enumeraterrrrrrrr#rrQr<rrr) rvr7rblocksrrrrrrs r)rEmu3VQVAEDownBlock.forwards(3OG !4!45 & W 5m D v{{#a',H$*$5$5g$>}$MM:G:M:M7J&$1$6$6zV^$\$f$fghjk$lM$*KK$8$G$JM$1$9$9*e$^$f$fghjkmnpq$rM$,$[T U]5 [UR5UlUR UlUR nURURS-n[R"5Ul [[UR55GH8n[R"5n[R"5n[R"5nURURU-n[UR S-5Hdn UR[UUUS95 UnXAR;dM0UR[!U55 UR[#X#55 Mf [R$"5n XZlXjlXzlUS:wa[-U5U lURR1SU 5 GM; g)Nrrrsr)rhrirrrrrrrRrupreversedrrrRrr[rrrrrrupsampleinsert) rvrbrSrrrrrrrrrws r)riEmu3VQVAEUpBlock.__init__si "6#<#<=$33))''&*C*CB*GG--/d&:&: ;&TU:BHG&M!|:8D GGNN1b !3=r+r7r-cb[URSSS25GHup4[URS-5HnURU"X5n[ UR 5S:dM3UnURU"X5nURupxpURXxX-5RSS5nUR U"U5SnURXyX5RSSSS5nXa-nM U[ UR5S- :wdMURU5nGM U$)Nrrrr r)rrrrrrrrr#rrQr<rr) rvr7r-rrrrrrrrs r)rEmu3VQVAEUpBlock.forwards(27OG !4!4q!89 & W 5m R v{{#a',H$*$5$5g$>}$[M:G:M:M7J&$1$6$6zV^$\$f$fghjk$lM$*KK$8$G$JM$1$9$9*e$^$f$fghjkmnpq$rM$,$  8 r+)rrrrrs@r)rrs-#"JU%6%6eFWFWr+rcJ^\rSrSrU4SjrS\R 4SjrSrU=r $)Emu3VQVAEEncoderic>[TU]5 URnURnURnUR nUR nU(aSU-OUnX&S-n[RRX2SSSS9Ul [U5Ul [X5Ul[RRSUSSS 9Ul[RRUUSSSS9Ul[%[&R("UR*55n [R,"5Ul[R,"5Ul[3U 5H)n [5Xw5n UR.R7U 5 M+ [3UR85H(n [;UUS 9n UR0R7U 5 M* g) Nr rrrrr"rT)r%r$rr&r~)rhrirr double_latentlatent_channelsrr$rRrconv_inr| down_blockrq middle_blockr'norm_outconv_outrmathlog2temporal_downsample_factorr time_convtime_res_stackrr?rrrE)rvrbrrrrrr rtemporal_down_blocksirr time_res_convrws r)riEmu3VQVAEEncoder.__init__ s ,, (( ,,  00#66.;q?*  b#99xx{qYZdef ,V40B**bxUYbf*g    (   #499V-N-N#OP mmo+,A.|JD NN ! !$ '-v,,-A8()M    & &} 5 .r+ pixel_valuesctURSnUR"S/URSSQ76nURU5nURU5nUR U5nUR U5nU[ R"U5-nURU5nUR"SU/URSSQ76nURSSSSS5nURH$nU"U5nU[ R"U5-nM& URH nU"U5nM URSSSSS5nU$)Nrrr rrr) r#r<rrrrr$rOrrrr)rvr temporal_dimr7rlayers r)rEmu3VQVAEEncoder.forward0s:#))!, #++BH1C1CAB1GH  \2  6 ))-8  m4 }55  m4 %--b,YATATUVUWAXY %--aAq!< NND /M U]]=9 9M#((E!-0M)&--aAq!< r+)rrrrrrr) rrrrrir$rrrrrs@r)rrs %6NE$4$4r+rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)Emu3VQVAEDecoderiNrbc>[T U]5 URnURURS-n[ R "5Ul[UR5H<n[URURS9nURRU5 M> [[R"UR 55n[ R "5Ul[U5H>n[%URUR5nUR"RU5 M@ [ R&"URUSSSS9Ul[+XUS9Ul[/U5UlURURS-n[3X#5Ul[ R&"UUR6SSSS9Ulg)Nrr~rrr)rSr)rhrirrrrRrrrrrErrrrrrrr3rrrqrrup_blockrrr r) rvrbrSrrrtemp_upsample_block_numrrrws r)riEmu3VQVAEDecoder.__init__Os| ))''&*C*CB*GG mmov,,-A8"22AWAWM    & &} 5 . #&dii0Q0Q&R"S./A,V-C-CVE[E[\D NN ! !$ '0yy  " "    1R`a(0 ''&*C*CA*FF,^F        r+r7r-c[R"X4SS9nURSSSSS5nURH nU"U5nM URH$nU"U5nU[R "U5-nM& URSSSSS5n[R "USSS9upUR"S/URSSQ76nUR"S/URSSQ76nURU5nURX5nURX5nURX5nU[R "U5-nURU5nU$)Nrr!r rrrr)r$r%rrrrOchunkr<r#rrrrr)rvr7r-hidden_quant_statesrs r)rEmu3VQVAEDecoder.forwardvsV#ii(E1M199!Q1aH((E"'(;"< )^^E"'(;"<  5==1D#E E $299!Q1aH&+kk2Eqa&P# %--bK=3F3Fqr3JK #++BH1C1CAB1GH  ]3 ))-F  mB  mB }55  m4 r+)rrrrrrr) rrrrrrir$rrrrrs@r)rrNs0% % NU\\r+raR The VQ-VAE model used in Emu3 for encoding/decoding images into discrete tokens. This model follows the "Make-a-scene: Scene-based text-to-image generation with human priors" paper from [ Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman](https://huggingface.co/papers/2203.13131). ) custom_introc^\rSrSr%\\S'SrSrSrSr Sr Sr /SQr Sr S\4U4SjjrS\R S \R 4S jrS \R 4S jrS rU=r$) Emu3VQVAEirb emuvideovqrT)rEr[rRrc[U[R[R45(a[RR UR SSS9 URbq[RRUR 5up#S[R"U5- n[RRURU*U5 gg[U[R5(a[RRUR [R"S5S9 URby[RRUR 5up#US:aS[R"U5- OSn[RRURU*U5 gg[U[R[R[R 45(aU[RR#UR S5 [RR#URS 5 g[U[R$5(adUR R&R)5 UR*b2UR R&UR*R-5 ggg) Nfan_outrelu)r nonlinearityr)arrr~) isinstancerRrrinitkaiming_normal_rrg_calculate_fan_in_and_fan_outrsqrtrrokaiming_uniform_ BatchNorm2drGr' constant_rrnormal_ padding_idxzero_)rvrBfan_inrbounds r) _init_weightsEmu3VQVAE._init_weightss fryy"))4 5 5 GG # #FMM PV # W{{&GGAA&--P DIIf--  ufe<' * * GG $ $V]]diil $ C{{&GGAA&--P 17!DIIf--  ufe<' N O O GG  fmmS 1 GG  fkk3 /  - - MM   & & (!!- ""6#5#56<<>..r+c>[TU]U5 Xl[U5Ul[ U5Ul[U5UlS[UR5S- -Ul [URURSSS9Ul[URURSSS9UlS[UR5S- -UlUR%5 UR'5 g)Nr r)rrrr6r7)rhrirbrencoderrdecoderrquantizerrvision_spatial_factorrrr quant_convpost_quant_convspatial_scale_factoreval post_initrs r)riEmu3VQVAE.__init__s   '/ '/ 08 %&3v/H/H+IA+M%N")  " "F$4$4)T]  /   f44)T] %&#f.G.G*H1*L$M!  r+ image_sizesc URS:HnU(aJURRnURupVpxUR S5R SUSSS5nOURupTpgnUR U5n U RSSSSS5n URU 5n U RSSSSS5n URU 5n U(aU RS5OU n [X5V V s/sHBupU S[U SUR- 52S[U SUR- 524PMD n n n U $s sn n f)Nrrrr r)ndimrbrr#r-repeatrrrrsqueezerrr)rvrris_imagerrrrrr7codes image_tokens single_imager/s r)encodeEmu3VQVAE.encodesP$$) {{==H2>2D2D /J&'11!4;;AxAqQLz) vocab_mapget eol_token_idimage_token_id)rvrs r)ri#Emu3ImageVocabularyMapping.__init__s)"%MM/:'mmI6r+c [URR5VVs/sHupURS5(dMUPM! snn5$s snnfNz<|visual tokensortedritems startswithrvnamevals r)r'Emu3ImageVocabularyMapping.image_tokenss<DNN,@,@,Bh,BytdooVfFgs,Bhiih A A c [URR5VVs/sHupURS5(dMUPM! snn5$s snnfr r rs r)image_tokens_str+Emu3ImageVocabularyMapping.image_tokens_strs<T^^-A-A-Ci-C tWgGht-CijjircxURVs0sHn[USS5URU_M! sn$s snf)NirK)rrr)rvtokens r)img2bpe"Emu3ImageVocabularyMapping.img2bpes;FJF[F[\F[UE"RL!4>>%#88F[\\\s%7cjURR5VVs0sHupX!_M snn$s snnfr)rr)rvr/vs r)bpe2img"Emu3ImageVocabularyMapping.bpe2img!s-!%!3!3!56!5!5666s /c[R"[URR 55S-[R S9nURR 5H up#X1U'M U$NrrL)r$zerosmaxrrerrrvmappingr/rs r)bpe2img_mapping_tensor1Emu3ImageVocabularyMapping.bpe2img_mapping_tensor%R++c$,,"3"3"56:%))LLL&&(DAAJ)r+c[R"[URR 55S-[R S9nURR 5H up#X1U'M U$r")r$r$r%rrerrr&s r)img2bpe_mapping_tensor1Emu3ImageVocabularyMapping.img2bpe_mapping_tensor,r*r+ img_batchr9c"URn[R"URSS4[RS9UR -nUR URS5n[R"XC/SS9nURU5$)Nrrr#cpurr!) devicer$rr#rrr,rVr%)rvr.r1eol_row img_tokenss r)convert_img2bpe*Emu3ImageVocabularyMapping.convert_img2bpe3su!!**iooa0!4EIIFIZIZZ00e1DE YY 4"= }}V$$r+cURnUSSS24nURURS5nURU5$)N.rr0)r1r(rV)rvr.r1r3s r)convert_bpe2img*Emu3ImageVocabularyMapping.convert_bpe2img:sG!!c3B3h' 00e1DE }}V$$r+)rr rN)rrrrrrirrrrrr(r,listr$rr4r7rrr+r)rr s7 jjkk]]77  %ell);% %%%%,,%r+rcN\rSrSr%\\S'SrSrS/rSS/r Sr Sr Sr Sr SrSrS rg ) Emu3PreTrainedModeliArbmodelTrpast_key_valuesr[FrN)rrrrrrrsupports_gradient_checkpointingr_skip_keys_device_placementrr_can_compile_fullgraph!_supports_param_buffer_assignmentrrrrr+r)r;r;AsO &*#$5m"DN!(-%"&r+r;cl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)Emu3RotaryEmbeddingiSrbc>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultinv_freqF) persistent)rhrihasattrrrEdictrrFmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrbr rope_init_fnattention_scalingregister_bufferrIoriginal_inv_freq)rvrbr1rIrws r)riEmu3RotaryEmbedding.__init__Ts  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rQrbrNrSrOrPrFr) rrrrrrir$no_gradrrrrrs@r)rCrCSs6/z//" ]]_ < [TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nr)rbF)rhri pad_token_idr vocab_sizerRrrk embed_tokensrrnum_hidden_layersrlayersrrnormrC rotary_embgradient_checkpointingrrus r)riEmu3TextModel.__init__|s  !.. ++LL):):F input_idsrFr2r= inputs_embedsr{rrIr9c 8USLUSL- (a [S5eUcURU5nU(a Uc [5nUcDUbUR5OSn [R "XUR S-URS9nUcURS5n[URUUUUUS9n Un URX5n URSURRHn U "U 4U UUUU S.UD6n M URU 5n [U US9$)Nz:You must specify exactly one of input_ids or inputs_embedsrr)r1)rb input_embedsrFr{r=r2)rFr2rzr{ry)last_hidden_stater=)r^rhr get_seq_lengthr$aranger#r1r-r rbrlrjrirkr)rvrorFr2r=rpr{rrIpast_seen_tokensr[r7ry decoder_layers r)rEmu3TextModel.forwardsK  -t"; <YZ Z  *.*;*;I*FM 0*nO  !CRC^==?de +0<< ]5H5H5K"KTaThTh,N  )33A6L(;;&))+%  & "oomJ![[)H4;;+H+HIM)*).-$7 MJ -0 &++  r+)rhrmrjrkrrlrg)NNNNNNN)rrrrrr__can_record_outputsrrirrrr$rrr rrrrrrrrrs@r)rcrcus*# z 151537+/5959$(8 E,,-8 !.8 u//0 8 "% 8   1 12 8 !!1!128 D>8 +,8  !8 8 r+rcc^\rSrSr%S/rSS0rSS/S/40r\\S'U4Sjr S r S r \ \ SS \\R S \\R"S \\R S\\S\\R&S\\R S\\S\\R S\\\R"4S\\S\4Sjj55rSrU=r$)Emu3ForCausalLMilm_head.weightlm_head colwise_repr7logitsrbc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gNFrf) rhrircr<rgrRrorkr}rrs r)riEmu3ForCausalLM.__init__sU  "6*  ++yy!3!3V5F5FUS  r+cXlgrr<rvrs r) set_decoderEmu3ForCausalLM.set_decoders r+cUR$rrrs r) get_decoderEmu3ForCausalLM.get_decoders zzr+rorFr2r=rplabelsrr{logits_to_keeprIr9c ~UR"SUUUUUUUS.U D6n U Rn [U [5(a [ U *S5OU n UR U SS2U SS245nSnUb)UR "SXURRS.U D6n[UUU RU RU RS9$)aI Example: ```python >>> from transformers import Emu3Processor, Emu3ForConditionalGeneration >>> import torch >>> import requests >>> from PIL import Image >>> model = Emu3ForCausalLM.from_pretrained("BAAI/Emu3-Chat-hf", torch_dtype=torch.bfloat16) >>> processor = Emu3Processor.from_pretrained("BAAI/Emu3-Chat-hf") >>> inputs = processor(text=["Can you write me a poem about winter."], return_tensors="pt").to(model.device) >>> generated_ids = model.generate(**inputs, max_new_tokens=100, do_sample=False) >>> processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ```rorFr2r=rprr{Nrrrglossrr=r7rdr) r<rsrrslicer} loss_functionrbrgrr=r7rd)rvrorFr2r=rprrr{rrIoutputsr7 slice_indicesrrs r)rEmu3ForCausalLM.forwards@,0:: , )%+') ,  ,  11 8B>SV8W8W~ot4]k mA}a,?@A  %%pVt{{OeOepiopD%#33!//))   r+)r}r<rg) NNNNNNNNr)rrrr_tied_weights_keys_tp_plan_pp_planrrrirrrrrr$rrr rrrrrrrrrrrs@r)r{r{s@*+=)H_-z:;H 151537+/59-1$(59348 E,,-8 !.8 u//0 8 "% 8   1 12 8 ))*8 D>8 !!1!128 c5<</08 +,8  8 8 r+r{cv^\rSrSrSS0rU4SjrSrSrSrSr S \ RS \ R4S jr S \ RS \ R4S jr\ RS \ RS\S\4Sj5rS\ RS\ RS\ R4Sjr\\SS\ RS \ RS \ R*S\\ R*S\\ RS\\S\\ RS\\S\\ RS\\S\\\44Sjj55rSrU=r $) Emu3Modeliztext_model.model text_modelc>[TU]U5 [RUR5Ul[ UR5Ul[UR5Ul UR5 gr) rhrirc _from_config text_configrr vq_configvqmodelrvocabulary_mapvocabulary_mappingrrs r)riEmu3Model.__init__sY  '44V5G5GH !1!12 "ilm>m n!,  224\B^A  s?BrrrcUSS2SS24RSX#S-5nURRU5nURR U5nU$)a Decodes generated image tokens from language model to continuous pixel values with VQGAN module via upsampling. Args: image_tokens (`torch.LongTensor` of shape `(batch_size, num_of_tokens)`): The tensors corresponding to the input images. height (`int`): Height of the generated image before upsampling. width (`int`): Width of the generated image before upsampling. Nrr)rrr7rr)rvrrr sequencesimages r)decode_image_tokensEmu3Model.decode_image_tokensWsV!CRC(--b&!)D ..>>yI  ##L1 r+rorprc JUcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnUR5nURS5RU5RUR 5nURSURS-nX$R5UR5:wa[SUSU35eU$)z Obtains multimodal placeholdr mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )rLr1rrrz6Image features and image tokens do not match: tokens: z , features )rr$tensorrr longr1allrr- expand_asrVr#numelr^)rvrorprspecial_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskEmu3Model.get_placeholder_maskjs  !.2K2K2M T44CC5::^k^r^rs3" "4!7!7!; !*.E.E.T.T!T +//1/99"=GG VYYZgZnZno)//2^5I5I!5LL  , 2 2 48L8L8N NHHXXcdtcuv "!r+rFr2r=rr{rIr9c 0USLUSL- (a [S5eUcUR5"U5nUbGURX#5n [R"U SS9n UR XU S9n UR X5nUR"SUUUUUU S.U D6n U $)aH image_sizes (`torch.LongTensor` of shape `(batch_size, 2)`): The sizes of the images in the batch, being (height, width) for each image. Image sizes can be obtained using [`AutoImageProcessor`]. See [`Emu3ImageProcessor.__call__`] for details ([]`Emu3Processor`] uses [`Emu3ImageProcessor`] for processing images). NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onerr!)rpr)rFr2r=rprr{r)r^rrr$r%rmasked_scatterr)rvrorrrFr2r=rprr{rI image_embedsrrs r)rEmu3Model.forwards*  -t"; <s    557 BM  #22<ML 99\q9L!%!:!:|";" *889KZM// )%+')   r+)rrr) NNNNNNNNN)!rrrr_checkpoint_conversion_mappingrirrrrr$rrrrrarrrrrrrr rrrrrrrrrrs@r)rrs&8,%G"64"U->->UM]M]"u/@/@uO_O_$ ]]0@0@#VY$"))":?:K:K"]b]n]n"0'+*.$(1537+/59$(59.##.''.\\ . !. . u//0 ."%.  1 12.D>.!!1!12.+,. u,, -..r+rc0^\rSrSrSrS/rSSSS.rU4SjrS rS r S \ R4S jr S r Sr\S5r\S5r\S5rSr\\S"S\R.S\R0S\R2S\\R2S\\R.S\\S\\R0S\\S\\R.S\\R.S\\\R24S\\ S \\!\"44Sjj55r#S#U4S jjr$S!r%U=r&$)$Emu3ForConditionalGenerationir|zmodel.text_modelz model.vqmodelr})z^text_model.modelz^vqmodelz^text_model.lm_headc>[TU]U5 [U5Ul[R "UR RUR RSS9Ul UR5 gr) rhrirr<rRrorrkrgr}rrs r)ri%Emu3ForConditionalGeneration.__init__sS  v& yy!3!3!?!?ASASA^A^ejk  r+c6URR5$r)r<rrs r)r1Emu3ForConditionalGeneration.get_input_embeddingsszz..00r+c:URRU5 gr)r<rrs r)r1Emu3ForConditionalGeneration.set_input_embeddingss ''.r+r9cUR$r)r}rs r)get_output_embeddings2Emu3ForConditionalGeneration.get_output_embeddingss ||r+c:URRU5 gr)r<rrs r)r(Emu3ForConditionalGeneration.set_decoders w'r+c6URR5$r)r<rrs r)r(Emu3ForConditionalGeneration.get_decoderszz%%''r+c.URR$r)r<rrs r)r'Emu3ForConditionalGeneration.text_modelszz$$$r+c.URR$r)r<rrs r)r$Emu3ForConditionalGeneration.vqmodelszz!!!r+c.URR$r)r<rrs r)r/Emu3ForConditionalGeneration.vocabulary_mappingszz,,,r+c :URR"S0UD6$rW)r<r)rvrIs r)r0Emu3ForConditionalGeneration.decode_image_tokensszz--777r+rorrrFr2r=rprr{rrrIc UR"SUUUUUUU S.U D6n U Sn[U [5(a [U *S5OU nUR USS2USS245nSnU b3UR "SUXR RRS.U D6n[UUU RU RU RS9$)aL image_sizes (`torch.LongTensor` of shape `(batch_size, 2)`): The sizes of the images in the batch, being (height, width) for each image. Image sizes can be obtained using [`AutoImageProcessor`]. See [`Emu3ImageProcessor.__call__`] for details ([]`Emu3Processor`] uses [`Emu3ImageProcessor`] for processing images). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import Emu3Processor, Emu3ForConditionalGeneration >>> import torch >>> import requests >>> from PIL import Image >>> model = Emu3ForConditionalGeneration.from_pretrained("BAAI/Emu3-Chat-hf", torch_dtype=torch.bfloat16) >>> processor = Emu3Processor.from_pretrained("BAAI/Emu3-Chat-hf") >>> conversation = [ ... { ... "role": "system", ... "content": [ ... {"type": "text", "text": "You are a helpful assistant."}, ... ], ... }, ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "Please describe the image."}, ... ], ... }, ... ] >>> prompt = processor.apply_chat_template(conversation, add_generation_prompt=True) >>> image = Image.open(requests.get("https://www.ilankelman.org/stopsigns/australia.jpg", stream=True).raw) >>> inputs = processor(images=[image], text=[prompt], return_tensors="pt").to(model.device, torch.bfloat16) >>> generated_ids = model.generate(**inputs, max_new_tokens=100, do_sample=False) >>> processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ```rrNrrr) r<rrrr}rrbrrgrr=r7rd)rvrorrrFr2r=rprr{rrrIrr7rrrs r)r$Emu3ForConditionalGeneration.forwards|**  )%+')      8B>SV8W8W~ot4]k mA}a,?@A  %%f9P9P9[9[_eD&#33!//))   r+c V>[T U]"U4UUUUUUUS.U D6n USS:waSU S'U $)N)r=rFrpr{r2rrrr)rhprepare_inputs_for_generation) rvror=rFrpr{r2rrrI model_inputsrws r)r:Emu3ForConditionalGeneration.prepare_inputs_for_generation@sZw<   +)')%%     !  !+/L (r+)r}r<) NNNNNNNNNNr)NNNNNTN)'rrrrrrrrirrrRrrrrpropertyrrrrrrr$rrrrr rrrrrrrrrrrrs@r)rrs*+/#(&" 1/ryy((%%""--8'+*.$(1537+/59$(59-134X ##X ''X \\ X !. X u//0 X "%X   1 12X D>X !!1!12X ))*X c5<</0X +,X  u,, -X X zr+r)rr{rcr;rr)Nr)r~)Ur functoolsrtypingrrrr$torch.nnrRtorch.nn.functionalrSr activationsr cache_utilsr r generationr integrationsr masking_utilsr modeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrconfiguration_emu3rrrr*r6rrrArrYr]r_rrrrrrrrr3r?rErRr[r'rirqr|rrrrrr;rCrcr{rr__all__rr+r)rs$. %,, !.)7/9OKF&II/KK(6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%'(%4C)BIIC)LY'J"))J(J( bii  *1*Z$ryy$D RYY 299bii:!299!H ."))&.(299.(b<(299<(~>)bii>)B V V299D88v7ryy7tCryyCLCryyCLl2l2l2^3%3%l'/''"<"))