PhdZddlmZmZmZddlZddlZddlmZddl mcm Z ddl m Z mZddlmZddlmZmZddlmZdd lmZdd lmZmZmZdd lmZmZd d lm Z m!Z!d dl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0ejbe2Z3dDde4fdZ5 dEdejlde4de4de4fdZ7dejldejpde4de9de4dejlf dZ:ddejvfdejld e4d!e4d"eejlejlffd%Z?d&ejld'ee4dejlfd(Z@Gd)d*e'ZAGd+d,e(ZBGd-d.e ZCGd/d0e%ZDGd1d2e)ZEGd3d4e&ZFeGd5d6e$ZGGd7d8eGZHGd9d:eGZIGd;deGZKGd?d@eGZLGdAdBe#ZMgdCZNy)Fz>> tokens = torch.tensor([[1, 2, 3], [4, 5, 6]]) >>> hashes = rolling_polynomial_hash(tokens, prime=31) >>> # hash[0] = 1*31^0 + 2*31^1 + 3*31^2 >>> # hash[1] = 4*31^0 + 5*31^1 + 6*31^2 dtypedevicer(dim)torchtensorint64r(arangeshapesum) token_tensorr$ prime_tensorpowers prime_powerss ]/var/www/html/saasai/venv/lib/python3.12/site-packages/transformers/models/blt/modular_blt.pyrolling_polynomial_hashr89sa.<<U[[ATATUL \\,,,R09L9L MF'L 99\L0b 99 token_ids group_sizemax_hashcZtj5|j\}}tj||dz tj|j }tj ||gd}|jd|d}t||} | |z} ddd| S#1swY SxYw)z1Hash token groups and map to range [0, max_hash].rr&r+N) r-no_gradr1zerosr/r(catunfoldr8) r:r;r$r< batch_sizeseq_lenpadding padded_tokenswindowshashes hash_valuess r7byte_group_hash_functionrIVs 'oo G++j*q. T]TdTde 7I"6A>  &&q*a8(%8x'     s BB  B*local_encoder_tokensencoder_hash_tok_embedding$encoder_hash_byte_group_nb_functionsencoder_hash_byte_group_sizeencoder_hash_byte_group_vocabreturncgd}|j|}d}t|D]@} || t|z} |D](} t|| | |} | ||zz} ||| z }|dz }*B|S)z=Compute token embeddings enhanced with hash-based embeddings.) ʚ;l21Ai oYlvtl.l}glAul 0lTlAK l|rr) embed_tokensrangelenrI)rJ local_encoderrKrLrMrNprimes embeddings embedding_idxfunc_nbr$r;hash_idsoffset_hash_idss r7compute_hash_embeddingsr\hs F++,@AJM=>wV,-6J/0DjRWYvwH&9V)VVO 4_E EJ Q M 7? r9F patch_ids num_patchessequence_lengthpatches_as_queries cross_attn_kr'cz|j\}}|j}|rp||z} |} tj||j dj dj |||} |j dj |||} no|} ||z} |j dj |||} tj||j dj dj |||} | | k(} |rdnd}| j ||} || | f}| j|k7rtd| jd|| j d} d| j|z }|j|jtjtj|j} | S) aR Prepare cross-attention mask for patch-based attention, following mllama's robust approach. This function creates masks that control which patches can attend to which other patches, with support for query/key role swapping and cross-attention multipliers. Args: patch_ids (torch.Tensor): Tensor of shape [batch_size, seq_len] containing patch ids. num_patches (int): Total number of patches. sequence_length (int): Length of the sequence. patches_as_queries (bool): If True, patches are used as queries, otherwise as keys. cross_attn_k (int): Cross-attention multiplier for repeating patches. dtype (torch.dtype): Data type for the output mask. Returns: Tuple[torch.Tensor, torch.Tensor]: - cross_attention_mask: 4D tensor [batch_size, 1, q_len, kv_len] r*rr)rr+zCross attention mask shape z doesn't match expected g?) r1r(r-r0 unsqueezeexpandrepeat_interleave ValueErrorto masked_fillboolfinfomin)r]r^r_r`rar'rBrCr(q_lenkv_len q_patch_ids kv_patch_idscross_attention_mask repeat_dimexpected_shapeinverted_cross_attn_masks r7#_prepare_patch_cross_attention_maskrts4$//J   Fl*  LLV 4 Yq\ Yr] VJ W 5  !**1-44ZgV |+))"-44Z+V LLV 4 > >q A K KA N U UV`bikv w  ',6)bJ/AA,T^A_!%0N!!^3)*>*D*D)EE]^l]m n  099!< #%9%<%z(process_patch_lengths..s6I&#f+Isr&r+r)sizeitemdivmodextendappendmaxr-r?r'r( enumerater.rTanyr2r1nonzero) rurvrB processedseqr{length full_chunks remaindermax_lenpaddedi last_nonzeros r7process_patch_lengthsrs##A&JI#'lF[[]F%+F4D%E "K MM+,{: ; i( #  6I66G [[*g.m6I6IR_RfRf gFy) 6 ',||F-BUBU^k^r^r'sF1mF m# $* ! Q##% Q7! ((Q(/779==?DDFJ =L=() Mr9c eZdZy)BltMLPN__name__ __module__ __qualname__r9r7rrr9rc eZdZy) BltRMSNormNrrr9r7rr rr9rc eZdZy)BltRotaryEmbeddingNrrr9r7rrrr9rc$eZdZdeffd ZxZS)BltTransformerLayer layer_idxct|t|||_t ||_t |j|j|_ t |j|j|_ y)N)configreps) super__init__BltSelfAttention self_attnrmlpr hidden_size rms_norm_epsinput_layernormpost_attention_layernormselfrr __class__s r7rzBltTransformerLayer.__init__s] )9M&>)&*<*<&BUBUV(263E3E6K^K^(_%r9)rrrintr __classcell__rs@r7rrs`#``r9rc eZdZdedeffd Z d dejdejdejdeffd Z xZ S) rrrc4t|||d|_y)NT)rr is_causalrs r7rzBltSelfAttention.__init__s +r9 hidden_statesattention_maskposition_embeddings use_cachec 0t|d||||||d|S)N)rrrrpast_key_valuescache_positionr)rforward) rrrrrrrkwargsrs r7rzBltSelfAttention.forward#s7w ') 3+)    r9)FNN) rrrrrrr-Tensorrirrrs@r7rrs]yS  ||   #\\     r9rceZdZdZd dededeeffd Z ddejdeejdee d eejd eejd e e f d ZxZS)BltCrossAttentionzL#b$2J2JDMMZddefhijJ',,S"d6N6NPTP]P]^hhijlmnL*+:+A+A dnn?OQ_>`,( LA ! #&&t~~6;;&&t~~6==%J p )@ ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,LL %  % ! \"))#ub9DDF kk+. !M1 L((r9ryNNNN)rrr__doc__rrrrr-rr LongTensorr rrrrs@r7rr8sFLyLSLxPS}L:>+/1559 3)||3)!) 63)"% 3) !. 3) !!1!12 3)+,3)r9rcjeZdZUeed<dZdZdZdgZe e dde e dddZ d Z d Zd Zy ) BltPreTrainedModelrFrr local_decoderindex layer_namer)r attentionsctdNzNo need to inherit it!AttributeErrorrmodules r7 _init_weightsz BltPreTrainedModel._init_weights566r9ctdrrrs r7_update_causal_maskz&BltPreTrainedModel._update_causal_maskrr9ctdrrrs r75_prepare_4d_causal_attention_mask_with_cache_positionzHBltPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positionrr9N)rrrr__annotations___supports_attention_backend_supports_flash_attn_supports_flex_attn_no_split_modulesrrr_can_record_outputsrrrrr9r7rrwsU "' ./'(;1Q`a$%5Q?[ 777r9rc|eZdZUeed<deeddiZdeffd Z dde e jde e jd e e jd e e jd e e jd e e d e e jde e jde ede e jdeefdZdZxZS)BltLocalEncoderrencoder_attentionsrrUrc t||d|_||_t j t |jDcgc]}t||c}|_ t||_ t j|j|j|jzd|_t j |j"|j|_t j |_|j(r |jnd}t |D]3}|j&j+t-|||j5|j/ycc}w)NFr in_features out_featuresbiasrrrr)rrgradient_checkpointingrnn ModuleListrSnum_hidden_layersrrr rotary_embLinearrrapatch_embedding_projection Embedding vocab_sizerRcross_attn_layerscross_attn_all_layersrr post_initrrr layers_to_addrs r7rzBltLocalEncoder.__init__s"  &+# mmEJ6KcKcEd eEd  3Ed e  -F;*,))**++f.A.AA+ ' LL):):F>|L +33 2 21 5 8P8P PRVR]R]RiRi  $(;;#D#DC! ,MD,B,B9,M-".+8#9- -)& ,.DD -1. ,222r9cF|jd}|jd}|jdjdd|jd}tj|||f|j |j }|j|d|dd}|ddd|ddf}|S) a Reduce variable length patches to single embedding per patch Note: this works with variable number of patches for different sequences in the batch It handles variable length patches by assuming that patch_lengths will be 0 for any extra patches on the *right*. Since there can be a variable number of patches this function also return the number of patches for each sequence in the batch. Any embeddings on the right that are not allocated to a patch (i.e. if the sum(patch_lengths[i]) < seq_len for any i) will be sent to a dummy patch, which is trimmed before returning. rr)r&ramaxF)srcr,rreduce include_selfN)r1rcrdr-r?r'r(scatter_reduce)rrmax_num_patchesr]rB embedding_dimreduced_embeddingss r7r!zBltLocalEncoder.patch_reduces#((+ %++B/ ''+222r=;N;Nr;RS "[[ - 8 @S@S\i\p\p 0>> ? 03CO3CQ0FG!!r9 NNNNNNNNNN)rrrr"rrrrrrr-rrrrr rrr!rrs@r7rrs! !!n-=QSbc421504/31537+/599=%),043E,,-43  -43u||, 43 !. 43 u//0 43"%43!!1!1243!) 643c]43ELL)43+,43l"r9rc:eZdZUeed<deffd Ze d deejdeejdeejdeejdeejdee d eejd eejd e e fd ZxZS)BltLocalDecoderrc t||d|_||_d|_t j t|jDcgc]}t||c}|_ t||_ t j|j|j|j zd|_t%|j|j&|_t j |_|j,r |jnd}t|D]3}|j*j/t1|||j5|j3ycc}w)NFTrrrrr)rrrrcross_attn_decoderr r rSr rrrr r hidden_size_globalrrarrrnormrrrrrrs r7rzBltLocalDecoder.__init__s%  &+# "&mmEJ6KcKcEd eEd  3Ed e  -F;*,))11++f.A.AA+ ' v11v7J7JK !#4:4P4P00VW }-I  " " ) )!9RXRdRde . ! fs E%rrrrrrrrrc $|jd} |} |j|}|j| |jd|jjz|jj }||j s| |z} |Mtj|jd|jjdj| d}|j| |} tj| |jj|j} t!|j"D]O\} }| dk(s|jj$r!|j&| d| ||d| \}}| |z} || f| |||d| } Q|j)| }|S) Nrrr*r)rrrr)r1rrrrarr3r-r0r(rcrdr r rrrrrrr5)rrrrrrrrrrrBrrrr#r$rlogitss r7rzBltLocalDecoder.forwards#((+ % 66|D #++  **1- 0H0HH$++JaJa   #D,C,C)L8M   ]003Mrtoken_embedding_projectionIdentityrrs r7rzBltGlobalTransformer.__init__Us   mmo v778I KK  269E F9,F; 66 = I.0ii00&2D2D5/D +/1kkmD + r9 input_embedsrrrrrc |j\}}} |j|} tj| |jj|j } |Mt j|jd|jjdj|d}|j| |} t|jD]\} } | | f| |||d|} | S)Nrrr*rr)r)r1rAr rrrr-r0r(rcrdr rr)rrCrrrrrrBrCrrrrr#s r7rzBltGlobalTransformer.forwardgs".!3!3 GQ77 E  -4;;3F3FQUQ^Q^_   \//2<;N;NOYYZ[\ccdnprs #oom\J!$++.HAu!$7- /-   M/r9r)rrrr rrrrrr-rrrrr rrrrs@r7r:r:Os &&^,LL)?)?AXAXYt{{66DKK>f>MUUW&3&9&9"1&=# O  ! ==. /%# >M"JJ_-]5H5HQ^QeQeM.m=MN #]F::r9c|jd}tjddgtj|jj dj |d}|jd}|ddddf}||kD}|jd}tj||jj dj|d} tj| |} tj| | gd} tj||gd} | | j||} |jdj}| ddd|f} tj|| |zfd}tj|ddddf|dz }tj|ddddfdz |fd}||z dz}|S)z Computes patch lengths from token entropies. Depending on whether a threshold is provided, the function uses either: - Thresholding the entropy values (when `threshold` is set). rrr&Nr*r)r+)r1r-r.longr(rcrepeatr0rd full_liker@rr2r)rNr_rIrJrB init_tokensoffset patch_maskrC token_indicessentinelpadded_indices padded_mask patch_startsmax_valid_patchespatch_start_ids last_token patch_endsrus r7rSz'BltPatcher.patch_lengths_from_entropiess__Q'  LL!Quzz):J:J K U UVW X _ _`jlm n ""1%ae$ * ""1% WY5E5EFPPQRSZZ[egij ??=':M8#?;!!1!12?;SM?;E??;#3-?;+,?;B 33r9rFcfeZdZdeffd Ze ddeejdeejdeejdeejdee deejd ee d eejd e ed efd ZdZdZdejded ejfdZxZS)BltModelrct||d|_||_t |j |_t|j|_ t|j|_ |jt|jz}|j |z}t#j$||j j&|_|jj*r[t-|j.|_|j0j3|j0j5D] }d|_ nd|_|j9y)NF)rrrrrencoder_configrUr: global_configr<r1decoder_configrrLrTrMrNr rrrKpatch_in_forwardrFpatcher_configpatchereval parameters requires_gradr)rrnum_embeddingstotal_vocab_sizeparamrs r7rzBltModel.__init__s  &+# ,V-B-BC"6v7K7K"L,V-B-BCDDs6KnKnGoo!??.P*,,,7GI^I^IjIj*k' ;; ' '%f&;&;
~ !!1!12~ +,~  !~ ~ @/0 iU\\ iC iTYT`T` ir9rlceZdZUeed<dZdZdgZdeffd Z dde e jde e jde e jd e e jd e e jd e e e je jfd e eeee j"fd e e j"de e jde ede e jdeee jfdeedee effdZxZS)BltForCausalLMrFmodelzlm_head.weightct|||j|_t||_t j |jj|jd|_ |jy)NFr?) rrrrlrr r rprrHr)rrrs r7rzBltForCausalLM.__init__sZ   ++f% yy!6!6!B!BFDUDU\ab  r9rrrrrpfull_text_row_masked_out_maskrrlabelsrrlogits_to_keeprrOc |jd|||||||| | d | }|j}t| tr t | dn| }|j |dd|ddfj }d}| |j|| |jfi| }t|||j|j|jS)N) rrrrprrrrr)lossr7rrrr) rr| isinstancerslicerHri loss_functionrr rrr)rrrrrrprrrrrrrroutputsr slice_indicesr7rs r7rzBltForCausalLM.forwards"$**  )%!5*G+')     11 8B>SV8W~ot4]k mA}a,?@AGGI  %4%%ffdooPPD%#33!//))   r9) NNNNNNNNNNNr)rrrrr_can_compile_fullgraphbase_model_prefix_tied_weights_keysrrr-rrtuplerrlistrhrirr rr rrrs@r7rrs "*+y151537=A;?UYKO59-1$(5934, E,,-, !., u//0 , !))9)9 : , 'u'7'78 , (0ellELL6P0Q'R, "%tE4E4E/F(F"GH,   1 12, ))*, D>, !!1!12, c5<</0, +,,  u,, -, r9r)rrlrFr)rQ)rrQi0u)Ortypingrrrr-torch.distributionstorch.nnr torch.nn.functional functionalr  cache_utilsrr masking_utilsr modeling_outputsr r modeling_utilsr processing_utilsr utilsrrr utils.genericrrcohere2.modeling_cohere2rrmllama.modeling_mllamarrrrrrrrconfiguration_bltrr r!r"r# get_loggerrloggerrr8rrIrrr\float32rir'rrtrrrrrrrrrr1r:rFrlr__all__rr9r7rsgC,, ./O5&@@?      H %::<\a||),9<UX$#,,#!# #+. # #' # $' # \\#T % K ||K K K  K  K ;; K  5<< %&K \))RU)[`[g[g)X ]  "  / `9` . 4<)0<)~7.77*p"(p"fL(L^2-2jF#FRfi!fiR9 &9 x r9