"SS\$5r?"S S!\&5r@"S"S#\%5rA"S$S%\#5rB"S&S'\'5rC\"S(S)\55rD"S*S+\D5rE"S,S-\ Rt5rF"S.S/\ Rt5rG"S0S1\ Rt5rH\\"S2S3\555rI"S4S5\ Rt5rJ"S6S7\ Rt5rK"S8S9\/5rL"S:S;\05rM"S<S=\-5rN"S>S?\+5rO"S@SA\,5rP"SBSC\.5rQ"SDSE\Q5rR"SFSG\Q\5rS/SHQrTg)IN) dataclass)OptionalUnion)Tensornn)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPast,BaseModelOutputWithPoolingAndCrossAttentionsCausalLMOutputWithPast ModelOutput)ModuleUtilsMixinPreTrainedModelget_parameter_dtype)auto_docstringcan_return_tuplelogging)check_model_inputs) EsmAttention EsmEmbeddings EsmEncoderEsmIntermediateEsmLayer EsmOutput EsmPoolerEsmSelfAttention EsmSelfOutput)LlamaAttentionLlamaDecoderLayerLlamaMLPLlamaPreTrainedModel LlamaRMSNormLlamaRotaryEmbedding) EvollaConfig SaProtConfigc(^\rSrSrU4SjrSrU=r$)EvollaSaProtEmbeddingsAc0>[TU]5 SUlgN)super__init__ position_idsselfconfig __class__s a/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/evolla/modular_evolla.pyr1EvollaSaProtEmbeddings.__init__Bs  )r2)__name__ __module__ __qualname____firstlineno__r1__static_attributes__ __classcell__r6s@r7r,r,As !!r9r,cVURSSS9up[R"U*U4SS9$)Nrdim)chunktorchcat)xx1x2s r7rotate_half_esmrKHs- WWQBW FB 99rc2YB ''r9cUSS2SS2SURS2SS24nUSS2SS2SURS2SS24nX-[U5U--$)N)shaperK)rHcossins r7apply_rotary_pos_emb_esmrQMsW aMaggbkM1$ %C aMaggbkM1$ %C G*S0 11r9c^\rSrSrSrS\4U4SjjrS SjrS\RS\RS\ \R\R44S jr S r U=r $) EvollaSaProtRotaryEmbeddingTz Rotary position embeddings based on those in [RoFormer](https://huggingface.co/docs/transformers/model_doc/roformer). Query and keys are transformed by rotation matrices which depend on their relative positions. rDc >[TU]5 SS[R"SUS[RS9R 5U- -- nUnUR SU5 SUlSUlSUl g)N?i'rrdtypeinv_freq) r0r1rFarangeint64floatregister_buffer_seq_len_cached _cos_cached _sin_cached)r4rDrYr6s r7r1$EvollaSaProtRotaryEmbedding.__init__[sl %ELLC%++$N$T$T$VY\$\]^ Z2#r9cjURUnX0R:wd$URRUR:waX0l[R "URUURS9R UR5n[R"X@R5n[R"XU4SS9RUR5nUR5SSSS2SS24UlUR5SSSS2SS24Ul URUR4$)NdevicerBrC)rNr^r_rdrFrZtype_asrYouterrGtorOrPr`)r4rH seq_dimensionseq_lentfreqsembs r7_update_cos_sin_tables2EvollaSaProtRotaryEmbedding._update_cos_sin_tablesfs''-( ** *d.>.>.E.E.Q#*  QWW]3AHHEMMdmm\AKK==1E))UN366qxx@C"wwytQ)9:D "wwytQ)9:D !1!111r9qkreturncURUSS9uUlUl[XRUR5[X RUR54$)NrM)rh)rmr_r`rQ)r4rorps r7forward#EvollaSaProtRotaryEmbedding.forwardvs[-1-H-HZ\-H-]*$* %Q(8(8$:J:J K $Q(8(8$:J:J K  r9)r_r^r`)r)r:r;r<r=__doc__intr1rmrFrtuplersr>r?r@s@r7rSrSTsR  C  2   %,, 5u||A[;\  r9rSc\rSrSrSSjrSrg)EvollaSaProtSelfAttentionNc[RR5 XlURUR -S:wa7[ US5(d&[SURSUR S35eUR Ul[URUR - 5Ul UR UR-Ul [R"URUR5Ul [R"URUR5Ul [R"URUR5Ul[R"UR 5UlU=(d [%USS5UlSUlUR&S:XdUR&S :XaHUR*Ul[R,"S UR*-S - UR5UlO(UR&S :Xa[1URS 9UlUR2UlX0lg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryrr(rotaryrC)rModuler1r5 hidden_sizenum_attention_headshasattr ValueErrorrvattention_head_size all_head_sizeLinearquerykeyvalueDropoutattention_probs_dropout_probdropoutgetattrr~rotary_embeddingsmax_position_embeddings Embeddingdistance_embeddingrS is_decoder layer_idx)r4r5r~rs r7r1"EvollaSaProtSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $"&  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #  ) )X 5%@TE]E]%^D " ++"r9)rrr5rrrrrrrr~rrrNN)r:r;r<r=r1r>r9r7ryrys#r9ryc\rSrSrSrg)EvollaSaProtSelfOutputrNr:r;r<r=r>rr9r7rrr9rc\rSrSrSrg)EvollaSaProtAttentionrNrrr9r7rrrr9rc\rSrSrSrg)EvollaSaProtIntermediaterNrrr9r7rrrr9rc\rSrSrSrg)EvollaSaProtOutputrNrrr9r7rrrr9rc\rSrSrSrg)EvollaSaProtLayerrNrrr9r7rrrr9rc\rSrSrSrg)EvollaSaProtEncoderrNrrr9r7rrrr9rc\rSrSrSrg)EvollaSaProtPoolerrNrrr9r7rrrr9rc0\rSrSr%\\S'S/rSrSrSr g)EvollaSaProtPreTrainedModelr5rTcURRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[R5(adUR R RSUS9 URb2UR R URR5 gg[U[R5(aJURR R5 UR R RS5 gg)zInitialize the weightsmeanstdNrV)r5initializer_range isinstancerrweightdatanormal_biaszero_r padding_idx LayerNormfill_r4modulers r7 _init_weights)EvollaSaProtPreTrainedModel._init_weightsskk++ fbii ( ( MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r9rN) r:r;r<r=r*__annotations___no_split_modules_supports_flash_attnrr>rr9r7rrs ,- *r9rc ^\rSrSrS\4U4SjjrSrSrSr\ SS\ \ RS\ \ RS \ \\ R\44S jj5rSS\ S \\S \ R$S \ R&S \ 4 SjjrSrU=r$)EvollaSaProtProteinEncoderr5cd>[TU]U5 [U5Ul[ U5Ulgr/)r0r1r, embeddingsrencoderr3s r7r1#EvollaSaProtProteinEncoder.__init__s(  08*62 r9c.URR$r/rword_embeddingsr4s r7get_input_embeddings/EvollaSaProtProteinEncoder.get_input_embeddingss...r9c$XRlgr/rr4rs r7set_input_embeddings/EvollaSaProtProteinEncoder.set_input_embeddingss */'r9cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrlayer attention prune_heads)r4heads_to_prunerheadss r7 _prune_heads'EvollaSaProtProteinEncoder._prune_headss< +002LE LL  u % / / ; ;E B3r9 input_idsattention_maskrqc0UR5nUupEURnUc[R"XE4US9nUR XS9nUR X#5nUR XxS9n U Sn [U U RU RU RS9$)Nrcrr)rr)last_hidden_state hidden_states attentionscross_attentions) sizerdrFonesrget_extended_attention_maskrrrrr) r4rr input_shape batch_size seq_lengthrd inputs_embedsextended_attention_maskencoder_outputssequence_outputs r7rs"EvollaSaProtProteinEncoder.forwards  nn& !, !!  !"ZZ*)A6RN)[ "&"B"B>"_,,},])!,;-)77&11,==   r9rrdrXcPUc [U5nUR5S:XaURR(dUb[R "S[ 5 UR5S:XaUSS2SSS2SS24nOqUR5S:XaAURR(a[R"X!U5nO*USS2SSSS24nO[SUSURS35eURUS9nS U- [R"U5R-nU$) a Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. NrzNThe `device` argument is deprecated and will be removed in v5 of Transformers.rz!Wrong shape for input_ids (shape z) or attention_mask (shape r}rWrV)rrDr5rwarningswarn FutureWarningr*create_extended_attention_mask_for_decoderrrNrgrFfinfomin)r4rrrdrXrs r7r6EvollaSaProtProteinEncoder.get_extended_attention_masks  ='-E""$)dkk.D.D! dfs    1 $&4Qa]&C #    !Q &{{%%*:*e*e+'+9D$9I*J'3K=@[\j\p\p[qqrs #:"<"<5"<"I#&)@#@EKKPUDVDZDZ"Z&&r9)rrr/r)r:r;r<r=r*r1rrrrrrFrrrwrrsrvrdr\rr>r?r@s@r7rrs3|3 /0C26 ELL) !.  uU\\"$PP Q   2rv2'$2'38:2'GL||2'chcncn2' 2'2'r9rc2^\rSrSrSU4SjjrSrSrU=r$)!EvollaSequenceCompressorAttentioni5cX>[TU]5 US-UlX0lX#-n[R "U5Ul[R "U5Ul[R"XSS9Ul [R"XS-SS9Ul [R"XASS9Ul g)NFrr) r0r1scalerrr norm_media norm_latentsrto_qto_kvto_out)r4rDdim_headr inner_dimr6s r7r1*EvollaSequenceCompressorAttention.__init__6s t^  $ ,,s+LL-IIc59 YYsM> ii U; r9c&URU5nURU5nURnURU5n[R "X4SS9nUR U5RSSS9upxURURS5URS5US5RSSSS5nURURS5URS5US5RSSSS5nURURS5URS5US5RSSSS5nXPR-n[R"XWRSS55n XRSSS 9R5- n U R upp[R""X5R%UR&5nUS S 2S S S S 24nUS S S 2S S 2S 4nUU-nU R)SU- R+5S 5n U R-SS9n[R"UU5nURSSSS5nUR/URS5URS5S5nUR1U5$) z Args: x (torch.Tensor): image features shape (b, n1, D) latent (torch.Tensor): latent features shape (b, n2, D); n2: num of latent tokens rMrCrrBrr(rTrDkeepdimNg)rrrrrFrGr rEviewrpermutermatmul transposeamaxdetachrNrrgrd masked_fillboolsoftmaxreshaper )r4rHlatentsmaskhrokv_inputrpvsimbsnhskdokdrmask_expones_expattnouts r7rs)EvollaSequenceCompressorAttention.forwardCs2 OOA ##G, JJ IIg 99a\r2zz(#)) 2*  FF166!9affQiB / 7 71a C FF166!9affQiB / 7 71a C FF166!9affQiB / 7 71a C Nll1kk"b12HHTH299;;99zz""%%dkk24q()aD()("ooq4xoo/6{{r{"ll4#kk!Q1%kk#((1+sxx{B7{{3r9)rrrrr r r)@r:r;r<r=r1rsr>r?r@s@r7rr5s <) ) r9rc2^\rSrSrSU4SjjrSrSrU=r$)EvollaFeedForwardioc>[TU]5 [X-5n[R"U5Ul[R "XSS9Ul[R"5Ul [R "X1SS9Ul gNFr) r0r1rvrrnormrfc1GELU activationfc2)r4rDmultr r6s r7r1EvollaFeedForward.__init__psZ  O LL% 99S%8'')99Y%8r9c ~URURURURU5555$r/)r5r4r2r1)r4rHs r7rsEvollaFeedForward.forwardys+xx1(>?@@r9)r4r2r5r1)r,r@s@r7r.r.os9AAr9r.c6^\rSrSrS\4U4SjjrSrSrU=r$)!EvollaSequenceCompressorResampleri}r5c >[TU]5 URRnURUl[ R"[R"UR U5SS9Ul [ R"/5Ul [UR5HanURR[ R"[!X!R"UR$S9['X!R(S9/55 Mc [ R*"UR5Ul[ R."X!R5Ulg)NT) requires_grad)rDr r)rDr6)r0r1protein_encoder_configrresampler_num_latents num_latentsr ParameterrFrandnr ModuleListlayersrangeresampler_depthappendrresampler_dim_headresampler_headsr.resampler_ff_multrr1rprotein_projector)r4r5protein_repr_dim_r6s r7r1*EvollaSequenceCompressorResampler.__init__~s !88DD!77||EKK0@0@BR$Scgh mmB' v--.A KK   9 0;T;T\b\r\r*.>E]E]^  /LL!3!34 !#+;=O=O!Pr9cdURSnURupE[R"X@R5R UR 5n[R "X&4SS9n[R"U5R URR 5nURSURSSS5-nUR UR5nURHupU "XU5U-nU "U5U-nM URU5n URU 5$)Nrr(rCrB) rNrFrrArgrdrGrrrXrErLr1) r4embedsrbr!rN latent_maskrrr'fftransformed_features r7rs)EvollaSequenceCompressorResampler.forwards LLO jj%5%5699$++F yy$,!4zz!} 3 34,,t$tyyQ'::**V\\* HD6D1G;GkG+G$#44W=yy,--r9)rrEr1rArL) r:r;r<r=r)r1rsr>r?r@s@r7r<r<}sQ|Q*..r9r<c\rSrSr%Sr\R \S'Sr\ \R \S'Sr \ \ \R S4\S'Sr \ \ \R S4\S'Sr g) EvollaProteinEncoderModelOutputiNsequence_compressor_outputr.rrr)r:r;r<r=rYrF FloatTensorrrrrrwrr>rr9r7rXrXsi59 1 1859x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r9rXct^\rSrSrS\4U4Sjjr\S\RS\R4Sj5r Sr U=r $)EvollaProteinEncoderir5cn>[TU]5 [URS9Ul[ US9Ulg)Nr5)r0r1rr?modelr<sequence_compressor_resamplerr3s r7r1EvollaProteinEncoder.__init__s. /v7T7TU -NV\-]*r9rrc URXS9nURnURXR5n[UURS9$)Nr)rYr)r_rr`rX)r4rrkwargsprotein_outputprotein_embeds sequence_reprs r7rsEvollaProteinEncoder.forwardsFiW'99::>Z .'4,>>  r9)r_r`) r:r;r<r=r)r1rrF LongTensorrZrsr>r?r@s@r7r\r\s?^|^  !1!1 5CTCT  r9r\cr^\rSrSrS S\\S\\S\\4U4SjjjrSrS SjrSr U=r $) #EvollaSequenceAlignerCrossAttentioniprotein_encoder_dimstructure_encoder_dimmsa_encoder_dimc>[TU]5 URUlURUlURS-Ul[ URUR- 5UlURUR -UlURnURnURn[R"URUR5Ul UbK[R"X R5Ul[R"X R5UlOSUlSUlUbK[R"X0R5Ul[R"X0R5UlOSUlSUlUbK[R"X@R5Ul[R"X@R5UlOSUlSUl[)UR5Ul[R,"U5Ul[R"URURUS9Ul[3URU5Ul[R6"[8R:"S/55Ul[R6"[8R:"S/55Ulg)Nrrr) r0r1rrrrvrr$aligner_attention_probs_dropout_probaligner_enable_biasaligner_ffn_multrrr key_protein value_protein key_structurevalue_structurekey_msa value_msa EvollaRMSNormattention_normrrout_projr.rTrBrFtensorgate_attentiongate_ffw) r4r5rkrlrmr enable_biasffn_multr6s r7r1,EvollaSequenceAlignerCrossAttention.__init__s !--#)#=#= --t3 #&t'7'7$:R:R'R#S !558P8PP'-'R'R$00 **YYt//1C1CD  *!yy)<>P>PQD !#+>@R@R!SD #D !%D  ,!#+@BTBT!UD #%99-BDVDV#WD !%D #'D  &99_6H6HIDLYY8J8JKDNDL!DN+D,<,<=zz">?  $"2"2D4D4D;W #D$4$4h? ll5<<+>? U\\3%%89 r9c XgU/n U V s/sH ocMU PM n n U (d [S5e[R"U SS9n URU5n UR U 5n UR bAUR b4URU5nUR U5n UR U5n OSn Sn URbAURb4URU5nURU5nURU5nOSnSnURbAURb4URU5nURU5nURU5nOSnSnXU/nUV s/sH ocMU PM nn [R"USS9nXU/nUV s/sH ocMU PM nn [R"USS9nU R5SSURUR4-nU R"U6R!SSSS5n UR5SSURUR4-nUR"U6R!SSSS5nUR5SSURUR4-nUR"U6R!SSSS5nXR"-n UcN[R$"URS5URS55RUR&5nUSS2SSS2S4U SS2SSSS24-n[R("U UR+SS 55nUUR-SS S 9R/5- nUR1SU- R35[R4"UR65R85n[:R<"SS9"U5n[R("UU5nUR!SSSS5R?5nUR5SS UR@4-nUR"U6nURCU5nU$s sn fs sn fs sn f) z query_states: text key_value_states: protein query_states: [bs, query_seq_len, dim] key_value_states: [bs, kv_seq_len, dim] query_attn_mask: [bs, query_seq_len] kv_attn_mask: [bs, kv_seq_len] Nz=At least one modality should be provided for cross attention.r(rCrBrrrrMTr)"rrFrGryrrrrsrgrtrurvrwrrrrrrrrdrrrrrrrrXrrSoftmax contiguousrrz)r4 query_statesprotein_key_value_statesstructure_key_value_statesmsa_key_value_statesquery_attn_maskprotein_kv_attn_maskstructure_kv_attn_maskmsa_kv_attn_mask kv_attn_maskrN query_layerkey_layer_proteinvalue_layer_proteinkey_layer_structurevalue_layer_structure key_layer_msavalue_layer_msa key_layer value_layernew_query_layer_shapenew_key_layer_shapenew_value_layer_shaper attn_weightsattention_scoresattention_probs context_layernew_context_layer_shapes r7cross_attention3EvollaSequenceAlignerCrossAttention.cross_attentionsK*-FVW #/A,>,J'?'B'B<'P $ $ 0 01I J "&"4"45M"N  $ "&     )d.B.B.N)C)F)F|)T &"&"4"45O"P $($8$89S$T !"& $( ! << #(B#7#:#:<#H  LL)=>M"nn-ABO M"O&]K ); 1Q ;IIiQ/ *?S "-?+Qq+ ?ii 3 + 0 0 23B 7  $ $  $ $; ! "&&(=>FFq!QPQR 'nn.s3  $ $  $ $7  NN$78@@Aq!L + 0 0 23B 7  $ $  $ $; ! "&&(=>FFq!QPQR !JJ.   "#jj):):1)=|?P?PQR?STWWXdXkXklO(D!T)9:\!TSWYZJZ=[[||K1D1DR1LM #l&7&7B&7&M&T&T&VV '33   % % '\5G5G)H)L)L **,-=> _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD  m4 qBL<@s"P?P?!Q*QQ Q c zUbvURupnUcc[R"X5RU R5U R X4S9R -RUR5nOSnUbyURun nnUce[R"U U5RU R5U R UU 4S9R -RUR5nOSnUbyURun nnUce[R"U U5RU R5U R UU 4S9R -RUR5nOSnUnUbUR5(d0UbUR5(dUbUR5(ayUnURUUUUUUUUS9n[R"UR5U-nUU-nUnURU5[R"UR5-nUU-nU$)N)r)rrrrrrrr) rNrFrrgrdexpandTanyrtanhr|rTr})r4rprotein_kv_statesstructure_kv_states msa_kv_statesrrrrprotein_batch_maskstructure_batch_maskmsa_batch_maskpast_key_valuer!protein_kv_seq_lenrDstructure_kv_seq_lenmsa_kv_seq_lenrresiduals r7rs+EvollaSequenceAlignerCrossAttention.forwardfsP  (*;*A*A 'BC#+JJr699:L:S:ST(//6H5M/NPPQ"&--.% $(  *,?,E,E )B$c%-JJr#78;;2556H6O6OP$++."1E+FHHI"]))*! $ $  */C/G/G/I/I#/4J4N4N4P4P).>.B.B.D.D$H 00*):+>%2 /%9'=!11 M"JJt':':;mKM$}4M$H GGM2UZZ 5NNM$}4Mr9)rrryrrTr|r}rrvrrrtrrzrrrwrsru)NNNNNNNNNN) r:r;r<r=rrvr1rrsr>r?r@s@r7rjrjsm.2/3)- 1:&c]1: (} 1: "# 1:1:fnn"#!GGr9rjc\rSrSrSrg)rxirNrrr9r7rxrxrr9rxc\rSrSrSrg)EvollaRotaryEmbeddingirNrrr9r7rrrr9rc\rSrSrSrg) EvollaMLPirNrrr9r7rrrr9rc\rSrSrSrg)EvollaAttentionirNrrr9r7rrrr9rc^\rSrSrS\S\4U4SjjrSS\RS\ \R\R4S\ \RS\ \RS \ \ S \ \ S \ \RS \ \RS \ \RS\ \RS\ \RS\ \RS\ \RS\ \R4SjjrSrU=r$)EvollaDecoderLayerir5rc>[TU]X5 US-[URUR-S5-S:Xa[ UUR S9Ulgg)Nr(r)rk)r0r1maxnum_hidden_layersaligner_num_add_layersrjradapterr4r5rr6s r7r1EvollaDecoderLayer.__init__sY + MS!9!9V=Z=Z!Z\]^ ^bc c>$*$6$6DL dr9rposition_embeddingsrr2r use_cachecache_positionrrrrrrrc UnURU5nUR"SUUUUUUUS.UD6unnUU-nUnURU5nURU5nUU-n[ US5(aUR UUU U UU U U S9nU$)N)rrr2rrrrr)rrrrrrrrr)input_layernorm self_attnpost_attention_layernormmlprr)r4rrrr2rrrrrrrrrrrcrrNs r7rsEvollaDecoderLayer.forwards$!,,];  >>  ')%)) 3     q!=0 !55mD /  =0 4 # # LL*"3$7+ /#5%9-) Mr9)r) NNNFNNNNNNNN)r:r;r<r=r)rvr1rFrrwrrhr rrsr>r?r@s@r7rrsL|2637*.$)59486:04597;15265||5#5<<#=>5!. 5 u//0 5 ! 5D>5!!1!125$ELL15&ell35  -5%U\\25'u||45!.5"%,,/55r9rc&\rSrSrSr/SQrSrSrg)EvollaPreTrainedModeliF)rr<rjcURRn[R"U5 [ U[ 5(adUR R5 URR5 URRRRS5 g[ U[5(a%URRRSUS9 gg)NrVrr)r5rr%rrrjr|rr}ryrrrr<rrrs r7r#EvollaPreTrainedModel._init_weights skk++**62 fA B B  ! ! ' ' ) OO ! ! #  ! ! ( ( - - 3 3C 8  A B B NN   ' 'Sc ' :Cr9rN)r:r;r<r=_supports_attention_backendrrr>rr9r7rrs"' ;r9rc!^\rSrSrS\4U4SjjrSrSr\\ SS\ RS\ \ RS\ \ RS \ \S \ \ RS \ \S \ \ RS \ \ RS\ \ RS\ \ RS\ \ RS\ \ RS\ \ RS\\\44Sjj55rSrU=r$) EvollaModelir5c 6>[TU]U5 URUlURUl[ R "URURUR5Ul[US9Ul [ R"[UR5Vs/sH n[UUS9PM sn5Ul[!URUR"S9Ul['US9Ul[+USS5UlUR/5 gs snf)Nr^)r5r)epsgradient_checkpointingF)r0r1 pad_token_idr vocab_sizerrr embed_tokensr\protein_encoderrDrFrrrErx rms_norm_epsr1r rotary_embrr post_initrs r7r1EvollaModel.__init__s  !.. ++LL&:L:LdN^N^_36Bmm "'v'?'?!@   "AI #!'"A    "&"4"4&:M:MN /v>&-f6NPU&V#  s#DcUR$r/rrs r7r EvollaModel.get_input_embeddings*s   r9cXlgr/rrs r7r EvollaModel.set_input_embeddings-s!r9rrr2past_key_valuesrrrprotein_input_idsprotein_attention_maskstructure_feats msa_featsrrrqc USLUSL- (a [S5eUcURU5nU(a Uc [5nUcDUbUR5OSn[R "XUR S-URS9nUcURS5nSnSnUbOU bLURUU S9nURn[R"S/UR S-URS9n[URUUUUS9nUnURUU5nURHnU"U4UUUUUUUU U UU U US . UD6nM UR!U5n[#UUS 9nU$) a protein_input_ids (torch.LongTensor): The input IDs for the protein sequence in structure-aware tokens. Should be of shape `(batch_size, protein_seq_length)` and type `torch.LongTensor`. protein_attention_mask (torch.Tensor): The attention mask for the protein sequence. Should be of shape `(batch_size, protein_seq_length)` and type `torch.Tensor`. structure_feats (torch.FloatTensor): The input IDs for purely structure-based features. Should be of shape `(batch_size, structure_seq_length, structure_feat_dim)` and type `torch.FloatTensor`. Dummy input for now. msa_feats (torch.FloatTensor): The input IDs for purely MSA-based features. Should be of shape `(batch_size, msa_seq_length, msa_feat_dim)` and type `torch.FloatTensor`. Dummy input for now. structure_batch_mask (torch.Tensor): The batch mask to decide which protein sequences are purely structure-based. Should be of shape `(batch_size)` and type `torch.Tensor`. Should be paired with `structure_feats`. Dummpy input for now. msa_batch_mask (torch.Tensor): The batch mask to decide which protein sequences are purely MSA-based. Should be of shape `(batch_size)` and type `torch.Tensor`. Should be paired with `msa_feats`. Dummpy input for now. Nz:You must specify exactly one of input_ids or inputs_embedsrr(rcrT)r5 input_embedsrrr) rr2rrrrrrrrrrr)rr)rrr get_seq_lengthrFrZrNrd unsqueezerrYr{r r5rrEr1r )r4rrr2rrrrrrrrrrrcpast_seen_tokens protein_featsrprotein_outputs causal_maskrr decoder_layeroutputs r7rsEvollaModel.forward0sB  -t"; <YZ Z   --i8M 0*nO  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L !  (-C-O"22+53O,FFM!&tf7H7N7Nq7Q.QZkZrZr!s (;;&))+  & #oom\J![[M)*).#-$7"/$3'#5%9- .M)& -0 (++  r9)rrrEr1rrrr) NNNNNNNNNNNNN)r:r;r<r=r)r1rrrrrFrhrrr rZrrrwr rsr>r?r@s@r7rrsw|*!"'+1537+/59$(598<9=7;157;15b##b!.bu//0 b "% b   1 12 bD>b!!1!12b$E$4$45b!) 6b"%"3"34bE--.b'u||4b!.b u-- .!bbr9rc^\rSrSrU4SjrSrSr\\SS\ RS\ \ RS\ \ RS\ \ RS \ RS \ \ RS \ \4S jj55rS rU=r$)EvollaForProteinText2Textic>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr0) r0r1rr_rrrrlm_headrr3s r7r1"EvollaForProteinText2Text.__init__sQ   (  ++yy!3!3T__5Q  r9c6URR5$r/)r_rrs r7r.EvollaForProteinText2Text.get_input_embeddingsszz..00r9c8URRU5$r/)r_rrs r7r.EvollaForProteinText2Text.set_input_embeddingsszz..u55r9rrrlabelsrrrc UR"SUUUUUUS.UD6n U Sn URU 5n Sn UbUR"SXURS.UD6n [ U U U R U R U RS9n U $)a| protein_input_ids (torch.LongTensor): The input IDs for the protein sequence. Should be of shape `(batch_size, protein_seq_length)` and type `torch.LongTensor`. protein_attention_mask (torch.Tensor): The attention mask for the protein sequence. Should be of shape `(batch_size, protein_seq_length)` and type `torch.Tensor`. Example: ```python >>> from transformers import EvollaProcessor, EvollaForProteinText2Text >>> model = EvollaForProteinText2Text.from_pretrained("westlake/Evolla-10B-hf") >>> processor = EvollaProcessor.from_pretrained("westlake/Evolla-10B-hf") >>> protein_information = { "aa_seq": "your amino acid sequence", "foldseek": "your foldseek sequence", } >>> question = "What is the function of this protein?" >>> message = [ {"role": "system", "content": "You are an AI expert that can answer any questions about protein."}, {"role": "user", "content": question}, ] >>> inputs = processor(proteins=[protein_information], messages_list=[message], return_tensors="pt", padding="longest") >>> outputs = model.generate(**inputs) >>> print(processor.batch_decode(outputs, skip_special_tokens=True)) ```)rrrrrrrN)logitsrr)lossrrrrr)r_r loss_functionrrrrr)r4rrrrrrrrcoutputsrrr lm_outputss r7rs!EvollaForProteinText2Text.forwardsT** )'/#9     m,  %%iVtibhiD+#33!//))  r9)rr_rr)r:r;r<r=r1rrrrrFrhrrrZrrsr>r?r@s@r7rrs16'+1559-1.29=$(?##?!.?  1 12 ? ))* ? !++ ?!) 6?D>??r9r)rrr)Ur dataclassesrtypingrrrFtorch.utils.checkpointrr cache_utilsr r generationr masking_utilsr modeling_outputsr rrrmodeling_utilsrrrutilsrrr utils.genericresm.modeling_esmrrrrrrrr r!llama.modeling_llamar"r#r$r%r&r'configuration_evollar)r* get_loggerr:loggerr,rKrQrrSryrrrrrrrrrrr.r<rXr\rjrxrrrrrrr__all__rr9r7rs !" .)/ UT 0   =   H %!]!( 2( "))( V# 0"))#D ]  L        *   */***_'!<_'D7  7 t A A'. '.T ?k? ? 299 $k"))k\ L  0    n >*>B;0;&@'@FP 5Pf Pr9