Sr?"SS\ Rx5r@"SS\ Rx5rA"SS \ Rx5rB"S!S"\A5rC\A\CS#.rD"S$S%\ Rx5rES&rF"S'S(\ Rx5rG"S)S*\ Rx5rH"S+S,\5rI"S-S.\ Rx5rJ"S/S0\ Rx5rK\/"S1S2\'55rL"S3S4\L5rM"S5S6\ Rx5rN"S7S8\ Rx5rO"S9S:\ Rx5rP\\/"S;S<\ 555rQ"S=S>\ Rx5rR"S?S@\ Rx5rS\"SA5"SBSC\ Rx55rT"SDSE\ Rx5rU"SFSG\ Rx5rVSHrWSbSIjrXSJ\RSK\YSL\R4SMjrZScSN\ RxSO\RSP\RSQ\RSR\\RSS\[ST\[SU\,\.4SVjjr\"SWSX\ Rx5r]"SYSZ\5r^\/"S[S\\'55r_"S]S^\_5r`"S_S`\_\5ra/SaQrbg)dN) dataclass)CallableOptionalUnion)Tensornn)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)!flash_attn_supports_top_left_maskis_flash_attn_available)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentionsBaseModelOutputWithPast,BaseModelOutputWithPoolingAndCrossAttentionsCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSModuleUtilsMixinPreTrainedModel find_pruneable_heads_and_indicesget_parameter_dtypeprune_linear_layer)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)check_model_inputs) EvollaConfig SaProtConfig)_flash_attention_forwardcURU5R5n[R"USS9R U5U-nUR 5U-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor r&dim)neinttorchcumsumtype_aslong) input_ids padding_idxmaskincremental_indicess b/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/evolla/modeling_evolla.py"create_position_ids_from_input_idsr8BsP << $ ( ( *D,,t3;;DADH  # # % 33cD^\rSrSrSrU4SjrSSjrSrSrU=r $)EvollaSaProtEmbeddingsRzN Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. c>[TU]5 [R"URUR UR S9UlUR(a/[R"UR URS9Ul OSUl [R"UR5Ul[USS5UlUR#S[$R&"UR(5R+S5SS9 UR UlUR S:Xa9[R"UR(UR UR,S9UlUR0UlUR2UlSUlg) N)r4epsposition_embedding_typeabsolute position_ids)r&F persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsemb_layer_norm_before LayerNormlayer_norm_eps layer_normDropouthidden_dropout_probdropoutgetattrr@register_bufferr/arangemax_position_embeddingsexpandr4position_embeddings token_dropout mask_token_idrBselfconfig __class__s r7rGEvollaSaProtEmbeddings.__init__Ws2 !||F,=,=v?Q?Q_e_r_rs  ' ' ll6+=+=6CXCXYDO"DOzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej  "..  ' ': 5')||..0B0BPTP`P`(D $$11#11 r9cUc*Ub[XR5nOURU5nUcURU5nUnUR(aUR XR :HRS5S5nSnURS5nXR :HRS5R5U- nUSU- -SU- SS2SS4- RUR5nURS:XaURU5n XY-nURbURU5nUb,XRRS5-RUR5nU$)NrCgQ?r&rA)r8r4&create_position_ids_from_inputs_embedsrLrZ masked_fillr[ unsqueezesumfloattodtyper@rYrP) r]r3attention_maskrB inputs_embeds embeddingsmask_ratio_train src_lengthsmask_ratio_observedrYs r7forwardEvollaSaProtEmbeddings.forwardpsp  $A)M]M]^ #JJ=Y   00;M#    #//>P>P1P0[0[\^0_adeJ) (,,R0K#,0B0B#B"G"G"K"Q"Q"SVa"a $,<(<=EXAXZ[]acgZg@hhll  J  ' ': 5"&":":<"H #9J ?? &4J  %$'?'?'CCGG HXHXYJr9cUR5SSnUSn[R"URS-X0R-S-[RUR S9nUR S5RU5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrCr&ridevicer)sizer/rVr4r2rtrerX)r]rk input_shapesequence_lengthrBs r7rc=EvollaSaProtEmbeddings.create_position_ids_from_inputs_embedss~$((*3B/ %a.||   q /4D4D"Dq"HPUPZPZcpcwcw %%a(// <[TU]5 SS[R"SUS[RS9R 5U- -- nUnUR SU5 SUlSUlSUl g)N?i'rrriinv_freq) rFrGr/rVint64rgrU_seq_len_cached _cos_cached _sin_cached)r]r,rr_s r7rG$EvollaSaProtRotaryEmbedding.__init__sl %ELLC%++$N$T$T$VY\$\]^ Z2#r9cjURUnX0R:wd$URRUR:waX0l[R "URUURS9R UR5n[R"X@R5n[R"XU4SS9RUR5nUR5SSSS2SS24UlUR5SSSS2SS24Ul URUR4$)NrtrCr+)rrrrtr/rVr1routerrrhrrr)r]r seq_dimensionseq_lentfreqsembs r7_update_cos_sin_tables2EvollaSaProtRotaryEmbedding._update_cos_sin_tabless''-( ** *d.>.>.E.E.Q#*  QWW]3AHHEMMdmm\AKK==1E))UN366qxx@C"wwytQ)9:D "wwytQ)9:D !1!111r9qkreturncURUSS9uUlUl[XRUR5[X RUR54$)Nr)r)rrrr)r]rrs r7rp#EvollaSaProtRotaryEmbedding.forwards[-1-H-HZ\-H-]*$* %Q(8(8$:J:J K $Q(8(8$:J:J K  r9)rrr)r)ryrzr{r|r}r.rGrr/rtuplerpr~rrs@r7rrsR  C  2   %,, 5u||A[;\  r9rc^\rSrSrS U4SjjrS S\R S\\RS\\RS\\RS\\RS\\ S \ \R 4S jjr S r U=r $)EvollaSaProtSelfAttentionc~>[TU]5 XlURUR-S:wa7[ US5(d&[ SURSURS35eURUl[URUR- 5UlURUR-Ul [R"URUR5Ul [R"URUR5Ul [R"URUR5Ul[R"UR 5UlU=(d [%USS5UlSUlUR&S:XdUR&S :XaHUR*Ul[R,"S UR*-S - UR5UlO(UR&S :Xa[1URS 9UlUR2UlX0lg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r@rA relative_keyrelative_key_queryrr&rotaryr+)rFrGr^rJnum_attention_headshasattr ValueErrorr.attention_head_size all_head_sizerLinearquerykeyvaluerQattention_probs_dropout_probrSrTr@rotary_embeddingsrWrHdistance_embeddingr is_decoder layer_idxr]r^r@rr_s r7rG"EvollaSaProtSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $"&  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #  ) )X 5%@TE]E]%^D " ++"r9 hidden_statesrj head_maskencoder_hidden_statesencoder_attention_maskoutput_attentionsrcURSSURUR4nURU5R U5R SS5nUSLn U (acUR U5R U5R SS5n URU5R U5R SS5n UnO`UR U5R U5R SS5n URU5R U5R SS5n XRS--nURS:XaURX5up[R"XR SS55n URS:XdURS :XGaCUR5Sn [R"U [RURS 9R SS5n[R"U [RURS 9R SS5nX- nUR!UUR"-S- 5nUR%UR&S 9nURS:Xa[R("S UU5nU U-n OHURS :Xa8[R("S UU5n[R("S U U5nU U-U-n UbX-n [*R,R/U SS9nUR1U5nUbUU-n[R"UR%U R&5U 5nUR3SSSS5R55nUR5SSUR64-nUR U5nU(aUU4OU4nUR8(aUS-nU$)NrrCr&rrrrrrsrzbhld,lrd->bhlrzbhrd,lrd->bhlrr+r N)rrrrview transposerrr@rr/matmulrurVr2rtrrWrhrieinsumr functionalsoftmaxrSpermute contiguousrr)r]rrjrrrr hidden_shape query_layeris_cross_attention key_layer value_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r7rp!EvollaSaProtSelfAttention.forward s&++A.D4L4LdNfNfg jj/44\BLLQPQR 3$> !67<<\JTTUVXYZI**%:;@@NXXYZ\]^K3N/44\BLLQPQRI**]388FPPQRTUVK "$<$ 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ??'Gr9)rrr^rrSrrrrWrr@rrrNNNNNNF)ryrzr{r|rGr/rr FloatTensorboolrrpr~rrs@r7rrs#F7;15=A>B,1O||O!!2!23OE--. O ((9(9: O !)):): ; O$D>O u|| OOr9rc.^\rSrSrU4SjrSrSrU=r$)EvollaSaProtSelfOutputi]c>[TU]5 [R"URUR5Ul[R "UR5Ulgr) rFrGrrrJdenserQrRrSr\s r7rGEvollaSaProtSelfOutput.__init__^sB YYv1163E3EF zz&"<"<= r9cRURU5nURU5nX-nU$rrrSr]r input_tensors r7rpEvollaSaProtSelfOutput.forwardc, =1  ]3 %4 r9rryrzr{r|rGrpr~rrs@r7rr]> r9rc^\rSrSrSrS U4SjjrSS\RS\\RS\\RS\\RS\\RS \\ S \ \R4U4S jjjr S r U=r$)EvollaSaProtFlashAttention2ijaJ EVOLLA_SA_PROT flash attention module. This module inherits from `EvollaSaProtSelfAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cb>[TU]XUS9 [5UlURUlg)N)r@r)rFrGr_flash_attn_uses_top_left_maskr dropout_probrs r7rG$EvollaSaProtFlashAttention2.__init__qs2 \ef /P.Q+"??r9rrjrrrrrc >U(dUcUb)[RS5 [TU] UUUUUU5$UR 5upxn UR UR U55n UR URU55n UR URU55n U Rn U RRS:waU RROSnU [R:Xa[R"5(aA[[S5(a[R "U5O[R""5nOR[UR$S5(aUR$R&nO UR R(Rn[RSUS35 U R+U5n U R+U5n U R+U5n XR,S--n UR.S :XaUR1X5upO9UR.S :XdUR.S :Xa[3S UR.S 35e[5U R7SSSS5U R7SSSS5U R7SSSS5UUUR8SUR:(a UR<OSUR>S9 nURAXxS5nUS4nUR8(aUS-nU$)NzEvollaSaProtFlashAttention2 does not support output_attentions, head_mask, or cross_attention. Falling back to the manual attention implementation. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.mpscpuget_autocast_dtype_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rrrrz%ESM flash attention does not support z embeddingsrrr&r rrb) query_length is_causal softmax_scalerSuse_top_left_maskrCr)!logger warning_oncerFrprutranspose_for_scoresrrrrirttyper/float32is_autocast_enabledrrget_autocast_gpu_dtyper^rweightrhrr@rrr)rrtrainingrrreshape)r]rrjrrrrbszq_len_rrr input_dtype device_type target_dtype attn_outputrr_s r7rp#EvollaSaProtFlashAttention2.forwardzs  59N9Z   U  7?%&!  &**, A// =0IJ --dhh}.EF // =0IJ "'' 1<1C1C1H1HE1Qk((--W\ %-- '((**u&:;;,,[9557 &?@@#{{BB #zz0066    >$  &..6K! \2I%..6K "$<$B,1]||]!!2!23]E--. ] ((9(9: ] !)):): ; ]$D>] u|| ]]r9r)eagerflash_attention_2cH^\rSrSrSU4SjjrSrSSjrSrU=r$)EvollaSaProtAttentionic>[TU]5 [UR"XS9Ul[ U5Ul[5Ul[R"URURS9Ul g)N)rr>) rFrG EVOLLA_SA_PROT_ATTENTION_CLASSES_attn_implementationr]routputset pruned_headsrrNrJrOr]r^rr_s r7rGEvollaSaProtAttention.__init__sZ 4V5P5PQRXn ,V4 Ef&8&8f>S>STr9c6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrr&r+)lenrr]rrr$rrrrr"rrunion)r]headsindexs r7 prune_heads!EvollaSaProtAttention.prune_headss  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r9c URU5nURUUUUUUS9n URU SU5n U 4U SS-n U $)Nrjrrrrrr&)rNr]r") r]rrjrrrrcache_positionhidden_states_ln self_outputsattention_outputrs r7rpEvollaSaProtAttention.forwardsh >>-8yy )"7#9/ !  ;;|A F#% QR(88r9)rNr"r$r]rNNNNFN) ryrzr{r|rGr,rpr~rrs@r7rrs,U;*"#r9rcnUS-S[R"U[R"S5- 5--$)zr This is the gelu implementation from the original EVOLLA_SA_PROT repo. Using F.gelu yields subtly wrong results. g?rg@)r/erfmathsqrt)rs r7gelur:s. s7cEIIa$))C.&899 ::r9cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)EvollaSaProtIntermediateic>[TU]5 [R"URUR 5Ulgr)rFrGrrrJintermediate_sizerr\s r7rG!EvollaSaProtIntermediate.__init__s, YYv1163K3KL r9rrc>URU5n[U5nU$r)rr:)r]rs r7rp EvollaSaProtIntermediate.forwards  =1 ]+ r9)r ryrzr{r|rGr/rrpr~rrs@r7r<r<s)MU\\ellr9r<c.^\rSrSrU4SjrSrSrU=r$)EvollaSaProtOutputi$c>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) rFrGrrr>rJrrQrRrSr\s r7rGEvollaSaProtOutput.__init__%sB YYv779K9KL zz&"<"<= r9cRURU5nURU5nX-nU$rrrs r7rpEvollaSaProtOutput.forward*rr9rrrs@r7rDrD$rr9rDcD^\rSrSrU4SjrSSjrSrSrU=r$)EvollaSaProtLayeri1c>[TU]5 URUlSUl[ U5UlUR UlURUlUR(a/UR (d[US35e[ U5Ul [U5Ul [U5Ul [R"UR UR"S9Ulg)Nr&z> should be used as a decoder model if cross attention is addedr>)rFrGchunk_size_feed_forward seq_len_dimr attentionradd_cross_attention RuntimeErrorcrossattentionr< intermediaterDr"rrNrJrOr\s r7rGEvollaSaProtLayer.__init__2s '-'E'E$.v6 ++#)#=#=  # #??"dV+i#jkk"7"?D 4V<(0 f&8&8f>S>STr9c nURUUUUS9nUSn UR(aUSSn OUSSn UR(aCUb@[US5(d[SUS35eUR U UUUUUS9n U Sn XSS-n UR U 5n U 4U -n UR(aU S -n U $) N)rjrrrr&rCrQz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r/r)rNrrAttributeErrorrQfeed_forward_chunk) r]rrjrrrrr0self_attention_outputsr3rcross_attention_outputs layer_outputs r7rpEvollaSaProtLayer.forwardAs "& )/ "0"  2!4 ??,Qr2G,QR0G ??4@4!122$=dVD`` '+&9&9 -#&;'="3 ':' # 7q9 " ==G../?@ /G+ ??'Gr9clURU5nURU5nURX15nU$r)rNrRr")r]r3attention_output_lnintermediate_outputrYs r7rV$EvollaSaProtLayer.feed_forward_chunkts9"nn-=>"//0CD{{#6I r9) rNrOrNrLrQrRrr"rMr5) ryrzr{r|rGrprVr~rrs@r7rJrJ1s- U$"#1fr9rJcL^\rSrSrU4Sjr\SSj5rSrU=r$)EvollaSaProtEncoderi{c0>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5Ul[R"URURS9Ul SUl gs snf)Nr>F)rFrGr^r ModuleListrangenum_hidden_layersrJlayerrNrJrOemb_layer_norm_aftergradient_checkpointing)r]r^rr_s r7rGEvollaSaProtEncoder.__init__|sr  ]]uVMeMeGf#gGf!$5f$=Gf#gh $&LL1C1CI^I^$_!&+#$hsBc U(aSOSn U(aSOSn U(aURR(aSOSn [UR5HbupU(aX4-n UbX=OSnU"UUUUUUS9nUSnU(dM3U US4-n URR(dMYU US4-n Md UR(aUR U5nU(aX4-n [ UU U U S9$)N)rrjrrrrrr&rlast_hidden_stater attentionscross_attentions)r^rO enumeratererfr)r]rrjrrrroutput_hidden_states return_dictr0all_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss r7rpEvollaSaProtEncoder.forwards #7BD$5b4%64;;;Z;Zr`d(4OA#$58H$H!.7.CilO(+-)&;'="3 M*!,M  &9]1=M[TU]5 [R"URUR5Ul[R "5Ulgr)rFrGrrrJrTanh activationr\s r7rGEvollaSaProtPooler.__init__s9 YYv1163E3EF '')r9rrc\USS2S4nURU5nURU5nU$)Nr)rr~)r]rfirst_token_tensor pooled_outputs r7rpEvollaSaProtPooler.forwards6+1a40 #56  6 r9)r~rrBrs@r7r{r{s($ U\\ellr9r{c0\rSrSr%\\S'S/rSrSrSr g)EvollaSaProtPreTrainedModelir^rJTcURRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[R5(adUR R RSUS9 URb2UR R URR5 gg[U[R5(aJURR R5 UR R RS5 gg)zInitialize the weightsrbmeanstdNr)r^initializer_range isinstancerrrdatanormal_biaszero_rHr4rNfill_)r]modulers r7 _init_weights)EvollaSaProtPreTrainedModel._init_weightsskk++ fbii ( ( MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r9rjN) ryrzr{r|r(__annotations___no_split_modules_supports_flash_attnrr~rjr9r7rrs ,- *r9rc ^\rSrSrS\4U4SjjrSrSrSr\ SS\ \ RS\ \ RS \ \\ R\44S jj5rSS\ S \\S \ R$S \ R&S \ 4 SjjrSrU=r$)EvollaSaProtProteinEncoderir^cd>[TU]U5 [U5Ul[ U5Ulgr)rFrGr;rlr`encoderr\s r7rG#EvollaSaProtProteinEncoder.__init__s(  08*62 r9c.URR$rrlrLr]s r7get_input_embeddings/EvollaSaProtProteinEncoder.get_input_embeddingss...r9c$XRlgrrr]rs r7set_input_embeddings/EvollaSaProtProteinEncoder.set_input_embeddingss */'r9cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrerNr,)r]heads_to_prunerer*s r7 _prune_heads'EvollaSaProtProteinEncoder._prune_headss< +002LE LL  u % / / ; ;E B3r9r3rjrc0UR5nUupEURnUc[R"XE4US9nUR XS9nUR X#5nUR XxS9n U Sn [U U RU RU RS9$)Nrr3rj)rjrrk) rurtr/onesrlget_extended_attention_maskrrrrmrn) r]r3rjrv batch_sizerrtrkextended_attention_maskencoder_outputssequence_outputs r7rp"EvollaSaProtProteinEncoder.forwards  nn& !, !!  !"ZZ*)A6RN)[ "&"B"B>"_,,},])!,;-)77&11,==   r9rvrtricPUc [U5nUR5S:XaURR(dUb[R "S[ 5 UR5S:XaUSS2SSS2SS24nOqUR5S:XaAURR(a[R"X!U5nO*USS2SSSS24nO[SUSURS35eURUS9nS U- [R"U5R-nU$) a Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. NrzNThe `device` argument is deprecated and will be removed in v5 of Transformers.r z!Wrong shape for input_ids (shape z) or attention_mask (shape rrr)rr,r^rwarningswarn FutureWarningr*create_extended_attention_mask_for_decoderrrrhr/finfomin)r]rjrvrtrirs r7r6EvollaSaProtProteinEncoder.get_extended_attention_mask s  ='-E""$)dkk.D.D! dfs    1 $&4Qa]&C #    !Q &{{%%*:*e*e+'+9D$9I*J'3K=@[\j\p\p[qqrs #:"<"<5"<"I#&)@#@EKKPUDVDZDZ"Z&&r9)rlrrr)ryrzr{r|r(rGrrrr#rr/rrrrrpr.rtrgrr~rrs@r7rrs3|3 /0C26 ELL) !.  uU\\"$PP Q   2rv2'$2'38:2'GL||2'chcncn2' 2'2'r9rc2^\rSrSrSU4SjjrSrSrU=r$)!EvollaSequenceCompressorAttentioni>cX>[TU]5 US-UlX0lX#-n[R "U5Ul[R "U5Ul[R"XSS9Ul [R"XS-SS9Ul [R"XASS9Ul g)NrFrr) rFrGscaler*rrN norm_media norm_latentsrto_qto_kvto_out)r]r,dim_headr* inner_dimr_s r7rG*EvollaSequenceCompressorAttention.__init__?s t^  $ ,,s+LL-IIc59 YYsM> ii U; r9c&URU5nURU5nURnURU5n[R "X4SS9nUR U5RSSS9upxURURS5URS5US5RSSSS5nURURS5URS5US5RSSSS5nURURS5URS5US5RSSSS5nXPR-n[R"XWRSS55n XRSSS 9R5- n U R upp[R""X5R%UR&5nUS S 2S S S S 24nUS S S 2S S 2S 4nUU-nU R)SU- R+5S 5n U R-SS9n[R"UU5nURSSSS5nUR/URS5URS5S5nUR1U5$) z Args: x (torch.Tensor): image features shape (b, n1, D) latent (torch.Tensor): latent features shape (b, n2, D); n2: num of latent tokens rr+rrCrr&r Tr,keepdimNg)rrr*rr/rrrrrurrrramaxdetachrrrhrtrdrrrr)r]rlatentsr5hrkv_inputrvsimbsnhskdokdrmask_expones_expattnouts r7rp)EvollaSequenceCompressorAttention.forwardLs2 OOA ##G, JJ IIg 99a\r2zz(#)) 2*  FF166!9affQiB / 7 71a C FF166!9affQiB / 7 71a C FF166!9affQiB / 7 71a C Nll1kk"b12HHTH299;;99zz""%%dkk24q()aD()("ooq4xoo/6{{r{"ll4#kk!Q1%kk#((1+sxx{B7{{3r9)r*rrrrrr)@rrs@r7rr>s <) ) r9rc2^\rSrSrSU4SjjrSrSrU=r$)EvollaFeedForwardixc>[TU]5 [X-5n[R"U5Ul[R "XSS9Ul[R"5Ul [R "X1SS9Ul gNFr) rFrGr.rrNnormrfc1GELUr~fc2)r]r,multrr_s r7rGEvollaFeedForward.__init__ysZ  O LL% 99S%8'')99Y%8r9c ~URURURURU5555$r)rr~rr)r]rs r7rpEvollaFeedForward.forwards+xx1(>?@@r9)r~rrr)rrs@r7rrxs9AAr9rc6^\rSrSrS\4U4SjjrSrSrU=r$)!EvollaSequenceCompressorResamplerir^c >[TU]5 URRnURUl[ R"[R"UR U5SS9Ul [ R"/5Ul [UR5HanURR[ R"[!X!R"UR$S9['X!R(S9/55 Mc [ R*"UR5Ul[ R."X!R5Ulg)NT) requires_grad)r,rr*)r,r)rFrGprotein_encoder_configrJresampler_num_latents num_latentsr Parameterr/randnrrblayersrcresampler_depthappendrresampler_dim_headresampler_headsrresampler_ff_multrNrrprotein_projector)r]r^protein_repr_dimrr_s r7rG*EvollaSequenceCompressorResampler.__init__s !88DD!77||EKK0@0@BR$Scgh mmB' v--.A KK   9 0;T;T\b\r\r*.>E]E]^  /LL!3!34 !#+;=O=O!Pr9cdURSnURupE[R"X@R5R UR 5n[R "X&4SS9n[R"U5R URR 5nURSURSSS5-nUR UR5nURHupU "XU5U-nU "U5U-nM URU5n URU 5$)Nrr&r+rC) rr/rrrhrtrrrrirrr) r]embedsr5brr latent_maskrrrfftransformed_features r7rp)EvollaSequenceCompressorResampler.forwards LLO jj%5%5699$++F yy$,!4zz!} 3 34,,t$tyyQ'::**V\\* HD6D1G;GkG+G$#44W=yy,--r9)rrrrr) ryrzr{r|r'rGrpr~rrs@r7rrsQ|Q*..r9rc\rSrSr%Sr\R \S'Sr\ \R \S'Sr \ \ \R S4\S'Sr \ \ \R S4\S'Sr g) EvollaProteinEncoderModelOutputiNsequence_compressor_outputrl.rrmrj)ryrzr{r|rr/rrrlrrrrmr~rjr9r7rrsi59 1 1859x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r9rct^\rSrSrS\4U4Sjjr\S\RS\R4Sj5r Sr U=r $)EvollaProteinEncoderir^cn>[TU]5 [URS9Ul[ US9Ulg)Nr^)rFrGrrmodelrsequence_compressor_resamplerr\s r7rGEvollaProteinEncoder.__init__s. /v7T7TU -NV\-]*r9r3rjc URXS9nURnURXR5n[UURS9$)Nr)rrl)r rlr r)r]r3rjkwargsprotein_outputprotein_embeds sequence_reprs r7rpEvollaProteinEncoder.forwardsFiW'99::>Z .'4,>>  r9)r r ) ryrzr{r|r'rGr#r/ LongTensorrrpr~rrs@r7rrs?^|^  !1!1 5CTCT  r9rcr^\rSrSrS S\\S\\S\\4U4SjjjrSrS SjrSr U=r $) #EvollaSequenceAlignerCrossAttentioniprotein_encoder_dimstructure_encoder_dimmsa_encoder_dimc>[TU]5 URUlURUlURS-Ul[ URUR- 5UlURUR -UlURnURnURn[R"URUR5Ul UbK[R"X R5Ul[R"X R5UlOSUlSUlUbK[R"X0R5Ul[R"X0R5UlOSUlSUlUbK[R"X@R5Ul[R"X@R5UlOSUlSUl[)UR5Ul[R,"U5Ul[R"URURUS9Ul[3URU5Ul[R6"[8R:"S/55Ul[R6"[8R:"S/55Ulg)Nrrrb) rFrGrJrrr.rr$aligner_attention_probs_dropout_probaligner_enable_biasaligner_ffn_multrrr key_protein value_protein key_structurevalue_structurekey_msa value_msa EvollaRMSNormattention_normrQrSout_projrrrr/tensorgate_attentiongate_ffw) r]r^rrrr enable_biasffn_multr_s r7rG,EvollaSequenceAlignerCrossAttention.__init__s !--#)#=#= --t3 #&t'7'7$:R:R'R#S !558P8PP'-'R'R$00 **YYt//1C1CD  *!yy)<>P>PQD !#+>@R@R!SD #D !%D  ,!#+@BTBT!UD #%99-BDVDV#WD !%D #'D  &99_6H6HIDLYY8J8JKDNDL!DN+D,<,<=zz">?  $"2"2D4D4D;W #D$4$4h? ll5<<+>? U\\3%%89 r9c XgU/n U V s/sH ocMU PM n n U (d [S5e[R"U SS9n URU5n UR U 5n UR bAUR b4URU5nUR U5n UR U5n OSn Sn URbAURb4URU5nURU5nURU5nOSnSnURbAURb4URU5nURU5nURU5nOSnSnXU/nUV s/sH ocMU PM nn [R"USS9nXU/nUV s/sH ocMU PM nn [R"USS9nU R5SSURUR4-nU R"U6R!SSSS5n UR5SSURUR4-nUR"U6R!SSSS5nUR5SSURUR4-nUR"U6R!SSSS5nXR"-n UcN[R$"URS5URS55RUR&5nUSS2SSS2S4U SS2SSSS24-n[R("U UR+SS 55nUUR-SS S 9R/5- nUR1SU- R35[R4"UR65R85n[:R<"SS9"U5n[R("UU5nUR!SSSS5R?5nUR5SS UR@4-nUR"U6nURCU5nU$s sn fs sn fs sn f) z query_states: text key_value_states: protein query_states: [bs, query_seq_len, dim] key_value_states: [bs, kv_seq_len, dim] query_attn_mask: [bs, query_seq_len] kv_attn_mask: [bs, kv_seq_len] Nz=At least one modality should be provided for cross attention.r&r+rCrrr rTr)"rr/rr$rrrrhrr r!r"rurrrrrrrtrrrrrdrrrirrSoftmaxrrr%)r] query_statesprotein_key_value_statesstructure_key_value_statesmsa_key_value_statesquery_attn_maskprotein_kv_attn_maskstructure_kv_attn_maskmsa_kv_attn_mask kv_attn_maskrrkey_layer_proteinvalue_layer_proteinkey_layer_structurevalue_layer_structure key_layer_msavalue_layer_msarrnew_query_layer_shapenew_key_layer_shapenew_value_layer_shaperj attn_weightsrrrrs r7cross_attention3EvollaSequenceAlignerCrossAttention.cross_attentionsK*-FVW #/A,>,J'?'B'B<'P $ $ 0 01I J "&"4"45M"N  $ "&     )d.B.B.N)C)F)F|)T &"&"4"45O"P $($8$89S$T !"& $( ! << #(B#7#:#:<#H  LL)=>M"nn-ABO M"O&]K ); 1Q ;IIiQ/ *?S "-?+Qq+ ?ii 3 + 0 0 23B 7  $ $  $ $; ! "&&(=>FFq!QPQR 'nn.s3  $ $  $ $7  NN$78@@Aq!L + 0 0 23B 7  $ $  $ $; ! "&&(=>FFq!QPQR !JJ.   "#jj):):1)=|?P?PQR?STWWXdXkXklO(D!T)9:\!TSWYZJZ=[[||K1D1DR1LM #l&7&7B&7&M&T&T&VV '33   % % '\5G5G)H)L)L **,-=> _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD  m4 qBL<@s"P?P?!Q*QQ Q c zUbvURupnUcc[R"X5RU R5U R X4S9R -RUR5nOSnUbyURun nnUce[R"U U5RU R5U R UU 4S9R -RUR5nOSnUbyURun nnUce[R"U U5RU R5U R UU 4S9R -RUR5nOSnUnUbUR5(d0UbUR5(dUbUR5(ayUnURUUUUUUUUS9n[R"UR5U-nUU-nUnURU5[R"UR5-nUU-nU$)N)ru)r.r/r0r1r2r3r4r5) rr/rrhrtrXTanyrAtanhr'rr()r]r.protein_kv_statesstructure_kv_states msa_kv_statesr2r3r4r5protein_batch_maskstructure_batch_maskmsa_batch_maskpast_key_valuerprotein_kv_seq_lenr,structure_kv_seq_lenmsa_kv_seq_lenrresiduals r7rp+EvollaSequenceAlignerCrossAttention.forwardosP  (*;*A*A 'BC#+JJr699:L:S:ST(//6H5M/NPPQ"&--.% $(  *,?,E,E )B$c%-JJr#78;;2556H6O6OP$++."1E+FHHI"]))*! $ $  */C/G/G/I/I#/4J4N4N4P4P).>.B.B.D.D$H 00*):+>%2 /%9'=!11 M"JJt':':;mKM$}4M$H GGM2UZZ 5NNM$}4Mr9)rrr$rSrr'r(rJr!rrrr%rrr"rr )NNNNNNNNNN) ryrzr{r|rr.rGrArpr~rrs@r7rrsm.2/3)- 1:&c]1: (} 1: "# 1:1:fnn"#!GGr9rRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$)r#ic>[TU]5 [R"[R "U55UlX lg)z, EvollaRMSNorm is equivalent to T5LayerNorm N)rFrGrrr/rrvariance_epsilon)r]rJr?r_s r7rGEvollaRMSNorm.__init__s/ ll5::k#:; #r9cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NrrCT)r) rirhr/r powrrsqrtrWr)r]rrvariances r7rpEvollaRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r9c^[URR5SUR3$)Nz, eps=)rrrrWrs r7 extra_reprEvollaRMSNorm.extra_reprs* ))*+6$2G2G1HIIr9)rWr)gư>) ryrzr{r|rGrpr_r~rrs@r7r#r#s$;JJr9r#cl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)EvollaRotaryEmbeddingir^c>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typer defaultrFrD)rFrGrrrddictgetrerWmax_seq_len_cachedoriginal_max_seq_lenr^r rope_init_fnattention_scalingrUroriginal_inv_freq)r]r^rtrr_s r7rGEvollaRotaryEmbedding.__init__s  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rlr^rirmrjrkrer) ryrzr{r|r'rGr/no_gradrrpr~rrs@r7rbrbs6/|//" ]]_ < [TU]5 XlURUlURUl[ R "URURURS9Ul[ R "URURURS9Ul [ R "URURURS9Ul [URUl g)Nr)rFrGr^rJr>rrmlp_bias gate_projup_proj down_projr hidden_actact_fnr\s r7rGEvollaMLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r9cURURURU55URU5-5nU$r)r}rr{r|)r]rr}s r7rpEvollaMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r9)rr^r}r{rJr>r|rrs@r7rxrxs0r9rxcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..NrCrr+)rr/rrs r7 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r9cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )rer)rrrrrB unsqueeze_dimq_embedk_embeds r7apply_rotary_pos_embrsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r9rn_reprcURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r&N)rrXr)rrbatchnum_key_value_headsslenhead_dims r7 repeat_kvr"s_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr9rrrrrjscalingrSrc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nrr rrC)r,ri)prr&)rnum_key_value_groupsr/rrrrrrr rhrirSrr) rrrrrjrrSr key_states value_statesr@ causal_maskrs r7eager_attention_forwardr.s3 ; ;[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-URS9Ul[R"UR URUR-URS9Ul[R"UR URUR-URS9Ul[R"UR UR-UR URS9Ulg)NrrTr)rFrGr^rrTrJrrrrrattention_dropoutrrrattention_biasq_projk_projv_projo_projr%s r7rGEvollaAttention.__init__KsI  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r9rrYrjrMr0rrc 4URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO URUR S.UD6unnUR""/UQSP76R%5nUR'U5nUU4$)NrCr&r)rrr0rrb)rSr)rrrrrrrrupdaterrr^r!rrrrrrr)r]rrYrjrMr0rrvrr.rrrr cache_kwargsattention_interfacerr@s r7rpEvollaAttention.forwardbs$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<'))!. )) ! )) !!1!12 ))+,)) u||U\\) *))))r9rc ^\rSrSrS\S\4U4SjjrSS\RS\ \R\R4S\ \RS\ \RS \ \ S \ \ S \ \RS \ \RS \ \RS\ \RS\ \RS\ \RS\ \RS\ \RS\ \R4SjjrSrU=r$)EvollaDecoderLayerir^rc>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul US-[URUR-S5-S:Xa[UURS9Ulgg)Nr^rr>r&r)r)rFrGrJr self_attnrxmlpr# rms_norm_epsinput_layernormpost_attention_layernormmaxrdaligner_num_add_layersradapterr%s r7rGEvollaDecoderLayer.__init__s !--(LV$,V-?-?VEXEXY(5f6H6HfNaNa(b% MS!9!9V=Z=Z!Z\]^ ^bc c>$*$6$6DL dr9rrYrjrBrM use_cacher0rGrHrIrJrKrLr2rc UnURU5nUR"SUUUUUUUS.UD6unnUU-nUnURU5nURU5nUU-n[ US5(aUR UUU U UU U U S9nU$)N)rrjrBrMrr0rYr)r.rGrHrIr2rJrKrLrj)rrrrrr)r]rrYrjrBrMrr0rGrHrIrJrKrLr2rrQrs r7rpEvollaDecoderLayer.forwards$!,,];  >>  ')%)) 3     q!=0 !55mD /  =0 4 # # LL*"3$7+ /#5%9-) Mr9)rrJrrrr) NNNFNNNNNNNN)ryrzr{r|r'r.rGr/rrrrr rrpr~rrs@r7rrs_ |  &2637*.$)59486:04597;15265||5#5<<#=>5!. 5 u//0 5 ! 5D>5!!1!125$ELL15&ell35  -5%U\\25'u||45!.5"%,,/5" u|| #55r9rch^\rSrSr%\\S'SrSr/SQrS/r Sr Sr Sr Sr Sr\\S.rU4S jrS rU=r$) EvollaPreTrainedModelir^r T)rrrpast_key_valuesF)rrmc>URRn[TU] U5 [ U[ 5(adUR R5 URR5 URRRRS5 g[ U[5(a%URRRSUS9 gg)Nrrbr)r^rrFrrrr'rr(r$rrrrrr)r]rrr_s r7r#EvollaPreTrainedModel._init_weightsskk++ f% fA B B  ! ! ' ' ) OO ! ! #  ! ! ( ( - - 3 3C 8  A B B NN   ' 'Sc ' :Cr9rj)ryrzr{r|r'rbase_model_prefixsupports_gradient_checkpointingr_skip_keys_device_placementr_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsrr~rrs@r7rrs] &*# $5"5N!"'+% ;;r9rc!^\rSrSrS\4U4SjjrSrSr\\ SS\ RS\ \ RS\ \ RS \ \S \ \ RS \ \S \ \ RS \ \ RS\ \ RS\ \ RS\ \ RS\ \ RS\ \ RS\\\44Sjj55rSrU=r$) EvollaModelir^c 6>[TU]U5 URUlURUl[ R "URURUR5Ul[US9Ul [ R"[UR5Vs/sH n[UUS9PM sn5Ul[!URUR"S9Ul['US9Ul[+USS5UlUR/5 gs snf)Nr rr>rgF)rFrGrKr4rIrrHrJ embed_tokensrprotein_encoderrbrcrdrrr#rrrb rotary_embrTrg post_initr%s r7rGEvollaModel.__init__s  !.. ++LL&:L:LdN^N^_36Bmm "'v'?'?!@   "AI #!'"A    "&"4"4&:M:MN /v>&-f6NPU&V#  s#DcUR$rrrs r7r EvollaModel.get_input_embeddings s   r9cXlgrrrs r7r EvollaModel.set_input_embeddingss!r9r3rjrBrrkrr0protein_input_idsprotein_attention_maskstructure_feats msa_featsrKrLrc USLUSL- (a [S5eUcURU5nU(a Uc [5nUcDUbUR5OSn[R "XUR S-URS9nUcURS5nSnSnUbOU bLURUU S9nURn[R"S/UR S-URS9n[URUUUUS9nUnURUU5nURHnU"U4UUUUUUUU U UU U US . UD6nM UR!U5n[#UUS 9nU$) a protein_input_ids (torch.LongTensor): The input IDs for the protein sequence in structure-aware tokens. Should be of shape `(batch_size, protein_seq_length)` and type `torch.LongTensor`. protein_attention_mask (torch.Tensor): The attention mask for the protein sequence. Should be of shape `(batch_size, protein_seq_length)` and type `torch.Tensor`. structure_feats (torch.FloatTensor): The input IDs for purely structure-based features. Should be of shape `(batch_size, structure_seq_length, structure_feat_dim)` and type `torch.FloatTensor`. Dummy input for now. msa_feats (torch.FloatTensor): The input IDs for purely MSA-based features. Should be of shape `(batch_size, msa_seq_length, msa_feat_dim)` and type `torch.FloatTensor`. Dummy input for now. structure_batch_mask (torch.Tensor): The batch mask to decide which protein sequences are purely structure-based. Should be of shape `(batch_size)` and type `torch.Tensor`. Should be paired with `structure_feats`. Dummpy input for now. msa_batch_mask (torch.Tensor): The batch mask to decide which protein sequences are purely MSA-based. Should be of shape `(batch_size)` and type `torch.Tensor`. Should be paired with `msa_feats`. Dummpy input for now. Nz:You must specify exactly one of input_ids or inputs_embedsrr&rrT)r^ input_embedsrjr0r) rjrBrMrr0rYrGrHrIrJrKrLr2)rlr)rrr get_seq_lengthr/rVrrtrerrr&rr^rrrr)r]r3rjrBrrkrr0rrrrrKrLrpast_seen_tokens protein_featsrJprotein_outputsrrrY decoder_layerr"s r7rpEvollaModel.forwardsB  -t"; <YZ Z   --i8M 0*nO  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L !  (-C-O"22+53O,FFM!&tf7H7N7Nq7Q.QZkZrZr!s (;;&))+  & #oom\J![[M)*).#-$7"/$3'#5%9- .M)& -0 (++  r9)rrgrrr4rrrI) NNNNNNNNNNNNN)ryrzr{r|r'rGrrr"r%r/rrrr rrrrrrpr~rrs@r7rrsw|*!"'+1537+/59$(598<9=7;157;15b##b!.bu//0 b "% b   1 12 bD>b!!1!12b$E$4$45b!) 6b"%"3"34bE--.b'u||4b!.b u-- .!bbr9rc^\rSrSrU4SjrSrSr\\SS\ RS\ \ RS\ \ RS\ \ RS \ RS \ \ RS \ \4S jj55rS rU=r$)EvollaForProteinText2Textizc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr) rFrGrr rIrrrJlm_headrr\s r7rG"EvollaForProteinText2Text.__init__{sQ   (  ++yy!3!3T__5Q  r9c6URR5$r)r rrs r7r.EvollaForProteinText2Text.get_input_embeddingsszz..00r9c8URRU5$r)r rrs r7r.EvollaForProteinText2Text.set_input_embeddingsszz..u55r9r3rjrklabelsrrrc UR"SUUUUUUS.UD6n U Sn URU 5n Sn UbUR"SXURS.UD6n [ U U U R U R U RS9n U $)a| protein_input_ids (torch.LongTensor): The input IDs for the protein sequence. Should be of shape `(batch_size, protein_seq_length)` and type `torch.LongTensor`. protein_attention_mask (torch.Tensor): The attention mask for the protein sequence. Should be of shape `(batch_size, protein_seq_length)` and type `torch.Tensor`. Example: ```python >>> from transformers import EvollaProcessor, EvollaForProteinText2Text >>> model = EvollaForProteinText2Text.from_pretrained("westlake/Evolla-10B-hf") >>> processor = EvollaProcessor.from_pretrained("westlake/Evolla-10B-hf") >>> protein_information = { "aa_seq": "your amino acid sequence", "foldseek": "your foldseek sequence", } >>> question = "What is the function of this protein?" >>> message = [ {"role": "system", "content": "You are an AI expert that can answer any questions about protein."}, {"role": "user", "content": question}, ] >>> inputs = processor(proteins=[protein_information], messages_list=[message], return_tensors="pt", padding="longest") >>> outputs = model.generate(**inputs) >>> print(processor.batch_decode(outputs, skip_special_tokens=True)) ```)r3rjrkrrrrN)logitsrrI)lossrrrrmrj)r r loss_functionrIrrrrm)r]r3rjrkrrrrrrrrr lm_outputss r7rp!EvollaForProteinText2Text.forwardsT** )'/#9     m,  %%iVtibhiD+#33!//))  r9)rr rIrS)ryrzr{r|rGrrr#r"r/rrrrrrpr~rrs@r7rrzs16'+1559-1.29=$(?##?!.?  1 12 ? ))* ? !++ ?!) 6?D>??r9r)rrr)Nr&)rb)cr8r dataclassesrtypingrrrr/rr activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_flash_attention_utilsrrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrrrrrprocessing_utilsr utilsr!r"r#r$ utils.genericr%configuration_evollar'r(r) get_loggerryr r8Moduler;rrrrrrr rr:r<rDrJr`r{rrrrrrrrr#rbrxrrr.rrgrrrrrr__all__rjr9r7rsT, !,, !.)7/h9L'RR/<J   H % 4 ^=RYY^=B( 2( "))( Vp pf RYY m";mb'4$ /BII/d;ryy  G2GT9 "))9 x  */***_'!<_'D7  7 t A A'. '.T ?k? ? 299 $k"))k\Y'JBIIJ(J(