"S&S'\ Rv5r?"S(S)\ Rv5r@"S*S+\ Rv5rA"S,S-\5rB"S.S/\ Rv5rC"S0S1\ Rv5rD"S2S3\ Rv5rE"S4S5\ Rv5rF\""S6S7\55rG\""S8S9\G55rH\""S:S;\G55rI"S<S=\ Rv5rJ\""S>S?9"S@SA\G55rK\""SBSC\G55rL\""SDSE\G55rM\""SFSG\G55rN/SHQrOg)LzPyTorch MRA model.N)Path)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)load)ACT2FN)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringis_cuda_platformis_ninja_availableis_torch_cuda_availablelogging) MraConfigc^[[5R5RRRS- S- mU4SjnU"/SQ5n[ SUSS9qg)Nkernelsmrac8>UVs/sHnTU- PM sn$s snfN)filesfile src_folders \/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/mra/modeling_mra.py append_root&load_cuda_kernels..append_root4s .34ed T!e444s)zcuda_kernel.cuzcuda_launch.cuztorch_extension.cpp cuda_kernelT)verbose)r__file__resolveparentr mra_cuda_kernel)r) src_filesr's @r(load_cuda_kernelsr20sQh'')0077>>JURJ5WXI=)TBOcH[UR55S:wa [S5e[UR55S:wa [S5eURS5S:wa [S5eURS5S:wa [S5eURS S 9RR S S 5nUR 5nUR5nUR 5n[RXAX#5upVUR S S 5S S 2S S 2S S S 24nXV4$) z0 Computes maximum values for softmax stability. z.sparse_qk_prod must be a 4-dimensional tensor.'indices must be a 2-dimensional tensor. z>The size of the second dimension of sparse_qk_prod must be 32.r z=The size of the third dimension of sparse_qk_prod must be 32.dimN) lensize ValueErrormaxvalues transpose contiguousintr0 index_max)sparse_qk_prodindicesquery_num_block key_num_block index_valsmax_valsmax_vals_scatters r( sparse_maxrM<s >   !Q&IJJ 7<<>aBCC1#YZZ1#XYY###+22<r?shapetorcharangelongrQreshape)maskrG block_size batch_sizeseq_len num_block batch_idxs r( sparse_maskr]Xs 499;1?@@ 7<<>aBCC zz!} a((]^^**J%I W\\!_EJJw~~VI << z :D !T'"W%8$>$>$@!C DD Kr3cfUR5upEnUR5upxnXS-S:wa [S5eX-S:wa [S5eURXEU-X65RSS5nURXHU-X65RSS5n[ UR55S:wa [S5e[ UR55S:wa [S5e[ UR55S :wa [S 5eURS 5S :wa [S 5eURS 5S :wa [S5eUR 5nUR 5nUR 5nUR 5n[RXUR 55$)z/ Performs Sampled Dense Matrix Multiplication. rzTquery_size (size of first dimension of dense_query) must be divisible by block_size.Pkey_size (size of first dimension of dense_key) must be divisible by block_size.r<r9r5z+dense_query must be a 4-dimensional tensor.)dense_key must be a 4-dimensional tensor.r6r7r r8z.The third dimension of dense_query must be 32.z,The third dimension of dense_key must be 32.) r>r?rVrBr=rCrDr0 mm_to_sparse) dense_query dense_keyrGrXrY query_sizer;_key_sizes r(raraos#."2"2"4JC ~~'A!#opp!kll%%j 2JJ\ffgikmnK!!**.DjV``aceghI ;   !#FGG 9>> !DEE 7<<>aBCCb IJJ~~aBGHH((*K$$&IkkmG  "G  ' '  NNr3cBUR5upVnXd-S:wa [S5eURS5U:wa [S5eURS5U:wa [S5eURXVU-XG5RSS5n[ UR55S :wa [S 5e[ UR55S :wa [S 5e[ UR55S:wa [S 5eURS5S :wa [S5eUR 5nUR 5nUR 5nUR 5n[RXX#5nURSS5RXSU-U5nU$)zH Performs matrix multiplication of a sparse matrix with a dense matrix. rr_r6zQThe size of the second dimension of sparse_query must be equal to the block_size.r zPThe size of the third dimension of sparse_query must be equal to the block_size.r<r9r5,sparse_query must be a 4-dimensional tensor.r`r7r8z8The size of the third dimension of dense_key must be 32.) r>r?rVrBr=rCrDr0sparse_dense_mm) sparse_queryrGrcrHrXrYrfr; dense_qk_prods r(riris!* 0J#!kllz)lmmz)kll!!**.DjV``aceghI <   1$GHH 9>> !DEE 7<<>aBCC~~aBSTT**,LkkmG  "G$$&I#33L9fM!++B3;;JZdHdfijM r3cXX-U-[R"XSS9-R5$)Nfloor rounding_mode)rSdivrU)rG dim_1_block dim_2_blocks r(transpose_indicesrss*  "k 1EIIgbi4j j p p rrr3cH\rSrSr\S5r\S5r\SSj5rSrg)MraSampledDenseMatMulcN[XX45nURXU5 X@lU$r#)rasave_for_backwardrX)ctxrbrcrGrXrFs r(forwardMraSampledDenseMatMul.forwards)%kgR kg>#r3cURup#nURnURS5U-nURS5U-n[XFU5n[ UR SS5XU5n [ XX65n XSS4$Nrr<r9) saved_tensorsrXr>rsrirB) rygradrbrcrGrXrHrI indices_Tgrad_key grad_querys r(backwardMraSampledDenseMatMul.backwards*-*;*;' ^^ %**1-;!q)Z7 %g N "4>>"b#99S`a$TIO T4//r3c.[RXX#5$r#)ruapply)rbrcrGrXs r( operator_call#MraSampledDenseMatMul.operator_calls$**;7WWr3r$Nr8 __name__ __module__ __qualname____firstlineno__ staticmethodrzrr__static_attributes__r$r3r(rurus> 00XXr3rucD\rSrSr\S5r\S5r\S5rSrg)MraSparseDenseMatMulcN[XX45nURXU5 X@lU$r#)rirxrH)ryrjrGrcrHrFs r(rzMraSparseDenseMatMul.forwards*( [ lY?-r3cURup#nURnURS5URS5-n[X5U5n[ UR SS5XqU5n[ XU5n U SUS4$r})r~rHr>rsrirBra) ryrrjrGrcrHrIrrrs r(rMraSparseDenseMatMul.backwards~+.+<+<( y--!q)\->->r-BB %g N "<#9#9"b#A9Tab!$7; 44//r3c.[RXX#5$r#)rr)rjrGrcrHs r(r"MraSparseDenseMatMul.operator_calls#)),\\r3r$Nrr$r3r(rrs> 00]]r3rc$\rSrSr\S5rSrg) MraReduceSumcUR5upEpg[UR55S:wa [S5e[UR55S:wa [S5eUR5u pvnUR5upEURSS9R XE-U5n[ R "URS5[ RURS9n[ R"XSS 9R5USS2S4U--R XE-5n [ R"XB-U4URURS9n U RSX5R XBU5n U R XBU-5n U $) Nr5rhr6r7r:rrOrmrn) r>r=r?sumrVrSrTrUrQrpzerosrP index_add) rjrGrHrIrYr[rXrer\ global_idxestempoutputs r(rMraReduceSum.operator_callsb/;/@/@/B, z |  " #q (KL L w||~ ! #FG G*//11! '  #''A'.66z7MzZ LLa 7>>Z IIgG D I I KiXY[_X_N`crNr r '*( ) {{  ): 6l>P>PYeYlYl <>FFzdno j,HI r3r$N)rrrrrrrr$r3r(rrsr3rcUR5upVnXb-nSn UbURXXU5RSS9n URXXX'5RSS9U SS2SS2S4S-- n URXXX'5RSS9U SS2SS2S4S-- n Ub/URXXX'5RSS9U SS2SS2S4S-- n OU[R"XX[R UR S9-n URXXX'5RSS9n URXXX'5RSS9n UbURXXX'5RSS9n [R"XRSS55[R"U5- n U RSSS9RnUb0U S U SS2SSS24U SS2SS2S4-S :R 5-- n XX4$) z' Compute low resolution approximation. Nr<r:r9ư>rOT)r;keepdims@g?)r>rVrrSonesfloatrQmeanmatmulrBmathsqrtr@rA)querykeyrXrWvaluerYrZhead_dimnum_block_per_row value_hat token_count query_hatkey_hatlow_resolution_logitlow_resolution_logit_row_maxs r(get_low_resolution_logitrs%*JJL!J-I ll:*MQQVXQY MM*VZZ_aZb 1d #d * ++jZRVV[]V^ 1d #d *    jZZ^^ce^fAq$J'$.I!5::jSXS^S^glgsgs#tt MM*V[[`b[c ++jZRWW\^W_   jZZ__df_gI << 3D3DR3LMPTPYPYZbPcc#7#;#;T#;#R#Y#Y   3;q$z+B[QRTUW[Q[E\+\`c*c)j)j)l#l l  .J UUr3cURupVnUS:a]US-n[R"XfURS9n [R"[R "X*S9US9n X SSS2SS24S--nUS:a:USS2SU2SS24S-USS2SU2SS24'USS2SS2SU24S-USS2SS2SU24'[R "URUS5USSS S 9n U Rn US :Xa@U RRSS 9Rn X SS2SS4:R5nX4$US :XaSnX4$[US35e)zR Compute the indices of the subset of components to be used in the approximation. rr6rQ)diagonalNg@r<TF)r;largestsortedfullr:sparsez# is not a valid approx_model value.) rRrSrrQtriltriutopkrVrGrAminrr?)r num_blocks approx_modeinitial_prior_first_n_blocksinitial_prior_diagonal_n_blocksrYtotal_blocks_per_rowreoffset temp_mask diagonal_mask top_k_valsrG thresholdhigh_resolution_masks r(get_block_idxesr7s +?*D*D'Ja&*0A5JJ3RfRmRmn  5::i'#JU[\ 3D!QJ6ORU6UU#a' $A%A$A1!D E K Q =!= =q@A !A'D(D'D!D E K Q#@$@#@@A$$Z4jbRV_dJ  Gf%%))b)188 4!T4-8P PWWY  ((  #  ((K=(KLMMr3c [c$[R"U5R5$UR 5uppX-n X-S:wa [ S5eX-nUR XU 5nUR XU 5nUR XU 5nUb*XSS2SS2S4-nXSS2SS2S4-nX#SS2SS2S4-nUS:Xa[XXcU5unnnnOAUS:Xa0[R"5 [XXc5unnnnSSS5 O [S5e[R"5 WW- n[UUUUU5unnSSS5 [RXWUS9[R"U 5- n[UUX5unnUU- nUb"USS [!UU5SS2SS2SS2S4- -- n[R""U5n[$RUUX.5n[&RUUX5nUS:XGax[R""WW- SW-- 5WSS2SSS24-n[R("UW5SS2SS2SSS24R+S S US 5R XU 5nUR-S S 9SS2SS2S4R+S S U5R X5nUR+S S U5R X5U- n UbU U-n [R""U U S:*R/5-5n!UU!SS2SS2S4-nUU!-n[R""U *U S:R/5-5n"UU"SS2SS2S4-nUU"-nUU-USS2SS2S4USS2SS2S4-S -- n#O$US:XaUUSS2SS2S4S -- n#O [S 5eUbU#USS2SS2S4-n#U#R XX5n#U#$!,(df  GN=f!,(df  GN=f)z( Use Mra to approximate self-attention. Nrz4sequence length must be divisible by the block_size.rrz&approx_mode must be "full" or "sparse")rXrrr<r:rz-config.approx_mode must be "full" or "sparse")r0rS zeros_likerequires_grad_r>r?rVrno_grad ExceptionrrurrrrMr]exprrrrepeatrr)$rrrrWrrrXrrrYnum_headrZr meta_batchrrrrrrelow_resolution_logit_normalizedrGrhigh_resolution_logitrKrLhigh_resolution_attnhigh_resolution_attn_outhigh_resolution_normalizerlow_resolution_attnlow_resolution_attn_outlow_resolution_normalizerlog_correctionlow_resolution_corrhigh_resolution_corr context_layers$ r(mra2_attentionr]sq&5577.3jjl+J'&Jq OPP- MM*x 8E ++j8 4C MM*x 8E Q4Z((At$$Q4Z((fUm  %V Rk+G  ]]_QiJR N +/KQ_ @AA *>A]*]'(7 +   ( + ) %% 2?? G @ (",,A7L]!qH14DD  5q;tU\C]^_abdegk^kCl?l8m m 99%:;3AAgu ".!;!;g'8"f II*-IICRfLff g!T1*% &  LL,i 8AtQ G VAq*a ( WZ( 3 # # # +Aq$J 7 > >q!Z P X XYc m "6<RSTVWY]S]>^#^ %?BV%V"14KK &q!Tz 25NqRSUYz5Z Z]a a   04NqRSUYz4Z]a4ab GHH %Q4Z(88 !))*RM S_ s1N?,O? O O c6^\rSrSrSrU4SjrSSjrSrU=r$) MraEmbeddingsizGConstruct the embeddings from word, position and token_type embeddings.c f>[TU]5 [R"URUR UR S9Ul[R"URS-UR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5S-5 [+USS5UlUR#S[$R."UR0R35[$R4UR0R6S 9S S 9 g) N) padding_idxr6eps position_ids)rr<position_embedding_typeabsolutetoken_type_idsrOF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferrSrTexpandgetattrrrrr>rUrQselfconfig __class__s r(rMraEmbeddings.__init__s? !||F,=,=v?Q?Q_e_r_rs#%<<0N0NQR0RTZTfTf#g %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=  ^U\\&:X:X-Y-`-`ah-ilm-mn'.v7PR\']$   KK))..0 4K\K\KcKc d  r3c`UbUR5nOUR5SSnUSnUcURSS2SU24nUcv[US5(a-URSS2SU24nUR USU5nUnO8[ R "U[ RURRS9nUcURU5nURU5n XI-n URS:XaURU5n X- n URU 5n URU 5n U $)Nr<rrrrOr)r>rhasattrrrrSrrUrQrrrrrr) r input_idsrr inputs_embeds input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrs r(rzMraEmbeddings.forwards:  #..*K',,.s3K ^  ,,Q  ^ J":  ' ': 5"&":":<"H   -J^^J/ \\*- r3)rrrrrr)NNNN rrrr__doc__rrzr __classcell__r s@r(rrsQ (  r3rc6^\rSrSrSU4SjjrSSjrSrU=r$)MraSelfAttentioni c >[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35e[ SLn[5(a0[5(a![5(aU(d [5 URUl[URUR- 5UlURUR-Ul["R$"URUR 5Ul["R$"URUR 5Ul["R$"URUR 5Ul["R,"UR.5UlUbUO UR2UlUR4S-UR6-Ul[;UR8[UR4S-S-55UlUR<UlUR>UlUR@Ul g![a#n[RSU35 SnAGNSnAff=f) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zGCould not load the custom kernel for multi-scale deformable attention: r8r6)!rrrnum_attention_headsr r?r0rrrr2rloggerwarningrDattention_head_size all_head_sizerLinearrrrrattention_probs_dropout_probrrr block_per_rowr[rrrr)rrr kernel_loadeder s r(rMraSelfAttention.__init__ s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  (t3 " $ $)9););@R@T@T]j n!#$*#=#= #&v'9'9F'J #PVPnPn $!88B>&BVBVVT^^S&2P2PTV2V[\1\-]^!--,2,O,O)/5/U/U,+ n!hijhklmm ns! I JI==Jc @URup4nURU5RUSURUR5R SS5nUR U5RUSURUR5R SS5nURU5RUSURUR5R SS5nSUS- -nUR5RSURS5RX0R-U5R5nSn URU :aX0RXIUR- 4n [R"U[R"XRS9/SS9n[R"U[R"XRS9/SS9n[R"U[R"XRS9/SS9n[!UR#5UR#5UR#5UR#5UR$UR&UR(UR*S 9n URU :aU SS2SS2SS2SUR24n U RX0RX@R5n U R-S SSS 5R/5n U R15SS UR24-n U R"U 6n U 4n U $) Nr<rr6?rr8rr:)rrrrr r9)rRrviewrr!rBrrsqueezerrVrDrScatrrQrrr[rrrpermuterCr>r")r hidden_statesattention_maskrYrZre query_layer key_layer value_layer gpu_warp_sizepad_sizernew_context_layer_shapeoutputss r(rzMraSelfAttention.forward.s!.!4!4 Q JJ} % T*b$":":DS"S%**,CD  "r3) r"rr!rrrrrr[rrrr#rrrrrrzrrrs@r(rr s!VF<<r3rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) MraSelfOutputinc(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNr) rrrr#rdenserrrrrrs r(rMraSelfOutput.__init__os` YYv1163E3EF f&8&8f>S>STzz&"<"<= r3r/ input_tensorreturncpURU5nURU5nURX-5nU$r#r>rrrr/r@s r(rzMraSelfOutput.forwardu5 =1  ]3 }'CD r3rr>r rrrrrrSTensorrzrrrs@r(r;r;n6> U\\RWR^R^r3r;c<^\rSrSrSU4SjjrSrSSjrSrU=r$) MraAttentioni|c|>[TU]5 [XS9Ul[ U5Ul[ 5Ulg)N)r)rrrrr;rset pruned_heads)rrrr s r(rMraAttention.__init__}s0 $V] #F+ Er3c6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr:)r=rrrr!rOrrrrrr>r"union)rheadsindexs r( prune_headsMraAttention.prune_headss  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r3cdURX5nURUSU5nU4USS-nU$Nrr)rr)rr/r0 self_outputsattention_outputr7s r(rzMraAttention.forwards>yy? ;;|A F#% QR(88r3)rrOrr#) rrrrrrUrzrrrs@r(rLrL|s" ;$r3rLcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)MraIntermediateic>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr#) rrrr#rintermediate_sizer> isinstance hidden_actstrr intermediate_act_fnrs r(rMraIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r3r/rAcJURU5nURU5nU$r#r>rcrr/s r(rzMraIntermediate.forwards& =1 00? r3rfrHrs@r(r]r]s(9U\\ellr3r]cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) MraOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr=) rrrr#r_rr>rrrrrrs r(rMraOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r3r/r@rAcpURU5nURU5nURX-5nU$r#rCrDs r(rzMraOutput.forwardrFr3rGrHrs@r(rjrjrJr3rjc8^\rSrSrU4SjrSSjrSrSrU=r$)MraLayeric>[TU]5 URUlSUl[ U5UlUR Ul[U5Ul[U5Ul gNr) rrchunk_size_feed_forward seq_len_dimrL attentionadd_cross_attentionr] intermediaterjrrs r(rMraLayer.__init__sW '-'E'E$%f-#)#=#= +F3' r3cURX5nUSnUSSn[URURURU5nU4U-nU$rX)rurfeed_forward_chunkrsrt)rr/r0self_attention_outputsrZr7 layer_outputs r(rzMraLayer.forwardsa!% !N1!4(,0  # #T%A%A4CSCSUe  /G+r3cJURU5nURX!5nU$r#)rwr)rrZintermediate_outputr|s r(rzMraLayer.feed_forward_chunks)"//0@A{{#6I r3)rvrursrwrrtr#) rrrrrrzrzrrrs@r(rprps( r3rpc:^\rSrSrU4SjrSSjrSrU=r$) MraEncoderic>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) rrrr ModuleListrangenum_hidden_layersrplayergradient_checkpointing)rrrer s r(rMraEncoder.__init__sR  ]]eFD\D\>]#^>]HV$4>]#^_ &+#$_sA%cU(aSOSn[UR5HupxU(aXa4-nU"X5n U SnM U(aXa4-nU(d[SX455$[UUS9$)Nr$rc3,# UH ocMUv M g7fr#r$).0vs r( %MraEncoder.forward..sX$Fq$Fs )last_hidden_stater/) enumeratertupler) rr/r0 head_maskoutput_hidden_states return_dictall_hidden_statesi layer_module layer_outputss r(rzMraEncoder.forwards#7BD(4OA#$58H$H!(GM)!,M 5 14D D X]$FXX X1++  r3)rrr)NNFTr9rs@r(rrs ,"   r3rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)MraPredictionHeadTransformicp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr=) rrrr#rr>r`rarbr transform_act_fnrrrs r(r#MraPredictionHeadTransform.__init__s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr3r/rAclURU5nURU5nURU5nU$r#)r>rrrgs r(rz"MraPredictionHeadTransform.forwards4 =1 --m< }5 r3)rr>rrHrs@r(rrs)UU\\ellr3rc4^\rSrSrU4SjrSrSrSrU=r$)MraLMPredictionHeadicH>[TU]5 [U5Ul[R "UR URSS9Ul[R"[R"UR55Ul URURl g)NF)bias) rrr transformrr#rrdecoder ParameterrSrrrs r(rMraLMPredictionHead.__init__sm 3F;yy!3!3V5F5FUS LLV->->!?@ !II r3c:URURlgr#)rrrs r( _tie_weights MraLMPredictionHead._tie_weightss II r3cJURU5nURU5nU$r#)rrrgs r(rzMraLMPredictionHead.forward s$}5  ]3 r3)rrr) rrrrrrrzrrrs@r(rrs &&r3rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)MraOnlyMLMHeadi'cB>[TU]5 [U5Ulgr#)rrr predictionsrs r(rMraOnlyMLMHead.__init__(s .v6r3sequence_outputrAc(URU5nU$r#r)rrprediction_scoress r(rzMraOnlyMLMHead.forward,s ,,_=  r3rrHrs@r(rr's(7!u||! !!r3rcJ\rSrSr%\\S'SrSrS\R4Sjr Sr g) MraPreTrainedModeli1rr!TmodulechURRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[R5(adUR R RSUS9 URb2UR R URR5 gg[U[R5(aJURR R5 UR R RS5 g[U[5(a%URR R5 gg)zInitialize the weightsg)rstdNr*)rinitializer_ranger`rr#weightdatanormal_rzero_rrrfill_r)rrrs r( _init_weights MraPreTrainedModel._init_weights8s.kk++ fbii ( ( MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S )  3 4 4 KK   " " $5r3r$N) rrrrr__annotations__base_model_prefixsupports_gradient_checkpointingrModulerrr$r3r(rr1s& &*#%BII%r3rc8^\rSrSrU4SjrSrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ S \ \ S\ \\44Sjj5rSrU=r$)MraModeliLc>[TU]U5 Xl[U5Ul[ U5UlUR5 gr#)rrrrrrencoder post_initrs r(rMraModel.__init__Ns9   '/!&)  r3c.URR$r#rrrs r(get_input_embeddingsMraModel.get_input_embeddingsXs...r3c$XRlgr#r)rrs r(set_input_embeddingsMraModel.set_input_embeddings[s */'r3cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrurU)rheads_to_prunerrSs r( _prune_headsMraModel._prune_heads^s< +002LE LL  u % / / ; ;E B3r3r r0rrrrrrrAc UbUOURRnUbUOURRnUbUb [S5eUb"UR X5 UR 5n O"UbUR 5SSn O [S5eU upUb UR O UR n Uc[R"X4U S9nUcr[URS5(a3URRSS2SU 24n U RX5nUnO$[R"U [RU S9nURX)5nUR!XPRR"5nURUUUUS9nUR%UUUUUS9nUS nU(d U4US S-$['UUR(UR*UR,S 9$) NzDYou cannot specify both input_ids and inputs_embeds at the same timer<z5You have to specify either input_ids or inputs_embedsrrrO)r rrr)r0rrrrr)rr/ attentionscross_attentions)rruse_return_dictr?%warn_if_padding_and_no_attention_maskr>rQrSrr rrrrrUget_extended_attention_mask get_head_maskrrrr/rr)rr r0rrrrrrrrYrrQrrextended_attention_maskembedding_outputencoder_outputsrs r(rzMraModel.forwardfs%9$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, %.%:!! @T@T  !"ZZ*)A6RN  !t(899*.//*H*HKZK*X'3J3Q3QR\3i0!A!&[ SY!Z150P0PQ_0m&&y++2O2OP ??%)' +  ,, 2!5# ' *!,#%(;; ;1-)77&11,==   r3)rrr)NNNNNNNN)rrrrrrrrrrrSrIboolrrrrzrrrs@r(rrLs/0C-11515/3,004/3&*J ELL)J !.J !. J u||, J ELL) J   -J 'tnJ d^J  u88 9J J r3rcZ^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ S\ \ S\ \\44Sjj5rSrU=r$)MraForMaskedLMizcls.predictions.decoder.weightzcls.predictions.decoder.biasc>[TU]U5 [U5Ul[ U5UlUR 5 gr#)rrrr!rclsrrs r(rMraForMaskedLM.__init__s4  F#!&) r3cBURRR$r#)rrrrs r(get_output_embeddings$MraForMaskedLM.get_output_embeddingssxx##+++r3cXRRlURURRlgr#)rrrr)rnew_embeddingss r(set_output_embeddings$MraForMaskedLM.set_output_embeddingss*'5$$2$7$7!r3r r0rrrrlabelsrrrAc U bU OURRn URUUUUUUUU S9n U Sn URU 5n Sn UbF[ 5nU"U R SURR 5UR S55n U (dU 4U SS-nU bU 4U-$U$[U U U RU RS9$)a{ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nr0rrrrrrrr<rlosslogitsr/r) rrr!rrr+rrr/r)rr r0rrrrrrrr7rrmasked_lm_lossloss_fctrs r(rzMraForMaskedLM.forwards&&1%[TU]5 [R"URUR5Ul[R "UR5Ul[R"URUR5Ul Xl gr#) rrrr#rr>rrr num_labelsout_projrrs r(rMraClassificationHead.__init__se YYv1163E3EF zz&"<"<=  &"4"4f6G6GH  r3c USS2SSS24nURU5nURU5n[URR"U5nURU5nUR U5nU$)Nr)rr>r rrar)rfeatureskwargsxs r(rzMraClassificationHead.forwardse Q1W  LLO JJqM 4;;)) *1 - LLO MM! r3)rr>rrrrs@r(rrs7r3rz MRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcF^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \ \ \ 44S jj5r SrU=r$)MraForSequenceClassificationic>[TU]U5 URUl[U5Ul[ U5UlUR5 gr#)rrrrr!r classifierrrs r(r%MraForSequenceClassification.__init__sA   ++F#/7 r3r r0rrrrrrrrAc .U bU OURRn URUUUUUUUU S9n U Sn URU 5n Sn UGbURRcUR S:XaSURlOoUR S:aNUR [R:XdUR [R:XaSURlOSURlURRS:XaI[5nUR S:Xa&U"U R5UR55n OU"X5n OURRS:Xa=[5nU"U RSUR 5URS55n O,URRS:Xa[5nU"X5n U (dU 4U SS-nU bU 4U-$U$[U U U R U R"S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationr<r)rrr!r problem_typerrPrSrUrDr r,rr+rrr/r)rr r0rrrrrrrr7rrrrrs r(rz$MraForSequenceClassification.forward"s&&1%[TU]U5 [U5Ul[R "UR UR 5Ul[R "UR S5UlUR5 grr) rrrr!rr#rpre_classifierrrrs r(rMraForMultipleChoice.__init__is_  F# ii(:(:Fr!rrReLUrrrr/r)rr r0rrrrrrr num_choicesr7 hidden_state pooled_outputrreshaped_logitsrrrs r(rzMraForMultipleChoice.forwardssV&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  (( ))%'!5#  qz $QT* ++M:  -0 / ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E("!//))   r3)rr!rr)rrrrrrrrSrIrrrrrzrrrs@r(rrgs-11515/3,004)-/3&*X ELL)X !.X !. X u||, X ELL) X   -X &X 'tnX d^X  u// 0X X r3rcF^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \ \ \ 44S jj5r SrU=r$)MraForTokenClassificationic0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr#) rrrrr!rrrrr#rrrrs r(r"MraForTokenClassification.__init__si   ++F#zz&"<"<= ))F$6$68I8IJ r3r r0rrrrrrrrAc U bU OURRn URUUUUUUUU S9n U Sn URU 5n UR U 5n Sn Ub[ 5nUbUR S5S:HnU R SUR5n[R"XR S5[R"UR5RU55nU"UU5n O2U"U R SUR5UR S55n U (dU 4U SS-nU bU 4U-$U$[U U U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr<rr)rrr!rrrr+rrSwheretensor ignore_indextype_asrr/r)rr r0rrrrrrrr7rrrr active_loss active_logits active_labelsrs r(rz!MraForTokenClassification.forwardsf"&1%[TU]U5 SUlURUl[U5Ul[ R "URUR5UlUR5 g)Nr6) rrrrr!rr#r qa_outputsrrs r(r MraForQuestionAnswering.__init__s[   ++F#))F$6$68I8IJ r3r r0rrrrstart_positions end_positionsrrrAc U bU OURRn URUUUUUUU U S9n U Sn URU 5n U R SSS9upUR S5nUR S5nSnUbUb[ UR55S:aUR S5n[ UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S- nU (dX4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrrr<r:)r*r6)r start_logits end_logitsr/r) rrr!r3splitr,r=r>clamprrr/r)rr r0rrrrr5r6rrr7rrr8r9 total_loss ignored_indexr start_lossend_lossrs r(rzMraForQuestionAnswering.forward's&1%rQs " AA*!9.llkk(   H % C&8.%OP%PsXENN33X0]5>>22].:%VP#)Z!"$%pf7BII7t`ryy`H BII 299B bii   ):! ! J$"))0!RYY!%%%2d !d d ND 'D D PBII* L #5L  L ^d -d d NF 2F F RJ 0J J Z r3