\&"S5S-9"S6S7\;55r?\&"S8S9\;55r@\&"S:S;\;55rA\&"S<S=\;55rB/S>QrCg)?zPyTorch RemBERT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) RemBertConfigc ^SSKnSSKnSSKn[ RRU5n[RSU35 URRU5n/n/n UH{umn [U4SjS55(aM"[RSTSU 35 URRUT5n URT5 U RU 5 M} [X5GHumn TR!S S 5mTR#S 5m[S T55(a)[RS S R%T535 MjUn THn UR'SU 5(aUR#SU 5nOU /nUSS:Xd USS:Xa [)U S5n OZUSS:Xd USS:Xa [)U S5n O;USS:Xa [)U S5n O%USS:Xa [)U S5n O[)XS5n [/U5S:dM[1US5nXn M W SSS:Xa [)U S5n OU S:XaUR3U 5n U R4U R4:wa&[7SU R4SU R4S 35e[RS!T35 [<R>"U 5U l GM U$![a [R S5 ef=f![*a8 [RSR-S R%T555 GMf=f![8a1nU=R:U R4U R44- sleSnAff=f)"z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from c3*># UH oT;v M g7fN).0denynames d/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/rembert/modeling_rembert.py -load_tf_weights_in_rembert..GsX(Wt|(Ws)adam_vadam_moutput_embeddingclszLoading TF weight z with shape zbert/zrembert//c3*# UH nUS;v M g7f))r)r*AdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepNr")r#ns r&r'r(Xs  n nsz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierz Skipping {}ri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )!renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variablesany load_variableappendzipreplacesplitjoin fullmatchgetattrAttributeErrorformatlenint transposeshape ValueErrorAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr>nptftf_path init_varsnamesarraysrXarraypointerm_name scope_namesnumer%s @r&load_tf_weights_in_rembertrn1sL ggoo01G KK8 BC''0I E F e X(WX X X  (l5'BC&&w5 T e!5) e||GZ0zz#      KK)CHHTN#34 5 F||,f55 hhy&9 %h 1~)[^w-F!'84Q=0KNf4L!'62Q#33!'84Q7*!'<8%g1~>G;1$+a.)!,+, #$<= (gx0G x LL'E }} + >'--@QRWR]R]Q^^i!jkk,  078''. c*d LS   Q   n&KK 4 4SXXd^ DE  FFw}}ekk2 2F  s6 K:K<AM!K9<=L>=L> M< ,M77M<c ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\\RS\ S \R4 S jjr S r U=r$) RemBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cv>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 g)N) padding_idxeps position_ids)rF) persistent)super__init__r Embedding vocab_sizeinput_embedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr\arangeexpandselfr` __class__s r&rzRemBertEmbeddings.__init__s !||   v::H[H[ $&<<0N0NPVPkPk#l %'\\&2H2H&JeJe%f"f&A&AvG\G\]zz&"<"<=   ELL)G)GHOOPWXej  input_idstoken_type_idsrv inputs_embedspast_key_values_lengthreturncUbUR5nOUR5SSnUSnUcURSS2XWU-24nUc8[R"U[RURR S9nUcUR U5nURU5nXH-n URU5n X- n URU 5n URU 5n U $)Nrwrdtypedevice) sizervr\zeroslongrrrrrr) rrrrvrr input_shape seq_lengthr embeddingsrs r&forwardRemBertEmbeddings.forwards  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !"[[EJJtO`O`OgOghN   00;M $ : :> J": "66|D) ^^J/ \\*- r)rrrrr)NNNNr)__name__ __module__ __qualname____firstlineno____doc__rzrr\ LongTensor FloatTensorrVTensorr__static_attributes__ __classcell__rs@r&rprpsQ (15593759&' E,,-!!1!12u//0    1 12  !$  rrpcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) RemBertPoolerc>[TU]5 [R"URUR5Ul[R "5Ulgr!)ryrzrLinear hidden_sizedenseTanh activationrs r&rzRemBertPooler.__init__s9 YYv1163E3EF '')r hidden_statesrc\USS2S4nURU5nURU5nU$)Nr)rr)rrfirst_token_tensor pooled_outputs r&rRemBertPooler.forwards6+1a40 #56  6 r)rr rrrrrzr\rrrrrs@r&rrs($ U\\ellrrc^\rSrSrS U4SjjrSS\R S\\RS\\RS\\RS\\ S\ S \\R S \ 4S jjr S r U=r$)RemBertSelfAttentionc>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5UlUR"UlX lg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())ryrzrnum_attention_headshasattrrYrVattention_head_size all_head_sizerrquerykeyvaluerattention_probs_dropout_probr is_decoder layer_idxrr`rrs r&rzRemBertSelfAttention.__init__s1     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F  %.*=>>+66::4>>J %*8*N*N'*8*M*M'&4#2D.- ."<+224>>BGGI-44T^^DKKK(j"d&>&>@X@XY1a   >*j"d&>&>@X@XY1a  )7It)<)C)C{DNN=M~<^*& ;&@DN--dnn=!<< Y5H5HR5PQ+dii8P8P.QQ  %/.@ --//0@b/I,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD o--r) rrrrrrrrrr!NNNNFN)rrrrrzr\rrrr booltuplerrrrs@r&rrs#07;15=A*."'15Q.||Q.!!2!23Q.E--. Q. ((9(9: Q. ! Q. Q.!.Q. Q.Q.rrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)RemBertSelfOutputi5c(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNrt) ryrzrrrrrrrrrrs r&rzRemBertSelfOutput.__init__6s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rr input_tensorrcpURU5nURU5nURX-5nU$r!rrrrrrs r&rRemBertSelfOutput.forward<5 =1  ]3 }'CD rrrrrrs@r&rr56> U\\RWR^R^rrc^\rSrSrSU4SjjrSrSS\RS\\RS\\RS\\RS\\ S \\ S \\RS \ \R4S jjr S rU=r$)RemBertAttentioniCc|>[TU]5 [XS9Ul[ U5Ul[ 5Ulg)Nr)ryrzrrroutputset pruned_headsrs r&rzRemBertAttention.__init__Ds0 (E '/ Erc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr)rUrrrrr rrrrr rrunion)rheadsindexs r& prune_headsRemBertAttention.prune_headsKs  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrrc lURUUUUUUUS9nURUSU5n U 4USS-n U $)Nrrrrrrrr)rr ) rrrrrrrr self_outputsattention_outputoutputss r&rRemBertAttention.forward^s\yy )"7)/)!  ;;|A F#% QR(88r)r r rr!r)rrrrrzrr\rrrr rrrrrrs@r&rrCs";,7;15=A*.,115||!!2!23E--.  ((9(9:  ! $D>!. u|| rrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertIntermediateiwc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr!) ryrzrrrintermediate_sizerr hidden_actstrr intermediate_act_fnrs r&rzRemBertIntermediate.__init__xs` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rrrcJURU5nURU5nU$r!rr!rrs r&rRemBertIntermediate.forwards& =1 00? rr$rrs@r&rrws(9U\\ellrrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) RemBertOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) ryrzrrrrrrrrrrrs r&rzRemBertOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rrrrcpURU5nURU5nURX-5nU$r!rrs r&rRemBertOutput.forwardrrrrrs@r&r(r(rrr(c.^\rSrSrSU4SjjrSS\R S\\RS\\RS\\RS\\RS\\ S \\ S \\R S \ \R 4S jjr S r SrU=r$) RemBertLayericr>[TU]5 URUlSUl[ X5UlUR UlURUlUR(a-UR (d[US35e[ XS9Ul [U5Ul [U5Ul g)Nrz> should be used as a decoder model if cross attention is addedr )ryrzchunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrYcrossattentionr intermediater(r rs r&rzRemBertLayer.__init__s '-'E'E$)&< ++#)#=#=  # #?? D6)g!hii"26"OD /7#F+ rrrrrencoder_attention_maskrrrrc PURUUUUUUS9n U Sn U SSn UR(aDUbA[US5(d[SUS35eUR U UUUUUUS9n U Sn XSS-n [ UR URURU 5n U 4U -n U $)N)rrrrrrrr4z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) r2rrrYr4rfeed_forward_chunkr0r1)rrrrrr7rrrself_attention_outputsrrcross_attention_outputs layer_outputs r&rRemBertLayer.forwards"& )/)) "0" 2!4(, ??4@4!122 =dVDDD '+&9&9 5#&;-"3-':' # 7q9  ;;G0  # #T%A%A4CSCSUe  /G+rcJURU5nURX!5nU$r!)r5r )rrintermediate_outputr<s r&r9RemBertLayer.feed_forward_chunks)"//0@A{{#6I r)r3r2r0r4r5rr r1r!)NNNNNFN)rrrrrzr\rrrr rrrr9rrrs@r&r.r.s ,$7;15=A>B*.,115.||.!!2!23.E--. . ((9(9: . !)):): ; .!.$D>.!.. u|| .brr.cF^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \ S \ S \ S \\R S\ \ \ 44Sjjr SrU=r$)RemBertEncoderic 0>[TU]5 Xl[R"UR UR 5Ul[R"[UR5Vs/sH n[XS9PM sn5Ul SUl gs snf)Nr F)ryrzr`rrr}rembedding_hidden_mapping_in ModuleListrangenum_hidden_layersr.layergradient_checkpointing)rr`irs r&rzRemBertEncoder.__init__ss  +-99V5P5PRXRdRd+e(]]uU[UmUmOn#oOn!L$EOn#op &+#$ps-Brrrrr7past_key_values use_cacheroutput_hidden_states return_dictrrc NUR(a/UR(aU(a[RS5 SnSn U(aB[ U[ 5(d-[RS5 Sn [ R"U5nURU5nU (aSOSn U(aSOSnU(aURR(aSOSn[UR5HgunnU (aX4-n UbUUOSnU"UUUUUUU5nUSnU(dM8UUS4-nURR(dM^UUS4-nMi U (aX4-n U (aUR5nU (d[S UUU UU455$[UUU UUS 9$) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.Tr"rrr<c3.# UH nUcMUv M g7fr!r")r#vs r&r')RemBertEncoder.forward..&s" As )last_hidden_staterLr attentionscross_attentions)rItrainingrB warning_oncerr r from_legacy_cacherDr`r3 enumeraterHto_legacy_cacherr)rrrrrr7rLrMrrNrOrreturn_legacy_cacheall_hidden_statesall_self_attentionsall_cross_attentionsrJ layer_modulelayer_head_mask layer_outputss r&rRemBertEncoder.forwards  & &4==##p" # Z??   \  #' 1CCOTO88G "6BD$5b4%64;;;Z;Zr`d(4OA|#$58H$H!.7.CilO(%&!M*!,M  &9]1=MBEI$("'%* 15R ||R !!2!23R E--. R ((9(9: R !)):): ; R "%e.?.?(@"ABR D>R  R #R R !.R  u?? @R R rrBcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertPredictionHeadTransformi;cp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) ryrzrrrrrrr r transform_act_fnrrrs r&rz'RemBertPredictionHeadTransform.__init__<s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrrrclURU5nURU5nURU5nU$r!)rrgrr%s r&r&RemBertPredictionHeadTransform.forwardEs4 =1 --m< }5 r)rrrgrrs@r&rere;s)UU\\ellrrecb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertLMPredictionHeadiLcn>[TU]5 [R"URUR 5Ul[R"UR UR5Ul[URUl [R"UR URS9Ul gr)ryrzrrroutput_embedding_sizerr|decoderr rrrrrs r&rz RemBertLMPredictionHead.__init__Msz YYv1163O3OP yy!=!=v?P?PQ  !2!23f&B&BH]H]^rrrcURU5nURU5nURU5nURU5nU$r!)rrrror%s r&rRemBertLMPredictionHead.forwardTs@ =1  6 }5  ]3 r)rrrorrrs@r&rlrlLs)_U\\ellrrlcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertOnlyMLMHeadi]cB>[TU]5 [U5Ulgr!)ryrzrl predictionsrs r&rzRemBertOnlyMLMHead.__init__^s 26:rsequence_outputrc(URU5nU$r!rv)rrxprediction_scoress r&rRemBertOnlyMLMHead.forwardbs ,,_=  rrzrrs@r&rtrt]s(;!u||! !!rrtc2\rSrSr%\\S'\rSrSr Sr Sr g)RemBertPreTrainedModeligr`rembertTc[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weightsg)meanstdNg?)rrrr5r^normal_r`initializer_ranger8zero_r{rsrfill_)rmodules r& _init_weights$RemBertPreTrainedModel._init_weightsns fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).rr"N) rrrrr__annotations__rnload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrr"rr&r~r~gs 0O!&*#*rr~a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introc"^\rSrSrSU4SjjrSrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ \ \ RS\ \S\ \S\ \S\ \S\ \ RS\\ \44Sjj5rSrU=r$) RemBertModelic>[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlUR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) ryrzr`rprrBencoderrpooler post_init)rr`add_pooling_layerrs r&rzRemBertModel.__init__sK   +F3%f- /@mF+d  rc.URR$r!rrrs r&get_input_embeddings!RemBertModel.get_input_embeddingss...rc$XRlgr!r)rrs r&set_input_embeddings!RemBertModel.set_input_embeddingss */'rcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrHr2r)rheads_to_prunerHrs r& _prune_headsRemBertModel._prune_headss< +002LE LL  u % / / ; ;E B3rrrrrvrrrr7rLrMrrNrOrrclU bU OURRn U bU OURRn U bU OURRn URR(aU bU OURR n OSn UbUb [ S5eUb"URX5 UR5nO"UbUR5SSnO [ S5eUunnUb URO URnSnU b:[U [5(dU SSRSOU R5nUc[R"UUU-4US9nUc$[R "U[R"US9nUR%X/5nURR(aEUbBUR5unnnUU4nUc[R"UUS9nUR'U5nOSnUR)XPRR*5nUR-UUUUUS 9nUR/UUUUUU U U U U US 9 nUSnUR0bUR1U5OSnU (d UU4US S-$[3UUUR4UR6UR8UR:S 9$) NFzDYou cannot specify both input_ids and inputs_embeds at the same timerwz5You have to specify either input_ids or inputs_embedsrr)rr)rrvrrr) rrrr7rLrMrrNrOrr)rT pooler_outputrLrrUrV)r`rrNuse_return_dictrrMrY%warn_if_padding_and_no_attention_maskrrrr rXget_seq_lengthr\onesrrget_extended_attention_maskinvert_attention_mask get_head_maskrGrrrrrLrrUrV)rrrrrvrrrr7rLrMrrNrOrrrrrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrxrs r&rRemBertModel.forwards$2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T!"  &"/599 "1%++B/$335 #  !"ZZ*jCY6Y)ZdjkN  !"[[EJJvVN150P0PQ_0m ;; ! !&;&G=R=W=W=Y :  7$68O#P %-).4HQW)X&.2.H.HI_.` +.2 +&&y++2O2OP ??%)'#9 + ,, 2"7#B+/!5#)'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   r)r`rrr)TNNNNNNNNNNNNNN)rrrrrzrrrrrr\rrrrrrrrrrrs@r&rrs /0C155959371559=A>BEI$(,0/3&*15m E,,-m !!1!12m !!1!12 m u//0 m E--. m   1 12m  ((9(9:m !)):): ;m "%e.?.?(@"ABm D>m $D>m 'tnm d^m !.m uBB C!m m rrc^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ S\ \ S\ \ S\\\44Sjj5rSSjr\S\ 4Sj5rSrU=r$)RemBertForMaskedLMicls.predictions.decoder.weightc>[TU]U5 UR(a[R S5 [ USS9Ul[U5UlUR5 g)NznIf you want to use `RemBertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr ryrzrrBwarningrrrtr,rrs r&rzRemBertForMaskedLM.__init__sR     NN1  $FeD %f- rcBURRR$r!r,rvrors r&get_output_embeddings(RemBertForMaskedLM.get_output_embeddings.xx##+++rc8XRRlgr!rrnew_embeddingss r&set_output_embeddings(RemBertForMaskedLM.set_output_embeddings1'5$rrrrrvrrrr7labelsrrNrOrc U bU OURRn URUUUUUUUUU U U S9 n U SnURU5nSnU bF[ 5nU"UR SURR 5U R S55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a{ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) rrrvrrrr7rrNrOrrwr<losslogitsrrU) r`rrr,rrr|rrrU)rrrrrvrrrr7rrrNrOrrxr{masked_lm_lossloss_fctr s r&rRemBertForMaskedLM.forward4s,&1% &IIrcg)z Legacy correction: RemBertForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr")r,s r& can_generateRemBertForMaskedLM.can_generate{s rr,r) NNNNNNNNNNNNr!)rrrr_tied_weights_keysrzrrrrr\rrrrrrrr classmethodrrrrs@r&rrsp:; ,6155959371559=A>B-1,0/3&*6 E,,-6 !!1!126 !!1!12 6 u//0 6 E--. 6   1 126  ((9(9:6 !)):): ;6 ))*6 $D>6 'tn6 d^6  un$ %6 6 p JTrrzS RemBERT Model with a `language modeling` head on top for CLM fine-tuning. c"^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ \ \ RS\ \ RS\ \S\ \S\ \S\ \S\\ \44Sjj5rSrU=r$)RemBertForCausalLMirc>[TU]U5 UR(d[R S5 [ USS9Ul[U5UlUR5 g)NzOIf you want to use `RemBertForCausalLM` as a standalone, add `is_decoder=True.`Frrrs r&rzRemBertForCausalLM.__init__sL    NNl m#FeD %f- rcBURRR$r!rrs r&r(RemBertForCausalLM.get_output_embeddingsrrc8XRRlgr!rrs r&r(RemBertForCausalLM.set_output_embeddingsrrrrrrvrrrr7rLrrMrrNrOrc UbUOURRnURUUUUUUUUU U U U US9 nUSnURU5nSnU b*UR"UU 4SURR 0UD6nU(dU4USS-nUbU4U-$U$[ UUURURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, RemBertForCausalLM, RemBertConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/rembert") >>> config = RemBertConfig.from_pretrained("google/rembert") >>> config.is_decoder = True >>> model = RemBertForCausalLM.from_pretrained("google/rembert", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```N) rrrvrrrr7rLrMrrNrOrr|r<)rrrLrrUrV) r`rrr, loss_functionr|rrLrrUrV)rrrrrvrrrr7rLrrMrrNrOkwargsrrxr{lm_lossr s r&rRemBertForCausalLM.forwardsR&1%BEI-1$(,0/3&*Q E,,-Q !!1!12Q !!1!12 Q u//0 Q E--. Q   1 12Q  ((9(9:Q !)):): ;Q "%e.?.?(@"ABQ ))*Q D>Q $D>Q 'tnQ d^Q " u77 8#Q Q rrz RemBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$) RemBertForSequenceClassificationic0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr! ryrz num_labelsrrrrclassifier_dropout_probrrrr;rrs r&rz)RemBertForSequenceClassification.__init__si   ++#F+ zz&"@"@A ))F$6$68I8IJ rrrrrvrrrrrNrOrc RU bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURR S:Xa=[5nU"U RSUR 5URS55nO,URR S:Xa[5nU"X5nU (dU 4U SS-nUbU4U-$U$[!UU U R"U R$S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrvrrrrNrOr regressionsingle_label_classificationmulti_label_classificationrwr<r)r`rrrr; problem_typerrr\rrVrsqueezerrrrrrU)rrrrrvrrrrrNrOrrrrrr s r&r(RemBertForSequenceClassification.forwards(&1%E 'tnE d^E  u.. /E E rrcR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$)RemBertForMultipleChoiceiMc>[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)Nr) ryrzrrrrrrrrr;rrs r&rz!RemBertForMultipleChoice.__init__OsV  #F+ zz&"@"@A ))F$6$6: rrrrrvrrrrrNrOrc ZU bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5n URU 5nURSU 5nSnUb[5nU"X5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrwrrr<r) r`rrXrrrrr;rrrrU)rrrrrvrrrrrNrO num_choicesrrrreshaped_logitsrrr s r&r RemBertForMultipleChoice.forwardYsX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ,, ))%'/!5#     ]3 / ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E("!//))   r)r;rrr)rrrrrzrrr\rrrrrrrrrrs@r&rrMs266:59481559-1,0/3&*X E--.X !!2!23X !!1!12 X u001 X E--. X   1 12X ))*X $D>X 'tnX d^X  u// 0X X rrcR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$)RemBertForTokenClassificationic.>[TU]U5 URUl[USS9Ul[ R "UR5Ul[ R"URUR5Ul UR5 gNFrrrs r&rz&RemBertForTokenClassification.__init__sk   ++#FeD zz&"@"@A ))F$6$68I8IJ rrrrrvrrrrrNrOrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrwr<r) r`rrrr;rrrrrrU)rrrrrvrrrrrNrOrrxrrrr s r&r%RemBertForTokenClassification.forwards$&1%2 'tn2 d^2  u++ ,2 2 rrcr^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \ 44Sjj5rSrU=r$)RemBertForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr ) ryrzrrrrrr qa_outputsrrs r&rz$RemBertForQuestionAnswering.__init__sU   ++#FeD ))F$6$68I8IJ rrrrrvrrstart_positions end_positionsrrNrOrc U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5nUR S5nSnUbUb[ UR55S:aUR S5n[ UR55S:aUR S5nURS5nURSU5 URSU5 [US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrrrwr) ignore_indexr<)r start_logits end_logitsrrU) r`rrrrOrrUrclamp_rrrrU)rrrrrvrrrrrrNrOrrxrrr total_loss ignored_indexr start_lossend_lossr s r&r#RemBertForQuestionAnswering.forwards&1% E--.> !!2!23> !!1!12 > u001 > E--. >   1 12> "%"2"23>   0 01> $D>> 'tn> d^>  u22 3> > rr) rrrrrrr.rr~rn)DrrrDtypingrrr\torch.utils.checkpointrtorch.nnrrr activationsr cache_utilsr r generationr modeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_rembertr get_loggerrrBrnModulerprrrrrr(r.rBrerlrtr~rrrrrrr__all__r"rr&r-sN " AA!5)9   .ll,0   H %Pf3 3n BII g.299g.V  0ryy0h "))  BII D-DN[ RYY[ ~RYY" bii "!!*_**.  M )M  M `e/eeP g /g  g T Q '=Q  Q hd 5d d N? $:? ? DK "8K K \ r