"SS\ Rr5r?"S S!\ Rr5r@"S"S#\ Rr5rA"S$S%\5rB"S&S'\ Rr5rC"S(S)\ Rr5rD"S*S+\ Rr5rE"S,S-\ Rr5rF"S.S/\ Rr5rG"S0S1\ Rr5rH"S2S3\ Rr5rI\0"S4S5\)55rJ\\0"S6S79"S8S9\/555rK\0"S:S79"S;S<\J55rL\0"S=S79"S>S?\J55rM\0"S@S79"SASB\J\55rN\0"SCSD\J55rO\0"SES79"SFSG\J55rP\0"SHS79"SISJ\J55rQ\0"SKSL\J55rR\0"SMSN\J55rS\0"SOSP\J55rT/SQQrUg)RzPyTorch BERT model.N) dataclass)OptionalUnion)version)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)#_prepare_4d_attention_mask_for_sdpa*_prepare_4d_causal_attention_mask_for_sdpa)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringget_torch_versionlogging) BertConfigc SSKnSSKnSSKn[ RRU5n[RSU35 URRU5n/n/n UH]up[RSU SU 35 URRXj5n URU 5 U RU 5 M_ [X5GHupU RS5n [!SU 55(a)[RS SR#U 535 MWUn U HnUR%S U5(aURS U5nOU/nUSS :Xd USS :Xa ['U S5n OZUSS:Xd USS:Xa ['U S5n O;USS:Xa ['U S5n O%USS:Xa ['U S5n O['XS5n [+U5S:dM[-US5nU Un M WSSS:Xa ['U S5n OUS :XaUR/U 5n U R0U R0:wa&[3SU R0SU R0S35e[RSU 35 [6R8"U 5U lGM U$![a [R S5 ef=f![(a, [RS SR#U 535 GMf=f![2a1nU=R4U R0U R04- sleSnAff=f)z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /c3*# UH nUS;v M g7f))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns ^/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/bert/modeling_bert.py *load_tf_weights_in_bert..Ts  n nsz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierr$i _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipsplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshape ValueErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr?nptftf_path init_varsnamesarraysnamerWarraypointerm_name scope_namesnumes r1load_tf_weights_in_bertrm7s ggoo01G KK8 BC''0I E F  (l5'BC&&w5 T e ! 5) zz#      KK)CHHTN#34 5 F||,f55 hhy&9 %h 1~)[^w-F!'84Q=0KNf4L!'62Q#33!'84Q7*!'<8%g1~>G;1$+a.)!#,+, #$<= (gx0G x LL'E }} + >'--@QRWR]R]Q^^i!jkk,  078''. Y*Z L   Q   Z&KK)CHHTN+; <=  FFw}}ekk2 2F  s6 J'K ,AL'!K 1LL L?,L::L?c ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\\RS\ S \R4 S jjr S r U=r$) BertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c.>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5Ul[#USS5UlUR'S[(R*"UR5R-S5SS9 UR'S [(R."UR0R35[(R4S 9SS9 g) N) padding_idxepsposition_embedding_typeabsolute position_ids)r$F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutrRruregister_bufferrZarangeexpandzerosrwsizelongselfr^ __class__s r1r~BertEmbeddings.__init__s/ !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsrzrw inputs_embedspast_key_values_lengthreturncdUbUR5nOUR5SSnUSnUcURSS2XWU-24nUcv[US5(a-URSS2SU24nUR USU5n U nO8[ R "U[ RURRS9nUcURU5nURU5n XJ-n URS:XaURU5n X- n URU 5n URU 5n U $)Nrxr$rzrr|devicerv)rrwhasattrrzrrZrrrrrrurrr) rrrzrwrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrs r1forwardBertEmbeddings.forwardsC  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !t-..*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J":  ' ': 5"&":":<"H   -J^^J/ \\*- r)rrrurrr)NNNNr)__name__ __module__ __qualname____firstlineno____doc__r~rrZ LongTensor FloatTensorrUTensorr__static_attributes__ __classcell__rs@r1rorosQ *15593759&' 'E,,-'!!1!12'u//0 '   1 12 ' !$ ' ''rroc^\rSrSrS U4SjjrSS\R S\\RS\\RS\\RS\\ S\\ S \\R S \ \R 4S jjr S r U=r$)BertSelfAttentionc>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5UlU=(d [#USS5UlUR$S:XdUR$S :XaGUR&Ul[R("S UR&-S - UR5UlUR,UlX0lg) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rurv relative_keyrelative_key_queryr=r$)r}r~rnum_attention_headsrrXrUattention_head_size all_head_sizerLinearquerykeyvaluerattention_probs_dropout_probrrRrurrdistance_embedding is_decoder layer_idxrr^rurrs r1r~BertSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++"r hidden_statesattention_mask head_maskencoder_hidden_statespast_key_valueoutput_attentionscache_positionrc` URupn URU5n U RUSURUR5R SS5n USLn Ub][ U[5(aFURRUR5n U (a URnOURnOUnU (aUOUnU (aQUbNW (aGWRURRnURURRnOUR!U5nURUSURUR5R SS5nUR#U5nURUSURUR5R SS5nUbNU (dUOSnWR%UUURSU05unnU (aSURUR'[&R("U UR SS55nUR*S:XdUR*S:XGaU RSURSnnUbB[&R,"US- [&R.UR0S 9RSS5nO>[&R2"U[&R.UR0S 9RSS5n[&R2"U[&R.UR0S 9RSS5nUU- nUR5UUR6-S- 5nUR9U R:S 9nUR*S:Xa[&R<"S U U5nUU-nOHUR*S:Xa8[&R<"S U U5n[&R<"S UU5nUU-U-nU[>R@"UR5- nUbUU-n[BRDRGUSS 9nURIU5nUbUU-n[&R("UU5nURKSSSS5RM5nURO5SSURP4-nURU5nUU4$)Nrxr$r=rTrrrr{zbhld,lrd->bhlrzbhrd,lrd->bhlrdimrr ))rWrviewrrrV isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdaterZmatmulrutensorrrrrrtor|einsummathsqrtr functionalsoftmaxrpermute contiguousrr)rrrrrrrr batch_sizer_ query_layeris_cross_attentionrcurr_past_key_valuecurrent_states key_layer value_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r1rBertSelfAttention.forwardsO%2$7$7! jj/ !&&z2t7O7OQUQiQijtt q 3$>  %.*=>>+66::4>>J %*8*N*N'*8*M*M'&4#2D.- ."<+224>>BGGI-44T^^DKKK0I!z2t7O7OQUQiQijtt1I**^4K%**B 8 8$:R:Ri1o )7It)<)C)C{DNN=M~<^*& ;&@DN--dnn=!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L)!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC o--r) rrrrrrrrrrurrNNNNNNFNrrrrr~rZrrrr booltuplerrrrs@r1rrs#<7;15=A*.,115d.||d.!!2!23d.E--. d. ((9(9: d. ! d.$D>d.!.d. u|| d.d.rrc^\rSrSrS U4SjjrSS\R S\\R S\\RS\\RS\\ S\\ S \\R S \ \R 4U4S jjjr S r U=r$)BertSdpaSelfAttentioniCc>[TU]XUS9 URUl[R "[ 55[R "S5:Ulg)Nrurz2.2.0)r}r~r dropout_probrparser"require_contiguous_qkvrs r1r~BertSdpaSelfAttention.__init__DsH \ef"??&-mm4E4G&H7==Y`Ka&a#rrrrrrrrrc >URS:wd U(dUb*[RS5 [TU]UUUUUUU5$UR 5upn UR U5RUSURUR5RSS5n USLn U (aUOUn Ub][U[5(aFURRUR5nU (a UR nOUR"nOUnU (aUOUn U (aQUbNW(aGWR$URR&nUR$URR(nOUR+U 5RUSURUR5RSS5nUR-U 5RUSURUR5RSS5nUbNU (dUOSnWR/UUURSU05unnU (aSURUR'UR0(aMU R2R4S:Xa3Ub0U R75n UR75nUR75nUR8=(a U (+=(a USL=(a U S:n[:R<R>RAU UUUURB(a URDOS US 9nURSS5nURGXURH5nUS4$) NrvaBertSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rxr$r=rTcuda) attn_mask dropout_p is_causal)%rurC warning_oncer}rrrrrrrVrrrrrrrrrrrrrrrtyperrrZrrscaled_dot_product_attentiontrainingrreshaper)rrrrrrrrbsztgt_lenrrrrrrrrr attn_outputrs r1rBertSdpaSelfAttention.forwardJs  ' ': 59JiNc   H 7?%! (,,.a JJ} % * *3D4L4LdNfNf g q qrsuv w 3$>2D.-  %.*=>>+66::4>>J %*8*N*N'*8*M*M'&4#2D.- ."<+224>>BGGI-44T^^DKKK(c2t779Q9QR1a   >*c2t779Q9QR1a  )7It)<)C)C{DNN=M~<^*& ;&@DN--dnn=  & &;+=+=+B+Bf+LQ_Qk%002K!,,.I%002K OOi,>(>i>UYCYi^ehi^i hh))FF   $+/==d''c G "++Aq1 !))#8J8JK D  r)rrrrrrs@r1rrCsb2615=A*.,115e!||e!!.e!E--. e! ((9(9: e! ! e!$D>e!!.e! u|| e!e!rrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)BertSelfOutputic(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNrs) r}r~rrrdenserrrrrrs r1r~BertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rr input_tensorrcpURU5nURU5nURX-5nU$Nrrrrrrs r1rBertSelfOutput.forward5 =1  ]3 }'CD rrrr rrrrr~rZrrrrrs@r1rr6> U\\RWR^R^rr)eagersdpac^\rSrSrSU4SjjrSrSS\RS\\RS\\RS\\RS\\ S \\ S \\RS \ \R4S jjr S rU=r$) BertAttentionic>[TU]5 [UR"UUUS9Ul[ U5Ul[5Ulg)Nr) r}r~BERT_SELF_ATTENTION_CLASSES_attn_implementationrroutputset pruned_headsrs r1r~BertAttention.__init__sF /0K0KL $;  %V, Erc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrr$r)rTrrrrr1rrrrr/rrunion)rheadsindexs r1 prune_headsBertAttention.prune_headss  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrrc lURUUUUUUUS9nURUSU5n U 4USS-n U $)Nrrrrrrrr$)rr/) rrrrrrrr self_outputsattention_outputoutputss r1rBertAttention.forwards\yy )"7)/)!  ;;|A F#% QR(88r)r/r1rrr)rrrrr~r7rZrrrr rrrrrrs@r1r+r+s";*7;15=A*.,115||!!2!23E--.  ((9(9:  ! $D>!. u|| rr+cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BertIntermediateic>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr ) r}r~rrrintermediate_sizerr hidden_actstrr intermediate_act_fnrs r1r~BertIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rrrcJURU5nURU5nU$r rrErrs r1rBertIntermediate.forwards& =1 00? rrHr&rs@r1r@r@s(9U\\ellrr@cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) BertOutputi c(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r}r~rrrBrrrrrrrrs r1r~BertOutput.__init__ s` YYv779K9KL f&8&8f>S>STzz&"<"<= rrrrcpURU5nURU5nURX-5nU$r r!r"s r1rBertOutput.forwardr$rr%r&rs@r1rLrL r'rrLc.^\rSrSrSU4SjjrSS\R S\\RS\\RS\\RS\\RS\\ S \\ S \\R S \ \R 4S jjr S r SrU=r$) BertLayericr>[TU]5 URUlSUl[ XS9UlUR UlURUlUR(a/UR (d[US35e[ USUS9Ul [U5Ul [U5Ul g)Nr$rz> should be used as a decoder model if cross attention is addedrvr)r}r~chunk_size_feed_forward seq_len_dimr+ attentionradd_cross_attentionrXcrossattentionr@ intermediaterLr/)rr^rrs r1r~BertLayer.__init__s '-'E'E$&vC ++#)#=#=  # #?? D6)g!hii"/PZfo"pD ,V4 ( rrrrrencoder_attention_maskrrrrc PURUUUUUUS9n U Sn U SSn UR(aDUbA[US5(d[SUS35eUR U UUUUUUS9n U Sn XSS-n [ UR URURU 5n U 4U -n U $)N)rrrrrrr$rYz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r:) rWrrrXrYrfeed_forward_chunkrUrV)rrrrrr\rrrself_attention_outputsr<r=cross_attention_outputs layer_outputs r1rBertLayer.forward's"& )/)) "0" 2!4(, ??4@4!122 =dVDDD '+&9&9 5#&;-"3-':' # 7q9  ;;G0  # #T%A%A4CSCSUe  /G+rcJURU5nURX!5nU$r )rZr/)rr<intermediate_outputras r1r^BertLayer.feed_forward_chunkWs)"//0@A{{#6I r)rXrWrUrYrZrr/rVr )NNNNNFN)rrrrr~rZrrrr rrrr^rrrs@r1rRrRs )"7;15=A>B*.,115.||.!!2!23.E--. . ((9(9: . !)):): ; .!.$D>.!.. u|| .`rrRcv^\rSrSrSU4SjjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \\ S \\ S \\ S \\R S\ \ \R \ 44Sjjr SrU=r$) BertEncoderi]c >[TU]5 Xl[R"[ UR 5Vs/sH n[XS9PM sn5UlSUl gs snf)NrTF) r}r~r^r ModuleListrangenum_hidden_layersrRlayergradient_checkpointing)rr^rirs r1r~BertEncoder.__init__^sS  ]]ERXRjRjLk#lLkqIf$BLk#lm &+#$msA#rrrrr\past_key_values use_cacheroutput_hidden_states return_dictrrc `U (aSOSn U(aSOSn U(aURR(aSOSnUR(a/UR(aU(a[R S5 SnSnU(a]URR (aB[U[5(d-[R S5 Sn[R"U5n[UR5HfunnU (aX4-n UbUUOSnU"UUUUUUUU S9nUSnU(dM7U US4-n URR(dM]UUS 4-nMh U (aX4-n U(aUR5nU (d[S UUU U U455$[UUU U US 9$) Nr.zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.T)r\rrrrr$r=c3.# UH nUcMUv M g7fr r.)r/vs r1r2&BertEncoder.forward..s" As )last_hidden_staterpr attentionscross_attentions)r^rXrmrrCrrrr rfrom_legacy_cache enumeraterlto_legacy_cacherr)rrrrrr\rprqrrrrsrall_hidden_statesall_self_attentionsall_cross_attentionsreturn_legacy_cachern layer_modulelayer_head_mask layer_outputss r1rBertEncoder.forwardds#7BD$5b4%64;;;Z;Zr`d  & &4==##p" # // ?TY8Z8Z   \  #' 1CCOTO(4OA|#$58H$H!.7.CilO(%'=."3- M*!,M  &9]1=MBEI$(,1/4&*15R ||R !!2!23R E--. R ((9(9: R !)):): ; R "%e.?.?(@"ABR D>R $D>R 'tnR d^R !.R  uU\\"$MM NR R rrgcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) BertPooleric>[TU]5 [R"URUR5Ul[R "5Ulgr )r}r~rrrrTanh activationrs r1r~BertPooler.__init__s9 YYv1163E3EF '')rrrc\USS2S4nURU5nURU5nU$)Nr)rr)rrfirst_token_tensor pooled_outputs r1rBertPooler.forwards6+1a40 #56  6 r)rrr&rs@r1rrs($ U\\ellrrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BertPredictionHeadTransformicp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) r}r~rrrrrrCrDr transform_act_fnrrrs r1r~$BertPredictionHeadTransform.__init__s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrrrclURU5nURU5nURU5nU$r )rrrrIs r1r#BertPredictionHeadTransform.forwards4 =1 --m< }5 r)rrrr&rs@r1rrs)UU\\ellrrc4^\rSrSrU4SjrSrSrSrU=r$)BertLMPredictionHeadicH>[TU]5 [U5Ul[R "UR URSS9Ul[R"[R"UR55Ul URURl g)NF)r9) r}r~r transformrrrrdecoder ParameterrZrr9rs r1r~BertLMPredictionHead.__init__sm 4V<yy!3!3V5F5FUS LLV->->!?@ !II rc:URURlgr )r9rrs r1 _tie_weights!BertLMPredictionHead._tie_weightss II rcJURU5nURU5nU$r )rrrIs r1rBertLMPredictionHead.forwards$}5  ]3 r)r9rr) rrrrr~rrrrrs@r1rrs &&rrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BertOnlyMLMHeadicB>[TU]5 [U5Ulgr )r}r~r predictionsrs r1r~BertOnlyMLMHead.__init__s /7rsequence_outputrc(URU5nU$r r)rrprediction_scoress r1rBertOnlyMLMHead.forwards ,,_=  rrr&rs@r1rrs(8!u||! !!rrc.^\rSrSrU4SjrSrSrU=r$)BertOnlyNSPHeadicn>[TU]5 [R"URS5UlgNr=)r}r~rrrseq_relationshiprs r1r~BertOnlyNSPHead.__init__s'  " &*<*[TU]5 [U5Ul[R "UR S5Ulgr)r}r~rrrrrrrs r1r~BertPreTrainingHeads.__init__s4 /7 " &*<*.  - - KK   " " $ MM   $ $S )  4 5 5 KK   " " $6rr.N) rrrrr%__annotations__rmload_tf_weightsbase_model_prefixsupports_gradient_checkpointing_supports_sdparrr.rr1rrs# -O&*#N%rrz0 Output type of [`BertForPreTraining`]. ) custom_introc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'S rg) BertForPreTrainingOutputi+ar loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrryr.)rrrrrrrrZrrrrrrryrr.rr1rr+s~ )-D(5$$ %,59x 1 129;?Xe&7&78?8[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlURUl URUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r}r~r^rorrgencoderrpoolerr.attn_implementationru post_init)rr^add_pooling_layerrs r1r~BertModel.__init__Ssg   (0"6* ,=j(4 #)#>#> '-'E'E$ rc.URR$r rrrs r1get_input_embeddingsBertModel.get_input_embeddingsfs...rc$XRlgr r)rrs r1set_input_embeddingsBertModel.set_input_embeddingsis */'rcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrlrWr7)rheads_to_prunerlr5s r1 _prune_headsBertModel._prune_headsls< +002LE LL  u % / / ; ;E B3rrrrzrwrrrr\rprqrrrrsrrcU bU OURRn U bU OURRn U bU OURRn URR(aU bU OURR n OSn UbUb [ S5eUb"URX5 UR5nO"UbUR5SSnO [ S5eUunnUb URO URnSnU b:[U [5(dU SSRSOU R5nUcs[URS5(a4URR SS2SU24nUR#UU5nUnO$[$R&"U[$R(US9nURUUUUUS 9nUc[$R*"UUU-4US 9nUR,S :H=(a( UR.S :H=(a USL=(a U (+nU(aTUR15S :Xa@URR(a[3UUUU5nO'[5UUR6US9nOUR9X/5nURR(avUbsUR5unnnUU4nUc[$R*"UUS 9nU(a*UR15S :Xa[5UUR6US9nOUR;U5nOSnUR=XPRR>5nURAUUUUUU U U U U US9 nUSnURBbURCU5OSn U (d UU 4USS-$[EUU URFURHURJURLS9$)NFzDYou cannot specify both input_ids and inputs_embeds at the same timerxz5You have to specify either input_ids or inputs_embedsrrrzr)rrwrzrr)rr)rvr=)r) rrrr\rprqrrrrsrr$)rx pooler_outputrprryrz)'r^rrruse_return_dictrrqrX%warn_if_padding_and_no_attention_maskrrrr rWget_seq_lengthrrrzrrZrronesrrurrrr|get_extended_attention_maskinvert_attention_mask get_head_maskrkrrrrprryrz)!rrrrzrwrrrr\rprqrrrrsrrrrrrrrembedding_outputuse_sdpa_attention_masksextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskencoder_outputsrrs! r1rBertModel.forwardts$2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T!"  &"/599 "1%++B/$335 #  !t(899*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z??%)'#9 +   !"ZZZBX5X(YbhiN  $ $ . &,, : &T! &&% ! $(:(:(<(A{{%%*T"$* +'+N"$4$:$:J+' '+&F&F~&c # ;; ! !&;&G=R=W=W=Y :  7$68O#P %-).4HQW)X&',B,F,F,HA,M3V*,<,B,BJ3/372L2LMc2d/.2 +&&y++2O2OP ,, 2"7#B+/!5#)'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   r)rr^rrrru)T)NNNNNNNNNNNNNN)rrrr_no_split_modulesr~rrrr!rrZrlistrrrrrrrrrs@r1rrDs*;7&/0C-11515/3,0048<9==A$(,0/3&*15S ELL)S !.S !. S u||, S ELL) S   -S  ( 5S !) 6S "$u'8'8"9:S D>S $D>S 'tnS d^S !.S uU\\"$PP Q!S S rrz Bert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ S\ \ S\ \ S\ \\ R\44Sjj5rSrU=r$)BertForPreTrainingi predictions.decoder.biascls.predictions.decoder.weightc>[TU]U5 [U5Ul[ U5UlUR 5 gr )r}r~rrrclsrrs r1r~BertForPreTraining.__init__s4  f% '/ rcBURRR$r rrrrs r1get_output_embeddings(BertForPreTraining.get_output_embeddingsxx##+++rcXRRlURURRlgr rrrr9rnew_embeddingss r1set_output_embeddings(BertForPreTraining.set_output_embeddings *'5$$2$7$7!rrrrzrwrrlabelsnext_sentence_labelrrrrsrc  U bU OURRn URUUUUUUU U U S9 n U SSupURX5unnSnUbvUbs[ 5nU"UR SURR 5UR S55nU"UR SS5UR S55nUU-nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForPreTraining.from_pretrained("google-bert/bert-base-uncased") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Nrrzrwrrrrrrsr=rx)rrrrry) r^rrrr rrrrry)rrrrzrwrrrrrrrrsr=rrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossr/s r1rBertForPreTraining.forward$sCV&1%L 'tnL d^L  uU\\"$<< =L L rrzP Bert Model with a `language modeling` head on top for CLM fine-tuning. c$^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ \ RS\ \ S\ \ S\ \ S\ \ S\ \ RS\\\ R\44 Sjj5rSrU=r$)BertLMHeadModelitzcls.predictions.decoder.biasrc>[TU]U5 UR(d[R S5 [ USS9Ul[U5UlUR5 g)NzLIf you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`Fr r}r~rrCwarningrrrrrrs r1r~BertLMHeadModel.__init__|sL    NNi jf> "6* rcBURRR$r r rs r1r %BertLMHeadModel.get_output_embeddingsr rcXRRlURURRlgr rrs r1r%BertLMHeadModel.set_output_embeddingsrrrrrzrwrrrr\rrprqrrrrsrrc UbUOURRnU bSn URUUUUUUUUU U U U UUS9nUSnURU5nSnU b(UR"UXRR 40UD6nU(dU4USS-nUbU4U-$U$[ UUURURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NF) rrzrwrrrr\rprqrrrrsrrr=)rlogitsrprryrz) r^rrr loss_functionrrrprryrz)rrrrzrwrrrr\rrprqrrrrsr loss_kwargsr=rrlm_lossr/s r1rBertLMHeadModel.forwards4&1%@ $D>@ 'tn@ d^@ !.!@ $ uU\\"$EE F%@ @ rr!c^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ S\ \ S\ \ S\ \\ R\44Sjj5rSSjr\S\ 4Sj5rSrU=r$)BertForMaskedLMirrc>[TU]U5 UR(a[R S5 [ USS9Ul[U5UlUR5 g)NzkIf you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr#r$rs r1r~BertForMaskedLM.__init__sR     NN1  f> "6* rcBURRR$r r rs r1r %BertForMaskedLM.get_output_embeddingsr rcXRRlURURRlgr rrs r1r%BertForMaskedLM.set_output_embeddingsrrrrrzrwrrrr\rrrrrsrc U bU OURRn URUUUUUUUUU U U S9 n U SnURU5nSnU bF[ 5nU"UR SURR 5U R S55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` N) rrzrwrrrr\rrrrsrrxr=rr,rry) r^rrrr rrrrry)rrrrzrwrrrr\rrrrrsr=rrrrr/s r1rBertForMaskedLM.forwards.&1% &IIrcg)z Legacy correction: BertForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr.)rs r1 can_generateBertForMaskedLM.can_generate7s rr) NNNNNNNNNNNNr )rrrrrr~r rr!rrZrrrrrrrC classmethodrFrrrs@r1r2r2sj46VW ,8-11515/3,0048<9=)-,0/3&*7 ELL)7 !.7 !. 7 u||, 7 ELL) 7   -7  ( 57 !) 67 &7 $D>7 'tn7 d^7  uU\\"N2 37 7 rJ Trr2zT Bert Model with a `next sentence prediction (classification)` head on top. cl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForNextSentencePredictioni@c>[TU]U5 [U5Ul[ U5UlUR 5 gr )r}r~rrrrrrs r1r~&BertForNextSentencePrediction.__init__Fs4  f% "6* rrrrzrwrrrrrrrsrc SU ;a,[R"S[5 U RS5nU bU OURR n UR UUUUUUUU U S9 n U Sn URU 5nSnUb2[5nU"URSS5URS55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$) a" labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForNextSentencePrediction.from_pretrained("google-bert/bert-base-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` rzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrr$rxr=r:) warningswarn FutureWarningpopr^rrrr rrrry)rrrrzrwrrrrrrrskwargsr=rseq_relationship_scoresrrr/s r1r%BertForNextSentencePrediction.forwardOsT !F * MM%  ZZ 56F%0%Q 'tnQ d^Q  uU\\"$?? @Q Q rrJz Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForSequenceClassificationicr>[TU]U5 URUlXl[ U5UlUR b UR O URn[R"U5Ul [R"URUR5Ul UR5 gr )r}r~ num_labelsr^rrclassifier_dropoutrrrrrrr<rrr^rZrs r1r~&BertForSequenceClassification.__init__s   ++ f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrrzrwrrrrrrrsrc RU bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURR S:Xa=[5nU"U RSUR 5URS55nO,URR S:Xa[5nU"X5nU (dU 4U SS-nUbU4U-$U$[!UU U R"U R$S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr$ regressionsingle_label_classificationmulti_label_classificationrxr=r:)r^rrrr< problem_typerYr|rZrrUr squeezer rrrrry)rrrrzrwrrrrrrrsr=rr,rrr/s r1r%BertForSequenceClassification.forwards(&1%E 'tnE d^E  uU\\"$<< =E E rrWcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForMultipleChoiceic0>[TU]U5 [U5UlURb URO UR n[ R"U5Ul[ R"URS5Ul UR5 g)Nr$) r}r~rrrZrrrrrrr<rr[s r1r~BertForMultipleChoice.__init__su  f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rrrrzrwrrrrrrrsrc ZU bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5n URU 5nURSU 5nSnUb[5nU"X5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr$rxrrr=r:) r^rrWrrrrr<r rrry)rrrrzrwrrrrrrrs num_choicesr=rr,reshaped_logitsrrr/s r1rBertForMultipleChoice.forwardsX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  )) ))%'/!5#     ]3 / ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E("!//))   r)rr<rrU)rrrrr~r!rrZrrrrrrrrrs@r1reres -11515/3,004)-,0/3&*X ELL)X !.X !. X u||, X ELL) X   -X &X $D>X 'tnX d^X  uU\\"$== >X X rrecl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForTokenClassificationincd>[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gNFr#)r}r~rYrrrZrrrrrrr<rr[s r1r~#BertForTokenClassification.__init__ps   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrrzrwrrrrrrrsrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrxr=r:) r^rrrr<r rrYrrry)rrrrzrwrrrrrrrsr=rr,rrr/s r1r"BertForTokenClassification.forward~s$&1%2 'tn2 d^2  uU\\"$99 :2 2 rrmc^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \R\ 44Sjj5r SrU=r$)BertForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gro) r}r~rYrrrrr qa_outputsrrs r1r~!BertForQuestionAnswering.__init__sU   ++f> ))F$6$68I8IJ rrrrzrwrrstart_positions end_positionsrrrrsrc $U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrr$rxr) ignore_indexr=)r start_logits end_logitsrry)r^rrrvrNrbrrTrclampr rrry)rrrrzrwrrrxryrrrrsr=rr,r|r}r ignored_indexr start_lossend_lossr/s r1r BertForQuestionAnswering.forwards&1% ELL)> !.> !. > u||, > ELL) >   -> "%,,/>   -> $D>> 'tn> d^>  uU\\"$@@ A> > rrt) r2rerJrrtrWrmrRr!rrrm)VrrrErN dataclassesrtypingrrrZtorch.utils.checkpoint packagingrrtorch.nnrr r activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsr r!r"r#configuration_bertr% get_loggerrrCrmModulerorrrr-r+r@rLrRrgrrrrrrrrrrr!r2rJrWrermrt__all__r.rr1rs!  !" AA!5)w9   .llLL*   H %FR=RYY=@@. @.Fl!-l!^ RYY  ! 2BII2j ryy   B*BJY "))Y x  "))"299.!bii!&bii& 9299 9%/%%4  :{:  :&  x #x  x v ` ,`  ` F W )?W  W ti)iiX \ $7\  \ ~ V $7V  V rg /g g TB !4B B JJ 2J J Z r