tjdZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZmZdd lmZmZmZdd lmZdd lmZmZdd lmZddlmZmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5ddl6m7Z7e0j8e9Z:Gddej;Z< dWdej;dej=dej=dej=dej=dzde>dzd e>d!e)e.fd"Z?Gd#d$ej;Z@Gd%d&ej;ZAGd'd(ej;ZBGd)d*ej;ZCGd+d,ej;ZDGd-d.ej;ZEGd/d0eZFGd1d2ej;ZGGd3d4ej;ZHGd5d6ej;ZIe/Gd7d8e'ZJe/d9:eGd;deJZLGd?d@ej;ZMGdAdBej;ZNe/dC:GdDdEeJZOe/dF:GdGdHeJZPe/dI:GdJdKeJZQe/dL:GdMdNeJZRe/GdOdPeJZSe/GdQdReJZTe/dS:GdTdUeJeZUgdVZVdS)XzPyTorch ELECTRA model.)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNget_activation)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentions)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) ElectraConfigc eZdZdZfdZ d dejdzdejdzdejdzdejdzd ed ej f d Z xZ S) ElectraEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jdd|dt%j|jt$jddS) N) padding_idxeps position_idsr'F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr/sizelongselfconfig __class__s j/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/electra/modeling_electra.pyr7zElectraEmbeddings.__init__;s5 !|F,=v?Tbhbuvvv#%<0NPVPe#f#f %'\&2H&J_%`%`"f&;AVWWWz&"<==   EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg      Nr input_idsr3r/ inputs_embedspast_key_values_lengthreturnc||}n|dd}|\}}||jdd|||zf}|t|drT|j|jdd} t j| d|} | ||}n+t j|t j |jj }|| |}| |} || z} | |} | | z} || } || } | S)Nr1r3rr')dimindex)r5device)rKr/hasattrr3rIshaperGgatherrJrLrZr<r@r>rArE) rNrSr3r/rTrU input_shape batch_size seq_lengthbuffered_token_type_idsr@ embeddingsr>s rQforwardzElectraEmbeddings.forwardMsu  #..**KK',,..ss3K!, J  ,QQQ0FVlIl0l-lmL  !t-.. m*.*=*D*D\EWXYEZ\^*_*_'*/,7NTU]i*j*j*j'!8!?!? J!W!W!&[ SWSdSk!l!l!l   00;;M $ : :> J J"%:: "66|DD"55 ^^J// \\*-- rR)NNNNr) __name__ __module__ __qualname____doc__r7rG LongTensor FloatTensorintTensorrc __classcell__rPs@rQr*r*8sQQ     (.2260426&' ((#d*((4/(&- ( (4/ ( !$ ( ((((((((rRr*modulequerykeyvalueattention_maskscalingrEkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nr1r rX)ptrainingr') rKrGmatmul transposer functionalsoftmaxrEr{ contiguous) rorprqrrrsrtrEru attn_weights attn_outputs rQeager_attention_forwardrys**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$rRc eZdZd fd Z d dejdejdzdedzdee de ejf d Z xZ S) ElectraSelfAttentionFNct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj |j|j |_ tj |j|j |_tj |j|j |_tj|j|_|j|_||_||_dSNrr:zThe hidden size (z6) is not a multiple of the number of attention heads ()rw)r6r7 hidden_sizenum_attention_headsr[ ValueErrorrOrjattention_head_size all_head_sizertrLinearrprqrrrCattention_probs_dropout_probrE is_decoder is_causal layer_idxrNrOrrrPs rQr7zElectraSelfAttention.__init__sG    : :a ? ?PVXhHiHi ?8F$688 4888  #)#= #&v'9F?? O*9*N'&=%C%CI{\`\j%k%k "I{(?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH L((rRFN)NN rdrerfr7rGrkrir rr!tuplercrlrms@rQrrs######648(, ')')|'))D0') ') +, ') u|  ')')')')')')')')rRrceZdZd fd Z d dejdejdzdejdzdedzdee d e ejf d Z xZ S) ElectraCrossAttentionFNct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj |j|j |_ tj |j|j |_tj |j|j |_tj|j|_||_||_dSr)r6r7rrr[rrOrjrrrtrrrprqrrrCrrErrrs rQr7zElectraCrossAttention.__init__s=    : :a ? ?PVXhHiHi ?8F$688 4888  #)#= #&v'9FB*4>:(?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH L((rRr)NNN) rdrerfr7rGrkrirrr!rrcrlrms@rQrrs######4;?376: 1)1)|1) %0471))D0 1) -t3 1) +, 1) u| 1)1)1)1)1)1)1)1)rRrcPeZdZfdZdejdejdejfdZxZS)ElectraSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr-) r6r7rrrdenserArBrCrDrErMs rQr7zElectraSelfOutput.__init__(sf Yv163EFF f&8f>STTTz&"<== rRr input_tensorrVc||}||}|||z}|SNrrErArNrrs rQrczElectraSelfOutput.forward.@ =11  ]33 }|'CDD rRrdrerfr7rGrkrcrlrms@rQrr'i>>>>> U\RWR^rRrceZdZd fd Z d dejdejdzdejdzdejdzdedzd ee d e ejfd Z xZ S)ElectraAttentionFNct||_|rtnt}|||||_t ||_dS)Nrr)r6r7is_cross_attentionrrrNroutput)rNrOrrrattention_classrPs rQr7zElectraAttention.__init__7s] "43E_//K_#OFi9UUU '// rRrrsrencoder_attention_maskrrurVc v|js|n|}|j|f|||d|\}}|||}||fS)N)rrsr)rrNr) rNrrsrrrruattention_outputrs rQrczElectraAttention.forward>sq04/FbLb)2 * "7)+ * *  * * &, ;;'7GG--rR)FNFNNNNrrms@rQrr6s00000048:>;?(, ..|.)D0. %047 . !& 1D 8 .  .+,. u| ........rRrcBeZdZfdZdejdejfdZxZS)ElectraIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r6r7rrrintermediate_sizerr hidden_actstrr intermediate_act_fnrMs rQr7zElectraIntermediate.__init__Usn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rRrrVcZ||}||}|Sr)rr)rNrs rQrczElectraIntermediate.forward]s, =11 00?? rRrrms@rQrrTs^99999U\elrRrcPeZdZfdZdejdejdejfdZxZS) ElectraOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r6r7rrrrrrArBrCrDrErMs rQr7zElectraOutput.__init__esf Yv79KLL f&8f>STTTz&"<== rRrrrVc||}||}|||z}|Srrrs rQrczElectraOutput.forwardkrrRrrms@rQrrdrrRrceZdZd fd Z d dejdejdzdejdzdejdzdedzdee d ejfd Z d Z xZ S) ElectraLayerNct|j|_d|_t ||j||_|j|_|j|_|jr1|jst|dt |d|d|_ t||_ t||_ dS)Nr'rz> should be used as a decoder model if cross attention is addedFT)rrr)r6r7chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rNrOrrPs rQr7zElectraLayer.__init__ts '-'E$)&F  $ $ ,$  $ $  q 1 ? 64@4!122  DdDDD )<(;%%& )) !0 )  )) % "A 6 0  #T%A4CSUe  rRc\||}|||}|Sr)rr)rNrintermediate_outputrs rQrzElectraLayer.feed_forward_chunks2"//0@AA{{#68HII rRrr) rdrerfr7rGrkrir rr!rcrrlrms@rQrrss,,,,,,,48:>;?(, %%|%)D0% %047 % !& 1D 8 %  %+,% %%%%NrRrceZdZfdZ d dejdejdzdejdzdejdzdedzdedzd e e d e eje zfd Z xZS) ElectraEncoderct|_tjfdt jD|_dS)Nc2g|]}t|S))r)r).0irOs rQ z+ElectraEncoder.__init__..s&#o#o#o!L1$E$E$E#o#o#orR)r6r7rOr ModuleListrangenum_hidden_layerslayerrMs `rQr7zElectraEncoder.__init__sW  ]#o#o#o#ouU[UmOnOn#o#o#opp rRNrrsrrr use_cacherurVc |t|jD]\}} | |||f||d|}t||r|ndS)N)rrlast_hidden_stater) enumeraterr) rNrrsrrrrrur layer_modules rQrczElectraEncoder.forwards )44  OA|(L%(> /   MM9+/8BOOd    rR)NNNNN)rdrerfr7rGrkrir boolrr!rrrcrlrms@rQrrsqqqqq48:>;?(,!%  | )D0  %047  !& 1D 8    $; +,  u| H H        rRrc(eZdZdZfdZdZxZS)ElectraDiscriminatorPredictionszEPrediction module for the discriminator, made up of two dense layers.cttj|j|j|_t |j|_tj|jd|_ ||_ dSNr') r6r7rrrrr r activationdense_predictionrOrMs rQr7z(ElectraDiscriminatorPredictions.__init__sf Yv163EFF ():;; " &*??  66 &&}55==bAA rRrdrerfrgr7rcrlrms@rQrrsMOOrRrc(eZdZdZfdZdZxZS)ElectraGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.cttd|_t j|j|j|_t j|j |j|_ dS)Ngelur-) r6r7r rrrAr:rBrrrrMs rQr7z$ElectraGeneratorPredictions.__init__sa (00f&;AVWWWYv163HII rRc||}||}||}|Sr)rrrA)rNgenerator_hidden_statesrs rQrcz#ElectraGeneratorPredictions.forwards< #:;;  66 }55 rRrrms@rQrrsRKKJJJJJrRrcFeZdZeZdZdZdZdZdZ dZ e e e dZfdZxZS)ElectraPreTrainedModelelectraT)r attentionscross_attentionscHt|t|trjt j|jtj|jj d dt j |j dSdS)Nr1r0) r6 _init_weightsrr*initcopy_r/rGrHr\rIzeros_r3)rNrorPs rQrz$ElectraPreTrainedModel._init_weights s f%%% f/ 0 0 / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h K- . . . . . / /rR)rdrerfr( config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrrlrms@rQr r sv L!&*#N"&%*1 /////////rRr z3 Output type of [`ElectraForPreTraining`]. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ e ejdzed<dZ e ejdzed<dS)ElectraForPreTrainingOutputa+ loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss of the ELECTRA objective. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Prediction scores of the head (scores for each token before SoftMax). Nlossrrr) rdrerfrgr!rGri__annotations__rrrrrRrQr r s&*D% d ")))'+FE  $+++59M5*+d299926Je'(4/66666rRr cbeZdZfdZdZdZeee dde j dzde j dzde j dzde j dzd e j dzd e j dzd e j dzd e e j dzd e dzdeedee j ezfdZdZxZS) ElectraModelcFt|t||_|j|jkr$t j|j|j|_t||_ ||_ d|_ | dS)NF)r6r7r*rbr:rrrembeddings_projectrencoderrOgradient_checkpointing post_initrMs rQr7zElectraModel.__init__&s    +F33  F$6 6 6&(i0EvGY&Z&ZD #%f--  &+# rRc|jjSrrbr<rNs rQget_input_embeddingsz!ElectraModel.get_input_embeddings3s ..rRc||j_dSrr,)rNrrs rQset_input_embeddingsz!ElectraModel.set_input_embeddings6s*/'''rRNrSrsr3r/rTrrrrrurVc r|du|duz rtd|jjr| | n |jj} nd} | r[|Y| |jjr6t t |jt |jnt |j}||nd} |||||| } t|dr| | } | ||| ||\}}|j | f||||| |d| } t| j| j S) Nz:You must specify exactly one of input_ids or inputs_embedsF)rOr)rSr/r3rTrUr')rsrembedding_outputrr)rsrrrrr/r)rrOrris_encoder_decoderrrget_seq_lengthrbr[r'_create_attention_masksr(rrr)rNrSrsr3r/rTrrrrrurUr2encoder_outputss rQrczElectraModel.forward9s  -t"; < [YZZ Z ; ! %.%:  @UIII  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  FUE`!?!?!A!A!Afg??%)'#9 +   4- . . I#667GHH 151M1M)#9-"7+ 2N2 2 ..'$,   )"7#9+%      9-?+;    rRc|jjrt|j|||}nt|j||}|t|j|||}||fS)N)rOrTrsr)rOrTrs)rOrTrsr)rOrrr)rNrsrr2rrs rQr5z$ElectraModel._create_attention_masks}s ; ! /{.- / NN7{.-N " -%>{.5&; &&& "555rR) NNNNNNNNN)rdrerfr7r.r0r%r&r"rGrklistrirrr!rrrcr5rlrms@rQr%r%$s~     ///000 *..2.2,0-1596::>!%> > <$&>  t+>  t+ > lT) > |d* >  %|d2> !& t 3> e/047> $;> +,>  u| A A> > > ^_ > B6666666rRr%c(eZdZdZfdZdZxZS)ElectraClassificationHeadz-Head for sentence-level classification tasks.c\ttj|j|j|_|j|jn|j}td|_ tj ||_ tj|j|j |_ dS)Nr )r6r7rrrrclassifier_dropoutrDr rrCrE num_labelsout_projrNrOr<rPs rQr7z"ElectraClassificationHead.__init__s Yv163EFF )/)B)NF % %TZTn )00z"455  &"4f6GHH rRc |dddddf}||}||}||}||}||}|S)Nr)rErrr>)rNfeaturesruxs rQrcz!ElectraClassificationHead.forwardsn QQQ111W  LLOO JJqMM OOA   LLOO MM!  rRrrms@rQr:r:sR77IIIIIrRr:cdeZdZdZdeffd Z d dejdejdzdejfdZ xZ S) ElectraSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`ElectraConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. rOcVtt|dd|_|jdkrtt j|_t|drW|j rPt|dr|j r|j dkr|j }n|j }t j |j ||_t|dd}|rt|nt j|_t j|_t|dr)|jdkrt j|j|_t j|_t|d r+|jdkr"t j|j|_dSdSdS) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)r6r7getattrrFNotImplementedErrorrIdentitysummaryr[rIrJr=rrr r first_dropoutrLrC last_dropoutrM)rNrO num_classesactivation_stringrPs rQr7zElectraSequenceSummary.__init__s #FNFCC   & && %{}} 6- . . F63J Fv788 1V=Z 1_e_pst_t_t$/ $0 9V%7EEDL#F,@$GGIZ$mN3D$E$E$E`b`k`m`m[]] 62 3 3 J8TWX8X8X!#F,H!I!ID KMM 61 2 2 Hv7RUV7V7V " 6+F G GD    H H7V7VrRNr cls_indexrVc:|jdkr|dddf}n-|jdkr|dddf}n|jdkr|d}n|jd kr|=tj|d ddddf|jd dz tj }nl|dd}|d |dz z| dfz}| d | d }n|jdkrt| |}||}||}||}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. rGNr1firstrmeanr'ryrV.r4)r1rH)rFrYrG full_liker\rL unsqueezerIrXrKr]rrOrRrQrrS)rNrrVrs rQrczElectraSequenceSummary.forwards   & &"111b5)FF  ' ) )"111a4(FF  & ( ("''A'..FF  + - - !O!#rr111*-!'+a/* &//33==bAA %,,Uimmoo6I-JmN`N`acNdNdMf-fgg "))"i88@@DDFF  & ( (% %##F++f%%((""6** rRr) rdrerfrgr(r7rGrirhrcrlrms@rQrDrDs2H}HHHHHH<VZ))".);@;Kd;R)  ))))))))rRrDz ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) ElectraForSequenceClassificationct||j|_||_t ||_t ||_|dSr) r6r7r=rOr%rr: classifierr*rMs rQr7z)ElectraForSequenceClassification.__init__ sb     + #F++ 3F;; rRNrSrsr3r/rTlabelsrurVc \|j|f||||dd|}|d} || } d} |Z|jjf|jdkr d|j_nN|jdkr7|jt jks|jt jkr d|j_n d|j_|jjdkrWt} |jdkr1| | | } n| | |} n|jjdkrGt} | | d |j| d } n*|jjdkrt} | | |} t| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trsr3r/rT return_dictrNr' regressionsingle_label_classificationmulti_label_classificationr1r!rrr)rr`rO problem_typer=r5rGrLrjrrrrrrrr) rNrSrsr3r/rTrarursequence_outputrr!loss_fcts rQrcz(ElectraForSequenceClassification.forward*s$'3dl ' ))%' ' ' ' ' #6a811  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'5C2=     rRNNNNNN)rdrerfr7r$r"rGrkrr!rrrcrlrms@rQr^r^s*..2.2,0-1&*9 9 <$&9  t+9  t+ 9 lT) 9 |d* 9  t#9 +,9  u| 7 79 9 9 ^9 9 9 9 9 rRr^z Electra model with a binary classification head on top as used during pretraining for identifying generated tokens. It is recommended to load the discriminator checkpoint into that model. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) ElectraForPreTrainingct|t||_t ||_|dSr)r6r7r%rrdiscriminator_predictionsr*rMs rQr7zElectraForPreTraining.__init__psP    #F++ )H)P)P& rRNrSrsr3r/rTrarurVc 0|j|f||||dd|}|d} || } d} |tj} |s|d| jddk} | d| jd| }|| }| ||} n=| | d| jd|} t| | |j|j S)am labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the ELECTRA loss. Input should be a sequence of tokens (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates the token is an original token, - 1 indicates the token was replaced. Examples: ```python >>> from transformers import ElectraForPreTraining, AutoTokenizer >>> import torch >>> discriminator = ElectraForPreTraining.from_pretrained("google/electra-base-discriminator") >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-discriminator") >>> sentence = "The quick brown fox jumps over the lazy dog" >>> fake_sentence = "The quick brown fox fake over the lazy dog" >>> fake_tokens = tokenizer.tokenize(fake_sentence, add_special_tokens=True) >>> fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt") >>> discriminator_outputs = discriminator(fake_inputs) >>> predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2) >>> fake_tokens ['[CLS]', 'the', 'quick', 'brown', 'fox', 'fake', 'over', 'the', 'lazy', 'dog', '[SEP]'] >>> predictions.squeeze().tolist() [0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0] ```TrcrNr1r'rh) rrprrrr\floatr rr)rNrSrsr3r/rTrarurdiscriminator_sequence_outputrr!rk active_loss active_logits active_labelss rQrczElectraForPreTraining.forwardxsPV'3dl ' ))%' ' ' ' ' #)DA(F%//0MNN  +--H),11"6S6YZ[6\]]abb & B0M0STU0V W WXc d &{ 3 x }/B/B/D/DEEx B0M0STU0V W WY_YeYeYgYghh*5C2=     rRrl)rdrerfr7r$r"rGrkrr!rr rcrlrms@rQrnrnhs*..2.2,0-1&*F F <$&F  t+F  t+ F lT) F |d* F  t#F +,F  u| : :F F F ^F F F F F rRrnz Electra model with a language modeling head on top. Even though both the discriminator and generator may be loaded into this model, the generator is the only model of the two to have been trained for the masked language modeling task. ceZdZddiZfdZdZdZee dde j dzde j dzd e j dzd e j dzd e j dzd e j dzd e e de e j ezfdZxZS)ElectraForMaskedLMgenerator_lm_head.weight)electra.embeddings.word_embeddings.weightc t|t||_t ||_t j|j|j |_ | dSr) r6r7r%rrgenerator_predictionsrrr:r9generator_lm_headr*rMs rQr7zElectraForMaskedLM.__init__sj    #F++ %@%H%H"!#6+@&BS!T!T rRc|jSrr}r-s rQget_output_embeddingsz(ElectraForMaskedLM.get_output_embeddings %%rRc||_dSrr)rNr<s rQset_output_embeddingsz(ElectraForMaskedLM.set_output_embeddingss!0rRNrSrsr3r/rTrarurVc p|j|f||||dd|}|d} || } || } d} |Ptj} | | d|jj|d} t| | |j |j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` TrcrNr1rh) rr|r}rrrrOr9rrr) rNrSrsr3r/rTrarur generator_sequence_outputprediction_scoresr!rks rQrczElectraForMaskedLM.forwards$#/$, # ))%' # # # # %! 667PQQ 223DEE  *,,H8-222t{7MNNPVP[P[\^P_P_``D$1?.9     rRrl)rdrerf_tied_weights_keysr7rrr$r"rGrkrr!rrrcrlrms@rQrxrxs256ab&&&111*..2.2,0-1&*) ) <$&)  t+)  t+ ) lT) ) |d* )  t#) +,)  u| ~ -) ) ) ^) ) ) ) ) rRrxz Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) ElectraForTokenClassificationcVt||j|_t||_|j|jn|j}tj||_ tj |j |j|_ | dSr)r6r7r=r%rr<rDrrCrErrr`r*r?s rQr7z&ElectraForTokenClassification.__init__s     +#F++ )/)B)NF % %TZTn z"455 )F$68IJJ rRNrSrsr3r/rTrarurVc \|j|f||||dd|}|d} || } || } d} |Ft} | | d|j|d} t | | |j|jS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrcrNr1rh) rrEr`rrr=rrr) rNrSrsr3r/rTrarurrsrr!rks rQrcz%ElectraForTokenClassification.forward!s '3dl ' ))%' ' ' ' ' #)DA(F%(, 5R(S(S%!>??  '))H8FKKDO< >)F$6:: rRNrSrsr3r/rTrarurVc R| |jdn |jd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}|j|f||||dd|} | d} || } || } | d|} d}|t }|| |}t|| | j| j S) a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr'r1rZTrcrrh) r\rrKrrr`rrrr)rNrSrsr3r/rTraru num_choicesrrj pooled_outputrreshaped_logitsr!rks rQrcz ElectraForMultipleChoice.forwardsT-6,Aioa((}GZ[\G] >G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  '3dl ' ))%' ' ' ' ' #6a8--o>> // ++b+66  '))H8OV44D("5C2=     rRrl)rdrerfr7r$r"rGrkrr!rrrcrlrms@rQrrs*..2.2,0-1&*N N <$&N  t+N  t+ N lT) N |d* N  t#N +,N  u| 8 8N N N ^N N N N N rRrzS ELECTRA Model with a `language modeling` head on top for CLM fine-tuning. cjeZdZddiZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde j dzde dzde dzde e j zdeedee j ezfdZxZS)ElectraForCausalLMryrzcLt||jstdt ||_t||_tj |j |j |_ |dS)NzOIf you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.`)r6r7rloggerwarningr%rrr|rrr:r9r}r*rMs rQr7zElectraForCausalLM.__init__s      n NNl m m m#F++ %@%H%H"!#6+@&BS!T!T rRc|jSrrr-s rQrz(ElectraForCausalLM.get_output_embeddingsrrRc||_dSrr)rNnew_embeddingss rQrz(ElectraForCausalLM.set_output_embeddings s!/rRNrrSrsr3r/rTrrrarrlogits_to_keeprurVc |d} |j|f||||||| | dd | } | j}t| trt | dn| }|||dd|ddf}d}||jd|||jj d| }t||| j | j | j | jS)a3 labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, ElectraForCausalLM, ElectraConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-generator") >>> config = ElectraConfig.from_pretrained("google/electra-base-generator") >>> config.is_decoder = True >>> model = ElectraForCausalLM.from_pretrained("google/electra-base-generator", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rsr3r/rTrrrrrd)rrar9)r!rrrrrr#)rrrrjslicer}r| loss_functionrOr9rrrrr)rNrSrsr3r/rTrrrarrrruoutputsr slice_indicesrr!s rQrczElectraForCausalLM.forward s)N  I=IT\  > ))%'"7#9+ > >  > >  1 8B>SV8W8Wk~ot444]k ''(B(B=QRQRQRTacdcdcdQdCe(f(fgg  %4%pVFt{OeppioppD0#3!/)$5     rR) NNNNNNNNNNr)rdrerfrr7rrr$r"rGrkr rrjrr!rrrcrlrms@rQrrs 56ab     &&&000*..2.2,0-1596:&*(,!%-.F F <$&F  t+F  t+ F lT) F |d* F  %|d2F !& t 3F  t#F F $;F el*F +,F  u| @ @F F F ^F F F F F rRr) rrxrrnrr^rr%r )Nrn)Wrgcollections.abcr dataclassesrrGrtorch.nnrrrr r activationsr r cache_utilsr rr generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsr r!r"r# utils.genericr$r%utils.output_capturingr&configuration_electrar( get_loggerrdrModuler*rkrrrrrrrrrrrrrr r r%r:rDr^rnrxrrrr__all__r#rRrQrs$$$$$$!!!!!! AAAAAAAAAA&&&&&&11111111CCCCCCCCCC))))))JJJJJJJJ999999                    GFFFFFFF&&&&&&666666 JIIIIIII555555000000  H % %===== ===N!%% I% <% % < % L4' % T\ %%'(%%%%:@)@)@)@)@)29@)@)@)HI)I)I)I)I)BII)I)I)Z        .....ry...<     ")        BI   >>>>>->>>D     RY   @bi&")$/////_///*   7 7 7 7 7+ 7 7   7v6v6v6v6v6)v6v6v6r 0`````RY```F F F F F F '=F F  F RQ Q Q Q Q 2Q Q Q h> > > > > /> > > B6 6 6 6 6 $:6 6 6 rC C C C C "8C C C L[ [ [ [ [ 5[ [ [ | ] ] ] ] ] /] ]  ] @   rR