tjlddlmZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m6Z6e/j7e8Z9Gddej:Z; d[dej:dej<dej<dej<dej<dzde=dzde=de(e-fd Z>Gd!d"ej:Z?Gd#d$ej:Z@Gd%d&ej:ZAGd'd(ej:ZBGd)d*ej:ZCGd+d,ej:ZDGd-d.eZEGd/d0ej:ZFGd1d2ej:ZGGd3d4ej:ZHGd5d6ej:ZIe.Gd7d8e&ZJe.d9:Gd;dd?e,ZLGd@dAej:ZMe.dB:GdCdDeJZNGdEdFej:ZOe.dG:GdHdIeJeZPe.GdJdKeJZQGdLdMej:ZRe.dN:GdOdPeJZSe.dQ:GdRdSeJZTe.GdTdUeJZUe.GdVdWeJZVe.GdXdYeJZWgdZZXdS)\)Callable) dataclassN)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) ErnieConfigceZdZdZfdZ d dejdzdejdzdejdzdejdzd ejdzd ed ej fd Z xZ S)ErnieEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cDttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jdd|dt%j|jt$jd|j|_|jr&tj|j|j|_dSdS) N) padding_idxeps position_idsr&F) persistenttoken_type_ids)dtype)super__init__nn Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr.sizelong use_task_idtask_type_vocab_sizetask_type_embeddingsselfconfig __class__s f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/ernie/modeling_ernie.pyr5zErnieEmbeddings.__init__<so !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg    "-   f(* V5PRXRd(e(eD % % % f fNr input_idsr2 task_type_idsr. inputs_embedspast_key_values_lengthreturnc||}n|dd}|\}} ||jdd|| |zf}|t|drT|j|jdd} t j| d|} | || }n+t j|t j |jj }|| |}| |} | | j }|| z} ||} | | z} |jrG|+t j|t j |jj }||}| |z } || } || } | S)Nr0r2rr&)dimindex)r3device)rJr.hasattrr2rHshaperFgatherrIrKr]r;r?tor=rLrNr@rD)rPrUr2rVr.rWrX input_shape batch_size seq_lengthbuffered_token_type_idsr? embeddingsr=rNs rSforwardzErnieEmbeddings.forwardPs  #..**KK',,..ss3K!, J  ,QQQ0FVlIl0l-lmL  !t-.. m*.*=*D*D\EWXYEZ\^*_*_'*/,7NTU]i*j*j*j'!8!?!? J!W!W!&[ SWSdSk!l!l!l   00;;M $ : :> J J&(()>)EFF "%:: "66|DD"55    /$ % KuzRVRcRj k k k #'#<#<]#K#K . .J^^J// \\*-- rT)NNNNNr) __name__ __module__ __qualname____doc__r5rF LongTensor FloatTensorintTensorrg __classcell__rRs@rSr)r)9sQQfffff,.226150426&'33#d*3(4/3'$. 3 &- 3 (4/ 3!$3 33333333rTr)modulequerykeyvalueattention_maskscalingrDkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nr0rr[)ptrainingr&) rJrFmatmul transposer6 functionalsoftmaxrDr contiguous) rsrtrurvrwrxrDry attn_weights attn_outputs rSeager_attention_forwardrs**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$rTc eZdZd fd Z d dejdejdzdedzdee de ejf d Z xZ S) ErnieSelfAttentionFNct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj |j|j |_ tj |j|j |_tj |j|j |_tj|j|_|j|_||_||_dSNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r{)r4r5r9num_attention_headsr^ ValueErrorrQrnattention_head_size all_head_sizerxr6LinearrtrurvrBattention_probs_dropout_probrD is_decoder is_causal layer_idxrPrQrrrRs rSr5zErnieSelfAttention.__init__sG    : :a ? ?PVXhHiHi ?8F$688 4888  #)#= #&v'9F?? O*9*N'&=%C%CI{\`\j%k%k "I{(?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH L((rTFN)NN rhrirjr5rFrormr rr tuplergrprqs@rSrrs######648(, ')')|'))D0') ') +, ') u|  ')')')')')')')')rTrceZdZd fd Z d dejdejdzdejdzdedzdee d e ejf d Z xZ S) ErnieCrossAttentionFNct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj |j|j |_ tj |j|j |_tj |j|j |_tj|j|_||_||_dSr)r4r5r9rr^rrQrnrrrxr6rrtrurvrBrrDrrrs rSr5zErnieCrossAttention.__init__s=    : :a ? ?PVXhHiHi ?8F$688 4888  #)#= #&v'9FB*4>:(?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH L((rTr)NNN) rhrirjr5rFrormr rr rrgrprqs@rSrrs######4;?376: 1)1)|1) %0471))D0 1) -t3 1) +, 1) u| 1)1)1)1)1)1)1)1)rTrcPeZdZfdZdejdejdejfdZxZS)ErnieSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr,) r4r5r6rr9denser@rArBrCrDrOs rSr5zErnieSelfOutput.__init__2sf Yv163EFF f&8f>STTTz&"<== rTr input_tensorrYc||}||}|||z}|SNrrDr@rPrrs rSrgzErnieSelfOutput.forward8@ =11  ]33 }|'CDD rTrhrirjr5rFrorgrprqs@rSrr1i>>>>> U\RWR^rTrceZdZd fd Z d dejdejdzdejdzdejdzdedzd ee d e ejfd Z xZ S)ErnieAttentionFNct||_|rtnt}|||||_t ||_dS)Nrr)r4r5is_cross_attentionrrrProutput)rPrQrrrattention_classrRs rSr5zErnieAttention.__init__@s] "41C[--I[#OFi9UUU %f-- rTrrwrencoder_attention_maskrryrYc v|js|n|}|j|f|||d|\}}|||}||fS)N)rrwr)rrPr) rPrrwrrrryattention_outputrs rSrgzErnieAttention.forwardGsq04/FbLb)2 * "7)+ * *  * * &, ;;'7GG--rT)FNFNNNNrrqs@rSrr?s......48:>;?(, ..|.)D0. %047 . !& 1D 8 .  .+,. u| ........rTrcBeZdZfdZdejdejfdZxZS)ErnieIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r4r5r6rr9intermediate_sizerr hidden_actstrr intermediate_act_fnrOs rSr5zErnieIntermediate.__init__]sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rTrrYcZ||}||}|Sr)rrrPrs rSrgzErnieIntermediate.forwardes, =11 00?? rTrrqs@rSrr\s^99999U\elrTrcPeZdZfdZdejdejdejfdZxZS) ErnieOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r4r5r6rrr9rr@rArBrCrDrOs rSr5zErnieOutput.__init__lsf Yv79KLL f&8f>STTTz&"<== rTrrrYc||}||}|||z}|Srrrs rSrgzErnieOutput.forwardrrrTrrqs@rSrrkrrTrceZdZd fd Z d dejdejdzdejdzdejdzdedzdee d ejfd Z d Z xZ S) ErnieLayerNct|j|_d|_t ||j||_|j|_|j|_|jr1|jst|dt |d|d|_ t||_ t||_ dS)Nr&rz> should be used as a decoder model if cross attention is addedFT)rrr)r4r5chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rPrQrrRs rSr5zErnieLayer.__init__zs '-'E$'&:KW`aaa +#)#=  # ? j D!h!h!hiii"0##' ###D  .f55!&)) rTrrwrrrryrYc |j||fd|i|\}}|} |jr=|;t|dstd|d|j|d||fd|i|\} }| } t |j|j|j| } | S)Nrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rrr^rrrfeed_forward_chunkrr) rPrrwrrrryself_attention_output_rcross_attention_output layer_outputs rSrgzErnieLayer.forwards$24>  $ $ ,$  $ $  q 1 ? 64@4!122  DdDDD )<(;%%& )) !0 )  )) % "A 6 0  #T%A4CSUe  rTc\||}|||}|Sr)rr)rPrintermediate_outputrs rSrzErnieLayer.feed_forward_chunks2"//0@AA{{#68HII rTrr) rhrirjr5rFrormr rr rgrrprqs@rSrrys******,48:>;?(, %%|%)D0% %047 % !& 1D 8 %  %+,% %%%%NrTrcBeZdZfdZdejdejfdZxZS) ErniePoolercttj|j|j|_tj|_dSr)r4r5r6rr9rTanh activationrOs rSr5zErniePooler.__init__sC Yv163EFF '))rTrrYcr|dddf}||}||}|S)Nr)rr)rPrfirst_token_tensor pooled_outputs rSrgzErniePooler.forwards@+111a40 #566  66 rTrrqs@rSrrs^$$$$$ U\elrTrcBeZdZfdZdejdejfdZxZS)ErniePredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) r4r5r6rr9rrrrr transform_act_fnr@rArOs rSr5z%ErniePredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTrTrrYc||}||}||}|Sr)rrr@rs rSrgz$ErniePredictionHeadTransform.forwards= =11 --m<< }55 rTrrqs@rSrrscUUUUUU\elrTrc$eZdZfdZdZxZS)ErnieLMPredictionHeadctt||_t j|j|jd|_t j tj |j|_ dS)NT)bias) r4r5r transformr6rr9r8decoder ParameterrFrIrrOs rSr5zErnieLMPredictionHead.__init__sj 5f==y!3V5FTRRR LV->!?!?@@ rTcZ||}||}|Sr)rrrs rSrgzErnieLMPredictionHead.forwards*}55  ]33 rTrhrirjr5rgrprqs@rSrrsLAAAAArTrceZdZfdZ d dejdejdzdejdzdejdzdedzdedzd e e d e eje zfd Z xZS) ErnieEncoderct|_tjfdt jD|_dS)Nc2g|]}t|S))r)r).0irQs rS z)ErnieEncoder.__init__..s&#m#m#mJv$C$C$C#m#m#mrT)r4r5rQr6 ModuleListrangenum_hidden_layerslayerrOs `rSr5zErnieEncoder.__init__sW  ]#m#m#m#mUSYSkMlMl#m#m#mnn rTNrrwrrr use_cacheryrYc |t|jD]\}} | |||f||d|}t||r|ndS)N)rr)last_hidden_stater) enumeraterr) rPrrwrrrrryr  layer_modules rSrgzErnieEncoder.forwards )44  OA|(L%(> /   MM9+/8BOOd    rT)NNNNN)rhrirjr5rFrormr boolrr rrrgrprqs@rSrrsooooo48:>;?(,!%  | )D0  %047  !& 1D 8    $; +,  u| H H        rTrcpeZdZeZdZdZdZdZdZ dZ e e e dZejfdZxZS)ErniePreTrainedModelernieT)r attentionscross_attentionsct|t|trt j|jdSt|trjt j|j tj |j j d dt j|jdSdS)zInitialize the weightsr0r/N)r4 _init_weightsrrinitzeros_rr)copy_r.rFrGr_rHr2)rPrsrRs rSrz"ErniePreTrainedModel._init_weightss f%%% f3 4 4 / K $ $ $ $ $  0 0 / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h K- . . . . . / /rT)rhrirjr' config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrFno_gradrrprqs@rSrr sL&*#N"&#(/ U]__////_/////rTra  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introcjeZdZdgZdfd ZdZdZeee dde j dzde j dzd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde dzde dzdeedee j ezfdZdZxZS) ErnieModelrTct|||_d|_t ||_t ||_|rt|nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r4r5rQgradient_checkpointingr)rfrencoderrpooler post_init)rPrQadd_pooling_layerrRs rSr5zErnieModel.__init__3s{     &+#)&11#F++ ->Hk&)))D  rTc|jjSrrfr;rPs rSget_input_embeddingszErnieModel.get_input_embeddingsDs ..rTc||j_dSrr2)rPrvs rSset_input_embeddingszErnieModel.set_input_embeddingsGs*/'''rTNrUrwr2rVr.rWrrrrryrYc n|du|duz rtd|jjr| | n |jj} nd} | r[| Y| |jjr6t t |jt |jnt |j} | | nd} ||||||| } | ||| || \}}|j | f|||| | |d| }|d}|j | |nd}t|||j S)  task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] Nz:You must specify exactly one of input_ids or inputs_embedsF)rQr)rUr.r2rVrWrX)rwrembedding_outputrr)rwrrrrr.)r pooler_outputr)rrQrris_encoder_decoderr r get_seq_lengthrf_create_attention_masksr-r.rr)rPrUrwr2rVr.rWrrrrryrXr9encoder_outputssequence_outputrs rSrgzErnieModel.forwardJs0  -t"; < [YZZ Z ; ! %.%:  @UIII  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  FUE`!?!?!A!A!Afg??%)''#9+  261M1M)#9-"7+ 2N2 2 ..'$,   )"7#9+%      *!,8< 8O O444UY ;-'+;    rTc|jjrt|j|||}nt|j||}|t|j|||}||fS)N)rQrWrwr)rQrWrw)rQrWrwr)rQrrr)rPrwrr9rrs rSr=z"ErnieModel._create_attention_maskss ; ! /{.- / NN7{.-N " -%>{.5&; &&& "555rT)T) NNNNNNNNNN)rhrirj_no_split_modulesr5r4r6r$r%r!rFror rrr rrrgr=rprqs@rSr*r*$s&"///000 *..2.2-1,0-1596:(,!%H H <$&H  t+H  t+ H |d* H lT) H |d*H  %|d2H !& t 3H H $;H +,H  u| K KH H H ^_ H T6666666rTr*z1 Output type of [`ErnieForPreTraining`]. ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dS)ErnieForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr) rhrirjrkrDrFrm__annotations__rErFrrrrTrSrCrCs  &*D% d ")))26u(4/6668<U.5<<<59M5*+d299926Je'(4/66666rTrCc$eZdZfdZdZxZS)ErniePreTrainingHeadsctt||_t j|jd|_dSNr|)r4r5r predictionsr6rr9seq_relationshiprOs rSr5zErniePreTrainingHeads.__init__sF 088 " &*eZdZdddZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde j dzde e de e j ezfdZxZS)ErnieForPreTrainingcls.predictions.bias'ernie.embeddings.word_embeddings.weightcls.predictions.decoder.biascls.predictions.decoder.weightct|t||_t ||_|dSr)r4r5r*rrJclsr/rOs rSr5zErnieForPreTraining.__init__sQ    '' (00 rTc$|jjjSrrZrMrr3s rSget_output_embeddingsz)ErnieForPreTraining.get_output_embeddingsx#++rTcT||jj_|j|jj_dSrrZrMrrrPnew_embeddingss rSset_output_embeddingsz)ErnieForPreTraining.set_output_embeddings%'5$$2$7!!!rTNrUrwr2rVr.rWlabelsnext_sentence_labelryrYc |j|f|||||dd| } | dd\} } || | \} }d}||t}|| d|jj|d}||dd|d}||z}t || || j| jS)a: task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trwr2rVr.rW return_dictNr|r0)rDrErFrr) rrZrrrQr8rCrr)rPrUrwr2rVr.rWrerfryoutputsr?rrPrQ total_lossloss_fctmasked_lm_lossnext_sentence_losss rSrgzErnieForPreTraining.forwards,^$*   ))'%'      *1!&48HH_m4\4\11  "5"A'))H%X&7&<&SV8W8Wk~ot444]k -=!!!(;<==  %4%pVFt{OeppioppD0#3!/)$5     rT) NNNNNNNNNNNr)rhrirjrpr5r]rcr#r!rFrolistrrnrr rrrgrprqs@rSrvrvPs+T(>     ,,,888*..2.2-1,0-1596:&*59!%-.= = <$&=  t+=  t+ = |d* = lT) = |d*=  %|d2= !& t 3=  t#= el+d2= $;= el*= +,=  u| @ @= = = ^= = = = = rTrvcTeZdZdddZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde j dzde j dzde e de e j ezfdZxZS)ErnieForMaskedLMrTrUrVct||jrtdt |d|_t||_| dS)NzlIf you want to use `ErnieForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrxryrOs rSr5zErnieForMaskedLM.__init__s~       NN1     %@@@ #F++ rTc$|jjjSrr\r3s rSr]z&ErnieForMaskedLM.get_output_embeddingsr^rTcT||jj_|j|jj_dSrr`ras rSrcz&ErnieForMaskedLM.set_output_embeddingsrdrTNrUrwr2rVr.rWrrreryrYc B|j|f|||||||dd| } | d} || } d}| Kt}|| d|jj| d}t || | j| jS)as task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rwr2rVr.rWrrrirNr0rDrrr) rrZrrrQr8rrr)rPrUrwr2rVr.rWrrreryrjr?rPrmrls rSrgzErnieForMaskedLM.forwards4$*   ))'%'"7#9      "!* HH_55  '))H%X&7&<&>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trhr&Nr0r|r)rrZrrrrr)rPrUrwr2rVr.rWreryrjrseq_relationship_scoresrnrls rSrgz&ErnieForNextSentencePrediction.forwardsZ$*   ))'%'        "&((="9"9!  '))H!)*A*F*Fr1*M*Mv{{[]!_!_ *#*!/)     rTNNNNNNN)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrs- *..2.2-1,0-1&*D D <$&D  t+D  t+ D |d* D lT) D |d*D  t#D +,D  u| : :D D D ^D D D D D rTrz Ernie Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForSequenceClassificationcdt||j|_||_t ||_|j|jn|j}tj ||_ tj |j |j|_ |dSr)r4r5 num_labelsrQr*rclassifier_dropoutrCr6rBrDrr9 classifierr/rPrQrrRs rSr5z'ErnieForSequenceClassification.__init__ms     + '' )/)B)NF % %TZTn z"455 )F$68IJJ rTNrUrwr2rVr.rWreryrYc |j|f|||||dd|} | d} || } || } d} |Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt} |jdkr1| | | } n| | |} n|jjdkrGt} | | d|j| d} n*|jjdkrt} | | |} t| | | j| j S) a^ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trhr&N regressionsingle_label_classificationmulti_label_classificationr0r)rrDrrQ problem_typerr3rFrKrnrsqueezerrrrrr)rPrUrwr2rVr.rWreryrjrrrDrls rSrgz&ErnieForSequenceClassification.forward|s0$*   ))'%'         ]33 //  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     rTr)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrfs-     *..2.2-1,0-1&*B B <$&B  t+B  t+ B |d* B lT) B |d*B  t#B +,B  u| 7 7B B B ^B B B B B rTrceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForMultipleChoicec4t|t||_|j|jn|j}t j||_t j |j d|_ | dS)Nr&) r4r5r*rrrCr6rBrDrr9rr/rs rSr5zErnieForMultipleChoice.__init__s    '' )/)B)NF % %TZTn z"455 )F$6:: rTNrUrwr2rVr.rWreryrYc T| |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}|j|f|||||dd|} | d} || } || } | d| } d}|t }|| |}t|| | j| j S)a9 input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr&r0Trhr) r_rrJrrDrrrrr)rPrUrwr2rVr.rWrery num_choicesrjrrreshaped_logitsrDrls rSrgzErnieForMultipleChoice.forwards`-6,Aioa((}GZ[\G] >G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  $*   ))'%'         ]33 // ++b+66  '))H8OV44D("!/)     rTr)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrs-     *..2.2-1,0-1&*U U <$&U  t+U  t+ U |d* U lT) U |d*U  t#U +,U  u| 8 8U U U ^U U U U U rTrceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSNFrx)r4r5rr*rrrCr6rBrDrr9rr/rs rSr5z$ErnieForTokenClassification.__init__.s     +%@@@ )/)B)NF % %TZTn z"455 )F$68IJJ rTNrUrwr2rVr.rWreryrYc ^|j|f|||||dd|} | d} || } || } d} |Ft} | | d|j|d} t | | | j| jS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrhrNr0r) rrDrrrrrrr)rPrUrwr2rVr.rWreryrjr?rrDrls rSrgz#ErnieForTokenClassification.forward<s,$*   ))'%'      "!*,,7711  '))H8FKKDO<rs*%$$$$$!!!!!! AAAAAAAAAA&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999                      GFFFFFFF&&&&&&666666MMMMMMMMMMMMIIIIIIII555555,,,,,,  H % %JJJJJbiJJJf!%% I% <% % < % L4' % T\ %%'(%%%%8@)@)@)@)@)@)@)@)FI)I)I)I)I)")I)I)I)X     bi   .....RY...:             ")   >>>>>+>>>B     ")   29"     BI         29   @/////?///2    E6E6E6E6E6%E6E6  E6P  77777 77   7& 9 9 9 9 9BI 9 9 9 ` ` ` ` ` .` `  ` F!!!!!ry!!! X X X X X +_X X  X vP P P P P +P P P f&&&&&ry&&& P P P P P %9P P  P f T T T T T %9T T  T ne e e e e 1e e e P? ? ? ? ? "6? ? ? DI I I I I 4I I I X   rT