tjddlmZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e%j-e.Z/Gddej0Z1ej2e1dZ3Gddej0Z4 d`dej0dej5dej5dej5dej5dzde6dzde6d e e#fd!Z7Gd"d#ej0Z8Gd$d%ej0Z9Gd&d'ej0Z:Gd(d)ej0Z;Gd*d+ej0Z<Gd,d-ej0Z=Gd.d/ej0Z>Gd0d1ej0Z?Gd2d3ej0Z@Gd4d5ej0ZAGd6d7eZBGd8d9ej0ZCGd:d;ej0ZDGd<d=ej0ZEGd>d?ej0ZFGd@dAej0ZGGdBdCej0ZHe$GdDdEeZIe$dFGeGdHdIe"ZJe$GdJdKeIZKe$dLGGdMdNeIZLe$GdOdPeIZMGdQdRej0ZNe$dSGGdTdUeIZOe$dVGGdWdXeIZPe$GdYdZeIZQe$Gd[d\eIZRe$Gd]d^eIZSgd_ZTdS)a)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)MobileBertConfigcDeZdZdfd ZdejdejfdZxZS)NoNormNcttjt j||_tjt j||_dSN) super__init__r Parametertorchzerosbiasonesweight)self feat_sizeeps __class__s p/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mobilebert/modeling_mobilebert.pyr'zNoNorm.__init__8sS LY!7!788 l5:i#8#899 input_tensorreturnc&||jz|jzSr%)r-r+)r.r4s r2forwardzNoNorm.forward=sdk)DI55r3r%__name__ __module__ __qualname__r'r)Tensorr7 __classcell__r1s@r2r#r#7sc:::::: 6EL6U\66666666r3r#) layer_normno_normc eZdZdZfdZ d dejdzdejdzdejdzdejdzdejf d Z xZ S) MobileBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|j|_|j|_|j|_t j|j|j|j|_ t j|j |j|_ t j|j |j|_ |jrdnd}|j|z}t j||j|_t!|j|j|_t j|j|_|dt/j|j dddS)N) padding_idxr r position_idsr F) persistent)r&r' trigram_inputembedding_size hidden_sizer Embedding vocab_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsLinearembedding_transformationNORM2FNnormalization_type LayerNormDropouthidden_dropout_probdropoutregister_bufferr)arangeexpand)r.configembed_dim_multiplierembedded_input_sizer1s r2r'zMobileBertEmbeddings.__init__GsF #1$3!-!|F,=v?Tbhbuvvv#%<0NPVPb#c#c %'\&2H&J\%]%]"$($6=qqA"14HH(* 2EvGY(Z(Z% !:;F J J"%88;PP ^^J// \\*-- r3)NNNN) r9r:r;__doc__r'r) LongTensor FloatTensorr<r7r=r>s@r2rBrBDsQQ     0.2260426 00#d*0(4/0&- 0 (4/ 0  00000000r3rBrhmodulequerykeyriattention_maskscalingr[kwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)NrGrjr rk)ptrainingr ) rmr)matmul transposerrpsoftmaxr[r contiguous) rxryrzrir{r|r[r} attn_weights attn_outputs r2eager_attention_forwardrs**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r3ceZdZfdZ d dejdejdejdejdzdeede ejf d Z xZ S) MobileBertSelfAttentionc4t||_|j|_t |j|jz |_|j|jz|_|jdz|_tj |j|j|_ tj |j|j|_ tj |j r|jn|j|j|_tj|j|_d|_dS)NrF)r&r'r_num_attention_headsinttrue_hidden_sizeattention_head_size all_head_sizer|rrTryrzuse_bottleneck_attentionrKrirYattention_probs_dropout_probr[ is_causalr.r_r1s r2r'z MobileBertSelfAttention.__init__s  #)#= #&v'>A['[#\#\ !58PP/5 Yv68JKK 9V4d6HIIY'-'F ^F # #FL^`d`r   z&"EFF r3N query_tensor key_tensor value_tensorr{r}r5c J|jdd}g|d|jR}||j|dd}||j|dd} ||j|dd} tj|j j t} | ||| | |f|j sdn |j j|jd|\} } | jg|dR} | | fS)NrGr rjrh)r[r|)shaperryviewrrzrir get_interfacer__attn_implementationrrr[rr|reshaper)r.rrrr{r}rr hidden_shape query_layer key_layer value_layerattention_interfacerrs r2r7zMobileBertSelfAttention.forwards]#("- CCbC$*BCC 4djj..3\BLLQPQRR -DHHZ((-|<FFq!LL 3djj..3\BLLQPQRR (?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH L((r3r% r9r:r;r'r)r<rwrrtupler7r=r>s@r2rrs,48 ))l)L)l ) )D0 ) +, ) u| ))))))))r3rcPeZdZfdZdejdejdejfdZxZS)MobileBertSelfOutputcLt|j|_tj|j|j|_t|j|j|j |_ |js tj |j |_ dSdSNr0)r&r'use_bottleneckrrTrdenserVrWlayer_norm_epsrXrYrZr[rs r2r'zMobileBertSelfOutput.__init__s $3Yv68OPP  !:;Fs@r2rrsnBBBBBU\ELUZUar3rceZdZfdZ d dejdejdejdejdejdzdeed e ejfd Z xZ S) MobileBertAttentionctt||_t ||_dSr%)r&r'rr.routputrs r2r'zMobileBertAttention.__init__s; +F33 *622 r3Nrrr layer_inputr{r}r5c ^|j||||fi|\}}|||}||fSr%)r.r) r.rrrrr{r}attention_outputrs r2r7zMobileBertAttention.forwardsZ*3     * *  * * &, ;;'7EE--r3r%rr>s@r2rrs3333348 ..l.L.l . \ . )D0 .+,. u| ........r3rcBeZdZfdZdejdejfdZxZS)MobileBertIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr%) r&r'rrTrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrs r2r'zMobileBertIntermediate.__init__sn Yv68PQQ f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r3rr5cZ||}||}|Sr%)rrr.rs r2r7zMobileBertIntermediate.forwards, =11 00?? r3r8r>s@r2rrs^99999U\elr3rcPeZdZfdZdejdejdejfdZxZS)OutputBottleneckc"ttj|j|j|_t|j|j|j |_ tj |j |_ dSr)r&r'rrTrrKrrVrWrrXrYrZr[rs r2r'zOutputBottleneck.__init__sm Yv68JKK  !:;Fs@r2rrsi>>>>> U\ELUZUar3rc^eZdZfdZdejdejdejdejfdZxZS)MobileBertOutputcft|j|_tj|j|j|_t|j |j|_ |js tj |j |_ dSt||_dSr%)r&r'rrrTrrrrVrWrXrYrZr[r bottleneckrs r2r'zMobileBertOutput.__init__,s $3Yv79PQQ  !:;F>,9J*JKKLL>>,9J*JKKL??<9JKKLr3r8r>s@r2rr+su77777 #(< DIL ejeq          r3rcBeZdZfdZdejdejfdZxZS)BottleneckLayercttj|j|j|_t|j|j|j |_ dSr) r&r'rrTrKintra_bottleneck_sizerrVrWrrXrs r2r'zBottleneckLayer.__init__DsY Yv163OPP  !:;Fs@r2rrCscuuuuu U\elr3rcNeZdZfdZdejdeejfdZxZS) Bottleneckct|j|_|j|_t ||_|jrt ||_dSdSr%)r&r'key_query_shared_bottleneckrrinput attentionrs r2r'zBottleneck.__init__Psf +1+M((.(G%$V,,  + 5,V44DNNN 5 5r3rr5c||}|jr|fdzS|jr||}||||fS||||fS)N)rrrr)r.rbottlenecked_hidden_statesshared_attention_inputs r2r7zBottleneck.forwardXso"&*ZZ %>%>"  ( ].014 4  - ]%)^^M%B%B "*,BMSmn n!=-A[\ \r3 r9r:r;r'r)r<rr7r=r>s@r2rrOsm55555]U\]eEL6I]]]]]]]]r3rcPeZdZfdZdejdejdejfdZxZS) FFNOutputcttj|j|j|_t|j|j|j |_ dSr) r&r'rrTrrrrVrWrrXrs r2r'zFFNOutput.__init__tsY Yv79PQQ  !:;Fs@r2rrssnppppp U\ELUZUar3rcBeZdZfdZdejdejfdZxZS)FFNLayerctt||_t ||_dSr%)r&r'r intermediaterrrs r2r'zFFNLayer.__init__s< 26::'' r3rr5c\||}|||}|Sr%)rr)r.rintermediate_outputrs r2r7zFFNLayer.forwards0"// >> $7GG r3r8r>s@r2rrs^((((( U\elr3rc jeZdZfdZ ddejdejdzdeedejfdZ xZ S) MobileBertLayerctj|_j|_t |_t |_t|_ |jrt|_ jdkrz,MobileBertLayer.__init__..s!%k%k%k1hv&6&6%k%k%kr3)r&r'rnum_feedforward_networksrrrrrrrrr ModuleListrangeffnrs `r2r'zMobileBertLayer.__init__s $3(.(G%,V4426::&v..   1(00DO  *Q . .}%k%k%k%kfFehiFi@j@j%k%k%kllDHHH / .r3Nrr{r}r5c &|jr||\}}}}n |gdz\}}}}|j|||||fi|\}} |} |jdkr|jD] } | | } || } || | |} | S)Nrr )rrrrrrr)r.rr{r}rrrrself_attention_outputrr ffn_modulerrs r2r7zMobileBertLayer.forwards   VBF//R_B`B` ?L*lKKCP/TUBU ?L*lK#14>      $ $  $ $  q1  (A - -"h @ @ #-:.>#?#?  "//0@AA{{#68H-XX r3r%) r9r:r;r'r)r<rwrrr7r=r>s@r2rrs m m m m m 48|)D0+,   r3rc feZdZfdZ ddejdejdzdeede e zfdZ xZ S) MobileBertEncodercttjfdt jD|_dS)Nc.g|]}tSr)rrs r2rz.MobileBertEncoder.__init__..s!#e#e#eOF$;$;#e#e#er3)r&r'rrrnum_hidden_layerslayerrs `r2r'zMobileBertEncoder.__init__sO ]#e#e#e#eU6KcEdEd#e#e#eff r3Nrr{r}r5c jt|jD]\}}|||fi|}t|S)N)last_hidden_state) enumeraterr)r.rr{r}i layer_modules r2r7zMobileBertEncoder.forwards\ )44  OA|(LMM ????r3r%) r9r:r;r'r)r<rwrrrrr7r=r>s@r2rrsggggg48 @ @| @)D0 @+, @  @ @ @ @ @ @ @ @r3rcBeZdZfdZdejdejfdZxZS)MobileBertPoolerct|j|_|jr&t j|j|j|_dSdSr%)r&r'classifier_activation do_activaterrTrKrrs r2r'zMobileBertPooler.__init__sY !7   K6#5v7IJJDJJJ K Kr3rr5c|dddf}|js|S||}tj|}|S)Nr)r rr)tanh)r.rfirst_token_tensor pooled_outputs r2r7zMobileBertPooler.forwardsO+111a40 !% % JJ'9::M!J}55M r3r8r>s@r2rrscKKKKK !U\ !el ! ! ! ! ! ! ! !r3rcBeZdZfdZdejdejfdZxZS)!MobileBertPredictionHeadTransformcXttj|j|j|_t |jtrt|j|_ n |j|_ td|j|j |_ dS)Nr?r)r&r'rrTrKrrrrr transform_act_fnrVrrXrs r2r'z*MobileBertPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D ! .v/AvG\]]]r3rr5c||}||}||}|Sr%)rrrXrs r2r7z)MobileBertPredictionHeadTransform.forwards= =11 --m<< }55 r3r8r>s@r2rrsc^^^^^U\elr3rcBeZdZfdZdejdejfdZxZS)MobileBertLMPredictionHeadcxtt||_t j|j|j|jz d|_ t j|j|jd|_ t j tj |j|_dS)NF)r+T)r&r'r transformrrTrMrKrJrdecoderr(r)r*r+rs r2r'z#MobileBertLMPredictionHead.__init__s :6BBYv0&2DvG\2\chiii y!68IPTUUU LV->!?!?@@ r3rr5c||}|tj|jj|jjgd}||jjz }|S)Nrrk) rrr)rorr-trr+rs r2r7z"MobileBertLMPredictionHead.forwardsh}55 %,,UY 8K8M8M8O8OQUQ[Qb7cij-k-k-kll ** r3r8r>s@r2rrscAAAAAU\elr3rcBeZdZfdZdejdejfdZxZS)MobileBertOnlyMLMHeadcptt||_dSr%)r&r'r predictionsrs r2r'zMobileBertOnlyMLMHead.__init__s/ 5f==r3sequence_outputr5c0||}|Sr%)r)r.rprediction_scoress r2r7zMobileBertOnlyMLMHead.forwards ,,_==  r3r8r>s@r2rrs^>>>>>!u|! !!!!!!!!r3rc\eZdZfdZdejdejdeejfdZxZS)MobileBertPreTrainingHeadsctt||_t j|jd|_dSNrj)r&r'rrrrTrKseq_relationshiprs r2r'z#MobileBertPreTrainingHeads.__init__ sF 5f== " &*s@r2r#r# stAAAAA 9u|9EL9UZ[`[gUh99999999r3r#cveZdZUeed<dZdZdZdZdZ dZ e e dZ ejfdZxZS)MobileBertPreTrainedModelr_ mobilebertT)r attentionsct|t|tr4t j|jt j|jdSt|trt j|jdSt|trQt j |j tj|j jdddSdS)zInitialize the weightsrGrFN)r& _init_weightsrr#initzeros_r+ones_r-rrBcopy_rEr)r]rr^)r.rxr1s r2r.z'MobileBertPreTrainedModel._init_weights$s f%%% ff % % i K $ $ $ Jv} % % % % %  : ; ; i K $ $ $ $ $  4 5 5 i Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h h h i ir3)r9r:r;r!__annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsr)no_gradr.r=r>s@r2r*r*s $&*#N"&(- U]__ i i i i_ i i i i ir3r*z6 Output type of [`MobileBertForPreTraining`]. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dS)MobileBertForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr,) r9r:r;rur?r)rwr3r@rArrr,rr3r2r>r>1s  &*D% d ")))26u(4/6668<U.5<<<59M5*+d299926Je'(4/66666r3r>ceZdZdZdfd ZdZdZeee dde j dzde j dzd e j dzd e j dzd e j dzd e ed eezfdZxZS)MobileBertModelz2 https://huggingface.co/papers/2004.02984 Tct|||_d|_t ||_t ||_|rt|nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r&r'r_gradient_checkpointingrBrtrencoderrpooler post_init)r.r_add_pooling_layerr1s r2r'zMobileBertModel.__init__Ps|     &+#.v66(00 2CM&v...  r3c|jjSr%rtrOr.s r2get_input_embeddingsz$MobileBertModel.get_input_embeddingsas ..r3c||j_dSr%rK)r.ris r2set_input_embeddingsz$MobileBertModel.set_input_embeddingsds*/'''r3Nrbr{rcrErdr}r5c |du|duz rtd|||||}t|j||}|j|fd|i|}|d} |j|| nd} t | | S)Nz:You must specify exactly one of input_ids or inputs_embeds)rbrErcrd)r_rdr{r{r)r pooler_output) ValueErrorrtr r_rFrGr) r.rbr{rcrErdr}embedding_outputencoder_outputsrrs r2r7zMobileBertModel.forwardgs  -t"; < [YZZ Z??%)' +  3;*)    '$,   )    *!,8< 8O O444UY )-'    r3)T)NNNNN)r9r:r;rur'rMrOrrrr)rvrwrrrrr7r=r>s@r2rCrCJs("///000 .237260426 $ $ #d*$ )D0$ (4/ $ &- $ (4/ $ +,$  + +$ $ $ ^_ $ $ $ $ $ r3rCz MobileBert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c8eZdZdddZfdZdZdZddedzd ej ffd Z e e dd e jdzd e jdzd e jdzde jdzde jdzde jdzde jdzdeed eezfdZxZS)MobileBertForPreTrainingcls.predictions.bias,mobilebert.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightct|t||_t ||_|dSr%)r&r'rCr+r#clsrHrs r2r'z!MobileBertForPreTraining.__init__sQ    )&11-f55 r3c$|jjjSr%r[rrrLs r2get_output_embeddingsz.MobileBertForPreTraining.get_output_embeddingsx#++r3cT||jj_|j|jj_dSr%r[rrr+r.new_embeddingss r2set_output_embeddingsz.MobileBertForPreTraining.set_output_embeddings%'5$$2$7!!!r3Nnew_num_tokensr5c||jjj|d|jj_t |SNT)rf transposed)rf_get_resized_lm_headr[rrr&resize_token_embeddingsr.rfr1s r2rlz0MobileBertForPreTraining.resize_token_embeddingssR%)%>%> H &~RV&?& & "ww..n.MMMr3rbr{rcrErdlabelsnext_sentence_labelr}c |j|f||||dd|} | dd\} } || | \} } d}||t}|| d|jj|d}|| dd|d}||z}t || | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForPreTraining.from_pretrained("google/mobilebert-uncased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Tr{rcrErd return_dictNrjrG)r?r@rArr,) r+r[rrr_rMr>rr,)r.rbr{rcrErdrnror}outputsrrr!r( total_lossloss_fctmasked_lm_lossnext_sentence_losss r2r7z MobileBertForPreTraining.forwards)R"$/  ))%'     *1!&48HH_m4\4\11  "5"A'))H%X&7&<&r7r=r>s@r2rVrVs)?*X ,,,888NNcDjNBLNNNNNN.237260426*.7;@ @ #d*@ )D0@ (4/ @ &- @ (4/ @  4'@ #-4@ +,@  / /@ @ @ ^@ @ @ @ @ r3rVc"eZdZdddZfdZdZdZddedzd ej ffd Z e e dd e jdzd e jdzd e jdzde jdzde jdzde jdzdeed eezfdZxZS)MobileBertForMaskedLMrWrXrYct|t|d|_t ||_||_|dSNF)rI)r&r'rCr+rr[r_rHrs r2r'zMobileBertForMaskedLM.__init__s]    )&EJJJ(00  r3c$|jjjSr%r]rLs r2r^z+MobileBertForMaskedLM.get_output_embeddings r_r3cT||jj_|j|jj_dSr%rarbs r2rdz+MobileBertForMaskedLM.set_output_embeddings rer3Nrfr5c||jjj|d|jj_t |Srhrjrms r2rlz-MobileBertForMaskedLM.resize_token_embeddingssR%)%>%> H &~RV&?& & "ww..n.MMMr3rbr{rcrErdrnr}c <|j|f||||dd|}|d} || } d} |Kt} | | d|jj|d} t | | |j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` TrqrNrGr?logitsrr,) r+r[rrr_rMrrr,) r.rbr{rcrErdrnr}rsrr!rvrus r2r7zMobileBertForMaskedLM.forwards$"$/  ))%'     "!* HH_55  '))H%X&7&<&s@r2r{r{sx)?*X ,,,888NNcDjNBLNNNNNN.237260426*.' ' #d*' )D0' (4/ ' &- ' (4/ '  4'' +,'   ' ' ' ^' ' ' ' ' r3r{cBeZdZfdZdejdejfdZxZS)MobileBertOnlyNSPHeadcttj|jd|_dSr%)r&r'rrTrKr&rs r2r'zMobileBertOnlyNSPHead.__init__Ds6  " &*s@r2rrCscAAAAA&U\&el&&&&&&&&r3rzZ MobileBert Model with a `next sentence prediction (classification)` head on top. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e e zfd Z xZS) #MobileBertForNextSentencePredictionct|t||_t ||_|dSr%)r&r'rCr+rr[rHrs r2r'z,MobileBertForNextSentencePrediction.__init__SsQ    )&11(00 r3Nrbr{rcrErdrnr}r5c (|j|f||||dd|}|d} || } d} |At} | | dd|d} t | | |j|jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```Trqr NrGrjr)r+r[rrrrr,) r.rbr{rcrErdrnr}rsrr(rwrus r2r7z+MobileBertForNextSentencePrediction.forward\sL"$/  ))%'       !%-!8!8!  '))H!)*@*E*Eb!*L*LfkkZ\oo!^!^ *#)!/)     r3r)r9r:r;r'rrr)rvrwrrrrr7r=r>s@r2rrMs .237260426*.; ; #d*; )D0; (4/ ; &- ; (4/ ;  4'; +,;  , ,; ; ; ^; ; ; ; ; r3rz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) #MobileBertForSequenceClassificationcdt||j|_||_t ||_|j|jn|j}tj ||_ tj |j |j|_ |dSr%)r&r' num_labelsr_rCr+classifier_dropoutrZrrYr[rTrK classifierrHr.r_rr1s r2r'z,MobileBertForSequenceClassification.__init__s     + )&11)/)B)NF % %TZTn z"455 )F$68IJJ r3Nrbr{rcrErdrnr}r5c |j|f||||dd|}|d} || } || } d} |Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt} |jdkr1| | | } n| | |} n|jjdkrGt} | | d|j| d} n*|jjdkrt} | | |} t| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trqr N regressionsingle_label_classificationmulti_label_classificationrGr)r+r[rr_ problem_typerrfr)rnrrsqueezerrrrrr,) r.rbr{rcrErdrnr}rsrrr?rus r2r7z+MobileBertForSequenceClassification.forwards$"$/  ))%'        ]33 //  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     r3r)r9r:r;r'rrr)r<rrrrr7r=r>s@r2rrs     *..2.2,0-1&*; ; <$&;  t+;  t+ ; lT) ; |d* ;  t#; +,;  u| 7 7; ; ; ^; ; ; ; ; r3rceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)MobileBertForQuestionAnsweringct||j|_t|d|_t j|j|j|_| dSr}) r&r'rrCr+rrTrK qa_outputsrHrs r2r'z'MobileBertForQuestionAnswering.__init__sj     +)&EJJJ)F$68IJJ r3Nrbr{rcrErdstart_positions end_positionsr}r5c F|j|f||||dd|} | d} || } | dd\} } | d} | d} d}||t |dkr|d}t |dkr|d}| d}|d|}|d|}t|}|| |}|| |}||zdz }t|| | | j | j S) NTrqrr rGrk) ignore_indexrj)r? start_logits end_logitsrr,) r+rsplitrrlenrmclamprrrr,)r.rbr{rcrErdrrr}rsrrrrrt ignored_indexru start_lossend_losss r2r7z&MobileBertForQuestionAnswering.forwards"$/  ))%'     "!*11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM']CCCH!,@@Jx M::H$x/14J+%!!/)     r3rx)r9r:r;r'rrr)r<rrrrr7r=r>s@r2rrs*..2.2,0-1/3-13 3 <$&3  t+3  t+ 3 lT) 3 |d* 3 ,3 |d*3 +,3  u| ; ;3 3 3 ^3 3 3 3 3 r3rceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) MobileBertForMultipleChoicec4t|t||_|j|jn|j}t j||_t j |j d|_ | dS)Nr ) r&r'rCr+rrZrrYr[rTrKrrHrs r2r'z$MobileBertForMultipleChoice.__init__;s    )&11)/)B)NF % %TZTn z"455 )F$6:: r3Nrbr{rcrErdrnr}r5c R| |jdn |jd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}|j|f||||dd|} | d} || } || } | d|} d} |t }|| |} t| | | j| j S)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr rGTrqr) rrrmr+r[rrrrr,)r.rbr{rcrErdrnr} num_choicesrsrrreshaped_logitsr?rus r2r7z#MobileBertForMultipleChoice.forwardHsT-6,Aioa((}GZ[\G] >G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  "$/  ))%'        ]33 // ++b+66  '))H8OV44D("!/)     r3r)r9r:r;r'rrr)r<rrrrr7r=r>s@r2rr8s     *..2.2,0-1&*N N <$&N  t+N  t+ N lT) N |d* N  t#N +,N  u| 8 8N N N ^N N N N N r3rceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) MobileBertForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSr})r&r'rrCr+rrZrrYr[rTrKrrHrs r2r'z)MobileBertForTokenClassification.__init__s     +)&EJJJ)/)B)NF % %TZTn z"455 )F$68IJJ r3Nrbr{rcrErdrnr}r5c \|j|f||||dd|}|d} || } || } d} |Ft} | | d|j|d} t | | |j|jS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrqrNrGr) r+r[rrrrrrr,) r.rbr{rcrErdrnr}rsrrr?rus r2r7z(MobileBertForTokenClassification.forwards "$/  ))%'     "!*,,7711  '))H8FKKDO<s@r2rrs     *..2.2,0-1&*' ' <$&'  t+'  t+ ' lT) ' |d* '  t#' +,'  u| 4 4' ' ' ^' ' ' ' ' r3r) r{rrrVrrrrrCr*)Nrh)Ucollections.abcr dataclassesrr)rtorch.nnrrrr r/ activationsr masking_utilsr modeling_layersr modeling_outputsrrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_mobilebertr! get_loggerr9loggerModuler#rXrVrBr<floatrrrrrrrrrrrrrrrrrr#r*r>rCrVr{rrrrrr__all__rr3r2rs.%$$$$$!!!!!! AAAAAAAAAA&&&&&&!!!!!!666666999999                    GFFFFFFF&&&&&&MMMMMMMMMMMMIIIIIIII555555666666  H % %66666RY666& 9 9IIIII29IIIf!%% I% <% % < % L4' % T\ %%'(%%%%81)1)1)1)1)bi1)1)1)h29".....")...8     RY        ry   ry0     bi   !]!]!]!]!]!]!]!]H             ry   )))))0)))X@@@@@ @@@(!!!!!ry!!!& ""!!!!!BI!!! 9 9 9 9 9 9 9 9iiiiiiii4  77777[77   7&C C C C C /C C C L _ _ _ _ _ 8_ _  _ DF F F F F 5F F F R&&&&&BI&&& G G G G G *CG G  G TM M M M M *CM M M `@ @ @ @ @ %>@ @ @ F^ ^ ^ ^ ^ ";^ ^ ^ B8 8 8 8 8 '@8 8 8 v   r3