tjdZddlmZddlZddlZddlmZddlmZm Z m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,ddl-m.Z.ddl/m0Z0e)j1e2Z3de4de4dej5fdZ6de4de4dej5fdZ7Gddej8Z9 dAdej8dej5d ej5d!ej5d"ej5dzd#e:dzd$e:d%e#e'fd&Z;Gd'd(ej8Z<Gd)d*ej8Z=Gd+d,eZ>Gd-d.ej8Z?e(Gd/d0e!Z@e(Gd1d2e@ZAe(d34Gd5d6e@ZBe(d74Gd8d9e@ZCe(Gd:d;e@ZDe(Gd<d=e@ZEe(Gd>d?e@ZFgd@ZGdS)Bz PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) )CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)get_activation)PreTrainedConfig)is_deepspeed_zero3_enabled)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)DistilBertConfign_posdimoutc:tr|ddl}|j|d5tjdkrt|||cdddS ddddS#1swxYwYdSt|||S)Nr) modifier_rankr!r"r#)r deepspeedzeroGatheredParameterstorch distributedget_rank_create_sinusoidal_embeddings)r!r"r#r's p/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/distilbert/modeling_distilbert.pycreate_sinusoidal_embeddingsr/>s&!##L ^ . .s! . D D T T ))++q0045csSSS T T T T T T T T0 T T T T T T T T T T T T T T T T T T-5csKKKKs4A>>BBc tjfdt|D}d|_t jtj|dddddf|dddddf<t jtj|dddddf|dddddf<||S)NcJg|]fdtDS)c Rg|]#}tjdd|dzzz z $S)i')nppower).0jr"poss r. z<_create_sinusoidal_embeddings...Js7\\\AcBHUAaL34F$G$GG\\\)range)r6r8r"s @r.r9z1_create_sinusoidal_embeddings..Js=uuuad\\\\\QVWZQ[Q[\\\uuur:Frr3r) r4arrayr; requires_gradr* FloatTensorsincosdetach_)r!r"r# position_encs ` r.r-r-Is8uuuuhmnshthtuuuvvLC$RVLADqD,A%B%BCCC14a4L$RVLADqD,A%B%BCCC14a4LKKMMM Jr:c veZdZdeffd Z d dejdejdzdejdzdejfdZxZ S) Embeddingsconfigcttj|j|j|j|_tj|j|j|_ tj |jd|_ tj |j |_ | dtj|jdddS)N) padding_idx-q=eps position_idsrF) persistent)super__init__r Embedding vocab_sizer" pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutregister_bufferr*arangeexpandselfrE __class__s r.rPzEmbeddings.__init__Ss !|F,=vzW]Wjkkk#%<0NPVPZ#[#[ fje<<<z&.11   EL)GHHOOPWXXej      r:N input_ids inputs_embedsrKreturnc|||}|d}|rt|dr|jddd|f}nNt j|tj|j}|d |}| |}||z}| |}| |}|S)NrrK)dtypedevicer) rTsizehasattrrKr*r[longre unsqueeze expand_asrVrWrY)r^r`rarK seq_lengthrV embeddingss r.forwardzEmbeddings.forward^s   00;;M"''**  t^,, N#0KZK@ $|JejQZQabbb +55a88BB9MM "66|DD"%88 ^^J// \\*-- r:)NN) __name__ __module__ __qualname__r rPr*Tensor LongTensorrm __classcell__r_s@r.rDrDRs  /       .204 <|d*&-   r:rDmodulequerykeyvalueattention_maskscalingrYkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)NrMr3rr")ptrainingr) rfr*matmul transposer functionalsoftmaxrYr contiguous) rvrwrxryrzr{rYr| attn_weights attn_outputs r.eager_attention_forwardr|s**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r:c |eZdZdeffd Z d dejdejdzdee de ejfdZ xZ S) DistilBertSelfAttentionrEct||_|j|_|j|_|j|jz|_|jdz|_|j|jzdkr td|jd|jdtj |j|j|_ tj |j|j|_ tj |j|j|_ tj |j|j|_ tj|j|_d|_dS) Nr~rzself.n_heads: z must divide self.dim:  evenly in_features out_featuresrF)rOrPrEn_headsr"attention_head_sizer{ ValueErrorrLinearq_link_linv_linout_linrXattention_dropoutrY is_causalr]s r.rPz DistilBertSelfAttention.__init__s  ~ :#'8t|#; /5  8dl "a ' 'ddlddSWS[dddee eY6:FJOOO Y6:FJOOO Y6:FJOOO yVZfjQQQ zF$<=== r:N hidden_statesrzr|rbc t|jdd}g|d|jR}||j|dd}||j|dd}||j|dd}tj|j j t} | |||||f|j sdn |j j|jd|\} } | jg|dR} || } | | fS)NrMrr3ru)rYr{)shaperrviewrrrr get_interfacerE_attn_implementationrrrYrr{reshaperr) r^rrzr| input_shape hidden_shape query_layer key_layer value_layerattention_interfacerrs r.rmzDistilBertSelfAttention.forwardsn $)#2#. CCbC$*BCC 5djj//4lCMMaQRSS 2DJJ}--2LAKKAqQQ 4djj//4lCMMaQRSS (?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH ll;// L((r:N) rnrorpr rPr*rqr>rrtuplermrsrts@r.rrs/248))|))D0)+, ) u|  ))))))))r:rcleZdZdeffd ZdejdejfdZdejdejfdZxZ S)FFNrEcrttj|j|_|j|_d|_tj|j|j |_ tj|j |j|_ t|j |_ dS)Nrrr)rOrPrrXrYchunk_size_feed_forward seq_len_dimrr" hidden_dimlin1lin2r activationr]s r.rPz FFN.__init__s zFN333 '-'E$I&*6CTUUU I&*;&*UUU ():;;r:inputrbcDt|j|j|j|Sr)rff_chunkrr)r^rs r.rmz FFN.forwards (8TVZVfhmnnnr:c||}||}||}||}|Sr)rrrrY)r^rxs r.rz FFN.ff_chunksG IIe   OOA   IIaLL LLOOr:) rnrorpr rPr*rqrmrrsrts@r.rrs</<<<<<<oU\oelooooelu|r:rc eZdZdeffd Z d dejdejdzdeede ejdffd Z xZ S) TransformerBlockrEc~t|j|jzdkr t d|jd|jdt ||_tj|jd|_ t||_ tj|jd|_ dS)Nrzconfig.n_heads z must divide config.dim rrH)normalized_shaperJ) rOrPr"rrr attentionrrW sa_layer_normrffnoutput_layer_normr]s r.rPzTransformerBlock.__init__s  : &! + +jv~jjW]Wajjjkk k088\6:5QQQv;;!#vzu!U!U!Ur:Nrrzr|rb.c |j|fd|i|\}}|||z}||}|||z}|S)Nrz)rrrr)r^rrzr|attention_output_ ffn_outputs r.rmzTransformerBlock.forwards-dn   )   !  --.>.NOOXX.// ++J9I,IJJ r:r) rnrorpr rPr*rqrrrrmrsrts@r.rrs V/ V V V V V V /3| t++,  u|S ! r:rc feZdZdeffd Z d dejdejdzdeede fdZ xZ S) TransformerrEctj|_tjfdt jD|_d|_dS)Nc.g|]}tS)r)r6rrEs r.r9z(Transformer.__init__.. s"#]#]#]$4V$<$<#]#]#]r:F)rOrPn_layersr ModuleListr;layergradient_checkpointingr]s `r.rPzTransformer.__init__ sa  ]#]#]#]#]eFOF\F\#]#]#]^^ &+###r:Nrrzr|rbc J|jD] }|||fi|} t|S)N)last_hidden_state)rr)r^rrzr| layer_modules r.rmzTransformer.forwardsP !J  L(LMM ????r:r) rnrorpr rPr*rqrrrrmrsrts@r.rrs,/,,,,,,/3 @ @| @ t+ @+, @  @ @ @ @ @ @ @ @r:rceZdZUeed<dZdZdZdZdZ dZ e e dZ ejdejffd ZxZS)DistilBertPreTrainedModelrE distilbertT)r attentionsrvc t|t|tr|jjr^t j|jj t|jj |jj tj|jj t j|jtj|jjdddSdS)zInitialize the weights.rMrLN)rO _init_weights isinstancerDrEsinusoidal_pos_embdsinitcopy_rVweightr/rUr"r* empty_likerKr[rr\)r^rvr_s r.rz'DistilBertPreTrainedModel._init_weights.s f%%% fj ) ) i{/  .50 ; ()C)JKK Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h h h i ir:)rnrorpr __annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsr*no_gradrModulerrsrts@r.rr s $&*#N"&)- U]__ iBI i i i i i_ i i i i ir:rc8eZdZdeffd ZdejfdZdefdZ dejfdZ dejfd Z e e e dd ejd zd ejd zd ejd zdejd zdeedeeejdfzf dZxZS)DistilBertModelrEct|t||_t ||_|dSr)rOrPrDrlr transformer post_initr]s r.rPzDistilBertModel.__init__AsR    $V,,&v.. r:rbc|jjSz1 Returns the position embeddings )rlrVr^s r.get_position_embeddingsz'DistilBertModel.get_position_embeddingsJs22r:new_num_position_embeddingsc||jjz }|dkrdStd|d||j_|jjj}tj |jj|jj |j_|jj r1t|jj|jj |jjntj5|dkr*tj||jjjd| <n+tj|d||jj_dddn #1swxYwY|jj|jdS) Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. rNz(Setting `config.max_position_embeddings=z`...r&)rErUloggerinforlrVrclonerrQr"rr/r*r Parametertore)r^rnum_position_embeds_diffold_position_embeddings_weights r.resize_position_embeddingsz*DistilBertModel.resize_position_embeddingsPs$?Ad#d  $q ( ( F `?Z```aaa.I +)-)L)S)Y)Y)[)[&.0l4;;^`d`k`o.p.p+ ; +  (k9t{TXTlTs       +a//]_]i6^^DO7>?YAY@Y?YZZBD67P8P7PQBBDO7>                 +..t{;;;;;s(AEEEc|jjSrrlrTrs r.get_input_embeddingsz$DistilBertModel.get_input_embeddingszs ..r:new_embeddingsc||j_dSrrr^rs r.set_input_embeddingsz$DistilBertModel.set_input_embeddings}s*8'''r:Nr`rzrarKr|.c |du|duz rtd||||}t|j||}|jd||d|S) input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds)rErarz)rrzr)rrlr rEr)r^r`rzrarKr|rls r.rmzDistilBertModel.forwards0  -t"; < [YZZ Z__Y |LL 2;$)     t $)     r:)NNNN)rnrorpr rPrrQrintrrrrrrr*rqrrrrrmrsrts@r.rr?sm/33333 (> 112CDD 001BCC  (():)?)?DUDZDZ[]D^D^)_)_agalalmoapapqqH$&4#.     r:NNNNN)rnrorp_tied_weights_keysr rPrrQrrrrrrrrr*rqrrrrrrrmrsrts@r.rrs 34bc 2/ 2 2 2 2 2 299999 Pc P P P P$ry$$$$.BI....*..2-1*.,0 1 1 <$&1  t+1 |d* 1  4' 1 lT) 1 +,1  % c 12 21 1 1 ^1 1 1 1 1 r:rz DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZdeffd ZdejfdZdefdZ e e dde j dzd e j dzd e j dzd e jdzd e j dzd eedeee j dfzfdZxZS)#DistilBertForSequenceClassificationrEct||j|_||_t ||_t j|j|j|_ t j|j|j|_ t j |j |_ |dSr)rOrP num_labelsrErrrrr"pre_classifier classifierrXseq_classif_dropoutrYrr]s r.rPz,DistilBertForSequenceClassification.__init__s     + )&11 i FJ??)FJ0ABBz&"<==  r:rbc4|jSrrrs r.rz;DistilBertForSequenceClassification.get_position_embeddings%rr:rc:|j|dSrrrs r.rz>DistilBertForSequenceClassification.resize_position_embeddings+rr:Nr`rzrarrKr|.c |jd ||||dd|}|d}|dddf} || } tj| } || } || } d} |Z|jjf|jdkr d|j_nN|jdkr7|j tj ks|j tj kr d|j_n d|j_|jjdkrWt} |jdkr1| | |} n| | |} n|jjdkrGt} | | d |j|d } n*|jjdkrt#} | | |} t%| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). TrrNr regressionsingle_label_classificationmulti_label_classificationrMrr)rr*rReLUrYr+rE problem_typer)rdr*rhrrsqueezerrrrrr) r^r`rzrarrKr|distilbert_output hidden_state pooled_outputr rloss_fcts r.rmz+DistilBertForSequenceClassification.forward9s",DO )'%      )+ $QQQT* ++M:: ! -00  ]33 //  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'+9(3     r:r$)rnrorpr rPrrQrrrrrr*rqrrrrrrrmrsrts@r.r'r'sB /      99999 Pc P P P P*..2-1*.,0 : : <$&:  t+: |d* :  4' : lT) : +,:  "E%,*;$< <: : : ^: : : : : r:r'c(eZdZdeffd ZdejfdZdefdZ e e dde j dzd e j dzd e j dzd e j dzd e j dzd e j dzdeedeee j dfzfdZxZS)DistilBertForQuestionAnsweringrEcbt|t||_t j|j|j|_|jdkrtd|jt j |j |_ | dS)Nr3z)config.num_labels should be 2, but it is )rOrPrrrrr"r) qa_outputsrrX qa_dropoutrYrr]s r.rPz'DistilBertForQuestionAnswering.__init__zs    )&11)FJ0ABB   ! !\IZ\\]] ]z&"344  r:rbc4|jSrrrs r.rz6DistilBertForQuestionAnswering.get_position_embeddingsrr:rc:|j|dSrrrs r.rz9DistilBertForQuestionAnswering.resize_position_embeddingsrr:Nr`rzrastart_positions end_positionsrKr|.c x|jd ||||dd|}|d} || } || } | dd\} } | d} | d} d} ||t |dkr|d}t |dkr|d}| d}|d|}|d|}tj |}|| |}|| |}||zd z } t| | | |j |j S) rTrrrrMrN) ignore_indexr3)r start_logits end_logitsrrr)rrYr=splitr5rlenrfclamprrrrr)r^r`rzrarArBrKr|r6rr rErF total_loss ignored_indexr9 start_lossend_losss r.rmz&DistilBertForQuestionAnswering.forwards2,DO )'%      *!,  ]33 //#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM* FFFH!,@@Jx M::H$x/14J+%!+9(3     r:)NNNNNN)rnrorpr rPrrQrrrrrr*rqrrrrrmrsrts@r.r;r;xsU /      99999 Pc P P P P*..2-1/3-1,0> > <$&>  t+> |d* > , > |d* > lT)> +,>  &elC.?(@ @> > > ^> > > > > r:r;ceZdZdeffd ZdejfdZdefdZ e e dde j dzd e j dzd e j dzd e jdzd e j dzd eedeee j dfzfdZxZS) DistilBertForTokenClassificationrEc6t||j|_t||_t j|j|_t j|j |j|_ | dSr) rOrPr)rrrrXrYr hidden_sizer+rr]s r.rPz)DistilBertForTokenClassification.__init__sx     +)&11z&.11 )F$68IJJ r:rbc4|jSrrrs r.rz8DistilBertForTokenClassification.get_position_embeddingsrr:rc:|j|dSrrrs r.rz;DistilBertForTokenClassification.resize_position_embeddingsrr:Nr`rzrarrKr|.c Z|j|f|||dd|}|d}||}||} d} |Ft} | | d|j|d} t | | |j|jS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrzrarKrrNrMr) rrYr+rrr)rrr) r^r`rzrarrKr|outputssequence_outputr rr9s r.rmz(DistilBertForTokenClassification.forwards"$/  )'%      "!*,,7711  '))H8FKKDO<>> from transformers import AutoTokenizer, DistilBertForMultipleChoice >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased") >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> choice0 = "It is eaten with a fork and a knife." >>> choice1 = "It is eaten while held in the hand." >>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True) >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels) # batch size is 1 >>> # the linear classifier still needs to be trained >>> loss = outputs.loss >>> logits = outputs.logits ```NrrMTrUrr) rrrfrr*rr3rYr+rrrr)r^r`rzrarrKr| num_choicesrVr7r8r reshaped_logitsrr9s r.rmz#DistilBertForMultipleChoice.forwardJsb-6,Aioa((}GZ[\G] >G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImq(   r=#5#5b#9#9=;M;Mb;Q;Q R R R  "$/  )'%      qz $QQQT* ++M:: ! -00  ]33 // ++b+66  '))H8OV44D("!/)     r:r$)rnrorpr rPrrQrrrrrr*rqrrrrrrrmrsrts@r.rYrY)sR /      99999 Pc P P P P*..2-1*.,0 U U <$&U  t+U |d* U  4' U lT) U +,U  #U5<+<%= =U U U ^U U U U U r:rY)rrYr;r'rOrr)Nru)H__doc__collections.abcrnumpyr4r*rtorch.nnrrrr r activationsr configuration_utilsr integrations.deepspeedr masking_utilsr modeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_distilbertr get_loggerrnrrrqr/r-rrDfloatrrrrrrrrr'r;rOrY__all__rr:r.rvs %$$$$$ AAAAAAAAAA&&&&&&))))))333333@@@@@@666666999999GFFFFFFF&&&&&& JIIIIIII555555666666  H % % LL#LELLLLL3U\&&&&&&&&`!%% I% <% % < % L4' % T\ %%'(%%%%85)5)5)5)5)bi5)5)5)p")*     1   F@@@@@")@@@0iiiiiiii<g g g g g /g g g T _ _ _ _ _ 5_ _  _ D ^ ^ ^ ^ ^ *C^ ^  ^ Bb b b b b %>b b b JG G G G G '@G G G Tw w w w w ";w w w t   r: