tj|dZddlZddlZddlmZddlmZmZmZmZddlm Z ddl m Z m Z ddlmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZmZddlm Z ej!e"Z#d(dZ$Gddej%Z&Gddej%Z'GddeZ(eGddeZ)eGdde)Z*edGdde)eZ+ed Gd!d"e)Z,eGd#d$e)Z-eGd%d&e)Z.gd'Z/dS))zPyTorch MPT model.N)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) MptConfigcjtjd|z dtj|ddd|}dt jt j|z}tjd|dztj|}|||z z}dtj d|z }|d|dd}||krAtj |ddddddf|ddddddfgdddd|df}||z}| dS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedeviceg?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopess b/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensorr4*sH L_,au{6 R R R W WXY[\^_ap q qE $)I*>*> ? ?? </!35;v V V V \ \ ^ ^D >$88 9D 59Q%% %F [[0!Q 7 7Fy((vaaaAsl3VAAAsssCK5HIqQQQRSRSRSU_V_U_adRde FNE ==  c |eZdZdZd dededzffd Z d dejdejde dzd ejdzfd Z xZ S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. Nconfig layer_idxc*t|j|_|j|_|j|_|j|jz|_|jj|_|j)dtj |j|jz z |_|jj |_ |jj |_ tj|jd|jzd|_tj|j|jd|_||_dS)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler$sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr9)selfr8r9 __class__s r3r>zMptAttention.__init__Fs !-~ $0(DL8 #/=   %!"TYt/?$,/N%O%O!OD $0;*3 Id.D4D0D5QQQ  $"2D4D5QQQ "r5 hidden_states position_biaspast_key_valuesattention_maskc |jdd\}}||}|jr"||j |j}|dd\} } } | |||j|jdd} | |||j|jdd} | |||j|jdd} || | | |j \} } tj | | dd|j z} ||n||z} |t|jdkr$t!dt|j| jd}t#d |d| z }t#d |d|z }|dd|d|df}| |z} |2| |tj| jj} t.j| d| j}t.j||j|j }tj || }|d ddd !||d}|"|}||fS) Nr)minmaxr rrz6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperKrIclampchunkreshaper@rC transposeupdater9r matmulrEget_seq_lengthlen ValueErrorrUsize masked_fillfinforrTrrsoftmaxr(todropoutrHrZpermute contiguousr#rL)rMrOrPrQrRkwargs batch_size seq_length mixed_qkv query_states key_states value_statesattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputs r3forwardzMptAttention.forwardVs "/!4RaR!8 JIIm,, = O!T]N NNI1:1J1J. j,#++J DLRVR_``jjklnopp '' J dm\\ffghjkll #++J DLRVR_``jjklnopp  &'6'='=j,X\Xf'g'g $J  < j6J6J2r6R6RSSVZVhh%4%r Tensorr r| __classcell__rNs@r3r7r7As##y#S4Z######()-.2 0)0)|0)|0) 0)  t+ 0)0)0)0)0)0)0)0)r5r7cVeZdZdeffd ZdejdejdejfdZxZS)MptMLPr8c(t|j}tj|d|zd|_tjd|_tjd|z|d|_|j j |_ dS)NFr;none) approximate) r=r>r?rrJup_projGELUact down_projrDrGhidden_dropout)rMr8r?rNs r3r>zMptMLP.__init__s ( ya+oEJJJ 7v...1{?KeLLL$0;r5rOresidualreturnc|||}||}tj||j|j}||z}|S)NrX)rrrFrjrrZ)rMrOrintermediate_outputoutputs r3r|zMptMLP.forwards^m!r rr|rrs@r3rrssr?rlayer_norm_epsilonnorm_1r<r@r,r7attnnorm_2rffnrDrG dropout_raterDropoutresid_attn_dropout)rMr8r9r?rNs r3r>zMptBlock.__init__s (  1JKKK   33  1JKKK  &>>".9"$*T->"?"?r5FrOrPrR layer_past use_cacheoutput_attentionsc ||}|} |||||\} } || | z}||}|} ||| } | | fS)N)rPrRrQ)rrrrr) rMrOrPrRrrrrmlayernorm_outputr attn_outputsryrs r3r|zMptBlock.forwards ;;}55 &*YY ')& &/& & " l// ==H ;;}55!*H55|##r5r})NFF) r~rrrrr>r rr boolr|rrs@r3rrs@@y@S4Z@@@@@@2$("'!$!$|!$|!$ !$ DL !$  !$ !$!$!$!$!$!$!$!$r5rc(eZdZUeed<dZdZdgZdS)MptPreTrainedModelr8 transformerTrN)r~rrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr5r3rrs2 %&*## r5rceZdZdeffd ZdZddZdejfdZ e dd ej dzd e dzd ejdzd ej dzd e dzde dzde dzde dzdeejdfezfdZxZS)MptModelr8ctj|_j|_t jj|j|_t j fdtj D|_ t|jj|_d|j_d|_|dS)Nc2g|]}t|S))r9)r).0ir8s r3 z%MptModel.__init__..s&$c$c$cqXf%B%B%B$c$c$cr5rF)r=r>r?r@r,r Embedding vocab_sizewte ModuleListrangen_layersblocksrrnorm_fr<gradient_checkpointing post_initrMr8rNs `r3r>zMptModel.__init__s    !-< 143CDDm$c$c$c$cERXRaLbLb$c$c$cdd   0f6OPPP  &+# r5c|jSr}r)rMs r3get_input_embeddingszMptModel.get_input_embeddingss xr5rNc&t||||Sr})r4)rMr,r-r.rs r3r4zMptModel.build_mpt_alibi_tensors%i.RXYYYr5new_embeddingsc||_dSr}rrMrs r3set_input_embeddingszMptModel.set_input_embeddingss !r5 input_idsrQrR inputs_embedsrroutput_hidden_states return_dictr.c ||n |jj}||n |jj}||n |jj}||n |jj}||t d| |j\} } n| |j\} } } nt d|jr%|jr|rt dd}|| |}|r|t|j}|} |rdnd}|rdnd}| |j|jj| j}t#|j||| t&j}|jD]1}|r|| fz}|| ||||| }|d } |r ||d fz}2|| } |r|| fz}|st/d | |||fDSt1| |||S) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r8rr)r8rrRrQ)rrRrrrPrrc3K|]}||V dSr}r)rvs r3 z#MptModel.forward..^s1ghgtgtgtgtgtr5)last_hidden_staterQrO attentions)r8rrrrrdr[rrZlogger warning_oncerr r4r,rArr rir rrrtupler)rMrrQrRrrrrrrmrnro_rOall_self_attentionsall_hidden_statesr/ causal_maskblockoutputss r3r|zMptModel.forwards42C1N--TXT_Tq$8$D $+Jj "+!6IIDKcdd d  "%._ "J  &(5(; %J AATUU U  & "4= " "##p"   HHY//M  ?0*$+>>>O% $5?bb4"6@BBD++DNDKrr4r rrr LongTensorr rrrr|rrs@r3rrsny,ZZZZ"5<"""".2(,.215!%)-,0#'f f #d*f f  t+ f '$. f $; f  $;f #Tkf D[f  u|S !$M Mf f f ^f f f f f r5rz The MPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc&eZdZddiZdeffd ZdejfdZe dd ej dzd e dzd ejdzd ejdzd ejdzde dzde dzde dzde dzde ejzdeejezfdZxZS)MptForCausalLMzlm_head.weightztransformer.wte.weightr8ct|t||_t j|j|jd|_| dSNFr;) r=r>rrrrJr?rlm_headrrs r3r>zMptForCausalLM.__init__ssa    #F++y!3V5FUSSS  r5rc||_dSr})rrs r3set_output_embeddingsz$MptForCausalLM.set_output_embeddings{s % r5NrrrQrRrlabelsrrrrlogits_to_keeprc | | n |jj} ||||||||| } | d} t| trt | dn| }|| dd|ddf}d}||jd|||jjd| }| s|f| ddz}||f|zn|St||| j | j | j S)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` NrQrRrrrrrr)logitsrrrlossrrQrOrr) r8rr isinstancerslicer loss_functionrrrQrOr)rMrrQrRrrrrrrrrmtransformer_outputsrO slice_indicesrrrs r3r|zMptForCausalLM.forward~s:@&1%SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%pVFt{OeppioppD FY!4QRR!88F)-)9TGf$$v E0/?-;*5     r5) NNNNNNNNNr)r~rr_tied_weights_keysrr>r rrrrr rrrrr|rrs@r3rrjse+,DEy&EL&&&&.2(,.2-1&*!%)-,0#'-.? ? #d*? ?  t+ ? |d* ?  t# ? $;?  $;? #Tk? D[? el*?  u| @ @? ? ? ^? ? ? ? ? r5ra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZdeffd ZdejfdZe ddej dzde dzdejdzd ejdzd ejdzd e dzd e dzd e dzde dzde eje zfdZxZS)MptForSequenceClassificationr8ct||j|_t||_t j|j|jd|_| dSr) r=r> num_labelsrrrrJr?scorerrs r3r>z%MptForSequenceClassification.__init__sk     +#F++Yv163D5QQQ  r5rc||_dSr})rrs r3rz2MptForSequenceClassification.set_output_embeddingss # r5NrrQrRrrrrrrrc | | n |jj} ||||||||| } | d} || } ||jd}n |jd}|jj|dkrt d|jjd}n|}||jjk| jtj }tj |jd| jtj }||z d}n)d}t|jjd| tj || j |f}d}|.|jjf|jdkr d |j_nN|jdkr7|jtjks|jtjkr d |j_n d |j_|jjd krWt-}|jdkr1|||}nb|||}nU|jjd krt1}|||}n*|jjd krt3}|||}| s|f| ddz}||f|zn|St5||| j| j| j S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrz=Cannot handle batch sizes > 1 if no padding token is defined.rV)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r8rrrr[ pad_token_idrdrirr r"r!argmaxrrrNr~ problem_typerrlongrrr+rrrrQrOr)rMrrQrRrrrrrrrmrrOrrnlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r3r|z$MptForSequenceClassification.forwards>&1%!F!Fr!J!J  !#    >*ZZZ    u|Jv}MMMOaab   {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8M$9$9$;$;V^^=M=MNNDD#8M6::DD)-JJJ+--x v66)-III,..x v66 F#%(;ABB(??F)-)9TGf$$v E/ /?-;*5     r5 NNNNNNNNN)r~rrrr>r rrrrr rrrr|rrs@r3rrsVy$EL$$$$.2(,.2-1&*!%)-,0#'e e #d*e e  t+ e |d* e  t# e $;e  $;e #Tke D[e  u| ? ?e e e ^e e e e e r5rceZdZdeffd Ze ddejdzdedzdej dzdej dzdej dzd e dzd e dzd e dzd e dzd e ej e zfdZ xZS)MptForTokenClassificationr8ct||j|_t||_t |dr|j|j}n!t |dr|j|j}nd}tj ||_ tj |j |j|_ |dS)Nclassifier_dropoutrg?)r=r>rrrhasattrrrrrrjrJr? classifierr)rMr8rrNs r3r>z"MptForTokenClassification.__init__Gs     +#F++ 6/ 0 0 %V5N5Z!'!:   V- . . %63H3T!'!6  !$ z"455 )F$68IJJ r5NrrQrRrrrrrrrc | | n |jj} ||||||||| } | d} || } || } d}|p|| j}|j\}}t}|| ||z|j | ||z}| s| f| ddz}||f|zn|St|| | j | j S)rNrrr)rrrOr)r8rrrjrrirr[rr#rrrOr)rMrrQrRrrrrrrdeprecated_argumentsrrOrrrnror rs r3r|z!MptForTokenClassification.forwardXsL>&1%rr rr rrrrr|rrs@r3rrEs9y".2(,.2-1&*!%)-,0#'B B #d*B B  t+ B |d* B  t# B $;B  $;B #TkB D[B  u| 4 4B B B ^B B B B B r5rceZdZfdZe d dejdzdejdzdejdzdejdzdejdzdedzd edzd edzd e e zfd Z xZ S)MptForQuestionAnsweringct|t||_t j|jd|_|dS)Nr) r=r>rrrrJr? qa_outputsrrs r3r>z MptForQuestionAnswering.__init__sY    #F++)F$6:: r5NrrRrstart_positions end_positionsrrrrc  ||n |jj}|||||||} | d} || } | dd\} }| d} |d}d}||t|dkr|d}t|dkr|d}| d}| d|}| d|}t|}|| |}|||}||zdz }|s| |f| ddz}||f|zn|St|| || j | j S) rN)rRrrrrrrrVr) ignore_indexr)r start_logits end_logitsrOr)r8rrrsplitr+rlrcrer\rrrOr)rMrrRrrrrrrrmrsequence_outputrrr  total_loss ignored_indexr  start_lossend_lossrs r3r|zMptForQuestionAnswering.forwards 4&1%rr r FloatTensorrrrr|rrs@r3rrs.237263715)-,0#'F F #d*F )D0F (4/ F )D0 F '$. F  $;F #TkF D[F  - -F F F ^F F F F F r5r)rrrrrrr)0rr$r rtorch.nnrrrrrr cache_utilsr r generationr masking_utilsr modeling_layersrmodeling_outputsrrrrrmodeling_utilsrutilsrrconfiguration_mptr get_loggerr~rr4Moduler7rrrrrrrr__all__rr5r3r4s LLLLLLLLLLLL$$$$$$........))))))//////999999.-----,,,,,,,,((((((  H % %.E)E)E)E)E)29E)E)E)PRY*6$6$6$6$6$)6$6$6$r%%%%%%%%G G G G G !G G G T N N N N N 'N N  N b    s s s s s #5s s   s lU U U U U 2U U U pP P P P P 0P P P f   r5