tjpudZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z dd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*e$j+e,Z-Gddej.Z/ d3dej0dej1dej1dej1dej1dzde2de2fdZ3Gd d!ej0Z4Gd"d#eZ5e"Gd$d%eZ6Gd&d'e6Z7e"Gd(d)e6Z8Gd*d+e6eZ9e"d,-Gd.d/e6Z:e"Gd0d1e6Z;gd2Ztj|d}||zdz }|dd|df}t||jzS)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)torchcumsumlongr'forwardr&)r)r-r.r/r*s r+r6z%OPTLearnedPositionalEmbedding.forward8sq   <A>>>L(>9A=CCEEL'+A+B+B(BCLww|dk9:::r,)rN) __name__ __module__ __qualname____doc__intr(r3 LongTensorr6 __classcell__r*s@r+r r -sFsF3FFFFFF'(04 ;;(;!$;&- ;;;;;;;;;;r,r modulequerykeyvaluer-scalingdropoutc tj||dd|z}|||z}tj|dtj|j}tj |||j }tj||} | dd } | |fS)N)r2dtypeptrainingrr%) r3matmul transposer functionalsoftmaxfloat32torIrErL contiguous) r@rArBrCr-rDrEkwargs attn_weights attn_outputs r+eager_attention_forwardrWJs<s}}R'<'<==GL!#n4 =((2U](SSVVW\WbccL=((6?([[L,|U33K''1--88::K  $$r,ceZdZdZ d dededzffd Z ddejde dzd ejdzd e d e ejejdze dzff d Z xZ S) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfig layer_idxc 4t||_|j|_|j|_|j|_|j |_ ||_ |(t d|j jd|j|jz|_d|_|j|jz|jkr t#d|jd|jd|jdz|_t'j|j|j|j |_t'j|j|j|j |_t'j|j|j|j |_t'j|j|j|j |_dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r'r(rZ hidden_size embed_dimnum_attention_heads num_headsattention_dropoutrE enable_biasr[logger warning_oncer*r7head_dim is_causal ValueErrorrDrLineark_projv_projq_projout_proj)r)rZr[rTr*s r+r(zOPTAttention.__init__ds  +3/ !-"     ,!8,,,    $.8  MDN *t~ = =8dn88%)^888 }d* iTEUVVV iTEUVVV iTEUVVV  $.$.tGWXXX r,F hidden_statespast_key_valuesr-output_attentionsreturnc T|\}}}|||jz} | |d|j|jdd} ||} ||} | |d|j|jdd} | |d|j|jdd} || | | |j \} } tj |j jt} | || | | |f|jsdn|jdd|\} }| ||d} || } | |fS)z#Input shape: Batch x Time x ChannelrGrr%Nr?g?)rErD)sizermrDviewrbrgrNrkrlupdater[r get_interfacerZ_attn_implementationrWrLrEreshaperSrn)r)rorpr-rqrTbsztgt_len_ query_states key_states value_statesattention_interfacerVrUs r+r6zOPTAttention.forwards(,,..Wa{{=11DL@ #((b$.$-PPZZ[\^_`` [[// {{=11 __S"dndmLLVVWXZ[\\ #((b$.$-PPZZ[\^_``  &'6'='=j,X\Xf'g'g $J (?(M K ,.E) ) %8$7      % $}>CC$, % %  % % ! \"))#w;;FFHH mmK00 L((r,N)NNF)r7r8r9r:rr;r(r3Tensorr booltupler6r=r>s@r+rYrYasGG !%!Y!Y!Y:!Y!Y!Y!Y!Y!YL)-.2"' .).)|.).) t+ .)  .) u|U\D0%$,> ?.).).).).).).).)r,rYceZdZddededzffd Z ddejdejdzdedzd e dzd ej dzd e e d ejfd Z xZS)OPTDecoderLayerNrZr[c*t|j|_t |||_|j|_|j|_t|j |_ tj |j|j |_tj|j|j|j|_tj|j|j|j|_tj |j|j |_dS)N)rZr[elementwise_affiner])r'r(r_r`rY self_attndo_layer_norm_beforerEr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normrjffn_dimrdfc1fc2final_layer_norm)r)rZr[r*s r+r(zOPTDecoderLayer.__init__s +%VyIII$*$?!~ #F$>?$&L Nv/S% % % !9T^V^&BTUUU9V^T^&BTUUU " T^PVPt u u ur,Fror-rp use_cacher/rTrrc |}|jr||}|jd||||d|\}}tj||j|j}||z}|js||}|j} |d| d}|}|jr| |}| |}| |}| |}tj||j|j}||z| }|js| |}|S)N)rorpr/r-rJrG)rrrrrOrErLshaperyrtrrrrru) r)ror-rprr/rTresidualr|hidden_states_shapes r+r6zOPTDecoderLayer.forwards!  $ E 55mDDM*4> '+%)       q --mt|VZVc-dd  =0 ( E 55mDDM,1%--b-2D2DR2H2HII    $ A 11-@@M// **=99 //  --mt|VZVc-dd !M1778KLL ( A 11-@@Mr,r)NNFN)r7r8r9rr;r(r3rr rr<rrr6r=r>s@r+rrsvvyvS4Zvvvvvv(/3(,!&04 33|3 t+3 3 $; 3 &- 3-.3 33333333r,rcFeZdZUeed<dZdZdgZdZdZ dZ dZ dZ e edZdS)OPTPreTrainedModelrZmodelTr)ro attentionsN)r7r8r9r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrrY_can_record_outputsrr,r+rrsc &*#*+"&N!("r,rceZdZdZdeffd Zeee dde j dzde j dzde dzde j dzd edzd e j dzd eed efd ZxZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rZchtj|_j|_j|_j|_j|_tj jj |j|_ tjj|_j jkr'tjjj d|_nd|_j jkr'tjj jd|_nd|_jr-js&tjjj|_nd|_tjfdt3jD|_d|_|dS)NFr]rc2g|]}t|S))r[)r).0irZs r+ z'OPTDecoder.__init__..8s&$s$s$sa_Vq%I%I%I$s$s$sr,)r'r(rE layerdrop pad_token_id padding_idxmax_position_embeddingsmax_target_positions vocab_sizer Embeddingword_embed_proj_dim embed_tokensr r_embed_positionsrj project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layerslayersgradient_checkpointing post_initr)rZr*s `r+r(zOPTDecoder.__init__s    ~ )!.$*$B! +L):Ftj |d}||zdz }|dd|df}t|j|||} | ||| } |j||}|| |j z} t!|jD]:\} }|jr tjg}||jkr,|| f| |||d |} ;|j|| } |j|| } t/| | S) Nz:You must specify exactly one of input_ids or inputs_embedsrG)rZrrdevicer1)rZrr-rp)r/)r-r/rpr)last_hidden_staterp)rirurrr rZget_seq_lengthr3onesrr4r5r rrrR enumeraterrLrandrrrr)r)rr-rprrr/rTpast_seen_tokens seq_length causal_mask pos_embedsroidx decoder_layerdropout_probabilitys r+r6zOPTDecoder.forward>sr  -t"; < [YZZ Z  !r9?2+>??I   --i88M  ?0*$+>>>O?N?Z?99;;;`a  !)M,?,BBJ"Z (;A(> S`SghhhN   <A>>>L(>9A=CCEEL'+;+<+<(<=L(;')+    )).:JYe)ff ? & OOM::M% m6J(K(KK #,DK"8"8   C} &+jnn#&77)M*) /#   MM  , 11-@@M   ' ,,];;M&++    r,NNNNNN)r7r8r9r:rr(rrrr3r<rr FloatTensorrrrrr6r=r>s@r+rrs)#y######J .2.2(,26!%04K K #d*K  t+K  K (4/ K $; K &-K +,K  !K K K ^_ K K K K K r,rceZdZdeffd ZdZdZee dde j dzde j dzde dzd e j dzd edzd e j dzd eed efdZxZS)OPTModelrZct|t||_|dSr)r'r(rdecoderrrs r+r(zOPTModel.__init__s@    !&))  r,c|jjSrrrr)s r+get_input_embeddingszOPTModel.get_input_embeddingss |((r,c||j_dSrrr)rCs r+set_input_embeddingszOPTModel.set_input_embeddingss$) !!!r,Nrr-rprrr/rTrrc x|jd||||||d|}t|j|j|j|jS)Nrr-r/rprr)rrprorr)rrrrpror) r)rr-rprrr/rTdecoder_outputss r+r6zOPTModel.forwardso4@4<4 )%+' 4 4 4 4 '-?+;)7&1     r,r)r7r8r9rr(rrrrr3r<rr rrrrrr6r=r>s@r+rrs!y )))***.2.2(,26!%04  #d*  t+   (4/  $;  &- +,  !   ^     r,rceZdZddiZfdZdZdZee dde j dzd e j dzd e dzd e j dzd e j dzd edzde j dzdee j zdeedeezfdZxZS)OPTForCausalLMzlm_head.weightz!model.decoder.embed_tokens.weightct|t||_t j|j|jd|_| dSNFr]) r'r(rrrrjrrlm_headrrs r+r(zOPTForCausalLM.__init__sc    f%% y!;V=NUZ[[[  r,c$|jjjSrrrrrs r+rz#OPTForCausalLM.get_input_embeddingsz!..r,c(||jj_dSrrrs r+rz#OPTForCausalLM.set_input_embeddings*/ '''r,Nrrr-rprlabelsrr/logits_to_keeprTrrc ~|jjd||||||d| } | j} t|trt | dn|} || dd| ddf} d}||jd| ||j j d| }t|| | j | j | jS)an labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```rN)logitsrrlossrrprorr)rrr isinstancer;slicerrS loss_functionrZrrrpror)r)rr-rprrrr/rrToutputsro slice_indicesrrs r+r6zOPTForCausalLM.forwardsJ,>4:+=, )%+' , , , ,  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AALLNN  %4%pVFt{OeppioppD%#3!/)     r,)NNNNNNNr)r7r8r9_tied_weights_keysr(rrrrr3r<rr rrr;rrrrr6r=r>s@r+rrsP*,OP///000.2.2(,26*.!%04-.< < #d*<  t+<  < (4/ <  4' < $;< &-< el*< +,<  ' '< < < ^< < < < < r,ra The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZdeffd Zee ddejdzdej dzde dzdej dzdejdzd e dzd ejdzd e e d eezfd ZdZdZxZS)OPTForSequenceClassificationrZct||j|_t||_t j|j|jd|_| dSr) r'r( num_labelsrrrrjrscorerrs r+r(z%OPTForSequenceClassification.__init__sj     +f%% Yv94?QVWWW  r,Nrr-rprrrr/rTrrc |j|f|||||d|} | j} || } ||jdd\} } n|jdd\} } |jj| dkrt d|jjd}n|}||jjk| jtj }tj |jd| jtj }||z d}n)d}t|jjd| tj | | j |f}d}|Z|jjf|jdkr d |j_nN|jdkr7|jtjks|jtjkr d |j_n d |j_|jjd krWt-}|jdkr1|||}n|||}n|jjd krGt1}||d|j|d}n*|jjd krt5}|||}t7||| j| j| j S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rpr-r/rrNr%rz=Cannot handle batch sizes > 1 if no padding token is defined.rG)rrIz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)rrrrrZrrirRrr3int32arangeargmaxrerfr*r7 problem_typerrIr5r;rsqueezerrurrrpror)r)rr-rprrrr/rTtransformer_outputsror batch_sizesequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fcts r+r6z$OPTForSequenceClassification.forward's&8Btz 8 +)%' 8 8 8 8 ,= M**  *3/"1"*= 'J*7*=bqb*A 'J ; # + a\]] ] ; # +!#    "%)AAEEfmUZU`aaL!L)!F!Fr!J!J  !#    >*ZZZ    u|Jv}MMMOaab   {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8M$9$9$;$;V^^=M=MNNDD#8M6::DD)-JJJ+--x 2 22t G GUWYY)-III,..x v66/ /?-;*5     r,c$|jjjSrrrs r+rz1OPTForSequenceClassification.get_input_embeddings|rr,c(||jj_dSrrrs r+rz1OPTForSequenceClassification.set_input_embeddingsrr,)NNNNNNN)r7r8r9rr(rrr3r<rr rrrrrr6rrr=r>s@r+rrsLy.237(,26*.!%04Q Q #d*Q )D0Q  Q (4/ Q  4' Q $;Q &-Q +,Q  1 1Q Q Q ^Q f///0000000r,rceZdZdeffd Zee ddejdzdej dzde dzdej dzdejdzd ejdzd e dzd ejdzd e e d eezfdZdZdZxZS)OPTForQuestionAnsweringrZct|t||_t j|jd|_|dSr$) r'r(rrrrjr qa_outputsrrs r+r(z OPTForQuestionAnswering.__init__sX    f%% )F$>BB r,Nrr-rprstart_positions end_positionsrr/rTrrc |j|f|||||d| } | j} || } | dd\} }| d} |d}d}||t |dkr|d}t |dkr|d}| d}|d| | j }|d| | j }t|}|| |}|||}||zdz }t|| || j | j S) a Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```rrrGr1Nr) ignore_indexr%)r start_logits end_logitsror)rrrsplitr rSlenrtclamprRrrrror)r)rr-rprrrrr/rTr rorrr total_loss ignored_indexr start_lossend_losss r+r6zOPTForQuestionAnswering.forwards\8Btz 8 +)%' 8 8 8 8 ,= //#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEHHWWO)//=AADDV]SSM']CCCH!,@@Jx M::H$x/14J+%!-;*5     r,c$|jjjSrrrs r+rz,OPTForQuestionAnswering.get_input_embeddingsrr,c(||jj_dSrrrs r+rz,OPTForQuestionAnswering.set_input_embeddingsrr,)NNNNNNNN)r7r8r9rr(rrr3r<rr rrrrrr6rrr=r>s@r+rrsby.237(,263715!%04Q Q #d*Q )D0Q  Q (4/ Q )D0 Q '$.Q $;Q &-Q +,Q  - -Q Q Q ^Q f///0000000r,r)rrrrr)r?)=r:collections.abcrr3rtorch.nnrrr activationsr cache_utilsr r generationr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrconfiguration_optr get_loggerr7rerr ModulerfloatrWrYrrrrrrr__all__rr,r+r;s,$$$$$$ AAAAAAAAAA!!!!!!........))))))//////BBBBBB999999 GFFFFFFF&&&&&&RRRRRRRRRRRR777777555555((((((  H % %;;;;;BL;;;H%% I% <% % < % L4' %  %%%%%.T)T)T)T)T)29T)T)T)nEEEEE0EEEP         { { { { { #{ { { |( ( ( ( ( !( ( ( VQ Q Q Q Q 'Q Q Q h    c0c0c0c0c0#5c0c0  c0Lb0b0b0b0b00b0b0b0J   r,