tj]dZddlmZddlZddlZddlmZddlmZm Z m Z ddl m Z ddlmZmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(e"j)e*Z+dZ,dZ- d0dej.dej/dej/dej/dej/dzde0dzde0deefdZ1Gdd ej.Z2d!Z3Gd"d#ej.Z4e Gd$d%eZ5e Gd&d'e5Z6e d()Gd*d+e5eZ7e d,)Gd-d.e5Z8gd/Z9dS)1zPyTorch CTRL model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs) CTRLConfigcNdtjdd|dzz|z z }||zS)Nri')torchpow)posi d_model_size angle_ratess d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr%-s0eiQ!V 'DEEEK  cttj|tj|dtj|tj|d|}tj|dddddf}tj|dddddf}tj||gd}|S)N)dtyperrrdim) r%rarangeint64to unsqueezesincoscat)positionr"r( angle_radssinescosines pos_encodings r$positional_encodingr82s XU[11144U;;EEaHH \55588??II!LLJ IjADqD) * *Ei 111add7+,,G9eW-2666L r&modulequerykeyvalueattention_maskscalingdropoutkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nr)rrr*)ptrainingr) sizermatmul transposer functionalsoftmaxr@rE contiguous) r:r;r<r=r>r?r@rA attn_weights attn_outputs r$eager_attention_forwardrNBs**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$r&c>eZdZdfd Z ddeefdZxZS)MultiHeadAttentionNc"t||_|j|_|j|_||_d|_t|j|jz |_ |j dz|_ tj |j|j|_tj |j|j|_tj |j|j|_tj |j|j|_dS)NTrC)super__init__confign_head num_headsn_embdr" layer_idx is_causalinthead_dimr?rLinearWqWkWvdenseselfrTrX __class__s r$rSzMultiHeadAttention.__init___s  "M"D->?? }d* )D-t/@AA)D-t/@AA)D-t/@AAYt0$2CDD r&rAc |jdd}g|d|jR}|||dd} |||dd} |||dd} ||| | |j\} } tj |j j t} | || | | |fd|jd|\} }| jg|dR} || } | |fS)Nr)rrr9)r@r?)shaper[r]viewrHr^r_updaterXr get_interfacerT_attn_implementationrNr?reshaperKr`)rbvkq layer_pastr>rA input_shape hidden_shape query_states key_states value_statesattention_interfacerMrLs r$forwardzMultiHeadAttention.forwardps~gcrcl 88b8$-88 wwqzz|44>>q!DD WWQZZ__\22<>q!DD  !'1'8'8\SWSa'b'b $J (?(M K ,.E) ) %8$7      % L % %  % % ! \*k);;;;;;FFHH jj-- L((r&NNN__name__ __module__ __qualname__rSrrru __classcell__rcs@r$rPrP^stEEEEEE, #)#)+,#)#)#)#)#)#)#)#)r&rPctjtj||tjtj||Srv)r Sequentialr\ReLU)r"dffs r$point_wise_feed_forward_networkrs5 =<55rwyy")CQ]B^B^ _ __r&c>eZdZdfd Z ddeefdZxZS) EncoderLayerNctt|||_t |j|j|_tj |jd|_ tj |jd|_ tj |j |_tj |j |_dS)NrXgư>eps)rRrSrPmulti_head_attentionrrWrffnr LayerNorm layernorm1 layernorm2Dropout resid_pdropdropout1dropout2ras r$rSzEncoderLayer.__init__s $6v$S$S$S!26=&*MM,v}$???,v}$??? 6#566  6#566 r&rAc ||}|j|||f||d|\}}||}||z}||} || } || } || z} | S)Nrnr>)rrrrrr) rbxrnr>rAnormedrM_out1out2 ffn_outputs r$ruzEncoderLayer.forwards##22    ")       QmmK00 ;t$$XXd^^ ]]:.. j  r&rvrwrxr}s@r$rrsn 7 7 7 7 7 7  +, r&rcHeZdZUeed<dZdZdZdZdZ e e dZ fdZ xZS)CTRLPreTrainedModelrT transformerT) hidden_states attentionsct|t|trDt j|jt|jj |j tj dSdSrv) rR _init_weights isinstance CTRLModelinitcopy_r7r8rT n_positionsr"rfloat)rbr:rcs r$rz!CTRLPreTrainedModel._init_weightssw f%%% fi ( (  J#%89RTZTginit%u%u       r&)ryrzr{r__annotations__base_model_prefix_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrP_can_record_outputsrr|r}s@r$rrsv %N"&%( r&rceZdZfdZdZdZeee dde j dzde dzde j dzde j dzd e j dzd e j dzd e dzd eed efdZxZS)rcVtj|_j|_t jjj|_ t j j |_ t j fdtjD|_t jjj|_|dt)j|jt,jd|dS)Nc2g|]}t|S)r)r).0r!rTs r$ z&CTRLModel.__init__..s&aaaa Vq A A Aaaar&rr7F) persistent)rRrSrWr"n_layer num_layersr Embedding vocab_sizewr embd_pdropr@ ModuleListrangehrlayer_norm_epsilon layernormregister_bufferr8rrr post_initrbrTrcs `r$rSzCTRLModel.__init__s    "M .f/??z&"344 aaaa5QWQ_K`K`aaabbfm9RSSS  /0BDDUW\Wbccpu    r&c|jSrvr)rbs r$get_input_embeddingszCTRLModel.get_input_embeddingss v r&c||_dSrvr)rbnew_embeddingss r$set_input_embeddingszCTRLModel.set_input_embeddingss r&N input_idspast_key_valuesr>token_type_ids position_ids inputs_embeds use_cacherAreturnc ||n |jj}||td|||}n|td|jdd} |jd} |j} |r|t |j}||nd} |@tj | | d| ztj | }| d}|2||} | tj |jz} nd} |!|jdkr|| d}t#|j|||| }|tj |jz}|j| |j |_|j|ddf}||z| z}||}|jD]}||f||d |}||}t1||r|nd S) a Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsr)r)rT)r(device)rTrr>rrrr(r)last_hidden_stater)rTr ValueErrorrrerr get_seq_lengthrr,longr/npsqrtr"ndimrfr r7r.r(r@rrr)rbrrr>rrrrrAro batch_sizer past_lengthtoken_type_embeds causal_mask pos_embedsrrs r$ruzCTRLModel.forwardsrD"+!6IIDKcdd d   FF9--MM  "TUU U#)#2#. "(+ %  ?0*$+>>>O:I:Uo44666[\   < [_{5RZ_ZdmstttL'11!44L  % $~ 6 6  ):!;!; ;   !   %.*=*A*A+00R@@N(;')+%     !2333 !-00mFY0ZZ&|QQQ7 % 25FF  ]33   AA** MM}55 &+/8BOOd    r&)NNNNNNN)ryrzr{rSrrrrrr LongTensorr FloatTensorboolrrrrur|r}s@r$rrsG&    .2(,37260426!%\ \ #d*\ \ )D0 \ (4/ \ &- \ (4/\ $;\ +,\  !\ \ \ ^_ \ \ \ \ \ r&rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc$eZdZddiZfdZee ddejdzde dzdej dzd ejdzd ejdzd ej dzd ejdzd e dzde ej zdeedefdZ dfd ZxZS)CTRLLMHeadModelzlm_head.weightztransformer.w.weightct|t||_t j|j|jd|_| dS)NTbias) rRrSrrrr\rWrlm_headrrs r$rSzCTRLLMHeadModel.__init__]s`    $V,,y0AMMM  r&Nrrrr>rrrlabelsrlogits_to_keeprArc T|j|f||||||d| } | d} t| trt| dn| } || dd| ddf}d}||j||fd|jji| }t||| j | j | j S)ag labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```rr>rrrrrNr)losslogitsrrr) rrrZslicer loss_functionrTrrrrr)rbrrr>rrrrrrrAtransformer_outputsr slice_indicesrrs r$ruzCTRLLMHeadModel.forwardes\/d.   +))%'      ,A. 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4% ;1 D&/?-;*5     r&Fc ptj|f|||d|}|dd|S)N)rris_first_iterationr)rRprepare_inputs_for_generationpop)rbrrrrrA model_inputsrcs r$rz-CTRLLMHeadModel.prepare_inputs_for_generations\ =uww<  +1       )4000r&) NNNNNNNNr)NNF)ryrzr{_tied_weights_keysrSrrrrr rrrZTensorrrrrurr|r}s@r$rrTsz+,BC.2(,37260426*.!%-.K K #d*K K )D0 K (4/ K &- K (4/K  4'K $;K el*K +,K  K K K ^K \SXr&ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZfdZee ddejdzdedzdej dzdejdzdejdzdej dzd ejdzd e dzd e e d e fd ZxZS)CTRLForSequenceClassificationct||j|_t||_t j|j|jd|_| dS)NFr) rRrS num_labelsrrrr\rW classifierrrs r$rSz&CTRLForSequenceClassification.__init__si     +$V,,)FM4?OOO r&Nrrr>rrrrrrArc |j|f||||||d| } | d} || } ||jdd\} }n|jdd\} }|jj| dkrt d|jjd}n|}||jjk| jtj }tj |jd| jtj }||z d}n)d}t |jjd | tj | | j |f}d}|Z|jjf|jdkr d |j_nN|jdkr7|jtjks|jtjkr d |j_n d |j_|jjd krWt+}|jdkr1|||}n|||}n|jjd krGt/}||d|j|d}n*|jjd krt3}|||}t5||| j| jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```rrNrrz=Cannot handle batch sizes > 1 if no padding token is defined.r)rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrr)rrrerT pad_token_idrr.rrint32r,argmaxlogger warning_oncercry problem_typerr(rrZrsqueezerrfrrrr)rbrrr>rrrrrrArrrrsequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fcts r$ruz%CTRLForSequenceClassification.forwardsv/d.   +))%'      ,A. //  *3/"1"*= 'J*7*=bqb*A 'J ; # + a\]] ] ; # +!#    "%)AAEEfmUZU`aaL!L)!F!Fr!J!J  !#    >*ZZZ    u|Jv}MMMOaab   {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8M$9$9$;$;V^^=M=MNNDD#8M6::DD)-JJJ+--x 2 22t G GUWYY)-III,..x v66' -;*5     r&)NNNNNNNN)ryrzr{rSrrrrr rrrrrrur|r}s@r$rrs5.2(,37260426*.!%Y Y #d*Y Y )D0 Y (4/ Y &- Y (4/Y  4'Y $;Y +,Y  "Y Y Y ^Y Y Y Y Y r&r)rrrr)Nr9):__doc__collections.abcrnumpyrrrtorch.nnrrrr r cache_utilsr r generationr masking_utilsr modeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrconfiguration_ctrlr get_loggerryrr%r8ModulerrrNrPrrrrrr__all__r&r$r#s$$$$$$ AAAAAAAAAA&&&&&&........))))))//////iiiiiiiiiiFFFFFFFF&&&&&& 877777555555******  H % %   ,!%% I% <% % < % L4' % T\ %%'(%%%%85)5)5)5)5)5)5)5)p```%%%%%29%%%P/(y y y y y #y y y x jjjjj)?jj  jZ   e e e e e $7e e   e P c b br&