tju{8dZddlZddlmZddlmZmZmZddlmZ ddl m Z ddl m Z ddlmZmZmZmZmZmZmZdd lmZdd lmZdd lmZmZmZdd lmZd dl m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ej(e)Z*Gdde"Z+Gdde%Z,Gdde!Z-Gdde#Z.eGddeZ/Gdde$Z0edGdd e/e Z1eGd!d"e/Z2Gd#d$ej3Z4ed%Gd&d'e/Z5eGd(d)e/Z6eGd*d+e/Z7Gd,d-ej3Z8eGd.d/e/Z9gd0Z:dS)1zPyTorch RoBERTa model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)gelu)GenerationMixin),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)BertCrossAttentionBertEmbeddings BertLayer BertModelBertSelfAttention) RobertaConfigc eZdZfdZ d dejdzdejdzdejdzdejdzdef d Ze d Z e d d Z xZ S)RobertaEmbeddingsct||`|`|j|_t j|j|j|j|_dS)N) padding_idx) super__init__ pad_token_idposition_embeddingsr"nn Embeddingmax_position_embeddings hidden_sizeselfconfig __class__s i/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/roberta/modular_roberta.pyr$zRobertaEmbeddings.__init__-sa       $!.#%<  *F,>DL\$ $ $    Nr input_idstoken_type_ids position_ids inputs_embedspast_key_values_lengthc*|:||||j|}n|||j}||}n|dd}|\}}|t |drl|j|j|j dd} tj | d|} | ||}n+tj |tj |jj}|||}||} || z} ||} | | z} || } || } | S)Nr2rr)dimindexdtypedevice)"create_position_ids_from_input_idsr"&create_position_ids_from_inputs_embedssizehasattrr2tor<expandshapetorchgatherzeroslongr3word_embeddingstoken_type_embeddingsr& LayerNormdropout) r,r1r2r3r4r5 input_shape batch_size seq_lengthbuffered_token_type_idsrI embeddingsr&s r/forwardzRobertaEmbeddings.forward8s  $#FFt/1G    $JJ=Z^Zjkk  #..**KK',,..ss3K!, J  !t-.. m*.*=*@*@AT*U*U*\*\]i]opq]rtv*w*w'*/,7NTU]i*j*j*j'!8!?!? J!W!W!&[ SWSdSk!l!l!l   00;;M $ : :> J J"%:: "66|DD"55 ^^J// \\*-- r0c|dd}|d}tj|dz||zdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr7rr:r)r?rDarangerGr< unsqueezerB)r4r"rLsequence_lengthr3s r/r>z8RobertaEmbeddings.create_position_ids_from_inputs_embedshs~$((**3B3/ %a.| !O_{:Q>ejYfYm   %%a((// <<r= __classcell__r.s@r/r r ,s      .2260426&' ..#d*.(4/.&- . (4/ . !$ ....`==\=" 8 8 8\ 8 8 8 8 8r0r ceZdZdS)RobertaSelfAttentionNr^r_r`r0r/rgrgDr0rgceZdZdS)RobertaCrossAttentionNrhrir0r/rlrlrjr0rlceZdZdS) RobertaLayerNrhrir0r/rnrnrjr0rncpeZdZeZdZdZdZdZdZ dZ e e e dZejfdZxZS)RobertaPreTrainedModelrobertaT) hidden_states attentionscross_attentionsct|t|trt j|jdSt|trjt j|j tj |j j d dt j|jdSdS)zInitialize the weightsr7)rr7N)r# _init_weights isinstance RobertaLMHeadinitzeros_biasr copy_r3rDrSrCrBr2)r,moduler.s r/rvz$RobertaPreTrainedModel._init_weightss f%%% fm , , / K $ $ $ $ $  1 2 2 / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h K- . . . . . / /r0)r^r_r`r config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrnrgrl_can_record_outputsrDno_gradrvrdres@r/rprps L!&*#N"&%*1 U]__////_/////r0rpc eZdZdfd ZxZS) RobertaModelTcLt||dSN)r#r$)r,r-add_pooling_layerr.s r/r$zRobertaModel.__init__s# v&&&&&r0)T)r^r_r`r$rdres@r/rrs=''''''''''r0rzS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introceZdZdddZfdZdZdZee dd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde j dzde j dzde e e j dzde dzdee jzdeede e jezfdZxZS)RobertaForCausalLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasct||jstdt |d|_t||_| dS)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr r#r$ is_decoderloggerwarningrrqrxlm_head post_initr+s r/r$zRobertaForCausalLM.__init__su      n NNl m m m#FeDDD $V,,  r0c|jjSrrdecoderr,s r/get_output_embeddingsz(RobertaForCausalLM.get_output_embeddings |##r0c||j_dSrrr,new_embeddingss r/set_output_embeddingsz(RobertaForCausalLM.set_output_embeddings- r0Nrr1attention_maskr2r3r4encoder_hidden_statesencoder_attention_masklabelspast_key_values use_cachelogits_to_keepkwargsreturnc l|d} |j|f||||||| | dd | } | j}t| trt | dn| }||dd|ddf}d}||jd|||jjd| }t||| j | j | j | j S)am token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rr2r3r4rrrr return_dict)logitsr vocab_size)lossrrrrrsrtri)rqlast_hidden_staterwrYslicer loss_functionr-rr rrrrsrt)r,r1rr2r3r4rrrrrrroutputsrr slice_indicesrrs r/rQzRobertaForCausalLM.forwards`  I@L  A ))%'"7#9+ A A  A A  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%pVFt{OeppioppD0#3!/)$5     r0) NNNNNNNNNNr)r^r_r`_tied_weights_keysr$rrrrrDrarbtupleboolrYTensorrrr rQrdres@r/rrs#N .     $$$....237260426:>;?*.BF!%-.O O #d*O )D0O (4/ O &- O (4/ O  %047O !& 1D 8O  4'O uU%6784?O $;O el*O +,O  u| @ @O O O ^O O O O O r0rc>eZdZdddZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde j dzde e dee jezfdZxZS)RobertaForMaskedLMrrrct||jrtdt |d|_t||_| dS)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrr+s r/r$zRobertaForMaskedLM.__init__.s~       NN1    $FeDDD $V,,  r0c|jjSrrrs r/rz(RobertaForMaskedLM.get_output_embeddings=rr0c||j_dSrrrs r/rz(RobertaForMaskedLM.set_output_embeddings@rr0Nr1rr2r3r4rrrrrc t|j|f||||||dd| } | d} || } d} |e|| j}t }|| d|jj|d} t| | | j | j S)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr2r3r4rrrrNr7rrrrrs) rqrrAr<rviewr-rr rrrs)r,r1rr2r3r4rrrrrsequence_outputprediction_scoresmasked_lm_lossloss_fcts r/rQzRobertaForMaskedLM.forwardCs:$,   ))%'"7#9      "!* LL99  YY0788F'))H%X&7&<&;?*.5 5 #d*5 )D05 (4/ 5 &- 5 (4/ 5  %0475 !& 1D 85  4'5 +,5  u| ~ -5 5 5 ^5 5 5 5 5 r0rc(eZdZdZfdZdZxZS)rxz*Roberta Head for masked language modeling.cttj|j|j|_tj|j|j|_tj|j|j |_ tj tj |j |_dS)N)eps)r#r$r'Linearr*denserJlayer_norm_eps layer_normrr ParameterrDrFr{r+s r/r$zRobertaLMHead.__init__s Yv163EFF ,v'9v?TUUUy!3V5FGG LV->!?!?@@ r0c ||}t|}||}||}|Sr)rrrrr,featuresrxs r/rQzRobertaLMHead.forwardsE JJx  GG OOA   LLOOr0r^r_r`__doc__r$rQrdres@r/rxrx}sR44AAAAAr0rxz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) RobertaForSequenceClassificationct||j|_||_t |d|_t ||_|dSNFr) r#r$ num_labelsr-rrqRobertaClassificationHead classifierrr+s r/r$z)RobertaForSequenceClassification.__init__sg     + #FeDDD 3F;; r0Nr1rr2r3r4rrrc |j|f||||dd|}|d} || } d} |t|| j}|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt} |jdkr1| | | } n| | |} n|jjdkrGt} | | d |j|d } n*|jjdkrt} | | |} t!| | |j|j S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr2r3r4rrNr regressionsingle_label_classificationmulti_label_classificationr7r)rqrrAr<r- problem_typerr;rDrGrYrsqueezerrrrrrrs r,r1rr2r3r4rrrrrrrs r/rQz(RobertaForSequenceClassification.forwards6$,  ))%'     "!*11  YYv}--F{'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     r0NNNNNN)r^r_r`r$rrrDrarbrrrrrrQrdres@r/rrs     .237260426*.C C #d*C )D0C (4/ C &- C (4/ C  4'C +,C  u| 7 7C C C ^C C C C C r0rceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) RobertaForMultipleChoicect|t||_t j|j|_t j|j d|_ | dS)Nr) r#r$rrqr'Dropouthidden_dropout_probrKrr*rrr+s r/r$z!RobertaForMultipleChoice.__init__sl    #F++ z&"<== )F$6:: r0Nr1r2rrr3r4rrc | |jdn |jd}|)|d|dnd} |)|d|dnd} |)|d|dnd} |)|d|dnd} |=|d|d|dnd} |j| f| | | | dd|}|d}||}||}|d|}d}|4||j}t}|||}t|||j |j S)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr7T)r3r2rr4rr) rCrr?rqrKrrAr<rr rrrs)r,r1r2rrr3r4r num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr pooled_outputrreshaped_logitsrrs r/rQz RobertaForMultipleChoice.forwardsV-6,Aioa((}GZ[\G] CLCXINN2,>,>???^bLXLdL--b,2C2CB2G2GHHHjnR`Rln11"n6I6I"6M6MNNNrvR`Rln11"n6I6I"6M6MNNNrv(   r=#5#5b#9#9=;M;Mb;Q;Q R R R  $,  *..,        ]33 // ++b+66  YY566F'))H8OV44D("!/)     r0r)r^r_r`r$rrrDrarbrrrrr rQrdres@r/rrs.22637*.0426P P #d*P (4/P )D0 P  4' P &- P (4/P +,P  u| 8 8P P P ^P P P P P r0rceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) RobertaForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSr)r#r$rrrqclassifier_dropoutrr'rrKrr*rrr,r-rr.s r/r$z&RobertaForTokenClassification.__init__Ps     +#FeDDD )/)B)NF % %TZTn z"455 )F$68IJJ r0Nr1rr2r3r4rrrc |j|f||||dd|}|d} || } || } d} |`|| j}t } | | d|j|d} t| | |j |j S)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNr7r) rqrKrrAr<rrrrrrrsrs r/rQz%RobertaForTokenClassification.forward^s2$,  ))%'     "!*,,7711  YYv}--F'))H8FKKDO<= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Trrrr7rWN) ignore_indexr)r start_logits end_logitsrrrs) rqrsplitr contiguouslenr?clamprrrrrs)r,r1rr2r3r4rrrrrrrr  total_loss ignored_indexr start_lossend_losss r/rQz#RobertaForQuestionAnswering.forwards0$,  ))%'     "!*11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM']CCCH!,@@Jx M::H$x/14J+%!!/)     r0)NNNNNNN)r^r_r`r$rrrDrarbrrrrrrQrdres@r/rrs".2372604263715> > #d*> )D0> (4/ > &- > (4/ > )D0> '$.> +,>  u| ; ;> > > ^> > > > > r0r)rrrrrrrrp);rrDtorch.nnr'rrrrry activationsr generationr modeling_outputsr r r r rrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrbert.modeling_bertrrrrrconfiguration_robertar get_loggerr^rr rgrlrnrprrrModulerxrrrrr__all__rir0r/r s AAAAAAAAAA&&&&&&)))))).-----&&&&&&@@@@@@@@@@------llllllllllllll000000  H % %\8\8\8\8\8\8\8\8~     ,        .        9   /////_///2'''''9'''  i i i i i /i i  i XR R R R R /R R R jBI, Q Q Q Q Q '=Q Q  Q h] ] ] ] ] 5] ] ] @C C C C C $:C C C L ,K K K K K "8K K K \   r0