tjXdZddlZddlmZmZmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZddlmZd d lmZmZmZmZmZmZmZmZGd d eZGd deZGddeZ GddeZ!GddeZ"GddeZ#GddeZ$GddeZ%gdZ&dS)zPyTorch CamemBERT model.N)BCEWithLogitsLossCrossEntropyLossMSELoss),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)Unpack)TransformersKwargsauto_docstring)can_return_tuple)RobertaForCausalLMRobertaForMaskedLMRobertaForMultipleChoiceRobertaForQuestionAnswering RobertaForSequenceClassificationRobertaForTokenClassification RobertaModelRobertaPreTrainedModelceZdZdZdS)CamembertPreTrainedModelrobertaN)__name__ __module__ __qualname__base_model_prefixm/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/camembert/modular_camembert.pyrr,s!r#rceZdZdS)CamembertModelN)rrr r"r#r$r&r&0sDr#r&c2eZdZdddZfdZee ddejdzdej dzdejdzd ejdzd ej dzd ej dzd ej dzd ejdzde e de ej ezfdZxZS)CamembertForMaskedLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasczt||`t|d|_dSNFadd_pooling_layersuper__init__ camembertr&rselfconfig __class__s r$r2zCamembertForMaskedLM.__init__:8    N%fFFF r#N input_idsattention_masktoken_type_ids position_ids inputs_embedsencoder_hidden_statesencoder_attention_masklabelskwargsreturnc t|j|f||||||dd| } | d} || } d} |e|| j}t }|| d|jj|d} t| | | j | j S)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)r:r;r<r=r>r? return_dictrNlosslogits hidden_states attentions) rlm_headtodevicerviewr6 vocab_sizer rIrJ)r5r9r:r;r<r=r>r?r@rAoutputssequence_outputprediction_scoresmasked_lm_lossloss_fcts r$forwardzCamembertForMaskedLM.forward@s:$,   ))%'"7#9      "!* LL99  YY0788F'))H%X&7&<&;?*.5 5 #d*5 )D05 (4/ 5 &- 5 (4/ 5  %0475 !& 1D 85  4'5 +,5  u| ~ -5 5 5 ^5 5 5 5 5 r#r(ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) "CamembertForSequenceClassificationczt||`t|d|_dSr-r0r4s r$r2z+CamembertForSequenceClassification.__init__{r8r#Nr9r:r;r<r=r@rArBc |j|f||||dd|}|d} || } d} |t|| j}|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt} |jdkr1| | | } n| | |} n|jjdkrGt} | | d |j|d } n*|jjdkrt} | | |} t!| | |j|j S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Tr:r;r<r=rDrN regressionsingle_label_classificationmulti_label_classificationrErF)r classifierrLrMr6 problem_type num_labelsdtyperWlongintrsqueezerrNrr rIrJ r5r9r:r;r<r=r@rArPrQrHrGrTs r$rUz*CamembertForSequenceClassification.forwards6$,  ))%'     "!*11  YYv}--F{'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     r#NNNNNN)rrr r2rrrWrXrYrrrZr[r rUr\r]s@r$r_r_zs#GGGGG .237260426*.C C #d*C )D0C (4/ C &- C (4/ C  4'C +,C  u| 7 7C C C ^C C C C C r#r_ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) CamembertForMultipleChoiceczt||`t|d|_dS)NTr.r0r4s r$r2z#CamembertForMultipleChoice.__init__s8    N%fEEE r#Nr9r;r:r@r<r=rArBc | |jdn |jd}|)|d|dnd} |)|d|dnd} |)|d|dnd} |)|d|dnd} |=|d|d|dnd} |j| f| | | | dd|}|d}||}||}|d|}d}|4||j}t}|||}t|||j |j S)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NrcrET)r<r;r:r=rDrF) shaperNsizerdropoutrgrLrMrr rIrJ)r5r9r;r:r@r<r=rA num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrP pooled_outputrHreshaped_logitsrGrTs r$rUz"CamembertForMultipleChoice.forwardsV-6,Aioa((}GZ[\G] CLCXINN2,>,>???^bLXLdL--b,2C2CB2G2GHHHjnR`Rln11"n6I6I"6M6MNNNrvR`Rln11"n6I6I"6M6MNNNrv(   r=#5#5b#9#9=;M;Mb;Q;Q R R R  $,  *..,        ]33 // ++b+66  YY566F'))H8OV44D("!/)     r#ro)rrr r2rrrWrXrYrrrZr[r rUr\r]s@r$rqrqs#FFFFF .22637*.0426P P #d*P (4/P )D0 P  4' P &- P (4/P +,P  u| 8 8P P P ^P P P P P r#rqceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) CamembertForTokenClassificationczt||`t|d|_dSr-r0r4s r$r2z(CamembertForTokenClassification.__init__&r8r#Nr9r:r;r<r=r@rArBc |j|f||||dd|}|d} || } || } d} |`|| j}t } | | d|j|d} t| | |j |j S)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrbrNrErF) rrwrgrLrMrrNrir rIrJrns r$rUz'CamembertForTokenClassification.forward,s2$,  ))%'     "!*,,7711  YYv}--F'))H8FKKDO<= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) TrbrrcrE)dimN) ignore_indexr)rG start_logits end_logitsrIrJ) r qa_outputssplitrm contiguouslenrvclamprr rIrJ)r5r9r:r;r<r=rrrArPrQrHrr total_loss ignored_indexrT start_lossend_losss r$rUz%CamembertForQuestionAnswering.forwardjs0$,  ))%'     "!*11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM']CCCH!,@@Jx M::H$x/14J+%!!/)     r#)NNNNNNN)rrr r2rrrWrXrYrrrZr[r rUr\r]s@r$rrcs'GGGGG .2372604263715> > #d*> )D0> (4/ > &- > (4/ > )D0> '$.> +,>  u| ; ;> > > ^> > > > > r#rceZdZdddZfdZee ddejdzdej dzd ejdzd ejdzd ej dzd ej dzd ej dzdejdzde e ej dzde dzde ej zdeede ej ezfdZxZS)CamembertForCausalLMr)r*r+czt||`t|d|_dSr-r0r4s r$r2zCamembertForCausalLM.__init__r8r#Nrr9r:r;r<r=r>r?r@past_key_values use_cachelogits_to_keeprArBc l|d} |j|f||||||| | dd | } | j}t| trt | dn| }||dd|ddf}d}||jd|||jjd| }t||| j | j | j | j S)aq token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, CamembertForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("almanach/camembert-base") >>> config = AutoConfig.from_pretrained("almanach/camembert-base") >>> config.is_decoder = True >>> model = CamembertForCausalLM.from_pretrained("almanach/camembert-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) r:r;r<r=r>r?rrrD)rHr@rO)rGrHrrIrJcross_attentionsr")rlast_hidden_state isinstancerlslicerK loss_functionr6rOrrrIrJr)r5r9r:r;r<r=r>r?r@rrrrArPrI slice_indicesrHrGs r$rUzCamembertForCausalLM.forwards`  I@L  A ))%'"7#9+ A A  A A  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%pVFt{OeppioppD0#3!/)$5     r#) NNNNNNNNNNr)rrr rVr2rrrWrXrYrZboolrlr[rrrrUr\r]s@r$rrs"M . GGGGG .237260426:>;?*.BF!%-.O O #d*O )D0O (4/ O &- O (4/ O  %047O !& 1D 8O  4'O uU%6784?O $;O el*O +,O  u| @ @O O O ^O O O O O r#r)rr(rqrr_rr&r)'__doc__rWtorch.nnrrrmodeling_outputsrrr r r r r processing_utilsrutilsrr utils.genericrroberta.modeling_robertarrrrrrrrrr&r(r_rqrrr__all__r"r#r$rs AAAAAAAAAA'&&&&&77777777------                    """""5"""     \   C C C C C -C C C LL L L L L )IL L L ^Y Y Y Y Y !9Y Y Y x; ; ; ; ; &C; ; ; |G G G G G $?G G G T] ] ] ] ] -] ] ] @   r#