tjddlmZddlZddlmZddlmZmZmZddlm Z ddl m Z m Z ddlmZmZmZddlmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3e,j4e5Z6Gddej7Z8 dKdej7dej9dej9dej9dej9dzde:dzde:de&e*fdZ;Gd d!ej7Z<Gd"d#ej7Z=Gd$d%ej7Z>Gd&d'ej7Z?Gd(d)ej7Z@Gd*d+ej7ZAGd,d-eZBGd.d/ej7ZCe+Gd0d1e$ZDGd2d3ej7ZEGd4d5ej7ZFe+d67Gd8d9eDZGe+Gd:d;eDZHGd<d=ej7ZIe+d>7Gd?d@eDZJe+GdAdBeDZKe+GdCdDeDZLe+GdEdFeDZMe+dG7GdHdIeDeZNgdJZOdS)L)CallableN)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNgelu)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)CamembertConfigc eZdZdZfdZ ddejdzdejdzdejdzdejdzd ed ej f d Z e d Z e dd Z xZS)CamembertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj |j|j |_ tj |j |_|dt!j|jdd|dt!j|jt jd|j|_tj|j|j|j|_dS) N) padding_idxeps position_idsr$F) persistenttoken_type_ids)dtype)super__init__nn Embedding vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangemax_position_embeddingsexpandzerosr,sizelongr)position_embeddingsselfconfig __class__s n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/camembert/modeling_camembert.pyr3zCamembertEmbeddings.__init__;sJ !|F,=v?Q_e_rsss%'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg    ".#%<  *F,>DL\$ $ $    Nr input_idsr0r, inputs_embedspast_key_values_lengthreturnc*|:||||j|}n|||j}||}n|dd}|\}}|t |drl|j|j|j dd} tj | d|} | ||}n+tj |tj |jj}|||}||} || z} ||} | | z} || } || } | S)Nr.r0rr$)dimindexr1device)"create_position_ids_from_input_idsr)&create_position_ids_from_inputs_embedsrGhasattrr0torXrEshaperBgatherrFrHr,r9r;rIr<r@) rKrPr0r,rQrR input_shape batch_size seq_lengthbuffered_token_type_idsr; embeddingsrIs rNforwardzCamembertEmbeddings.forwardOs  $#FFt/1G    $JJ=Z^Zjkk  #..**KK',,..ss3K!, J  !t-.. m*.*=*@*@AT*U*U*\*\]i]opq]rtv*w*w'*/,7NTU]i*j*j*j'!8!?!? J!W!W!&[ SWSdSk!l!l!l   00;;M $ : :> J J"%:: "66|DD"55 ^^J// \\*-- rOc|dd}|d}tj|dz||zdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr.r$rWr)rGrBrCrHrX unsqueezerE)rQr)r_sequence_lengthr,s rNrZz:CamembertEmbeddings.create_position_ids_from_inputs_embedss~$((**3B3/ %a.| !O_{:Q>ejYfYm   %%a((// <<>L=((6?([[L,|U33K''1--88::K  $$rOc eZdZd fd Z d dejdejdzdedzdee de ejf d Z xZ S) CamembertSelfAttentionFNct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj |j|j |_ tj |j|j |_tj |j|j |_tj|j|_|j|_||_||_dSNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r)r2r3r7num_attention_headsr[ ValueErrorrLrkattention_head_size all_head_sizerr4Linearr|r}r~r>attention_probs_dropout_probr@ is_decoder is_causal layer_idxrKrLrrrMs rNr3zCamembertSelfAttention.__init__sG    : :a ? ?PVXhHiHi ?8F$688 4888  #)#= #&v'9F?? O*9*N'&=%C%CI{\`\j%k%k "I{(?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH L((rOFN)NN rprqrrr3rBrvrur rrtuplerdrxrys@rNrrs######648(, ')')|'))D0') ') +, ') u|  ')')')')')')')')rOrceZdZd fd Z d dejdejdzdejdzdedzdee d e ejf d Z xZ S) CamembertCrossAttentionFNct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj |j|j |_ tj |j|j |_tj |j|j |_tj|j|_||_||_dSr)r2r3r7rr[rrLrkrrrr4rr|r}r~r>rr@rrrs rNr3z CamembertCrossAttention.__init__s=    : :a ? ?PVXhHiHi ?8F$688 4888  #)#= #&v'9FB*4>:(?(M K ,.E) ) %8$7      % $}@CC$,.L % %  % % ! \*k);;;;;;FFHH L((rOr)NNN) rprqrrr3rBrvrur rrrrdrxrys@rNrrs######4;?376: 1)1)|1) %0471))D0 1) -t3 1) +, 1) u| 1)1)1)1)1)1)1)1)rOrcPeZdZfdZdejdejdejfdZxZS)CamembertSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr*) r2r3r4rr7denser<r=r>r?r@rJs rNr3zCamembertSelfOutput.__init__Nsf Yv163EFF f&8f>STTTz&"<== rOr input_tensorrSc||}||}|||z}|SNrr@r<rKrrs rNrdzCamembertSelfOutput.forwardT@ =11  ]33 }|'CDD rOrprqrrr3rBrvrdrxrys@rNrrMi>>>>> U\RWR^rOrceZdZd fd Z d dejdejdzdejdzdejdzdedzd ee d e ejfd Z xZ S)CamembertAttentionFNct||_|rtnt}|||||_t ||_dS)Nrr)r2r3is_cross_attentionrrrKroutput)rKrLrrrattention_classrMs rNr3zCamembertAttention.__init__\s] "45Gc11Mc#OFi9UUU )&11 rOrrrencoder_attention_maskrrrSc v|js|n|}|j|f|||d|\}}|||}||fS)N)rrr)rrKr) rKrrrrrrattention_outputrs rNrdzCamembertAttention.forwardcsq04/FbLb)2 * "7)+ * *  * * &, ;;'7GG--rO)FNFNNNNrrys@rNrr[s22222248:>;?(, ..|.)D0. %047 . !& 1D 8 .  .+,. u| ........rOrcBeZdZfdZdejdejfdZxZS)CamembertIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r2r3r4rr7intermediate_sizerr hidden_actstrr intermediate_act_fnrJs rNr3zCamembertIntermediate.__init__ysn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rOrrScZ||}||}|Sr)rr)rKrs rNrdzCamembertIntermediate.forwards, =11 00?? rOrrys@rNrrxs^99999U\elrOrcPeZdZfdZdejdejdejfdZxZS)CamembertOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r2r3r4rrr7rr<r=r>r?r@rJs rNr3zCamembertOutput.__init__sf Yv79KLL f&8f>STTTz&"<== rOrrrSc||}||}|||z}|Srrrs rNrdzCamembertOutput.forwardrrOrrys@rNrrrrOrceZdZd fd Z d dejdejdzdejdzdejdzdedzdee d ejfd Z d Z xZ S)CamembertLayerNct|j|_d|_t ||j||_|j|_|j|_|jr1|jst|dt |d|d|_ t||_ t||_ dS)Nr$rz> should be used as a decoder model if cross attention is addedFT)rrr)r2r3chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rKrLrrMs rNr3zCamembertLayer.__init__s '-'E$+Ff>O[deee +#)#=  # ? j D!h!h!hiii"4##' ###D  2&99%f-- rOrrrrrrrSc |j||fd|i|\}}|} |jr=|;t|dstd|d|j|d||fd|i|\} }| } t |j|j|j| } | S)Nrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rrr[rrrfeed_forward_chunkrr) rKrrrrrrself_attention_output_rcross_attention_output layer_outputs rNrdzCamembertLayer.forwards$24>  $ $ ,$  $ $  q 1 ? 64@4!122  DdDDD )<(;%%& )) !0 )  )) % "A 6 0  #T%A4CSUe  rOc\||}|||}|Sr)rr)rKrintermediate_outputrs rNrz!CamembertLayer.feed_forward_chunks2"//0@AA{{#68HII rOrr) rprqrrr3rBrvrur rrrdrrxrys@rNrrs......,48:>;?(, %%|%)D0% %047 % !& 1D 8 %  %+,% %%%%NrOrc(eZdZdZfdZdZxZS)CamembertLMHeadz,Camembert Head for masked language modeling.cttj|j|j|_tj|j|j|_tj|j|j |_ tj tj |j |_dSr)r2r3r4rr7rr<r= layer_normr6decoder ParameterrBrFbiasrJs rNr3zCamembertLMHead.__init__s Yv163EFF ,v'9v?TUUUy!3V5FGG LV->!?!?@@ rOc ||}t|}||}||}|Sr)rr rrrKfeaturesrxs rNrdzCamembertLMHead.forwardsE JJx  GG OOA   LLOOrOrprqrrrsr3rdrxrys@rNrrsR66AAAAArOrcpeZdZeZdZdZdZdZdZ dZ e e e dZejfdZxZS)CamembertPreTrainedModelrobertaT)r attentionscross_attentionsct|t|trt j|jdSt|trjt j|j tj |j j d dt j|jdSdS)zInitialize the weightsr.r-N)r2 _init_weightsrrinitzeros_rr'copy_r,rBrCr]rEr0)rKr{rMs rNrz&CamembertPreTrainedModel._init_weightss f%%% fo . . / K $ $ $ $ $  3 4 4 / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h K- . . . . . / /rO)rprqrrr% config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrBno_gradrrxrys@rNrrs"L!&*#N"&',3 U]__////_/////rOrceZdZfdZ d dejdejdzdejdzdejdzdedzdedzd e e d e eje zfd Z xZS) CamembertEncoderct|_tjfdt jD|_dS)Nc2g|]}t|S))r)r).0irLs rN z-CamembertEncoder.__init__.. s&#q#q#qAN6Q$G$G$G#q#q#qrO)r2r3rLr4 ModuleListrangenum_hidden_layerslayerrJs `rNr3zCamembertEncoder.__init__sX  ]#q#q#q#qQVW]WoQpQp#q#q#qrr rONrrrrr use_cacherrSc |t|jD]\}} | |||f||d|}t||r|ndS)N)rr)last_hidden_stater) enumeraterr) rKrrrrrrrr layer_modules rNrdzCamembertEncoder.forward s )44  OA|(L%(> /   MM9+/8BOOd    rO)NNNNN)rprqrrr3rBrvrur boolrrrrrdrxrys@rNrrssssss48:>;?(,!%  | )D0  %047  !& 1D 8    $; +,  u| H H        rOrcBeZdZfdZdejdejfdZxZS)CamembertPoolercttj|j|j|_tj|_dSr)r2r3r4rr7rTanh activationrJs rNr3zCamembertPooler.__init__'sC Yv163EFF '))rOrrScr|dddf}||}||}|SNr)rr()rKrfirst_token_tensor pooled_outputs rNrdzCamembertPooler.forward,s@+111a40 #566  66 rOrrys@rNr%r%&s^$$$$$ U\elrOr%a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introcVeZdZddgZdfd ZdZdZeee dde j dzd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde dzde dzdeedee j ezfdZdZxZS)CamembertModelr'rTct|||_d|_t ||_t ||_|rt|nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r2r3rLgradient_checkpointingr'rcrencoderr%pooler post_init)rKrLadd_pooling_layerrMs rNr3zCamembertModel.__init__Ds{     &+#-f55'// 1BLof---  rOc|jjSrrcr9rKs rNget_input_embeddingsz#CamembertModel.get_input_embeddingsUs ..rOc||j_dSrr7)rKr~s rNset_input_embeddingsz#CamembertModel.set_input_embeddingsXs*/'''rONrPrr0r,rQrrrrrrSc j|du|duz rtd|jjr| | n |jj} nd} | r[|Y| |jjr6t t |jt |jnt |j}||nd} |||||| } | ||| ||\}}|j | f||||| |d| } | j }|j | |nd}t||| jS) Nz:You must specify exactly one of input_ids or inputs_embedsF)rLr)rPr,r0rQrR)rrembedding_outputrr)rrrrrr,)r  pooler_outputr)rrLrris_encoder_decoderr r get_seq_lengthrc_create_attention_masksr2r r3rr)rKrPrr0r,rQrrrrrrRr=encoder_outputssequence_outputr,s rNrdzCamembertModel.forward[s  -t"; < [YZZ Z ; ! %.%:  @UIII  0)4 8V4$L $D$D$DlZ^ZeFfFfFfggg!555  FUE`!?!?!A!A!Afg??%)'#9 +  261M1M)#9-"7+ 2N2 2 ..'$,   )"7#9+%      *;8< 8O O444UY ;-'+;    rOc|jjrt|j|||}nt|j||}|t|j|||}||fS)N)rLrQrr)rLrQr)rLrQrr)rLrrr)rKrrr=rrs rNrAz&CamembertModel._create_attention_maskss ; ! /{.- / NN7{.-N " -%>{.5&; &&& "555rO)T) NNNNNNNNN)rprqrr_no_split_modulesr3r9r;r"r#rrBrvr r#rrrrrdrArxrys@rNr/r/5s/0@A"///000 *..2.2,0-1596:(,!%? ? <$&?  t+?  t+ ? lT) ? |d* ?  %|d2? !& t 3? ? $;? +,?  u| K K? ? ? ^_ ? B6666666rOr/c>eZdZdddZfdZdZdZee dde j dzd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde j dzde e dee jezfdZxZS)CamembertForMaskedLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasct||jrtdt ||_t|d|_| dS)NzpIf you want to use `CamembertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr5 r2r3rloggerwarningrlm_headr/rr4rJs rNr3zCamembertForMaskedLM.__init__s~       NN1   'v.. %fFFF  rOc|jjSrrPrr8s rNget_output_embeddingsz*CamembertForMaskedLM.get_output_embeddings |##rOc||j_dSrrRrKnew_embeddingss rNset_output_embeddingsz*CamembertForMaskedLM.set_output_embeddings- rONrPrr0r,rQrrlabelsrrSc t|j|f||||||dd| } | d} || } d} |e|| j}t }|| d|jj|d} t| | | j | j S)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr0r,rQrr return_dictrNr.losslogitsrr) rrPr\rXrrrLr6rrr)rKrPrr0r,rQrrrZroutputsrCprediction_scoresmasked_lm_lossloss_fcts rNrdzCamembertForMaskedLM.forwards:$,   ))%'"7#9      "!* LL99  YY0788F'))H%X&7&<&;?*.5 5 #d*5 )D05 (4/ 5 &- 5 (4/ 5  %0475 !& 1D 85  4'5 +,5  u| ~ -5 5 5 ^5 5 5 5 5 rOrGc(eZdZdZfdZdZxZS)CamembertClassificationHeadz-Head for sentence-level classification tasks.c4ttj|j|j|_|j|jn|j}tj||_ tj|j|j |_ dSr) r2r3r4rr7rclassifier_dropoutr?r>r@ num_labelsout_projrKrLrhrMs rNr3z$CamembertClassificationHead.__init__s Yv163EFF )/)B)NF % %TZTn z"455  &"4f6GHH rOc |dddddf}||}||}tj|}||}||}|Sr*)r@rrBtanhrjrs rNrdz#CamembertClassificationHead.forward"sj QQQ111W  LLOO JJqMM JqMM LLOO MM!  rOrrys@rNrfrfsR77IIIIIrOrfz Camembert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) "CamembertForSequenceClassificationct||j|_||_t ||_t |d|_|dSNFrL) r2r3rirLrf classifierr/rr4rJs rNr3z+CamembertForSequenceClassification.__init__3sg     + 5f==%fFFF  rONrPrr0r,rQrZrrSc |j|f||||dd|}|d} || } d} |t|| j}|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt} |jdkr1| | | } n| | |} n|jjdkrGt} | | d |j|d } n*|jjdkrt} | | |} t!| | |j|j S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr0r,rQr\rNr$ regressionsingle_label_classificationmulti_label_classificationr.r])rrrr\rXrL problem_typerir1rBrHrkrsqueezerrrrrr rKrPrr0r,rQrZrr`rCr_r^rcs rNrdz*CamembertForSequenceClassification.forward>s6$,  ))%'     "!*11  YYv}--F{'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     rONNNNNN)rprqrrr3r!rrBrtrurrrrvrrdrxrys@rNroro,s     .237260426*.C C #d*C )D0C (4/ C &- C (4/ C  4'C +,C  u| 7 7C C C ^C C C C C rOroceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) CamembertForMultipleChoicect|tj|j|_tj|jd|_t|d|_ | dS)Nr$TrL) r2r3r4r>r?r@rr7rrr/rr4rJs rNr3z#CamembertForMultipleChoice.__init__sq    z&"<== )F$6::%fEEE  rONrPr0rrZr,rQrrSc | |jdn |jd}|)|d|dnd} |)|d|dnd} |)|d|dnd} |)|d|dnd} |=|d|d|dnd} |j| f| | | | dd|}|d}||}||}|d|}d}|4||j}t}|||}t|||j |j S)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nr$r.T)r,r0rrQr\r]) r]rrGrr@rrr\rXrrrr)rKrPr0rrZr,rQr num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr`r,r_reshaped_logitsr^rcs rNrdz"CamembertForMultipleChoice.forwardsV-6,Aioa((}GZ[\G] CLCXINN2,>,>???^bLXLdL--b,2C2CB2G2GHHHjnR`Rln11"n6I6I"6M6MNNNrvR`Rln11"n6I6I"6M6MNNNrv(   r=#5#5b#9#9=;M;Mb;Q;Q R R R  $,  *..,        ]33 // ++b+66  YY566F'))H8OV44D("!/)     rOr{)rprqrrr3r!rrBrtrurrrrvrrdrxrys@rNr}r}s.22637*.0426P P #d*P (4/P )D0 P  4' P &- P (4/P +,P  u| 8 8P P P ^P P P P P rOr}ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e ej e zfd ZxZS) CamembertForTokenClassificationcZt||j|_|j|jn|j}t j||_t j|j |j|_ t|d|_ | dSrq)r2r3rirhr?r4r>r@rr7rrr/rr4rks rNr3z(CamembertForTokenClassification.__init__s     +)/)B)NF % %TZTn z"455 )F$68IJJ%fFFF  rONrPrr0r,rQrZrrSc |j|f||||dd|}|d} || } || } d} |`|| j}t } | | d|j|d} t| | |j |j S)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrtrNr.r]) rr@rrr\rXrrrirrrrzs rNrdz'CamembertForTokenClassification.forwards2$,  ))%'     "!*,,7711  YYv}--F'))H8FKKDO<= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Trtrr$r.riN) ignore_indexr)r^ start_logits end_logitsrr) rrsplitryrlenrGclamprrrr)rKrPrr0r,rQrrrr`rCr_rr total_loss ignored_indexrc start_lossend_losss rNrdz%CamembertForQuestionAnswering.forward:s0$,  ))%'     "!*11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M-33A}EEO)//=AAM']CCCH!,@@Jx M::H$x/14J+%!!/)     rO)NNNNNNN)rprqrrr3r!rrBrtrurrrrvrrdrxrys@rNrr.s".2372604263715> > #d*> )D0> (4/ > &- > (4/ > )D0> '$.> +,>  u| ; ;> > > ^> > > > > rOrzU Camembert Model with a `language modeling` head on top for CLM fine-tuning. ceZdZdddZfdZdZdZee dd e j dzd e j dzd e j dzd e j dzd e j dzde j dzde j dzde j dzde e e j dzde dzdee jzdeede e jezfdZxZS)CamembertForCausalLMrHrIrJct||jstdt ||_t|d|_| dS)NzQIf you want to use `CamembertLMHeadModel` as a standalone, add `is_decoder=True.`FrLrMrJs rNr3zCamembertForCausalLM.__init__su      p NNn o o o&v.. %fFFF  rOc|jjSrrRr8s rNrSz*CamembertForCausalLM.get_output_embeddingsrTrOc||j_dSrrRrVs rNrXz*CamembertForCausalLM.set_output_embeddingsrYrONrrPrr0r,rQrrrZrrlogits_to_keeprrSc l|d} |j|f||||||| | dd | } | j}t| trt | dn| }||dd|ddf}d}||jd|||jjd| }t||| j | j | j | j S)aq token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, CamembertForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("almanach/camembert-base") >>> config = AutoConfig.from_pretrained("almanach/camembert-base") >>> config.is_decoder = True >>> model = CamembertForCausalLM.from_pretrained("almanach/camembert-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rr0r,rQrrrrr\)r_rZr6)r^r_rrrr)rr rrkslicerP loss_functionrLr6rrrrr)rKrPrr0r,rQrrrZrrrrr`r slice_indicesr_r^s rNrdzCamembertForCausalLM.forwards`  I@L  A ))%'"7#9+ A A  A A  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%pVFt{OeppioppD0#3!/)$5     rO) NNNNNNNNNNr)rprqrrrdr3rSrXr!rrBrtrurr#rkrvrrrrdrxrys@rNrr}s#N .     $$$....237260426:>;?*.BF!%-.O O #d*O )D0O (4/ O &- O (4/ O  %047O !& 1D 8O  4'O uU%6784?O $;O el*O +,O  u| @ @O O O ^O O O O O rOr)rrGr}rrorr/r)Nrz)Pcollections.abcrrBtorch.nnr4rrrrr activationsr r cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericr!r"utils.output_capturingr#configuration_camembertr% get_loggerrprNModuler'rvfloatrrrrrrrrrrrr%r/rGrfror}rrr__all__rrOrNrs,%$$$$$ AAAAAAAAAA&&&&&&''''''''CCCCCCCCCC))))))JJJJJJJJ999999                    GFFFFFFF&&&&&&666666@@@@@@@@@@IIIIIIII555555444444  H % %g8g8g8g8g8")g8g8g8`!%% I% <% % < % L4' % T\ %%'(%%%%8@)@)@)@)@)RY@)@)@)FI)I)I)I)I)biI)I)I)X     ")   ........:     BI        bi   >>>>>/>>>Bbi,////////2     ry   @     bi       |6|6|6|6|6-|6|6  |6~R R R R R 3R R R j"), Q Q Q Q Q )AQ Q  Q h] ] ] ] ] !9] ] ] @C C C C C &>C C C LK K K K K $<K K K \ i i i i i 3_i i  i X   rO