tjPVdZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZmZerdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZm Z ddl!m"Z"ddl#m$Z$ddlm%Z%ddl&m'Z'e%j(e)Z*dZ+dZ,dZ-Gddej.Z/Gddej.Z0Gddej.Z1Gddej.Z2Gddej.Z3Gd d!ej.Z4Gd"d#eZ5Gd$d%ej.Z6Gd&d'ej.Z7Gd(d)ej.Z8Gd*d+ej.Z9Gd,d-ej.Z:Gd.d/ej.Z;Gd0d1ej.ZeGd8d9e=Z?ed:5Gd;de=ZAed?5Gd@dAe=ZBedB5GdCdDe=ZCeGdEdFe=ZDeGdGdHe=ZEeGdIdJe=ZFgdKZGdS)LzPyTorch FNet model.) dataclass)partialN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)auto_docstringis_scipy_available)linalg)ACT2FN)GradientCheckpointingLayer) BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)logging) FNetConfigc|jd}|d|d|f}|tj}tjd|||S)z4Applies 2D matrix multiplication to 3D input arrays.rNzbij,jk,ni->bnk)shapetypetorch complex64einsum)xmatrix_dim_onematrix_dim_two seq_lengths d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/fnet/modeling_fnet.py_two_dim_matmulr)5sNJ#KZK*$<=N uA <(!^^ L LLc$t|||SN)r))r$r%r&s r(two_dim_matmulr->s 1nn = ==r*c|}tt|jddD]#}tj||}$|S)z Applies n-dimensional Fast Fourier Transform (FFT) to input array. Args: x: Input n-dimensional array. Returns: n-dimensional Fourier transform of input n-dimensional array. rN)axis)reversedrangendimr!fft)r$outr/s r(fftnr5CsO Cqvqrr*++,,immCdm++ Jr*c*eZdZdZfdZddZxZS)FNetEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|j|_tj|j|_|dt)j|jdd|dt)j|jt(jddS) N) padding_idxeps position_idsrF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsLinear projectionDropouthidden_dropout_probdropoutregister_bufferr!arangeexpandzerosr<sizelongselfconfig __class__s r(rDzFNetEmbeddings.__init__VsN !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTT)F$68JKKz&"<==   EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg      r*Nc||}n|dd}|d}||jddd|f}|mt|dr2|jddd|f}||d|}|}n+t j|t j|jj}|| |}| |} || z} | |} | | z } | | } | | } || } | S)Nr>rr@rrBdevice)rYr<hasattrr@rWr!rXrZrarIrMrKrNrQrT) r\ input_idsr@r< inputs_embeds input_shaper'buffered_token_type_ids buffered_token_type_ids_expandedrM embeddingsrKs r(forwardzFNetEmbeddings.forwardjse  #..**KK',,..ss3K ^  ,QQQ  ^ J J"%:: "66|DD)) ^^J// __Z00 \\*-- r*)NNNN)__name__ __module__ __qualname____doc__rDri __classcell__r^s@r(r7r7SsRQQ     (!!!!!!!!r*r7c*eZdZfdZdZdZxZS)FNetBasicFourierTransformcrt||dSr,)rCrD_init_fourier_transformr[s r(rDz"FNetBasicFourierTransform.__init__s3  $$V,,,,,r*cl|js'ttjjd|_dS|jdkrtr|dtj tj |j tj |dtj tj |jtj tt|j|j|_dSt%jdt|_dSt|_dS) N)rdimdft_mat_hiddenrA dft_mat_seq)r%r&zpSciPy is needed for DFT matrix calculation and is not found. Using TPU optimized fast fourier transform instead.)use_tpu_fourier_optimizationsrr!r3r5fourier_transformrJr rUtensorr dftrGr"tpu_short_seq_lengthr-rzryrwarning)r\r]s r(rsz1FNetBasicFourierTransform._init_fourier_transforms*3 *%,UY^%H%H%HD " " "  +t 3 3!## .$$$el6:f>P3Q3QY^Yh&i&i&i$$!5< 6;V0W0W_d_n#o#o#o*1"43CTXTg***&&&**.&&&%)D " " "r*c<||j}|fSr,)r|real)r\ hidden_statesoutputss r(riz!FNetBasicFourierTransform.forwards" ((77<zr*)rjrkrlrDrsrirnros@r(rqrqsV-----***.r*rqc$eZdZfdZdZxZS)FNetBasicOutputcttj|j|j|_dSNr:)rCrDrrNrGrOr[s r(rDzFNetBasicOutput.__init__s9 f&8f>STTTr*c6|||z}|Sr,)rNr\r input_tensors r(rizFNetBasicOutput.forwards|m'CDD r*rjrkrlrDrirnros@r(rrsLUUUUUr*rc$eZdZfdZdZxZS)FNetFourierTransformctt||_t ||_dSr,)rCrDrqr\routputr[s r(rDzFNetFourierTransform.__init__s; -f55 %f-- r*cn||}||d|}|f}|SNr)r\r)r\r self_outputsfourier_outputrs r(rizFNetFourierTransform.forwards7yy// \!_mDD!#r*rros@r(rrsG..... r*rcBeZdZfdZdejdejfdZxZS)FNetIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr,) rCrDrrPrGintermediate_sizedense isinstance hidden_actstrrintermediate_act_fnr[s r(rDzFNetIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r*rreturncZ||}||}|Sr,)rrr\rs r(rizFNetIntermediate.forwards, =11 00?? r*rjrkrlrDr!Tensorrirnros@r(rrs^99999U\elr*rcPeZdZfdZdejdejdejfdZxZS) FNetOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) rCrDrrPrrGrrNrOrRrSrTr[s r(rDzFNetOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r*rrrc||}||}|||z}|Sr,)rrTrNrs r(rizFNetOutput.forwards@ =11  ]33 }|'CDD r*rros@r(rrsi>>>>> U\RWR^r*rc*eZdZfdZdZdZxZS) FNetLayerct|j|_d|_t ||_t ||_t||_ dSNr) rCrDchunk_size_feed_forward seq_len_dimrfourierr intermediaterrr[s r(rDzFNetLayer.__init__s^ '-'E$+F33 ,V44 (( r*c||}|d}t|j|j|j|}|f}|Sr)rrfeed_forward_chunkrr)r\rself_fourier_outputsr layer_outputrs r(rizFNetLayer.forwardsN#||M::-a00  #T%A4CSUc   /r*c\||}|||}|Sr,)rr)r\rintermediate_outputrs r(rzFNetLayer.feed_forward_chunks0"//??{{#6GG r*)rjrkrlrDrirrnros@r(rrsV)))))   r*rc&eZdZfdZddZxZS) FNetEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tS)r).0_r]s r( z(FNetEncoder.__init__.. s!#_#_#_!If$5$5#_#_#_r*F) rCrDr]r ModuleListr1num_hidden_layerslayergradient_checkpointingr[s `r(rDzFNetEncoder.__init__s`  ]#_#_#_#_uVE]?^?^#_#_#_`` &+###r*FTc|rdnd}t|jD] \}}|r||fz}||}|d}!|r||fz}|std||fDSt||S)Nrrc3K|]}||V dSr,r)rvs r( z&FNetEncoder.forward..s"XXq!-----XXr*)last_hidden_stater) enumeratertupler)r\routput_hidden_states return_dictall_hidden_statesi layer_module layer_outputss r(rizFNetEncoder.forward s"6@BBD(44 - -OA|# I$58H$H!(L77M)!,MM  E 1]4D D  YXX]4E$FXXXXX XN_````r*)FTrros@r(rrsT,,,,, aaaaaaaar*rcBeZdZfdZdejdejfdZxZS) FNetPoolercttj|j|j|_tj|_dSr,)rCrDrrPrGrTanh activationr[s r(rDzFNetPooler.__init__"sC Yv163EFF '))r*rrcr|dddf}||}||}|Sr)rr)r\rfirst_token_tensor pooled_outputs r(rizFNetPooler.forward's@+111a40 #566  66 r*rros@r(rr!s^$$$$$ U\elr*rcBeZdZfdZdejdejfdZxZS)FNetPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) rCrDrrPrGrrrrrtransform_act_fnrNrOr[s r(rDz$FNetPredictionHeadTransform.__init__2s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr*rrc||}||}||}|Sr,)rrrNrs r(riz#FNetPredictionHeadTransform.forward;s= =11 --m<< }55 r*rros@r(rr1scUUUUUU\elr*rc$eZdZfdZdZxZS)FNetLMPredictionHeadctt||_t j|j|j|_t j tj |j|_ dSr,) rCrDr transformrrPrGrFdecoder Parameterr!rXbiasr[s r(rDzFNetLMPredictionHead.__init__Csc 4V<<y!3V5FGG LV->!?!?@@ r*cZ||}||}|Sr,)rrrs r(rizFNetLMPredictionHead.forwardIs*}55  ]33 r*rros@r(rrBsLAAAAA r*rc$eZdZfdZdZxZS)FNetOnlyMLMHeadcptt||_dSr,)rCrDr predictionsr[s r(rDzFNetOnlyMLMHead.__init__Ps/ /77r*c0||}|Sr,)r)r\sequence_outputprediction_scoress r(rizFNetOnlyMLMHead.forwardTs ,,_==  r*rros@r(rrOsG88888!!!!!!!r*rc$eZdZfdZdZxZS)FNetOnlyNSPHeadcttj|jd|_dSNru)rCrDrrPrGseq_relationshipr[s r(rDzFNetOnlyNSPHead.__init__[s6  " &*r=) rC _init_weightsrr7initcopy_r<r!rVrrWzeros_r@)r\moduler^s r(rz!FNetPreTrainedModel._init_weightsws f%%% fn - - / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h K- . . . . . / /r*) rjrkrlr__annotations__base_model_prefixsupports_gradient_checkpointingrrnros@r(rrqsR &*#/////////r*rz0 Output type of [`FNetForPreTraining`]. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dS)FNetForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsr) rjrkrlrmrr! FloatTensorrrrrrrr*r(rr~s  &*D% d ")))26u(4/6668<U.5<<<59M5*+d299999r*rceZdZdZdfd ZdZdZe ddej dzdej dzd ej dzd ej dzd e dzd e dzd e e zfdZxZS) FNetModelz The model can behave as an encoder, following the architecture described in [FNet: Mixing Tokens with Fourier Transforms](https://huggingface.co/papers/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. Tct|||_t||_t ||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rCrDr]r7rhrencoderrpooler post_init)r\r]add_pooling_layerr^s r(rDzFNetModel.__init__ss     (00"6** ,=Gj(((4  r*c|jjSr,rhrIr\s r(get_input_embeddingszFNetModel.get_input_embeddingss ..r*c||j_dSr,r)r\values r(set_input_embeddingszFNetModel.set_input_embeddingss*/'''r*Nrcr@r<rdrrrc X||n |jj}||n |jj}||td||}|\} } n3|"|dd}|\} } ntd|jjr%| dkr|jj| krtd||jn|j} |gt|j dr1|j j ddd| f} | | | } | }n!tj |tj| }| ||||}|||| }|d }|j||nd}|s||f|d dzSt#|||j S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer>z5You have to specify either input_ids or inputs_embedsrxzThe `tpu_short_seq_length` in FNetConfig should be set equal to the sequence length being passed to the model when using TPU optimizations.r@r`)rcr<r@rd)rrrr)r pooler_outputr)r]rr ValueErrorrYr{rrarbrhr@rWr!rXrZr r rr)r\rcr@r<rdrrkwargsre batch_sizer'rarfrgembedding_outputencoder_outputsrrs r(rizFNetModel.forwards&%9$D $+Jj &1%cdd d  "#..**K%0 "J  &',,..ss3K%0 "J TUU U K 5 d"" 0J>>;  &/%:!! @T  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z??%)' +   ,, !5#'   *!,8< 8O O444UY  J#]3oabb6II I)-')7    r*)T)NNNNNN)rjrkrlrmrDrrr r! LongTensorrboolrrrirnros@r(r r s ///000.2260426,0#'D D #d*D (4/D &- D (4/ D #Tk D D[D   D D D ^D D D D D r*r z FNet Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. ceZdZdddZfdZdZdZe ddej dzd ej dzd ej dzd ej dzd ej dzd ej dzde dzde dzde e zfdZ xZS)FNetForPreTrainingcls.predictions.bias&fnet.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightct|t||_t ||_|dSr,)rCrDr rrclsr r[s r(rDzFNetForPreTraining.__init__ sQ    f%% '// r*c$|jjjSr,r%rrrs r(get_output_embeddingsz(FNetForPreTraining.get_output_embeddingsx#++r*cT||jj_|j|jj_dSr,r%rrrr\new_embeddingss r(set_output_embeddingsz(FNetForPreTraining.set_output_embeddings%'5$$2$7!!!r*Nrcr@r<rdlabelsnext_sentence_labelrrrc  ||n |jj}|||||||} | dd\} } || | \} }d}||t }|| d|jj|d}||dd|d}||z}|s| |f| ddz}||f|zn|St|| || jS)aH labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForPreTraining.from_pretrained("google/fnet-base") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Nr@r<rdrrrur>)rrrr) r]rrr%rviewrFrr)r\rcr@r<rdr0r1rrrrrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r(rizFNetForPreTraining.forwards_L&1%rurlogitsr) r]rrr%rr4rFrr)r\rcr@r<rdr0rrrrrrr7r6rs r(rizFNetForMaskedLM.forwardws$&1%:K[b[pqqqqr*NNNNNNN)rjrkrlr:rDr(r.r r!rrrrrirnros@r(r<r<`s6)?*R ,,,888*..2,0-1&*,0#'(r(r<$&(r t+(rlT) (r |d* (r  t# (r#Tk(rD[(r  (r(r(r^(r(r(r(r(rr*r<zT FNet Model with a `next sentence prediction (classification)` head on top. ceZdZfdZe d dejdzdejdzdejdzdejdzdejdzdedzd edzd ee zfd Z xZ S) FNetForNextSentencePredictionct|t||_t ||_|dSr,)rCrDr rrr%r r[s r(rDz&FNetForNextSentencePrediction.__init__r>r*Nrcr@r<rdr0rrrc ~||n |jj}|||||||} | d} || } d} |At } | | dd|d} |s| f| ddz}| | f|zn|St | | | jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForNextSentencePrediction.from_pretrained("google/fnet-base") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ```Nr3rr>rurB)r]rrr%rr4rr)r\rcr@r<rdr0rrrrrseq_relationship_scoresr8r6rs r(riz%FNetForNextSentencePrediction.forwardsH&1% 1` a classification loss is computed (Cross-Entropy). Nr3r regressionsingle_label_classificationmulti_label_classificationr>rurB)r]rrrTrO problem_typerNrBr!rZintrsqueezerr4rrr)r\rcr@r<rdr0rrrrrrCrr6rs r(riz%FNetForSequenceClassification.forwards$&1%r3rurB) r]rrr4rYrrTrOrrr)r\rcr@r<rdr0rrr num_choicesrrrCreshaped_logitsrr6rs r(rizFNetForMultipleChoice.forwardQsT&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  )) )%'!5#      ]33 // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E(d?ZaZoppppr*rD) rjrkrlrDr r!rrrrrirnros@r(rXrXEs*..2,0-1&*,0#'MqMq<$&Mq t+MqlT) Mq |d* Mq  t# Mq#TkMqD[Mq * *MqMqMq^MqMqMqMqMqr*rXceZdZfdZe d dejdzdejdzdejdzdejdzdejdzdedzd edzd ee zfd Z xZ S) FNetForTokenClassificationc6t||j|_t||_t j|j|_t j |j |j|_ | dSr,rMr[s r(rDz#FNetForTokenClassification.__init__sy     +f%% z&"<== )F$68IJJ r*Nrcr@r<rdr0rrrc ||n |jj}|||||||} | d} || } || } d} |Ft } | | d|j|d} |s| f| ddz}| | f|zn|St| | | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nr3rr>rurB) r]rrrTrOrr4rNrr)r\rcr@r<rdr0rrrrrrCrr6rs r(riz"FNetForTokenClassification.forwards &1%rv) ignore_indexru)r start_logits end_logitsr) r]rrresplitrV contiguouslenrYclamprrr)r\rcr@r<rdrfrgrrrrrrCrjrkr5 ignored_indexr6 start_lossend_lossrs r(riz FNetForQuestionAnswering.forwards&1%rsK!!!!!! AAAAAAAAAA&&&&&&77777777!!!!!!999999                      .-----666666******  H % %MMM>>>    88888RY888v##### ###Lbi     29        ry           *6aaaaa")aaa8         ")"     29   !!!!!bi!!!&&&&&bi&&& 9 9 9 9 929 9 9 9 / / / / // / / /  :::::{::   :$c c c c c #c c c L Z Z Z Z Z ,Z Z  Z z?r?r?r?r?r)?r?r?rD J J J J J $7J J  J Z GgGgGgGgGg$7GgGg  GgTYqYqYqYqYq/YqYqYqx7d7d7d7d7d!47d7d7dtB B B B B 2B B B J   r*