tjAdZddlZddlmZddlmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZmZddlm Z ej!e"Z#Gddej$Z%Gddej$Z&Gddej$Z'Gddej$Z(Gddej$Z)Gddej$Z* d|dej$dej+dej+d ej+d!ej+dzd"e,dzd#e,d$eefd%Z-Gd&d'ej$Z.Gd(d)ej$Z/Gd*d+ej$Z0Gd,d-ej$Z1Gd.d/ej$Z2Gd0d1ej$Z3Gd2d3ej$Z4eGd4d5e Z5Gd6d7ej$Z6 d}d9ej+d:e,d;e7dzdZ: d~d9ej+d?e7e9zd;e7dzd=e9fd@Z;GdAdBej$Z<GdCdDej$Z=GdEdFej$Z>GdGdHej$Z?GdIdJej$Z@edKLeGdMdNe ZAGdOdPe5ZBedQLeGdRdSe ZCedTLGdUdVe5ZDedWLeGdXdYe ZEedZLGd[d\e5ZFed]LeGd^d_e ZGed`LeGdadbe ZHed`LeGdcdde ZIdeejJjKdfej+dgej+fdhZLddiej+djej+dzdgej+fdkZMGdldme5ZNednLeGdodpe ZOGdqdre5ZPedsLeGdtdue ZQGdvdwej$ZRedxLGdydze5ZSgd{ZTdS)zPyTorch PatchTSMixer model.N)Callable) dataclass)PreTrainedModel) ModelOutput)initialization)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONS)Unpack)NegativeBinomialOutput NormalOutputStudentTOutput)TransformersKwargsauto_docstringcan_return_tuplelogging)PatchTSMixerConfigc2eZdZdZdedeffd ZdZxZS)PatchTSMixerGatedAttentionz Module that applies gated attention to input data. Args: in_size (`int`): The input size. out_size (`int`): The output size. in_sizeout_sizecttj|||_tjd|_dS)Ndim)super__init__nnLinear attn_layerSoftmax attn_softmax)selfrr __class__s t/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/patchtsmixer/modeling_patchtsmixer.pyrz#PatchTSMixerGatedAttention.__init__/sG )GX66J2...c`|||}||z}|SN)r#r!)r$inputs attn_weights r&forwardz"PatchTSMixerGatedAttention.forward4s0''(?(?@@ +% r')__name__ __module__ __qualname____doc__intrr, __classcell__r%s@r&rr&sd//s////// r'rc>eZdZdZdeffd ZdejfdZxZ S)PatchTSMixerBatchNormzP Compute batch normalization over the sequence length (time) dimension. configcttj|j|j|_dS)Neps)rrr BatchNorm1dd_modelnorm_eps batchnormr$r6r%s r&rzPatchTSMixerBatchNorm.__init__@s7 FOLLLr'r*c|dd}||}|ddS)a Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` r) transposer=)r$r*outputs r&r,zPatchTSMixerBatchNorm.forwardDs@!!!Q''''1%%%r' r-r.r/r0rrtorchTensorr,r2r3s@r&r5r5;ssM1MMMMMM &el & & & & & & & &r'r5cheZdZdZdeffd ZededejfdZ de j fdZ xZ S)PatchTSMixerPositionalEncodingz' Class for positional encoding r6ct|jr|||_dSt jtj|j |j |_dSr)) rruse_positional_encoding_init_pe position_encr ParameterrDzeros num_patchesr;r>s r&rz'PatchTSMixerPositionalEncoding.__init__Vsh   ) ^ $ f 5 5D    " U[9KV^-\-\ ] ]D   r'returnc|jdkr5tjtj|j|jd}nD|jdkr!tj|j|j}tjd|j d}tj tjd|jdtj d|jz z}tj ||z|dddddf<tj||z|dddddf<||z }||d zz }tj|d }nt#|jd |S) NrandomT requires_gradsincosrrr@g@ FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)positional_encoding_typerrLrDrandnrNr;rMarange unsqueezeexpmathlogsincosmeanstd ValueError)r6rKpositiondiv_terms r&rJz'PatchTSMixerPositionalEncoding._init_pe^s  *h 6 6< F4F(W(WgklllLL  , 8 8 ;v'96>JJL|Av'9::DDQGGHya!C!CQXHYHY\b\jHjFk!kllH$)Ih.A$B$BLADqD !$)Ih.A$B$BLADqD !',*;*;*=*==L'<+;+;+=+=+BCL< EJJJLL2CCC r' patch_inputc||jz}|Sr))rK)r$rd hidden_states r&r,z&PatchTSMixerPositionalEncoding.forwardrs"T%66 r')r-r.r/r0rr staticmethodrrLrJrDrEr,r2r3s@r&rGrGQs^1^^^^^^+ \&5<r'rGc>eZdZdZdeffd ZdejfdZxZ S)PatchTSMixerNormLayerzeNormalization block Args: config (`PatchTSMixerConfig`): Configuration. r6c t|j|_d|jvrt ||_dSt j|j|j |_dS)Nbatchr8) rrnorm_mlplowerr5normr LayerNormr;r<r>s r&rzPatchTSMixerNormLayer.__init__sl  fo++-- - --f55DIII V^IIIDIIIr'r*cTd|jvrwtj||jd|jdz|jd|jdf}||}tj||j}n||}|S)a Args: inputs (`torch.Tensor` of shape `((batch_size, num_channels, num_patches, d_model))`): Input to the normalization layer. Returns: `torch.Tensor` of shape `((batch_size, num_channels, num_patches, d_model))` rkrrr@r)rlrmrDreshapeshapern)r$r*inputs_reshapeds r&r,zPatchTSMixerNormLayer.forwards dm))++ + +#mLOfl1o5LOLOO#ii88O]?FLAAFFYYv&&F r'rCr3s@r&ririxssJ1JJJJJJelr'ric4eZdZfdZdejfdZxZS)PatchTSMixerMLPc<t||jz}tj|||_tj|j|_tj|||_ tj|j|_ dSr)) rrexpansion_factorrr fc1Dropoutdropoutdropout1fc2dropout2)r$ in_features out_featuresr6 num_hiddenr%s r&rzPatchTSMixerMLP.__init__sv  6#:: 9[*55 6>22 9Z66 6>22 r'r*c|tj||}||}||}|S)z Args: inputs (`torch.Tensor` of shape `((batch_size, num_channels, num_patches, d_model))`): Input to the MLP layer. Returns: `torch.Tensor` of the same shape as `inputs` )r{r functionalgelurxr|r})r$r*s r&r,zPatchTSMixerMLP.forwardsXr}11$((62B2BCCDD&!!v&& r')r-r.r/rrDrEr,r2r3s@r&rurusU33333 el        r'ruc>eZdZdZdeffd ZdejfdZxZ S)$PatchTSMixerChannelFeatureMixerBlockzThis module mixes the features in the channel dimension. Args: config (`PatchTSMixerConfig`): Configuration. r6ctt||_|j|_t |j|j||_|jr"t|j|j|_ dSdSNr~rr6rr) rrrirn gated_attnrunum_input_channelsmlpr gating_blockr>s r&rz-PatchTSMixerChannelFeatureMixerBlock.__init__s )&11  +"12       :1F>L=((6?([[L,|U33K''1--88::K  $$r'ceZdZdZ ddededed ed ed ed edzffd Z dde j de j dzde j dzdedzde e de e j e j dze e j dzff dZxZS)PatchTSMixerAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsrz is_decoderbias is_causalr6c t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).r)r)rrrrrzhead_dimr6rarrrrr k_projv_projq_projout_proj) r$rrrzrrrr6r%s r&rzPatchTSMixerAttention.__init__s "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r' hidden_stateskey_value_statesroutput_attentionsrrOc |du}|jdd}g|d|jR}|||dd} |r|n|} g| jddd|jR} || | dd} || | dd} tj|j j t}||| | | |f|j sdn|j |j|d|\}}|jg|dR}||}||dfS)z#Input shape: Batch x Time x ChannelNrrr@r)rzrr)rrrrviewrArrr get_interfacer6_attn_implementationrrrzrrqrr)r$rrrrris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs r&r,zPatchTSMixerAttention.forward0s.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT -?R))]B^)#2#.BBDMBB[[0055h??II!QOO {{>2277AAKKAqQQ (?(M K ,.E) ) %8$7      % $}>CC$,L/ % %  % % ! \*k);;;;;;FFHH mmK00 L$..r')rFTFN)NNF)r-r.r/r0r1floatboolrrrDrEr r tupler,r2r3s@r&rrsJGG  ,0CCCC C  C  CC#T)CCCCCCD15.2). 0/0/|0/ ,-0/ t+ 0/  $; 0/-.0/ u|U\D0% 2E2LL M0/0/0/0/0/0/0/0/r'rc.eZdZdZdeffd ZdZxZS)PatchMixerBlockzxThis module mixes the patch dimension. Args: config (`PatchTSMixerConfig`): Configuration. r6ctt||_|j|_|j|_t |j|j||_|jr t|j|j|_ |jr=t|j |j |j||_t||_dSdS)Nrr)rrrzr6)rrrirn self_attnrrurNrrrrr;self_attn_headsrzself_attn_layer norm_attnr>s r&rzPatchMixerBlock.__init__ks )&11 ) +"*+      t :6CU`f`r s s sD    ;#8 . 0 $$$D 36::DNNN ; ;r'c|}||}|jrY|j\}}}}|||z||}||d\}} } |||||}|dd}||}|jr||}|dd}|jr| ||z}||z} | S)z Args: hidden_state (`torch.Tensor`): Input tensor. Returns: `torch.Tensor`: Transformed tensor. F)rr@r) rnrrrrqrrArrrr) r$rfr batch_sizen_varsrNr;hidden_state_reshapedx_attn_rs r&r,zPatchMixerBlock.forwards yy.. > N7C7I 4J W$0$8$8f9Lk[b$c$c !//0EY^/__LFAq^^J WMMF$--a33 xx -- ? ;,,\::L$--a33 > A>>,*?@@LX% r'r-r.r/r0rrr,r2r3s@r&rrcs^;1;;;;;;4!!!!!!!r'rc>eZdZdZdeffd ZdejfdZxZ S)FeatureMixerBlockzThis module mixes the hidden feature dimension. Args: config (`PatchTSMixerConfig`): Configuration. r6ctt||_|j|_t |j|j||_|jr"t|j|j|_ dSdSr) rrrirnrrur;rrrr>s r&rzFeatureMixerBlock.__init__s )&11  +"      l :6>\b\j k k kD    l lr'hiddenc|}||}||}|jr||}||z}|S) Args: hidden (`torch.Tensor` of shape `(batch_size, num_patches, d_model)`): Input tensor to the layer. Returns: `torch.Tensor`: Transformed tensor. )rnrrr)r$rrrs r&r,zFeatureMixerBlock.forwardsW6""&!! ? /&&v..Fx r'rCr3s@r&rrssl1llllll elr'rc>eZdZdZdeffd ZdejfdZxZ S)PatchTSMixerLayerz The `PatchTSMixer` layer that does all three kinds of mixing. Args: config (`PatchTSMixerConfig`): Configuration. r6ctt||_t ||_|j|_|jdkrt||_dSdS)Nr6 mix_channel) rrr patch_mixerr feature_mixermoderchannel_feature_mixerr>s r&rzPatchTSMixerLayer.__init__sw *&999.f===K ;- ' ')MU[)\)\)\D & & & ( 'r'rc|jdkr||}||}||}|S)rr)rrrr)r$rs r&r,zPatchTSMixerLayer.forwardsO 9 % %//77F!!&))##F++ r'rCr3s@r&rrss ]1 ] ] ] ] ] ]elr'rc6eZdZdZdeffd ZddefdZxZS)PatchTSMixerBlockzThe main computing framework of the `PatchTSMixer` model. Args: config (`PatchTSMixerConfig`): Configuration. r6ctj}tjfdt |D|_dS)Nc0g|]}tS)r)r).0rr6s r& z.PatchTSMixerBlock.__init__.. s%$a$a$a!%6f%E%E%E$a$a$ar')rr num_layersr ModuleListrangemixers)r$r6rr%s ` r&rzPatchTSMixerBlock.__init__sU & m$a$a$a$auU_O`O`$a$a$abb r'Foutput_hidden_statescvg}|}|jD]$}||}|r||%|r||fS|dfS)as Args: hidden_state (`torch.Tensor`): The input tensor. output_hidden_states (`bool`, *optional*, defaults to False.): Whether to output the hidden states as well. Returns: `torch.Tensor`: The embedding. `list`: List of all hidden states if `output_hidden_states` is set to `True`. N)rappend)r$rfrall_hidden_states embeddingmods r&r,zPatchTSMixerBlock.forward sh ; 4 4CII# 4!((333  #// /d? "r'F) r-r.r/r0rrrr,r2r3s@r&rrsvc1cccccc##$########r'rc0eZdZdZddeffd ZdZxZS)PatchTSMixerForPredictionHeadzqPrediction Head for Forecasting Args: config (`PatchTSMixerConfig`): Configuration. Nr6ct|j|_|j|jt j|j|_|-t j|j |j z|j |_ n'| |j |j z|_ t jd|_dS)N start_dim)rrprediction_channel_indicessortrry head_dropout dropout_layerr rNr;prediction_lengthbase_forecast_blockget_parameter_projectionFlattenflatten)r$r6distribution_outputr%s r&rz&PatchTSMixerForPredictionHead.__init__-s *0*K'  * 6  + 0 0 2 2 2Z(;<<  &')y&2Dv~2UX^Xp'q'qD $ $':'S'S"V^3((D $zB/// r'c|}|}|}t|trt d|D}n|dd}j@t|trt fd|D}n|djf}|S)ar Args: hidden_features (`torch.Tensor` of shape `(batch_size, num_patch, d_model)` in `flatten` mode or `(batch_size, n_vars, num_patch, d_model)` in `common_channel`/`mix_channel` mode.): Input hidden features. Returns: `torch.Tensor` of shape `(batch_size, prediction_length, nvars)`. c3BK|]}|ddVdS)rrN)rA)rzs r& z8PatchTSMixerForPredictionHead.forward..Ps0CCQQ[[R00CCCCCCr'rrNc36K|]}|djfVdS).N)r)rrr$s r&rz8PatchTSMixerForPredictionHead.forward..Vs0 [ [Q3(G#G!H [ [ [ [ [ [r'.)rrr isinstancerrArr$hidden_featuresforecasts` r&r,z%PatchTSMixerForPredictionHead.forward?s,,77,,_==++O<< h & & 2CC(CCCCCHH))"b11H  * 6(E** J [ [ [ [RZ [ [ [[[#C)H$HIr'r)rr3s@r&rr%sc001000000$r'rc0eZdZdZddeffd ZdZxZS)PatchTSMixerLinearHeadzLinear head for Classification and Regression. Args: config (`PatchTSMixerConfig`): Configuration. Nr6ct|j|_|j|_|j|j}nd}||_|0t j|j|j z|z|j |_ n*| |j|j z|z|_ |jt j d|_nt j d|_t j|j|_dS)Nrrr)rrhead_aggregation output_rangerNrrr r;r num_targets projectionrrrryrrz)r$r6r mul_factorr%s r&rzPatchTSMixerLinearHead.__init__es  & 7"/  " *+JJJ#6  & i!::ZG"DOO 2JJ!::ZGDO  " *:333DLL:333DLz&"566 r'c|dd}|jdkr |d}nH|jdkr|dj}n!|jdkr|d}|jr||}||}||}|jE|j >tj ||j d |j d z z|j d z}|S) ai Args: hidden_features (`torch.Tensor` of shape `(batch_size x num_patch x d_model)` in `flatten` mode or `(batch_size x n_vars x num_patch x d_model)` in `common_channel`/`mix_channel` mode.): Input hidden features. Returns: `torch.Tensor` of shape `(batch_size x num_targets)`. rruse_last).rmax_poolravg_poolNrr) rArmaxvaluesr_rrzrrrrDsigmoid)r$r s r&r,zPatchTSMixerLinearHead.forwards*33B;;  J . .-g6OO  "j 0 0-11b199@OO  "j 0 0-22r2::O < <"ll?;;O,,77///::  $ ,43D3P o..$2CA2FIZ[\I]2]^aearstauu r'r)rr3s@r&rr]sc7717777778       r'rcdeZdZUeed<dZdZdZdZe j fdZ xZ S)PatchTSMixerPreTrainedModelr6model past_values)timeFczt|t|tr0|jjdkrt j|jdddSdSt|tr>t j |j j t j |j jdSdS)zInitialize weightsrQrg?)r_r`N)r _init_weightsr rGr6rVinitnormal_rKr5zeros_r=rones_weight)r$rr%s r&r$z)PatchTSMixerPreTrainedModel._init_weightss f%%% f< = = 0{3x?? V0sDDDDDD@?  5 6 6 0 K(- . . . Jv'. / / / / / 0 0r') r-r.r/r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointingrDno_gradr$r2r3s@r&rrsq #O &+#U]__ 0 0 0 0_ 0 0 0 0 0r'rc.eZdZdZdeffd ZdZxZS)PatchTSMixerPretrainHeadzcPretraining head. Args: config (`PatchTSMixerConfig`): Configuration. r6cttj|j|_tj|j|j|_ dSr)) rrrryrrr r; patch_length base_pt_blockr>s r&rz!PatchTSMixerPretrainHead.__init__sM Z(;<<Yv~v7JKKr'cZ||}||}|S)a Args: hidden_features (`torch.Tensor` of shape `(batch_size x num_patch x d_model)` in `flatten` mode or `(batch_size x n_vars x num_patch x d_model)` in `common_channel`/`mix_channel` mode.): Input hidden features. Returns: `torch.Tensor` of shape `(batch_size x n_vars x num_patch x patch_length)`. )rr4r s r&r,z PatchTSMixerPretrainHead.forwards/,,_==%%o66r'rr3s@r&r1r1seL1LLLLLL       r'r1Fr* mask_ratiounmasked_channel_indiceschannel_consistent_masking mask_valuec|dks|dkrtd|d|j\}}}}|j} t|d|z z} |r0t j|d|| } | d|d} nt j|||| } t j|||| } d| ddddd| f<t j| d} t j| d}t j | d| } | dddd|} |d| dd|ddddf<| | |}|| d fS) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] rrz Mask ratio z has to be between 0 and 1.deviceNrr)rindex.r) rarrr<r1rDrandrepeatonesargsortgatherrY masked_fillr)r*r6r7r8r9r num_channelssequence_length num_featuresr<len_keepnoisemask ids_shuffle ids_restore inputs_masks r&random_maskingrNs4A~~qNzNNNOOO>Dl;J o| ]F?a*n566H!U :q/&III Q a00 :|_VTTT :j, O O ODDAAAyy-2...K- 444K <"K 8 8 8D >>"   $ $Q1l ; ;D+23QQQ(!!!QQQ ./$$TYY[[*==K V $$r'num_forecast_mask_patchesct|tr|g}d|D}|j\}}}}tj||||j} g} d} t |} t||D]V\} }| dks| |krtd| dt||z| z }| | ||g| |z } Wt| d} | |kr| dd|| z z| dd<n#| |kr| d d| |z z| d d<d}| D]\}}}||z}d | ||d d | d f<|}tj | jd}| |} | d  d d d |} |d| d d |d d d d f<|| |}|| d fS) aForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` cg|]}dS)r)rrs r&rz$forecast_masking...sAAA!AAAAr'r;rznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.c|dSNr@rR)xs r&z"forecast_masking..@s !A$r')rr@rrNr>)r r1rrrDrMr<sumziprarsortedrandpermrYr@rDr)r*rOr7r9forecast_mask_ratiosrrErFrGrJt_list total_length total_ratior3ratiotemp_lenbatch1 patch_lenrbatch2permrMs r&forecast_maskingresU0+S11@%>$?!AA'@AAA>Dl;J o| ;z< W W WD FL*++K"#<>RSS!! e 1    ? ?q\qqq zE)K788 |UH5666 F / / /Fj  ay|zL'@Aq !  " "r 1  )BCr 1 F"( 1h("./VF]AAA z{{ *+ >$*Q- ( (D :D >>"   $ $Q1l ; ;D+23QQQ(!!!QQQ ./$$TYY[[*==K V $$r'c>eZdZdZdeffd ZdejfdZxZ S)PatchTSMixerPatchifyz A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` r6ct|j|_|j|_|j|_|j|jkr t d|jd|jdt|j|j|jz |jzdz|_|j|j|jdz zz}|j|z |_ dS)NzSequence length (z+) has to be greater than the patch length ()r) rrcontext_lengthrFr3 patch_striderarrNsequence_start)r$r6new_sequence_lengthr%s r&rzPatchTSMixerPatchify.__init__as %4"/"/  4#4 4 4yD$8yyeievyyy    4d6GHH4K\\aearruvv"/$2CtGWZ[G[2\\"25HHr'r!c,|jd}||jkrtd|d|jd|dd|jdddf}|d|j|j}|dd}|S)a! Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` rzInput sequence length (z%) doesn't match model configuration (rN) dimensionrstepr) rrrFrarlunfoldr3rkrAr)r$r!rFrBs r&r,zPatchTSMixerPatchify.forwardrs&+B/ d2 2 2x/xx`d`txxx QQQ 3 5 5qqq89$2C$J[\\!!"b))4466 r'rCr3s@r&rgrgYssI1IIIIII"5<r'rgc>eZdZdZdeffd ZdejfdZxZ S)PatchTSMixerMaskinga Class to perform random or forecast masking. Parameters: config (`PatchTSMixerConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points r6ct|j|_|j|_|j|_|j|_|j|_|j|_|jt|j|_dSdSr)) rrrandom_mask_ratior8 mask_typerOr7r9rYr>s r&rzPatchTSMixerMasking.__init__s !'!9*0*K'))/)I&(.(G% +  ( 4,243P,Q,QD ) ) ) 5 4r'rdc2|jdkr,t||j|j|j|j\}}nI|jdkr&t ||j|j|j\}}ntd|jd| }||fS)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input Return: masked_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points rQ)r*r6r7r8r9r )r*rOr7r9zInvalid mask type .) rvrNrur7r8r9rerOrar)r$rd masked_inputrJs r&r,zPatchTSMixerMasking.forwards >X % %!/"1)-)F+/+J? """ L$$^z ) )!1"*.*H)-)F? """ L$$C$.CCCDD Dyy{{T!!r'rCr3s@r&rsrsss   R1 R R R R R R!"5<!"!"!"!"!"!"!"!"r'rsc eZdZdZdeffd ZdejdejdeejejejffdZ xZ S)PatchTSMixerStdScalerz Standardize features by calculating the mean and scaling along the first dimension, and then normalizes it by subtracting from the mean and dividing by the standard deviation. r6ctt|dr|jnd|_t|dr|jnd|_t|dr|jnd|_dS)N scaling_dimrkeepdimT minimum_scalegh㈵>)rrhasattrr}rr~rr>s r&rzPatchTSMixerStdScaler.__init__sy )0)G)GN6%%Q)0)C)CMv~~ 5s r&rzPatchTSMixerMeanScaler.__init__s )0)G)GN6%%Q)0)C)CMv~~ 5s r&rzPatchTSMixerNOPScaler.__init__'sW )0)G)GN6%%Q)0)C)CMv~~ r'NrrrOctj|d|j|j}tj|d|j|j}|||fS)a Parameters: data (`torch.Tensor` of shape `(batch_size, sequence_length, num_input_channels)`): input for Batch norm calculation Returns: tuple of `torch.Tensor` of shapes (`(batch_size, sequence_length, num_input_channels)`,`(batch_size, 1, num_input_channels)`, `(batch_size, 1, num_input_channels)`) FrR)rr~)rDrr_rr~r)r$rrrrs r&r,zPatchTSMixerNOPScaler.forward,slE:::??DHVZVb?cct5999>>48UYUa>bbS%r'r)rr3s@r&rr"sN1NNNNNN MQ  L 6;lT6I u|U\5<7 8        r'rzS Base class for `PatchTSMixerEncoderOutput`, with potential hidden states. ) custom_introc\eZdZUdZdZejdzed<dZe ejdzed<dS)PatchTSMixerEncoderOutputa- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, d_model)`): Hidden-state at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer. Nlast_hidden_stater) r-r.r/r0rrD FloatTensorr*rrrRr'r&rr=sT37u(4/66659M5*+d299999r'rc peZdZdZdeffd Zee d dej de dzde fdZ xZ S) PatchTSMixerEncoderz Encoder for PatchTSMixer which inputs patched time-series and outputs patched embeddings. Args: config (`PatchTSMixerConfig`): Configuration. r6c,t|tj|j|j|_|jrt||_ nd|_ t||_ | dS)Nr) rrrr r3r;patcherrIrGpositional_encoderrmlp_mixer_encoder post_initr>s r&rzPatchTSMixerEncoder.__init__Xs    y!4fnEE  ) +&DF&S&S&SD # #&*D #!2&!A!A!A r'Nr!rrOc ||n |jj}||}|j||}|||\}}t ||S)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. Returns: `torch.FloatTensor` of shape `(batch_size, n_vars, num_patches, d_model)` Nr)rr)r6rrrrr)r$r!rrpatchesrrs r&r,zPatchTSMixerEncoder.forwardes.%9$D $+Jj  ,,{++  " .--g66G+/+A+A'`t+A+u+u(=(;L\ijjjjr'r))r-r.r/r0rrrrrDrErrr,r2r3s@r&rrOs 1      -1!k!k\!k#Tk!k # !k!k!k^!k!k!k!k!kr'rzG Base class for model's outputs, with potential hidden states. ceZdZUdZdZejdzed<dZe ejdzed<dZ ejdzed<dZ ejdzed<dZ ejdzed<dZ ejdzed<dS) PatchTSMixerModelOutputa last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, d_model)`): Hidden-state at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer. patch_input (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, patch_length)`): Patched input data to the model. mask (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches)`, *optional*): Bool Tensor indicating True in masked patches and False otherwise. loc (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*): Gives the mean of the context window per channel. Used for revin denorm outside the model, if revin enabled. scale (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*): Gives the std dev of the context window per channel. Used for revin denorm outside the model, if revin enabled. NrrrdrJrr)r-r.r/r0rrDrr*rrrdrJrrrRr'r&rrs"37u(4/66659M5*+d2999,0K"T)000%)D% d ")))$(C T !(((&*E5 t #*****r'rz= The PatchTSMixer Model for time-series forecasting. c eZdZd dedeffd Zee d dej dej dzdedzd e fd Z xZ S) PatchTSMixerModelFr6 mask_inputct|t||_t ||_|durt ||_nd|_|jdkrt||_ n=|jdks |jdurt||_ nt||_ | dS)z mask_input (bool, *optional*, defaults to `False`): Whether to mask the input using the [`PatchTSMixerMasking`] module. TNr_r`)rrrencoderrgpatchingrsmaskingrrscalerr{rr)r$r6rr%s r&rzPatchTSMixerModel.__init__s    *622 ,V44   .v66DLLDL >V # #088DKK ^u $ $$(>(>/77DKK/77DK r'Nr! observed_maskrrOc ||n |jj}d}|tj|}|||\}}}||} | } |j|| \} }|| |} t| tr t| } t| j | j | |||S)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. observed_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Nr)rrrdrJrr)r6rrDrrrrrr rrrrr) r$r!rrrrJscaled_past_valuesrr patched_x enc_inputencoder_outputs r&r,zPatchTSMixerModel.forwards0%9$D $+Jj   !OK88M)-[-)P)P&CMM"455  < #"ll955OIt !5&   ne , , H6GN&,>(6!     r'rNN) r-r.r/rrrrrrDrErr,r2r3s@r&rrs 1t2.2,0 5 5 \5 |d*5 #Tk 5 ! 5 5 5 ^5 5 5 5 5 r'rz> Output type of [`PatchTSMixerForPreTrainingOutput`]. ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dS) PatchTSMixerForPreTrainingOutputa@ loss (*optional*, returned when `y` is provided, `torch.FloatTensor` of shape `()`): Total loss prediction_outputs (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, patch_length)`): Prediction output from the pretrain head. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, d_model)`): Backbone embeddings before passing through the head. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer. Nlossprediction_outputsrr r-r.r/r0rrDrr*rrrrrRr'r&rr  &*D% d ")))37)D077726u(4/66659M5*+d299999r'rz. `PatchTSMixer` for mask pretraining. c eZdZdeffd Zee d dejdejdzde dzde d e f d Z xZ S) PatchTSMixerForPretrainingr6ct|t|d|_t ||_|j|_|dS)NT)rr)rrrr r1head masked_lossrr>s r&rz#PatchTSMixerForPretraining.__init__"sd    &v$??? ,F;;; !- r'NTr!rr return_lossrOc p||n |jj}|jdur!tjd}n tjd}||||}t|tr t|}| |j }|dur|||j } nd} |jdurO| M| d|jz|jd zz } t!| ||j |j S) aT past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. observed_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). return_loss (`bool`, *optional*): Whether to return the loss in the `forward` call. NTnone reductionr_rrrrrrrrr)r6rrrDrMSELossr r rrrrrdr_rJrWrr) r$r!rrrrr model_outputx_hatloss_vals r&r,z"PatchTSMixerForPretraining.forward+sY6%9$D $+Jj   t # #8##f#55DD8##f#55Dzz '!5"  lE * * B2LAL ,899 $  tE<#;<]9]+^__H' :#7DD|/|7I E   !,1D1D#tL-@@H/99H 22\5EE ,1D1D#tE=99H  * 6"3(G#GHC &sD,K'KLEE"C &E.$*<&4     r'c |j}||d|d}|j|j|j|jfdt |D}tj|d}t|S) a Generate sequences of sample predictions from a model with a probability distribution head. Args: past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Past values of the time series that serves as context in order to predict the future. observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Return: [`SamplePatchTSMixerPredictionOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of samples, prediction_length, num_input_channels)`. NF)r!rrrrc8g|]}SrRsamplerrrs r&rz6PatchTSMixerForPrediction.generate..s%NNNQ<&&((NNNr'rrr) rrrrrrrrDstackr)r$r!rroutputssamplesrs @r&generatez"PatchTSMixerForPrediction.generatebs2 $8$#'!&    /<<  &GKw}=  ONNN%8L2M2MNNN+g1---1GDDDDr')NNNTr))r-r.r/r0rrrrrDrErrr,r/rrr2r3s@r&rrsJ  1<.2-1,0 k k \k |d*k |d* k #Tk k  k  )k k k ^k ZU]__.2-E-E\-E|d*-E , -E-E-E_-E-E-E-E-Er'rzK Output type of [`PatchTSMixerForTimeSeriesClassificationOutput`]. ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dS)-PatchTSMixerForTimeSeriesClassificationOutputaP loss (*optional*, returned when `y` is provided, `torch.FloatTensor` of shape `()`): Total loss. prediction_outputs (`torch.FloatTensor` of shape `(batch_size, num_labels)`): Prediction output from the classification head. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, d_model)`): Backbone embeddings before passing through the head. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. NrrrrrrRr'r&rrrr'rc eZdZdZdeffd Zee d dej dej dzde dzd e d e f d Z xZ S) 'PatchTSMixerForTimeSeriesClassificationz `PatchTSMixer` for classification application. Args: config (`PatchTSMixerConfig`): Configuration. Returns: `None`. r6c&t|t||_t ||_|jdvr!t|j|j |_ nd|_ | dS)Nrr`r_Tr;rN) rrrr rrrInjectScalerStatistics4Dr;rN inject_scalerr>s r&rz0PatchTSMixerForTimeSeriesClassification.__init__s    &v.. *     >2 2 2 8]c]o p p pD   $D  r'NTr! target_valuesrrrOc tj}||n |jj}|||}t |tr t|}|j ,| |j |j |j |_ | |j }||dur |||} nd} t| ||j |jS)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. target_values (`torch.FloatTensor` of shape `(batch_size, target_len, num_input_channels)` for forecasting, `(batch_size, num_targets)` for regression, or `(batch_size,)` for classification, *optional*): Target values of the time series, that serve as labels for the model. The `target_values` is what the Transformer needs during training to learn to output, given the `past_values`. Note that, this is NOT required for a pretraining task. For a forecasting task, the shape is be `(batch_size, target_len, num_input_channels)`. Even if we want to forecast only specific channels by setting the indices in `prediction_channel_indices` parameter, pass the target data with all channels, as channel Filtering for both prediction and target will be manually applied before the loss computation. For a classification task, it has a shape of `(batch_size,)`. For a regression task, it has a shape of `(batch_size, num_targets)`. return_loss (`bool`, *optional*): Whether to return the loss in the `forward` call. NrrTr)rDrCrossEntropyLossr6rr r rrrrrrrrr) r$r!rrrrrrrrs r&r,z/PatchTSMixerForTimeSeriesClassification.forwardsJx((**%9$D $+Jj zz !5"   lE * * B2LAL   (-1->->. $"(.?..L *  ,899  $)<)<tE=11HHH<$*<&4     r'r)r-r.r/r0rrrrrDrErrr,r2r3s@r&rrs   1      .2,0 C C \C |d*C #Tk C  C  7C C C ^C C C C C r'rz= Output type of [`PatchTSMixerForRegressionOutput`]. ceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dS)PatchTSMixerForRegressionOutputaM loss (*optional*, returned when `y` is provided, `torch.FloatTensor` of shape `()`): Total loss. regression_outputs (`torch.FloatTensor` of shape `(batch_size, num_targets)`): Prediction output from the regression head. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, d_model)`): Backbone embeddings before passing through the head. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. Nrregression_outputsrr) r-r.r/r0rrDrr*rrrrrRr'r&rrrr'rc`eZdZd dededeffd Zdejdejdejfd ZxZS) rr@r;rN expansioncDttj|dz||z|_tj||z||_tjdd|z|_tjd|zd|_||_dSrT) rrrr inverse_trans_expansioninverse_trans_compressionmap_scale_expansionmap_scale_compressionrN)r$r;rNrr%s r&rz!InjectScalerStatistics4D.__init__'s ')y1i'>Q'R'R$)+9w3F)P)P&#%9QI #>#> %'Yq9}a%@%@"&r'r*rrc.|dd}|d}|dd|jd}|dd}|d}|dd|jd}t j||gd}||}||}t j||gd}||}| |}|S)a Args: inputs (`torch.Tensor` of shape `(batch_size, num_input_channels, num_patch, d_model)`) loc (`torch.Tensor` of shape `(batch_size, 1, num_input_channels)`) scale (`torch.Tensor` of shape `(batch_size, 1, num_input_channels)`) Returns: `torch.Tensor` of shape `(batch_size, num_input_channels, num_patch, d_model)` rrrr) rArYr@rNrDcatr#r$r!r")r$r*rrr_stdev concat_statss r&r,z InjectScalerStatistics4D.forward0s}}R$$~~b!!{{1a!1155B''## Q4#3Q77y$B777 // == 11,?? FL1r:::--f55//77 r')r@) r-r.r/r1rrDrEr,r2r3s@r&rr&s'''#'#''''''elelr'rz4 `PatchTSMixer` for regression application. c eZdZdeffd Zee d dejdejdzde dzde d e f d Z ej dejd e fd ZxZS) PatchTSMixerForRegressionr6c^t|t||_|j|_|j|_|j|_|jdkrd|_n_tttd}| |j}|||j |_ntd|j|j dvr!t|j|j|_nd|_t%||j|_|dS)Nrrrrrrr)rrrr rrrrr r rrrarrr;rNrrrr)r$r6rrr%s r&rz"PatchTSMixerForRegression.__init__Ts7    &v.. K #)#= $*$?! ;%  '+D $ $,&%;'' # 366v7QRRL'+72 2 2 8]c]o p p pD   $D * $ 8    r'NTr!rrrrOc &jdkrtjd}n"jdkrt}nt d||n jj}||}t|tr t|}j , |j |j |j|_ |j }||d urjrjd kr't#j|d krt'd j|} tfd |D}|| |} t+| } n|||} nd} t-| ||j |jS)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. target_values (`torch.FloatTensor` of shape `(batch_size, target_len, num_input_channels)` for forecasting, `(batch_size, num_targets)` for regression, or `(batch_size,)` for classification, *optional*): Target values of the time series, that serve as labels for the model. The `target_values` is what the Transformer needs during training to learn to output, given the `past_values`. Note that, this is NOT required for a pretraining task. For a forecasting task, the shape is be `(batch_size, target_len, num_input_channels)`. Even if we want to forecast only specific channels by setting the indices in `prediction_channel_indices` parameter, pass the target data with all channels, as channel Filtering for both prediction and target will be manually applied before the loss computation. For a classification task, it has a shape of `(batch_size,)`. For a regression task, it has a shape of `(batch_size, num_targets)`. return_loss (`bool`, *optional*): Whether to return the loss in the `forward` call. rr_rrrNrrTrrzDtarget_values cannot be negative for negative_binomial distribution.c3XK|]$}|djjV%dS)rN)rr6r)ritemr$s r&rz4PatchTSMixerForRegression.forward..s6WWdiiDK,CDDWWWWWWr')rrrr)rrrrrar6rr r rrrrrrrrrDany Exceptionrrrr) r$r!rrrrrrrrrs ` r&r,z!PatchTSMixerForRegression.forwardysH 9  :///DD Y%  DDQRR R%9$D $+Jj zz !5"   lE * * B2LAL   (-1->->. $"(.?..L *  ,899  $)<)<' 6+/BBBuyQ^abQbGcGcB#$jkkk#7DDUKK WWWWQVWWWWW4 m<<+H554}55H.$*<&4     r'c,|j}||dd}|j|jfdt |D}t j|dd||jj }t|S) a  Generate sequences of sample predictions from a model with a probability distribution head. Args: past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Past values of the time series that serves as context in order to predict the target values. Return: [`SamplePatchTSMixerRegressionOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of samples, num_targets)`. NF)r!rrc8g|]}SrRrr s r&rz6PatchTSMixerForRegression.generate..s2   &'L   ! !   r'rrrr ) rrrrrrDr rr6rr)r$r!rr r rs @r&rz"PatchTSMixerForRegression.generates" $8$#!&   /<rFsA "! $$$$$$!!!!!! 777777******&&&&&&BBBBBB555555&&&&&&UUUUUUUUUURRRRRRRRRRRR::::::  H % %*&&&&&BI&&&,$$$$$RY$$$N.....BI...bbi.-----29---n!%% I% <% % < % L4' % T\ %%'(%%%%:R/R/R/R/R/BIR/R/R/jCCCCCbiCCCL***** ***Z##### ###L&#&#&#&#&# &#&#&#R55555BI555pDDDDDRYDDDN00000/000*ryD-1', 7%7% L7%7%#Tk7%!% 7%  7%7%7%7%|-1 A%A% LA%#czA%#TkA% A%A%A%A%J-----29---b9"9"9"9"9"")9"9"9"z 0 0 0 0 0BI 0 0 0H3;3;3;3;3;RY3;3;3;n     BI   6   : : : : : : :   :9k9k9k9k9k59k9k9kx  +++++k++   +4 Q Q Q Q Q 3Q Q  Q h  :::::{::   :$ F F F F F !<F F  F R  +++++k++   +0  ///////  /  ///////  /#u"/##%,####**5<*%,:M*chco****0GEGEGEGEGE ;GEGEGET  :::::K::   :$` ` ` ` ` .I` ` ` F  :::::k::   :$%%%%%ry%%%P ^E^E^E^E^E ;^E^E  ^EB   r'