tjIdZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l mZdd lmZdd lmZdd lmZmZdd lmZddlmZmZmZddlmZmZmZm Z ddl!m"Z"e j#e$Z% dqdej&dej'dej'dej'dej'dzde(dzde(deefdZ)Gddej&Z*Gddej&Z+ drd!ej'd"e(d#e,dzd$e-d%e.f d&Z/ dsd!ej'd'e,e.zd#e,dzd%e.fd(Z0Gd)d*ej&Z1Gd+d,ej&Z2Gd-d.ej&Z3eGd/d0eZ4Gd1d2ej&Z5Gd3d4ej&Z6Gd5d6e4Z7ed78eGd9d:eZ8ed;8eGd<d=eZ9ed>8eGd?d@eZ:edA8eGdBdCeZ;edD8eGdEdFeZ<edG8eGdHdIeZ=dJej>j?dKej'dLej'fdMZ@dtdNej'dOej'dzdLej'fdPZAGdQdRej&ZBGdSdTej&ZCGdUdVej&ZDGdWdXej&ZEeGdYdZe4ZFGd[d\ej&ZGed]8Gd^d_e4ZHGd`daej&ZIedb8Gdcdde4ZJede8Gdfdgej&ZKedh8Gdidje4ZLGdkdlej&ZMedm8Gdndoe4ZNgdpZOdS)uzPyTorch PatchTST model.N)Callable) dataclass)nn)initialization)ACT2CLS)is_deepspeed_zero3_enabled)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)NegativeBinomialOutput NormalOutputStudentTOutput) ModelOutputTransformersKwargsauto_docstringlogging)PatchTSTConfigmodulequerykeyvalueattention_maskscalingdropoutkwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nrdim)ptrainingr) sizetorchmatmul transposer functionalsoftmaxrr( contiguous) rrrrrrrr attn_weights attn_outputs l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/patchtst/modeling_patchtst.pyeager_attention_forwardr3's**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$ceZdZdZ ddededed ed ed ed edzffd Z dde j de j dzde j dzdedzde e de e j e j dze e j dzff dZxZS)PatchTSTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderbias is_causalconfigc t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).r#r:)super__init__r7r8rhead_dimr< ValueErrorrr9r;rLineark_projv_projq_projout_proj) selfr7r8rr9r:r;r< __class__s r2rAzPatchTSTAttention.__init__Gs "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r4 hidden_stateskey_value_statesroutput_attentionsr returnc |du}|jdd}g|d|jR}|||dd} |r|n|} g| jddd|jR} || | dd} || | dd} tj|j j t}||| | | |f|j sdn|j |j|d|\}}|jg|dR}||}||dfS)z#Input shape: Batch x Time x ChannelNr"rr$r)rrrM)shaperBrGviewr,rErFr get_interfacer<_attn_implementationr3r(rrreshaper/rH)rIrKrLrrMr is_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacer1r0s r2forwardzPatchTSTAttention.forwardfs.T9$)#2#. 88b8$-88 {{=1166|DDNNqRSTT -?R))]B^)#2#.BBDMBB[[0055h??II!QOO {{>2277AAKKAqQQ (?(M K ,.E) ) %8$7      % $}>CC$,L/ % %  % % ! \*k);;;;;;FFHH mmK00 L$..r4)rFTFN)NNF)__name__ __module__ __qualname____doc__intfloatboolrrAr*Tensorrr tupler^ __classcell__rJs@r2r6r6DsJGG  (,CCCC C  C  CC%CCCCCCD15.2). 0/0/|0/ ,-0/ t+ 0/  $; 0/-.0/ u|U\D0% 2E2LL M0/0/0/0/0/0/0/0/r4r6c>eZdZdZdeffd ZdejfdZxZ S)PatchTSTBatchNormzP Compute batch normalization over the sequence length (time) dimension. r<cttj|j|j|_dS)Neps)r@rAr BatchNorm1dd_modelnorm_eps batchnormrIr<rJs r2rAzPatchTSTBatchNorm.__init__s7 FOLLLr4inputsc|dd}||}|ddS)a Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` rr$)r,rr)rIrtoutputs r2r^zPatchTSTBatchNorm.forwards@!!!Q''''1%%%r4 r_r`rarbrrAr*rfr^rhris@r2rkrksrM~MMMMMM &el & & & & & & & &r4rkFrt mask_ratiounmasked_channel_indiceschannel_consistent_masking mask_valuec|dks|dkrtd|d|j\}}}}|j} t|d|z z} |r0t j|d|| } | d|d} nt j|||| } t j|||| } d| ddddd| f<t j| d} t j| d}t j | d| } | dddd|} |d| dd|ddddf<| | |}|| d fS) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] rrz Mask ratio z has to be between 0 and 1.deviceNr"r%)r&index.r) rCrPr~rcr*randrepeatonesargsortgather unsqueeze masked_fillre)rtrxryrzr{ batch_size num_channelssequence_length num_featuresr~len_keepnoisemask ids_shuffle ids_restore inputs_masks r2random_maskingrs4A~~qNzNNNOOO>Dl;J o| ]F?a*n566H!U :q/&III Q a00 :|_VTTT :j, O O ODDAAAyy-2...K- 444K <"K 8 8 8D >>"   $ $Q1l ; ;D+23QQQ(!!!QQQ ./$$TYY[[*==K V $$r4num_forecast_mask_patchesct|tr|g}d|D}|j\}}}}tj||||j} g} d} t |} t||D]V\} }| dks| |krtd| dt||z| z }| | ||g| |z } Wt| d} | |kr| dd|| z z| dd<n#| |kr| d d| |z z| d d<d}| D]\}}}||z}d | ||d d | d f<|}tj | jd}| |} | d  d d d |} |d| d d |d d d d f<|| |}|| d fS) aForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` cg|]}dS)r).0_s r2 z$forecast_masking..sAAA!AAAAr4r}rznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.c|dS)Nr$r)xs r2z"forecast_masking..s !A$r4)rr$r"rNr) isinstancercrPr*zerosr~sumziprCappendsortedrandpermrrrre)rtrryr{forecast_mask_ratiosrrrrrt_list total_length total_ratio patch_lengthratiotemp_lenbatch1 patch_lenrbatch2permrs r2forecast_maskingrsU0+S11@%>$?!AA'@AAA>Dl;J o| ;z< W W WD FL*++K"#<>RSS!! e 1    ? ?q\qqq zE)K788 |UH5666 F / / /Fj  ay|zL'@Aq !  " "r 1  )BCr 1 F"( 1h("./VF]AAA z{{ *+ >$*Q- ( (D :D >>"   $ $Q1l ; ;D+23QQQ(!!!QQQ ./$$TYY[[*==K V $$r4c>eZdZdZdeffd ZdejfdZxZ S)PatchTSTPatchifyz A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` r<ct|j|_|j|_|j|_|j|jkr t d|jd|jdt|j|j|jz |jzdz|_|j|j|jdz zz}|j|z |_ dS)NzSequence length (z+) has to be greater than the patch length ()r) r@rAcontext_lengthrr patch_striderCmax num_patchessequence_start)rIr<new_sequence_lengthrJs r2rAzPatchTSTPatchify.__init__5s %4"/"/  4#4 4 4yD$8yyeievyyy    4d6GHH4K\\aearruvv"/$2CtGWZ[G[2\\"25HHr4 past_valuesc,|jd}||jkrtd|d|jd|dd|jdddf}|d|j|j}|dd}|S)a! Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` zInput sequence length (z%) doesn't match model configuration (r>N) dimensionr)step) rPrrCrunfoldrrr,r/)rIrrrvs r2r^zPatchTSTPatchify.forwardFs&+B/ d2 2 2x/xx`d`txxx QQQ 3 5 5qqq89$2C$J[\\!!"b))4466 r4rwris@r2rr-srI~IIIIII"5<r4rc>eZdZdZdeffd ZdejfdZxZ S)PatchTSTMaskinga Class to perform random or forecast masking. Parameters: config (`PatchTSTConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points r<ct|j|_|j|_|j|_|j|_|j|_|j|_|jt|j|_dSdSN) r@rArandom_mask_ratiorz mask_typerryr{rrss r2rAzPatchTSTMasking.__init__js !'!9*0*K'))/)I&(.(G% +  ( 4,243P,Q,QD ) ) ) 5 4r4 patch_inputc2|jdkr,t||j|j|j|j\}}nI|jdkr&t ||j|j|j\}}ntd|jd| }||fS)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input Return: masked_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points random)rtrxryrzr{forecast)rtrryr{zInvalid mask type .) rrrryrzr{rrrCre)rIr masked_inputrs r2r^zPatchTSTMasking.forwardus >X % %!/"1)-)F+/+J? """ L$$^z ) )!1"*.*H)-)F? """ L$$C$.CCCDD Dyy{{T!!r4rwris@r2rr]sr   R~ R R R R R R!"5<!"!"!"!"!"!"!"!"r4rcJeZdZdZdeffd ZddejdedzfdZ xZ S) PatchTSTEncoderLayerz PatchTST encoder layer r<c  t|j|_t|j|j|j||_|jdkrtj |jntj |_ |j dkrt||_nH|j dkr&tj|j|j|_nt%|j d|jr|jdkrtj |jntj |_|j dkrt||_nH|j dkr&tj|j|j|_nt%|j dtjtj|j|j|jt3|j|jdkrtj |jntj tj|j|j|j|_|jdkrtj |jntj |_|j dkrt||_nH|j dkr&tj|j|j|_nt%|j d|j|_dS)N)r7r8rr<rrr layernormrmz$ is not a supported norm layer type.r?) r@rAchannel_attentionr6rpnum_attention_headsattention_dropout self_attn path_dropoutrDropoutIdentity dropout_path1 norm_typerknorm_sublayer1 LayerNormrqrC dropout_path2norm_sublayer2 SequentialrDffn_dimr:ractivation_function ff_dropoutff dropout_path3norm_sublayer3pre_normrss r2rAzPatchTSTEncoderLayer.__init__s !'!9*n0,    AG@SVW@W@WRZ(;<<<]_]h]j]j  { * *"3F";";D     , ,"$,v~6?"S"S"SD   0VVVWW W  ! \DJDWZ[D[D[F,?!@!@!@acalananD ;..&7&?&?##![00&(l6>v&W&W&W## F$4!Z!Z!Z[[[- Ifnfn6; G G G F. / 1 1-3->-B-BBJv( ) ) ) Ifnfn6; G G G   AG@SVW@W@WRZ(;<<<]_]h]j]j  { * *"3F";";D     , ,"$,v~6?"S"S"SD   0VVVWW W r4N hidden_staterMc~|j\}}}}|||z||}|jrG||||\}}} |||z}nF|||\}}} ||||z}|||||}|jr|dd }|||z||}|jrG|| ||\}} } || |z}nF|||\}} } | || |z}|||||}|dd }|||z||}|jr?|| | ||z}n>||| | |z}|||||}|f} |r| |jr|| fn|fz } | S)a Parameters: hidden_state (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)`, *required*): Past values of the time series output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers Return: `torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)` )rKrMr$r)rPrQrrrrrTrr,r/rrrrr) rIrrMrnum_input_channelsrrpr1r0rchannel_attn_weightsoutputss r2r^zPatchTSTEncoderLayer.forwards%DPCU@ &$((6H)H/[bcc = _+/>>"11,??Sd,:,, (Kq($*<*<[*I*IILL,0>>*>O,:,, (Kq ..|d>P>PQ\>]>]/]^^L$++J8JO]dee   ! E'11!Q77BBDDL',,Z/-IK]_fggL} c7;~~"&"5"5l"C"CWh8F884 11 ,d.@.@.M.MM 8<~~".BS8F884 11 $22<$BTBTU`BaBa3abb (// OM_ahiiL'11!Q77BBDDL$((6H)H/[bcc = i($*<*P>PQUQXQXYeQfQf>g>g/ghhL$++J8JO]dee /  k t?Uj &:;;\h[j jGr4r) r_r`rarbrrAr*rfrer^rhris@r2rrs0(~0(0(0(0(0(0(dQQELQTD[QQQQQQQQr4rceZdZUeed<dZdZdZdZdZ dZ dZ e j dejffd Zd d ZxZS) PatchTSTPreTrainedModelr<modelr)timeFTrct|t|tr)t |jj|jj|jjz |jjzdz}|jj r tj |j d|dz }| |j|}trwddl}|j|jd5|jdkrtj|j|ddddS#1swxYwYdStj|j|dSdS)z$ Initialize weights rg{Gz?)stdrN) modifier_rank)r@ _init_weightsrPatchTSTPositionalEncodingrr<rrr use_cls_tokeninitnormal_ cls_token_init_per deepspeedzeroGatheredParameters position_encnumelcopy_)rIrrrrrJs r2rz%PatchTSTPreTrainedModel._init_weights/s f%%% f8 9 9 >DK. 0HIIDKLdd)*,-.K{( ! V-48888q !??4; DDL)++ >    ^66v7JZ^6__FF*0022Q66 6#6 EEEFFFFFFFFFFFFFFFFFF 6. =====% > >s48D99D=D=cBt|tr ||_dSdSr)rPatchTSTEncodergradient_checkpointing)rIrrs r2_set_gradient_checkpointingz3PatchTSTPreTrainedModel._set_gradient_checkpointingIs, f 0 0 2,1F ) ) ) 2 2r4)F)r_r`rar__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr*no_gradrModulerrrhris@r2rr$s #O &+#NU]__>BI>>>>>_>222222222r4rc:eZdZdeffd ZdejfdZxZS)PatchTSTEmbeddingr<ct|j|_|j|_|jr&t j|j|j|_dSt j |_t|jD]9}|j t j|j|j:dSr) r@rArshare_embeddingrrDrrpinput_embedding ModuleListranger)rIr<rrJs r2rAzPatchTSTEmbedding.__init__Os "(";%5   \#%9V-@&.#Q#QD #%=??D 6455 \ \$++BIf6I6>,Z,Z[[[[ \ \r4rcjd}|jkrtdjd|djr}n2fdt |D}t j|d}|S)a% Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input for embedding return: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, d_model)` rz&The defined number of input channels (zQ) in the config has to be the same as the number of channels in the batch input (rc \g|](}j|dd|ddddf)Sr)r)rirrIs r2rz-PatchTSTEmbedding.forward..msEnnnq1$.q1+aaaAAAqqqj2IJJnnnr4r%)rPrrCrrrr*stack)rIrr embeddingss`` r2r^zPatchTSTEmbedding.forward[s).q1 !8 8 8j9PjjTfjjj    8--k::JJnnnnnTYZlTmTmnnnJZQ777Jr4 r_r`rarrAr*rfr^rhris@r2rrNsh \~ \ \ \ \ \ \5<r4rcpeZdZdZdedeffd Zedededej fdZ de j fdZ xZS) rz' Class for positional encoding r<rct|j|_|j|_|jr8t jt jddd|j|_ |dz }| |||_ |j dkrt j |j nt j|_ dS)Nrr)r@rArrr Parameterr*rrprrrpositional_dropoutrrrIr<rrJs r2rAz#PatchTSTPositionalEncoding.__init__ws #1"(";   \%+aAv~*N*NOODN 1 K MM&+>>6<5NQR5R5RBJv0 1 1 1XZXcXeXe r4rNc|jdkr0tjtj||jd}n:|jdkrtj||j}tjd|d}tj tjd|jdtj d|jz z}tj ||z|dddddf<tj ||z|dddddf<||z }||d zz }tj|d }nt!|jd |S) NrT requires_gradsincosrrr$g@ FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)positional_encoding_typerrr*randnrprarangerexpmathlogsincosmeanrrC)r<rrpositiondiv_terms r2rz#PatchTSTPositionalEncoding._init_pesy  *h 6 6< K(P(P`deeeLL  , 8 8 ;{FNCCL|A{33==a@@Hya!C!CQXHYHY\b\jHjFk!kllH$)Ih.A$B$BLADqD !$)Ih.A$B$BLADqD !',*;*;*=*==L'<+;+;+=+=+BCL< EJJJLL2CCC r4rcX|jr|||jddddfz}|j|jddddfz}||jd|jdd}tj||fd}n|||jz}|S)Nrrr"r$r%) rrrrexpandrPrr*cat)rIrr cls_tokensrs r2r^z"PatchTSTPositionalEncoding.forwards   T11+@QRSRTRTVWVWVWRW@X2XYYK):2A2qqq5)AAI"))+*;A*>@WY[]_``J 9j+%>AFFFLL 22;AR3RSSLr4)r_r`rarbrrcrA staticmethodrrrr*rfr^rhris@r2rrrs  ~  C       cbl\& 5<        r4rc `eZdZdZdedeffd Z d dejde dzde dzd e fd Z xZ S) rz PatchTST Encoder r<rcBtd|_t|_t ||_tjfdtj D|_ | dS)NFc.g|]}tSr)r)rrr<s r2rz,PatchTSTEncoder.__init__..s"$k$k$ka%9&%A%A$k$k$kr4) r@rArrembedderrpositional_encoderrrrnum_hidden_layerslayers post_initrs ` r2rAzPatchTSTEncoder.__init__s    &+#*&11 "