tj4 HddlZddlZddlmZddlZddlmZmZddlmZddl m Z ddl m Z e j eZedGd d ejZeZed Gd d ejZedGddejZedGddejZedGddejZedGddejZGddejZGddejZGddejZGd d!ejZGd"d#ejZGd$d%ejZGd&d'ejZGd(d)eZGd*d+ejZ id,ed-ed.d/d0fd1ed2ed3ed4d5ifd6ed7ed8d5ifd9ed:ej!d;eded?ed@ej#dAedBej$ej%eeej&ej'ej(e dCZ)ee)Z*dDZ+e+d3Z,e+d2Z-e+d,Z.e+d1Z/e+d6Z0e+d?Z1e+dEZ2e+d>Z3e+d=Z4dS)FN) OrderedDict)Tensornn)use_kernel_forward_from_hub)logging)is_torchdynamo_compilingGeluTanhcJeZdZdZd deffd ZdedefdZdedefdZxZ S) GELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. Fuse_gelu_tanh_pythonct|r|j|_dSt jt jjd|_dS)Ntanh) approximate) super__init___gelu_tanh_pythonact functoolspartialr functionalgelu)selfr __class__s V/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/activations.pyrzGELUTanh.__init__(sP   Q-DHHH ();PPPDHHHinputreturnc |dzdtjtjdtjz |dtj|dzzzzzSN??@Hm?g@torchrmathsqrtpipowrrs rrzGELUTanh._gelu_tanh_python/sPs{cEJtytw/G/G5S[^c^ghmor^s^sSsKs/t$u$uuvvrc,||SNrr+s rforwardzGELUTanh.forward2xxrF) __name__ __module__ __qualname____doc__boolrrrr/ __classcell__rs@rr r sQQTQQQQQQwvw&wwwwVrr NewGELUc"eZdZdZdedefdZdS)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 rrc d|zdtjtjdtjz |dtj|dzzzzzSr r%r+s rr/zNewGELUActivation.forwardAsPU{cEJtytw/G/G5S[^c^ghmor^s^sSsKs/t$u$uuvvrNr2r3r4r5rr/rrr;r;:sH wVwwwwwwwrr;GeLUcJeZdZdZd deffd ZdedefdZdedefdZxZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 Fuse_gelu_pythonct|r|j|_dStjj|_dSr-)rr _gelu_pythonrrrr)rrBrs rrzGELUActivation.__init__Ns?   *(DHHH})DHHHrrrcf|dzdtj|tjdz zzS)Nr!r"r#)r&erfr'r(r+s rrDzGELUActivation._gelu_pythonUs-s{cEIedinn.D$E$EEFFrc,||Sr-r.r+s rr/zGELUActivation.forwardXr0rr1) r2r3r4r5r6rrrDr/r7r8s@rrArAEs********G&GVGGGGVrrASiLUc"eZdZdZdedefdZdS)SiLUActivationa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rrc@tj|Sr-)rrsilur+s rr/zSiLUActivation.forwardfs}!!%(((rNr=r>rrrJrJ\s@)V)))))))rrJFastGELUc"eZdZdZdedefdZdS)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rrcZd|zdtj|dzdd|z|zzzzzS)Nr!r"g3E?r$)r&rr+s rr/zFastGELUActivation.forwardps;U{cEJu|/CsXX]M]`eMeGe/f$g$gghhrNr=r>rrrOrOjsHiViiiiiiirrO QuickGELUc"eZdZdZdedefdZdS)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rrc6|tjd|zzS)NgZd;?)r&sigmoidr+s rr/zQuickGELUActivation.forwardzsu}UU]3333rNr=r>rrrSrSts@4V4444444rrSc<eZdZdZdedeffd ZdedefdZxZS)ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxc||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorrrrXrY)rrXrYrs rrzClippedGELUActivation.__init__sV 99PcPP#PPPQQ Q rxrc\tjt||j|jSr-)r&cliprrXrY)rr]s rr/zClippedGELUActivation.forwards z$q''48TX666r) r2r3r4r5floatrrr/r7r8s@rrWrW~sw  E77F77777777rrWc2eZdZdZfdZdedefdZxZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cttjdtjz |_dS)N)rrr'r(r)precomputed_constantrrs rrzAccurateGELUActivation.__init__s7 $(Ia$'k$:$:!!!rrrc ~d|zdtj|j|dtj|dzzzzzS)Nr!rr$)r&rrer*r+s rr/zAccurateGELUActivation.forwardsCU{a%*T-F%RZ]b]fglno]p]pRpJp-q"r"rrssr)r2r3r4r5rrr/r7r8s@rrbrbsn;;;;;tVtttttttttrrbcBeZdZdZfdZdedefdZdedefdZxZS)MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish ctttjj|_dSr-)rrrrmishrrfs rrzMishActivation.__init__s) =%rrrcj|tjtj|zSr-)r&rrrsoftplusr+s r _mish_pythonzMishActivation._mish_pythons'uz"-"8"8"?"?@@@@rc,||Sr-r.r+s rr/zMishActivation.forwardr0r) r2r3r4r5rrror/r7r8s@rrjrjs &&&&&A&AVAAAAVrrjc"eZdZdZdedefdZdS)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. rrc|Sr-r>r+s rr/zLinearActivation.forwards rNr=r>rrrrrrs@VrrrceZdZdZddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?c||z |tjdz}ddtj|zzS)Nr#r!r")divr'r(r&rF)rrmusigmas rr/zLaplaceActivation.forwards@  3!788cEIe,,,--rN)rvrwr2r3r4r5r/r>rrrurus2......rruceZdZdZdZdS)ReLUSquaredActivationz^ Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668 cltj|}tj|}|Sr-)rrrelur&square)rr relu_appliedsquareds rr/zReLUSquaredActivation.forwards+}))%00 ,|,,rNr|r>rrr~r~s-rr~ceZdZdZdZdS)SqrtSoftplusActivationuFsqrt(softplus(x)) — the router scoring function used by DeepSeek V4.cdtj|Sr-)rrrnr(r+s rr/zSqrtSoftplusActivation.forwards$}%%e,,11333rNr|r>rrrrs)PP44444rrceZdZfdZxZS)ClassInstantierct|}t|tr|n|if\}}|di|S)Nr>)r __getitem__ isinstancetuple)rkeycontentclskwargsrs rrzClassInstantier.__getitem__sL''%%c**!+GU!;!;Ngg'2 Vs}}V}}r)r2r3r4rr7r8s@rrrs8rrcjeZdZdZddddejdffd Zdedefd Zdedefd Z d edefd Z xZ S) XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?r!gưFc ttjt jt jt j||d|_ tjt jt jt j||z |d|_ | dt j||| dt j||||_ t||_t||_d|_ ddl}tjj|_d} ddlm} | |j|_|dz }n,#t2$r} |d| d z }|j|_Yd} ~ nd} ~ wwxYwt4|dS#t2$r)} t4d | d Yd} ~ dSd} ~ wwxYw) N)dtyperbetaepszUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.z CUDA-fused xIELU not available (u) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)rrr Parameterr&logexpm1tensor unsqueezealpha_palpha_nregister_bufferwith_vector_loadsr` _beta_scalar _eps_scalar_xielu_cuda_obj xielu.opsclassesxieluXIELUtorch.compilerr _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_once) r alpha_p_init alpha_n_initrrrrrmsgrerrrs rrzXIELUActivation.__init__sf |EIek%,|[`:a:a:a.b.b$c$c$m$mno$p$pqq | Iek%,|d/B%"P"P"PQQ R R \ \]^ _ _    VU\$e%D%D%DEEE UELE$B$B$BCCC!2!$KK ::#     #(=#6#<#<#>#>D 2C 7999999&4nT5E&F&F#?? 7 7 7tStttt&*&6###### 7    $ $ $ $ $      j3jjj          sB8.H' GH G1G,'H,G11H IH>>Ir]rc ntj|j}|jtj|jz}t j|dk||z|z|j|zzt jt j ||j |z |z|j|zzS)Nr) rrrnrrrr&whererrXr)rr]rrs r _xielu_pythonzXIELUActivation._xielu_pythons-((66)bm44T\BBB{ E aK!Odi!m + [1dh// 0 01 4 ?$)a- O   rcp|j}|dkr-|d}|dk-|dkr*|dd|d}||jkr!t d||j|j||j |j |j |j |j |j|j}||S)zDFirewall function to prevent torch.compile from seeing .item() callsrhrrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimrviewsizerrrr/rtorrrrr)rr]original_shaperesults rrzXIELUActivation._xielu_cuda"seeggkk AAeeggkk 5577Q;;r1affRjj))A QW $ $   q    %-- LOOAG $ $ LOOAG $ $      "  {{>***rrc|jD|jr=ts||Std||S)Nz:torch._dynamo is compiling, using Python version of xIELU.)ris_cudar rrrrr+s rr/zXIELUActivation.forward;s^   + ++-- b**5111##$`aaa!!%(((r) r2r3r4r5r&bfloat16rrrrr/r7r8s@rrrs  n((((((T v &    +V+++++2)V)))))))))rrrgelu_10i )rXrY gelu_fastgelu_new gelu_pythonrBTgelu_pytorch_tanhgelu_python_tanhr gelu_accurate hardswishlaplace leaky_relulinearrl quick_gelurrelu2relu6)rUrL sqrtsoftplusswishrprelurc |tvr t|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationrasOF""'((h#4hhSWX^XcXcXeXeSfSfhhiiirrL)5rr' collectionsrr&rrintegrations.hub_kernelsrutilsrutils.import_utilsr get_loggerr2rModuler PytorchGELUTanhr;rArJrOrSrWrbrjrrrur~rrr Hardswish LeakyReLUReLUReLU6SigmoidrHTanhPReLUACT2CLSrrrrrrrrrLrl linear_actr>rrrs ###### AAAAAA888888  H % %Z((ry)(0Y''wwwww ww('wV$$RY%$,V$$ ) ) ) ) )RY ) )%$ )Z((iiiiiii)(i[))44444")44*)477777BI7772 t t t t tRY t t t RY"ry . . . . . . . .BI44444RY444kZ)Z)Z)Z)Z)biZ)Z)Z)z  N  %s2'>'>? # !   N%6$=>     $:D#AB +    ",    N %  BG   "! " RX# $z * W G X 1   4  ! !jjjn]++ >* % %~f N; ' ' "N#677 ^L ) ) ~f~f ^H % % r