tjjddlmZddlmZddlmZddlmZddlm Z ddl m Z m Z m Z mZdd lmZdd lmZe r8dd lZeje Z eje Z ejeZ d8d9dZd:dZd;dZd;dZdZd Ze rVej d!ed"d#$ej!d!eej"d!ee%dd-Z) d?e(d.d d d.d/d@d7Z*d S)A) annotations)Callable)wraps)logging)GeneralInterface)is_torch_availableis_torch_greater_or_equalis_torch_less_or_equalis_torchdynamo_compiling)deepgemm_bf16_experts_forward)sonicmoe_experts_forwardNFinput torch.Tensorweightbiastorch.Tensor | None is_transposedboolreturnc&|r>i 3 344<r)rzerosr,rNr> enumeratetolistmm)rrrKoutputstartiends r _grouped_mm_fallbackrXs[A Au|SXS^ _ _ _F EDKKMM**3 C<<  uSy!6!9&s2CDDDD Mr"c|dksJdt|j|dksJdt|j|dksJdt|j|d|dks6Jd|dd |d|d|dks6Jd |dd |d|jt jt jfvsJd |jt j|d|d|j |j S)zRShape/dtype inference stub for `_grouped_mm_fallback` required by `torch.compile`.rz+input must be 2D (S, input_dim), got shape zBweight must be 3D (num_experts, input_dim, output_dim), got shape r z*offs must be 1D (num_experts,), got shape rz offs length z must match number of experts zinput_dim mismatch: input has z , weight has z$offs must be an integer tensor, got rM) r*tupleshaper,r>rint32int64emptyrNrrrKs r _grouped_mm_fallback_fakeras 99;;!   _5QVQ\K]K]__    ::<<1   bUSYS_M`M`bb    88::???\tzIZIZ\\??? 99Q<<6;;q>> ) ) )+v$))A,,+v+vflfqfqrsftft+v+v ) ) ) ::a==FKKNN * * *UAUUV[[QR^^UU + * * :%+u{3 3 3 35h\`\f5h5h 3 3 3 ;uzz!}}fkk!nnU\QVQ\ ] ] ]]r"cd||d|d|d|_dS)zjSaves input and weight for backward; offs is stored directly as it is a non-differentiable integer tensor.rr rN)save_for_backwardrK)ctxinputsrTs r "_grouped_mm_fallback_setup_contextrfs/&)VAY///ayCHHHr"c|j\}}tj|}tj|}d}t|jD]r\}}||kr tj|||||j|||tj|||j||||||}s||dfS)zuBackward pass for `_grouped_mm_fallback`. Computes grad_input and grad_weight per expert group; offs has no gradient.rrON) saved_tensorsr zeros_likerQrKrRrST) rd grad_outputrr grad_input grad_weightrUrVrWs r _grouped_mm_fallback_backwardrns%ME6!%((J"6**K ECHOO--..3 C<<  U3Y'*U3Y:OPPPP uSy!#[s%;QPPPP {D ((r"z!transformers::grouped_mm_fallbackz4(Tensor input, Tensor weight, Tensor offs) -> Tensor) mutates_argsschema) setup_contextcBtr|jtjksx|jjdkrGt ddr6|dzdks<|dzdks!|jjdkrt ddrdS|jjd krttj j d r(tj |jd kSttd ratd dr(tj |jd kStj |jdkSdSttj j d pttd S)a Check if torch.nn.functional.grouped_mm or torch._grouped_mm can be used based on availability and compatibility with torch.compile. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. Returns: `bool`: True if grouped_mm can be used, False otherwise. cpuz2.10.0T) accept_devrz2.8.0Fcuda grouped_mm)r _grouped_mmz2.9) r)r r>rbfloat16rNtyper data_ptrhasattrnn functionalrwget_device_capabilityr r`s r _can_use_grouped_mmr sx " # #(. (F(F =  & & "8 = = = ' __   #q ( (ENN,<,rzops transformersgrouped_mm_fallbackr`s r rzrz:s$5&$//P 58& 5 5 P8&11%((6<2H2H&W[1\\ \ UM * * P$UXXfl%;%;V$OOO O 9 ! 5 5eV$ 5 O OOr"c|rt|||}n&t||dd|}||||S)aGrouped linear layer supporting optional bias and transposed weights. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim) if `is_transposed`, else of shape (num_experts, output_dim, input_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. bias (`torch.Tensor`, *optional*): Bias tensor of shape (num_experts, output_dim). Default is `None`. is_transposed (`bool`, *optional*, defaults to `False`): Whether the weight tensor is transposed. Returns: `torch.Tensor`: Output tensor of shape (S, output_dim). rr)rz transposer)rrrKrrrs r _grouped_linearrYsc0F%d333%!1!1"b!9!9EEE   Jr"c|j}|d}|d}|d}|d}|d} tj| \} } || |z} || } |jdvr| n| }tj||j d|j dz }tj |dtj }| |j k d}| |j dz |jr|j}|jr |j| nd}n|j}|jr |j| nd}| |dt+| ||||j }|jr||}n||}|j}|jr |j| nd}t+|||||j }|| dz}||dtj| }tj| d| || <||}||||d }||j S) Nrr)rtmpsr )binsminmax)r*r>)rgr+)rNr))!rNr,r.rsortr}floatinthistcr0cumsumr]rclamp_r1r2r3r4r5r6 masked_fill_rrr7r8r9r: empty_likearanger;r<r=r>)r#r%r&r'rNr?r@rArCrD expert_ids_gpermselected_hidden_states_gsample_weights_g histc_inputtokens_per_expertoffsets sentinel_maskrErFrGrHinv_permrIs r grouped_mm_experts_forwardrs   !F  $$I##A&&J##B''J#**2..N$$R((JJ//L$,TY->?%d+ +1+*G*G,$$&&&\M]M]M_M_K Kd6FASWScfgSghhhl,!5;GGGG""T%55@@DDMD,q0111 }U,BF-Y$0>>UY<=A]T$+L99PT))-=== "2G/aeasH  })##H--;;x((~;?=Rd),77dO"G/QUQcH .88<<>>HTN)L '++J :NNRRWXRYY  ! !-"5 6 66r"c2eZdZdZeeeedZd fd Z xZ S) ExpertsInterfacez;Interface for registering custom experts forward functions.)deepgemm batched_mmrxsonicmoeexperts_implementationstrdefaultrrc|tdn|dkr||vrtd|dt||S)zfReturn the requested `experts_implementation`. Also strictly check its validity, and raise if invalid.Na You tried to access the `ExpertsInterface` with a `config._experts_implementation` set to `None`. This is expected if you use an Expert Module as a standalone Module. If this is not the case, something went wrong with the dispatch of `config._experts_implementation`eager`zL` is not a valid experts implementation registered in the `ExpertsInterface`)logger warning_onceKeyErrorsuperget)r#rr __class__s r get_interfacezExpertsInterface.get_interfaces ! )   N     $w . .3IQU3U3Ux*xxx ww{{17;;;r")rrrrrr) __name__ __module__ __qualname____doc__rrJrr_global_mappingr __classcell__)rs@r rrs]EE200, O < < < < < < < < < dfd }| ||S|S)aDecorator to modify experts class to support different experts implementations. Args: experts_class (`type[torch.nn.Module]`, *optional*): The experts class to modify. If not provided, returns a decorator that can be applied to the class. experts_interface (`ExpertsInterface`, *optional*, defaults to `ALL_EXPERTS_FUNCTIONS`): The experts interface to use for dispatching the forward method. is_concatenated (`bool`, *optional*, defaults to `True`): Whether the expert weights are stored in concatenated layout [gate;up] or interleaved layout [gate0, up0, gate1, up1, ...]. is_transposed (`bool`, *optional*, defaults to `False`): Whether the expert weights are stored in transposed format. has_bias (`bool`, *optional*, defaults to `False`): Whether the expert layers include bias terms or not. has_gate (`bool`, *optional*, defaults to `True`): Whether the experts use a gating mechanism or not. Whether it has gate_up_proj weights or just up_proj weights. Returns: `type[torch.nn.Module]`: The modified experts class. rrrc|j|jt fd}tfd}t|ds t|_||_||_|S)Nch||g|Ri|||_|_|_|_|_dSN)configr1r3rr) r#rargskwargsr3r1rr original_inits r __init__z=use_experts_implementation..wrapper..__init__4sP M$ 8 8 8 8 8 8 8 DK$DM$DM!.D #2D r"c\|jj}||g|Ri|Sr)rr_experts_implementation)r#rrexperts_forwardroriginal_forwards r forwardz.wrapper..forward=s>/==dk>acsttO"?49$999&99 9r"r7)rrrrrr7) rrrrrrr3r1rrs @@r wrapperz+use_experts_implementation..wrapper0s%. (0 }   3 3 3 3 3 3 3 3   3   : : : : : !  :}m44 <(;M %!)  ' r"N)rrrrro)rrrrr3r1rs ````` r use_experts_implementationrsV>2 w}%%% Nr")NF) rrrrrrrrrr) r#r$r%rr&rr'rrr)rrrrrKrrr)rrrrrKrrr) rrrrrKrrrrrrr)rrrrr)rrrrrrrrr3rr1rrr)+ __future__rcollections.abcr functoolsrutilsr utils.genericrutils.import_utilsr r r r rrrrr_dynamoassume_constant_result get_loggerrrr!rJrXrarfrnlibrary custom_op register_fakeregister_autogradrrzrrrALL_EXPERTS_FUNCTIONSrrror"r rs)#"""""$$$$$$,,,,,, 433333......ZLLL !& D DE^ _ _"]AABXYY  H % %\!% D=7=7=7=7F4 ^ ^ ^ ^ )))&  M+E   M CE^___ M##+%8$-W-W-W-W`PPPPF!% #####Le7e7e7e7P<<<<<'<<<2)(** " " " "37;+@ ;;;;;;;;r"