tj dZddlmZddlZddlZddlmZddlmZm Z m Z ddl m Z ddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZd d lmZeje Z!dZ"Gddej#Z$Gddej#Z%dej&de'de'dej&fdZ(Gddej#Z)Gddej#Z*Gddej#Z+Gddej#Z, dHd"ej&d#e'd$e'd%e-d&e-dej&f d'Z.Gd(d)ej#Z/Gd*d+ej#Z0eGd,d-eZ1Gd.d/ej#Z2ed01eGd2d3eZ3ed41Gd5d6e1Z4eGd7d8e1Z5ed91Gd:d;e1Z6eGd<d=e1Z7ed>1Gd?d@e1Z8eGdAdBe1Z9eGdCdDe1Z:eGdEdFe1Z;gdGZ Contains helpers for `FunnelRelMultiheadAttention `. cls_token_type_idrrNct||_tj|j|_tj|j|_d|_dSr5) rr rrr*r+ sin_dropout cos_dropout pooling_multr-s r0r z!FunnelAttentionStructure.__init__DsZ  :f&;<<:f&;<<!r1r3attention_masktoken_type_idscZd|_|dx|_}|||j|j}|||nd}|jjr;tj | |dz |dz gdnd}||||fS)zCReturns the attention inputs associated to the inputs of the model.rN)rrrr) rHsizeseq_lenget_position_embedsdtypedevicetoken_type_ids_to_matr separate_clsr functionalpadnew_ones)r.r3rIrJrMposition_embedstoken_type_matcls_masks r0init_attention_inputsz.FunnelAttentionStructure.init_attention_inputsMs!.!3!3A!6!66 w227M>>BI z952>>>ES%( (|Aw!|S TZ[[[^^_deeHEh'Q,&?@AHwhlGaKEK`fgggjjkpqqJ!A+K!!!!T'*Xd^;H((8)<)<==I((8)<)<==I 9i"8bAAAI,q'VLLLOOPUVVCJ#% $Q (>?? c c !##.2++!%!5!5c;!G!GJ;?3F"//VZq/QQG%aaag.):Q)>(?@@G,0K,DTF1R4LL&QRQSQS*M9g}SSqS'9:A>> >##A#; r1rrrrdc||}|d|dz }||jdz}|||zz}|d|dz }tj||dz | tj|jS)zV Build the relative positional vector between `pos` and `pooled_pos`. Nrrarr`)shaper=rflongrP) r.rrrrd ref_point num_removemax_distmin_dists r0roz%FunnelAttentionStructure.relative_poss{  JqMCF* Z-a00 zF22a=3r7*|HhlVG5:VYV`aaaar1tensoraxisc|dStttfrD]}||}|St|ttfr%t |fd|DS|jzjjrjjrtdddntddd}ttdgz|gz}jjrPttdgztddgz}tj |||g}||S)zT Perform pooling by stride slicing the tensor along the given axis. Nc3DK|]}|VdSr5) stride_pool).0xrr.s r0 z7FunnelAttentionStructure.stride_pool..s3JJa 0 0D 9 9JJJJJJr1rarCr)r) isinstancelisttuplertypendimrrRrslicer=rk)r.rrax axis_slice enc_slice cls_slices` ` r0rz$FunnelAttentionStructure.stride_poolsk >4 dT5M * *  6 6))&"55M fudm , , K4<<JJJJJ6JJJJJ J  #'+": qt{?W qE$A   ]bcgimop]q]q 5;;-$.*=>> ; # GuT{{md2eD!nn5EEFFIYy 16:FFFFi  r1meanmodecdStttfr'tfdDSjjr@jjrddddfn}tjddddf|gdj }|dkrddddddfn|dkrdddddddfdfdkr$tj d ned kr$tj d n;d kr&tj  d  ntd |dkrddddddfS|dkr dddfSS)z3Apply 1D pooling to a tensor of size [B x T (x H)].Nc3HK|]}VdS))rrN) pool_tensor)rrrr.rrs r0rz7FunnelAttentionStructure.pool_tensor..s9ccWX 0 0d6 0 R Rccccccr1rarrbrCrrT)r ceil_modemaxminz0The supported modes are 'mean', 'max' and 'min'.r)rrrrrrRrr=rkrrrS avg_pool2d max_pool2dNotImplementedError)r.rrrsuffixrs```` r0rz$FunnelAttentionStructure.pool_tensors >4 fudm , , d4<<ccccccc\bccccc c ; # ?'+{'?KVAAAssF^^VFYqqq"1"u v6A>>>F{ 199AAAtQQQ,-FF QYYAAAtQQQM*F! 6>>]--ffVW[-\\FF U]]]--ffVW[-\\FF U]]m..wvY].^^^FF%&XYY Y 199!!!Q1*% % QYY!!!Q$<  r1attention_inputsc|\}}}}|jjr|jjdkr)||ddd|ddz}||d}||d}|||jj}n|xjdzc_|jjdkr||d}||ddg}||ddg}||d}|||jj}||||f}||fS)zTPool `output` and the proper parts of `attention_inputs` before the attention layer.r^NrCrrrr)r pool_q_onlyrerr pooling_typerH)r.outputrrVrWrIrXs r0pre_attention_poolingz.FunnelAttentionStructure.pre_attention_poolingseEUA ; " M{)\99"&"2"2?2A23F"J"J_]^]_]_M`"`!--na@@N''!44H%%f4;3K%LLFF    "  {)\99"&"2"2?A"F"F!--nq!fEEN''1a&99H!--n5-IIN%%f4;3K%LLF+^^XV'''r1cP|\}}}}|jjr|xjdzc_|jjdkr)|dd||dddz}||d}||d}||d}||||f}|S)zFPool the proper parts of `attention_inputs` after the attention layer.rCr^Nrrrr)rrrHrerr)r.rrVrWrIrXs r0post_attention_poolingz/FunnelAttentionStructure.post_attention_pooling3sDTA ; " J    "  {)\99"1"1""58H8HYZY[Y[I\^_8`8`"`!--na@@N''!44H!--n5-IIN+^^XVr1r8Nr)rrC)r:r;r<__doc__rDint__annotations__rr r=r>rrYrQrOrPrrNrnrorstrrrrr?r@s@r0rBrB=ss!|!!!!!!!/3.2 KK|K t+K t+ K u|  KKKK((EL(U\((((N(N(#(;N(8= N( u| tD$67 7N(N(N(N(` el      b b bc bSV b_d_k b b b b! uU\22T%,5GG!E#Jc*!  !!!!Drs$$lU5<%884 ;MM$UX$kn$ $$$$L((-el(;( u|U5<00 1((((,  uU\7J  uUZUaOb         r1rBpositional_attn context_lenrdrc|j\}}}}tj|||||g}|dddd|dddf}tj||||||z g}|dd|f}|S)N.)rr=reshape)rrrd batch_sizen_headrM max_rel_lens r0_relative_shift_gatherrAs/>/D,JmOj&+W^5_``O%aaaEFFAAAo6OmOj&'S^afSf5ghhO%c??!#F,D!E!Ei&uEEE i&99 i&99  U[&&1A%B%BCC  U[&&1A%B%BCC  U['661J%K%KLL  U[&&1A%B%BCC ek1ff2E&F&FGG6F?G<<,wF4IJJJFCK( r1c:|jjdkr|\}}}}|j|jz} |j} t jd|| z| } | |dddfz} | |dddfz} t jd| |t jd| |z}n|jd|krdnd}||j|dz }|j|jz}|j} t jd|| }t jd||z|}t|||}|||z}|S) z5Relative attention score for the positional encodingsr^zbinh,dnh->bindNz bind,jd->bnijrrCz td,dnh->tnhzbinh,tnh->bnit) rrerrrr=einsumrrr)r.rVrrrXr{r}r|r~uw_r q_r_attentionq_r_attention_1q_r_attention_2rrdrvr_heads r0relative_positional_attentionz9FunnelRelMultiheadAttention.relative_positional_attentionisQ ; % 5 5#2 CS%  *A-C"L)96A:sKKM+c!!!T'l:O+bDk9O$l?OSQQTYT`%UUOO aK77AAQE  01%!)bnisrCrrarb) rrrr=rrrpsplitwhere) r.rWrrXrrMrrtoken_type_biasdiff_token_typesame_token_typetoken_type_attns r0relative_token_type_attentionz9FunnelRelMultiheadAttention.relative_token_type_attentions  !1+9+?( G[=4:- ,'7(9JDN[['4077V\RS_V]_j8kll+0;r+R+R+R(+ O22>3GHH/J`J`aoauJvJv     x 'Or1Fquerykeyvalueroutput_attentions.c"|\}}}} |j\} } } |jd} |jj|jj}}||| | ||}||| | ||}||| | ||}||jz}|j |jz}tj d||z|}| ||| | }| ||| }||z|z}|j}|}|-|t d|ddddfz zz }tj|d|}||}tj d||}||| | ||z}||}|||z}|r||fn|fS)Nrzbind,bjnd->bnijra)rcrOzbnij,bjnd->bind)rrrrrviewrrrrr=rrrrOfloatINFsoftmaxrrrr+r))r.rrrrrrVrWrIrXrrM_rrrrrrr content_scorerr attn_scorerO attn_probattn_vecattn_outrs r0r7z#FunnelRelMultiheadAttention.forwards!EUA!& GQil +T[-?U##((WffMMS!!&&z;OOU##(([&&QQ$*$=4:- %68I6RR <<_fVackll<<^VU]^^#_4F  %%''  %#cQ4 1N1T1T1V1V-V&WWJM*"EBBB **955 < 19fEE>>("2"2:wQW"X"XYY&&x00!122&7F ""fYFr1r5F)r:r;r<rrr rrr=r>rboolr7r?r@s@r0rrQs)|)#)$)))))).((((T<#( 3G3G|3G\3G| 3G   - 3G  3G u|S !3G3G3G3G3G3G3G3Gr1rcLeZdZdeddffd ZdejdejfdZxZS)FunnelPositionwiseFFNrrNcttj|j|j|_t|j|_ tj |j |_ tj|j|j|_ tj |j |_tj|j|j|_dSr5)rr rrr'd_innerlinear_1r hidden_actactivation_functionr*activation_dropoutlinear_2r+r,r&r(r)r-s r0r zFunnelPositionwiseFFN.__init__s  &.&.AA #)&*;#< "$*V-F"G"G &.&.AA z&"788 ,v~v7LMMr1hiddenc||}||}||}||}||}|||zSr5)rrr r r,r))r.r hs r0r7zFunnelPositionwiseFFN.forwardso MM& ! !  $ $Q ' '  # #A & & MM!   LLOOvz***r1r9r@s@r0rrsyN|NNNNNNN+el+u|++++++++r1rc jeZdZdededdffd Z d dejdejd ejd ede f d Z xZ S) FunnelLayerrrrNctt|||_t ||_dSr5)rr r attentionrffn)r.rrr/s r0r zFunnelLayer.__init__s= 4V[II(00r1Frrrrc||||||}||d}|r ||dfn|fS)Nrrr)rr)r.rrrrrattnrs r0r7zFunnelLayer.forwardsQ~~eS%1AUf~gg$q'""$5DQ  F9Dr1r) r:r;r<rrr r=r>rrr7r?r@s@r0rrs1|1#1$111111#( E E| E\ E| E  E  E E E E E E E Er1rceZdZdeddffd Z ddejdejdzd ejdzd ed ed edee zfd Z xZ S) FunnelEncoderrrNct|_t|_t jfdtjD|_ dS)Nctg|]3\}tjfdt|D4S)c0g|]}tSr)rrrrs r0 z5FunnelEncoder.__init__...s#[[[A{6;??[[[r1)r ModuleListrl)r block_sizerrs @r0rz*FunnelEncoder.__init__.. sX   +K [[[[[zIZIZ[[[\\   r1) rr rrBattention_structurerr enumerate block_sizesblocksr-s `r0r zFunnelEncoder.__init__sy  #;F#C#C m    /89K/L/L      r1FTr3rIrJroutput_hidden_states return_dictc ||}|j|||}|}|r|fnd} |rdnd} t|jD]\} } |d|jjrdndk} | o| dk} | r|j||\}}t| D]\}}t|jj | D]x}|dko|dko| }|r|}|jj r|n|x}}n|x}x}}||||||}|d}|r|j |}|r | |ddz} |r| |fz} y|std|| | fDSt|| | S) NrIrJrrrCrrc3K|]}||V dSr5rrrs r0rz(FunnelEncoder.forward..B(aaqSTS`S`S`S`S`aar1last_hidden_state hidden_states attentions)type_asr rYr!r#rLrrRrrl block_repeatsrrrr )r.r3rIrJrr$r%rr all_hidden_statesall_attentionsrblock pooling_flag pooled_hidden layer_indexlayer repeat_index do_poolingrrr layer_outputs r0r7zFunnelEncoder.forwardsC(// >>3II ))J   0DN],,$0:d"+DK"8"8 J J K!;;q>>$+2J-QQQPQRL';K!OL 262J2`2`,33/ /'0&6&6 J J" U$)$+*CK*P$Q$QJJL".!"3!\+:J!\P\J!5 -04 0G&Zff]Zee.444e#(5U$Oaaaaa aGXesttttr1NNFFT r:r;r<rr r=r>rrr r7r?r@s@r0rrs  |         /3.2"'%* 0u0u|0u t+0u t+ 0u  0u # 0u0u  0u0u0u0u0u0u0u0ur1rTFrr target_lenrRrc J|dkr|S|r|ddddf}|ddddf}tj||d}|rU|r)tj|ddd|dz ddf}|ddd|dz f}tj||gd}n|ddd|f}|S)z{ Upsample tensor `x` to match `target_len` by repeating the tokens `stride` time on the sequence length dimension. rN)repeatsrcrrb)r=repeat_interleaverrSrTrk)rrr=rRrclsrs r0upsamplerBFs {{2A2h aaaeH  $QA > > >F(  L]&&v1a!Q/JKKF+Z!^++,C=a000;J;' Mr1ceZdZdeddffd Z ddejdejd ejdzd ejdzd ed ed edee zfdZ xZ S) FunnelDecoderrrNct|_t|_t jfdtjD|_ dS)Nc0g|]}tdS)rr)rrrs r0rz*FunnelDecoder.__init__..as#$f$f$f[%;%;$f$f$fr1) rr rrBr rrrlnum_decoder_layerslayersr-s `r0r zFunnelDecoder.__init__]sf  #;F#C#C m$f$f$f$fU6KdEeEe$f$f$fgg r1FT final_hiddenfirst_block_hiddenrIrJrr$r%ct|dt|jjdz z|jd|jj|jj}||z} |r| fnd} |rdnd} |j| ||} |j D]1} | | | | | |}|d} |r | |ddz} |r| | fz} 2|std| | | fDSt| | | S) NrCr)rr=rRrrr'rrc3K|]}||V dSr5rr)s r0rz(FunnelDecoder.forward..r*r1r+) rBlenrr"rrRrr rYrHrr )r.rIrJrIrJrr$r%upsampled_hiddenr r1r2rr7r:s r0r7zFunnelDecoder.forwardcsW$ T[4559:)/211    "$66)=GVII40:d3II ))J   [ B BE 59I]noooL!!_F  C!/,qrr2B!B# B$5 $A! baaV->$Oaaaaa aGXesttttr1r;r<r@s@r0rDrD\sh|hhhhhhh/3.2"'%* 'u'ul'u"L'u t+ 'u  t+ 'u  'u#'u'u  'u'u'u'u'u'u'u'ur1rDcPeZdZdZdeddffd ZdejdejfdZxZ S)FunnelDiscriminatorPredictionszEPrediction module for the discriminator, made up of two dense layers.rrNct||_tj|j|j|_tj|jd|_dSr)rr rrrr'densedense_predictionr-s r0r z'FunnelDiscriminatorPredictions.__init__sS  Yv~v~>> " &.! < <r1discriminator_hidden_statesc||}t|jj|}||d}|S)Nra)rRr rrrSsqueeze)r.rTr-logitss r0r7z&FunnelDiscriminatorPredictions.forwardsQ #>?? t{56}EE &&}55==bAA r1) r:r;r<rrr r=r>r7r?r@s@r0rPrPswOO=|======= 5<ELr1rPcXeZdZUeed<dZejfdZxZ S)FunnelPreTrainedModelrfunnelcft||jj}|ddkrt |ddj|jj7|jj \}}tj dt||zz }n |jj}tj|j|t |ddtj|jddSdS|dkrtj|j|jj tj|j|jj tj|j|jj tj|j|jj tj|j|jj dS|d krv|jjdn |jj}tj|jj||jj2tj|jj|jjdSdSdS) Nrraweightr_)stdrgr)br)r _init_weightsr/r:findgetattrrinitializer_stdr\rnpsqrtrinitnormal_ constant_runiform_rinitializer_rangerrrrr%rzeros_)r.module classnamefan_outfan_inr]r/s r0r_z#FunnelPreTrainedModel._init_weightss f%%%$- >>( # #r ) )vx..:;.6&,m&9OGV'#fw.>(?(?"?@@CC+5C V]4444vvt,,8v{C0000098 7 7 7 M&/T[-J K K K K M&/T[-J K K K K M&/T[-J K K K K M&/T[-J K K K K M&*dk.K L L L L L L , , ,4<##$+B]C L/6C @ @ @ @%1= F29&:P:\]^^^^^ - ,>=r1) r:r;r<rrbase_model_prefixr=no_gradr_r?r@s@r0rYrYsg  U]____________r1rYcPeZdZdededdffd ZdejdejfdZxZ S)FunnelClassificationHeadrn_labelsrNc ttj|j|j|_tj|j|_tj|j||_ dSr5) rr rrr' linear_hiddenr*r+r, linear_out)r.rrsr/s r0r z!FunnelClassificationHead.__init__s^ Yv~v~FFz&"788 )FNH==r1r c||}tj|}||}||Sr5)rur=tanhr,rv)r.r s r0r7z FunnelClassificationHead.forwardsG##F++F##f%%v&&&r1) r:r;r<rrr r=r>r7r?r@s@r0rrrrsx>|>s>t>>>>>> 'el'u|''''''''r1rrz2 Output type of [`FunnelForPreTraining`]. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ e ejdzed<dZ e ejdzed<dS)FunnelForPreTrainingOutputa1 loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss of the ELECTRA-style objective. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Prediction scores of the head (scores for each token before SoftMax). NlossrWr-r.) r:r;r<rr|r= FloatTensorrrWr-rr.rr1r0r{r{s&*D% d ")))'+FE  $+++59M5*+d299926Je'(4/66666r1r{z The base Funnel Transformer Model transformer outputting raw hidden-states without upsampling head (also called decoder) or any task-specific head on top. ceZdZdeddffd ZdejfdZdejddfdZe dde j dzd e j dzd e j dzd e j dzd e j dzd e dzde dzde dzde ezfdZxZS)FunnelBaseModelrrNct|t||_t ||_|dSr5)rr rr6rencoder post_initr-s r0r zFunnelBaseModel.__init__sQ    *622$V,,  r1c|jjSr5r6r%r.s r0get_input_embeddingsz$FunnelBaseModel.get_input_embeddings ..r1new_embeddingsc||j_dSr5rr.rs r0set_input_embeddingsz$FunnelBaseModel.set_input_embeddings*8'''r1r2rIrJ position_idsr3rr$r%c  <||n |jj}||n |jj}||n |jj}||t d|+||||} n.||dd} nt d||jn|j} |tj | | }|!tj | tj | }| ||}| ||||||} | S)NDYou cannot specify both input_ids and inputs_embeds at the same timera5You have to specify either input_ids or inputs_embedsrPr`r3rIrJrr$r%)rrr$r% ValueError%warn_if_padding_and_no_attention_maskrLrPr=onesrrr6r) r.r2rIrJrr3rr$r%kwargs input_shaperPencoder_outputss r0r7zFunnelBaseModel.forwardsV2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN  !"[EJvVVVN OO ,, ))/!5# '  r1NNNNNNNNr:r;r<rr rr!rrrr=r>rrr r7r?r@s@r0rrsW|/bl////92<9D9999*..2.2,0-1)-,0#'..<$&. t+. t+ . lT) . |d* . $;.#Tk.D[.  ...^.....r1rceZdZdeddffd ZdejfdZdejddfdZe dde j dzd e j dzd e j dzd e j dzd e dzd e dzde dzde ezfdZxZS) FunnelModelrrNct|||_t||_t ||_t||_| dSr5) rr rrr6rrrDdecoderrr-s r0r zFunnelModel.__init__)sg     *622$V,, $V,,  r1c|jjSr5rrs r0rz FunnelModel.get_input_embeddings3rr1rc||j_dSr5rrs r0rz FunnelModel.set_input_embeddings6rr1r2rIrJr3rr$r%c ||n |jj}||n |jj}||n |jj}||t d|+||||} n.||dd} nt d||jn|j} |tj | | }|!tj | tj | }| ||}| ||||d|} || d | d |jjd ||||| } |sEd } | d f}|r| d z } || d | | zfz}|r| d z } || d | | zfz}|St!| d |r| j| jznd|r| j| jznd S)Nrrarrr`rTrrr)rIrJrIrJrr$r%rCr+)rrr$r%rrrLrPr=rrrr6rrr"r r-r.)r.r2rIrJr3rr$r%rrrPrdecoder_outputsidxoutputss r0r7zFunnelModel.forward9sf2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN  !"[EJvVVVN OO ,, ))/!%# '  ,,(+.q1$+2I!2LM))/!5#'   C&q)+G# Qq!_Q%7/#:N%N$PP  Qq!_Q%7/#:N%N$PPN-a0#?8?;XXXTeo2_5OOOko     r1)NNNNNNNrr@s@r0rr'sT|/bl////92<9D9999*..2.2-1)-,0#'H H <$&H  t+H  t+ H |d* H  $; H #TkH D[H   H H H ^H H H H H r1rz Funnel Transformer model with a binary classification head on top as used during pretraining for identifying generated tokens. ceZdZdeddffd Ze ddejdzdejdzdejdzdejdzd ejdzd edzd edzd edzde e zfd Z xZ S)FunnelForPreTrainingrrNct|t||_t ||_|dSr5)rr rrZrPdiscriminator_predictionsrr-s r0r zFunnelForPreTraining.__init__sP    !&)) )G)O)O& r1r2rIrJr3labelsrr$r%c ||n |jj}||||||||} | d} || } d} |t j}|s|d| jddk}| d| jd|}||}|||} n=|| d| jd|} |s| f| ddz}| | f|zn|St| | | j | j S)a" labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the ELECTRA-style loss. Input should be a sequence of tokens (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates the token is an original token, - 1 indicates the token was replaced. Examples: ```python >>> from transformers import AutoTokenizer, FunnelForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("funnel-transformer/small") >>> model = FunnelForPreTraining.from_pretrained("funnel-transformer/small") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> logits = model(**inputs).logits ```NrIrJr3rr$r%rrarr|rWr-r.) rr%rZrrrrrrr{r-r.)r.r2rIrJr3rrr$r%rrTdiscriminator_sequence_outputrWr|loss_fct active_loss active_logits active_labelsrs r0r7zFunnelForPreTraining.forwardsB&1%rrr{r7r?r@s@r0rrs)|*..2.2-1&*)-,0#'C C <$&C  t+C  t+ C |d* C  t# C  $;C #TkC D[C  + +C C C ^C C C C C r1rceZdZddiZdeddffd ZdejfdZdej ddfd Z e dd e j dzd e j dzd e j dzd e j dzde j dzdedzdedzdedzdeezfdZxZS)FunnelForMaskedLMzlm_head.weightz(funnel.embeddings.word_embeddings.weightrrNct|t||_t j|j|j|_| dSr5) rr rrZrrr'r"lm_headrr-s r0r zFunnelForMaskedLM.__init__sZ    !&)) y1BCC  r1c|jSr5rrs r0get_output_embeddingsz'FunnelForMaskedLM.get_output_embeddingss |r1rc||_dSr5rrs r0set_output_embeddingsz'FunnelForMaskedLM.set_output_embeddingss % r1r2rIrJr3rrr$r%c ||n |jj}||||||||} | d} || } d} |Kt }|| d|jj|d} |s| f| ddz}| | f|zn|St| | | j| j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrrarr) rr%rZrrrr"r r-r.)r.r2rIrJr3rrr$r%rrr,prediction_logitsmasked_lm_lossrrs r0r7zFunnelForMaskedLM.forwards&&1%rrr r7r?r@s@r0rrsc*,VW|ry&BL&T&&&&*..2.2-1&*)-,0#'/ / <$&/  t+/  t+ / |d* /  t# /  $;/ #Tk/ D[/   / / / ^/ / / / / r1rz Funnel Transformer Model with a sequence classification/regression head on top (two linear layer on top of the first timestep of the last hidden state) e.g. for GLUE tasks. ceZdZdeddffd Ze ddejdzdejdzdejdzdejdzd ejdzd edzd edzd edzde e zfd Z xZ S)FunnelForSequenceClassificationrrNct||j|_||_t ||_t ||j|_|dSr5) rr num_labelsrrrZrr classifierrr-s r0r z(FunnelForSequenceClassification.__init__(se     + %f-- 266;LMM r1r2rIrJr3rrr$r%c ||n |jj}||||||||} | d} | dddf} || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j| d}n*|jjdkrt}|| |}|s| f| ddz}||f|zn|St|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationrar)rr%rZr problem_typerrOr=rrrrVrrrrr-r.)r.r2rIrJr3rrr$r%rrr, pooled_outputrWr|rrs r0r7z'FunnelForSequenceClassification.forward2s&&1%rrrr7r?r@s@r0rr!s)|*..2.2-1&*)-,0#'B B <$&B  t+B  t+ B |d* B  t# B  $;B #TkB D[B  ) )B B B ^B B B B B r1rceZdZdeddffd Ze ddejdzdejdzdejdzdejdzd ejdzd edzd edzd edzde e zfd Z xZ S)FunnelForMultipleChoicerrNct|t||_t |d|_|dSr)rr rrZrrrrr-s r0r z FunnelForMultipleChoice.__init__zsQ    %f-- 261== r1r2rIrJr3rrr$r%c J||n |jj}| |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||} | d} | dddf} || }|d| }d}|t}|||}|s|f| ddz}||f|zn|St||| j | j S)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrarrr) rr%rrrLrZrrr r-r.)r.r2rIrJr3rrr$r%r num_choicesrr,rrWreshaped_logitsr|rrs r0r7zFunnelForMultipleChoice.forwards&&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImq(   r=#5#5b#9#9=;M;Mb;Q;Q R R R  ++ ))'/!5#  $AJ)!!!Q$/ // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     r1r) r:r;r<rr rr=r>rrr r7r?r@s@r0rrxs|*..2.2-1&*)-,0#'; ; <$&;  t+;  t+ ; |d* ;  t# ;  $;; #Tk; D[;  * *; ; ; ^; ; ; ; ; r1rceZdZdeddffd Ze ddejdzdejdzdejdzdejdzd ejdzd edzd edzd edzde e zfd Z xZ S)FunnelForTokenClassificationrrNc6t||j|_t||_t j|j|_t j |j |j|_ | dSr5) rr rrrZrr*r+r,rr#rrr-s r0r z%FunnelForTokenClassification.__init__sy     +!&)) z&"788 )F$68IJJ r1r2rIrJr3rrr$r%c ||n |jj}||||||||} | d} || } || } d} |Ft }|| d|j|d} |s| f| ddz}| | f|zn|St| | | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrarr) rr%rZr,rrrrrr-r.)r.r2rIrJr3rrr$r%rrr,rWr|rrs r0r7z$FunnelForTokenClassification.forwards"&1%rrrr7r?r@s@r0rrs |       *..2.2-1&*)-,0#'. . <$&.  t+.  t+ . |d* .  t# .  $;. #Tk. D[.  & &. . . ^. . . . . r1rceZdZdeddffd Ze ddejdzdejdzdejdzdejdzd ejdzd ejdzd edzd edzd edzde e zfdZ xZ S)FunnelForQuestionAnsweringrrNct||j|_t||_t j|j|j|_| dSr5) rr rrrZrrr# qa_outputsrr-s r0r z#FunnelForQuestionAnswering.__init__se     +!&)) )F$68IJJ r1r2rIrJr3start_positions end_positionsrr$r%c | | n |jj} |||||||| } | d} || } | dd\}}|d}|d}d}||t|dkr| d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | jS) Nrrrrarb) ignore_indexrC)r| start_logits end_logitsr-r.)rr%rZrrrV contiguousrMrLsquezeclamprrr-r.)r.r2rIrJr3rrrr$r%rrr,rWrr total_loss ignored_indexr start_lossend_lossrs r0r7z"FunnelForQuestionAnswering.forward s&1%rrrr7r?r@s@r0rrs+|*..2.2-1/3-1)-,0#'; ; <$&;  t+;  t+ ; |d* ; , ; |d*;  $;; #Tk; D[;  - -; ; ; ^; ; ; ; ; r1r) rrrrrrrrrY)TF)=r dataclassesrnumpyrcr=rtorch.nnrrrr re activationsr modeling_outputsr r r rrrmodeling_utilsrutilsrrrconfiguration_funnelr get_loggerr:loggerrModulerrBr>rrrrrrrrBrDrPrYrrr{rrrrrrrr__all__rr1r0rsk('!!!!!! AAAAAAAAAA&&&&&&!!!!!!.-----9999999999......  H % % ry"A A A A A ryA A A H EL s SV [`[g     MGMGMGMGMG")MGMGMG`+++++BI+++&EEEEE")EEE&<u<u<u<u<uBI<u<u<u@di | .1AE\` \,.u.u.u.u.uBI.u.u.ub     RY    _____O___> ' ' ' ' 'ry ' ' '   7 7 7 7 7 7 7   7 ?????+??  ?DZ Z Z Z Z 'Z Z Z z M M M M M 0M M  M `B B B B B -B B B J N N N N N &;N N  N bE E E E E 3E E E P; ; ; ; ; #8; ; ; |G G G G G !6G G G T   r1