tj|dZddlZddlZddlmcmZddlmZddlmZddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZdd lmZmZd dlmZejeZGddej Z!Gddej Z"Gddej Z#Gddej Z$Gddej Z%Gddej Z&Gddej Z'Gddej Z(Gdd ej Z)Gd!d"ej Z*Gd#d$ej Z+eGd%d&eZ,eGd'd(e,Z-ed)*Gd+d,e,eZ.gd-Z/dS).zPyTorch CPMAntN)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)auto_docstringlogging) CpmAntConfigc>eZdZdZdeffd ZdejfdZxZ S)CpmAntLayerNormz~ We use Root Mean Square (RMS) Layer Normalization, please see https://huggingface.co/papers/1910.07467 for details." configct|j|_|j|_t jtj|j|_ dSN) super__init__eps hidden_sizedim_normr Parametertorchemptyweightselfr __class__s h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/cpmant/modeling_cpmant.pyrzCpmAntLayerNorm.__init__)sN :* l5;v/A#B#BCC hidden_statescp|d|jkrtd|j}|t jddd}|t j ||j zz||j z}|S)f Args: hidden_states (`torch.Tensor` of shape `(batch, seq_len, dim_in)`) z'hidden_states.size(-1) != self.dim_normT)dimkeepdim) sizerAssertionErrordtypetorfloat32powmeanrsqrtrr)r!r% old_dtypevariances r#forwardzCpmAntLayerNorm.forward0s   b ! !T] 2 2 !JKK K!'  ##EM2266q99>>2t>TT&X5H)I)IIMMiXX[_[ff r$) __name__ __module__ __qualname____doc__rrrTensorr6 __classcell__r"s@r#rr$srD|DDDDDD U\        r$rceZdZd deffd Z ddejdejdejdejd edzd e dzd edzfd Z xZ S)CpmAntAttentionNrct|j|_|j|_|j|_||_tj |j|j|jzd|_ tj |j|j|jzd|_ tj |j|j|jzd|_ tj |j|jz|jd|_ tjd|_|j,tj|j|_dSd|_dS)NFbiasr(r*)p)rrr dim_modelnum_attention_heads num_headsdim_head layer_idxrLinear project_q project_k project_v attention_outrSoftmaxsoftmax dropout_pDropoutdropoutr!rrIr"s r#rzCpmAntAttention.__init__>s +3 "4>4>DM3QX]^^^4>4>DM3QX]^^^4>4>DM3QX]^^^Yt~ 'Et~\abbbx''B'//   ' 8++f.>+??DLLLDLLLr$Fhidden_q hidden_kvattention_mask position_biasoutput_attentionspast_key_values use_cachec |d} |d} |d} ||} ||} ||}| | | |j|jdddd} | | | |j|jdddd} || | |j|jdddd}|4|| ||j \} }| d} tj | | ddtj|jz }||z}tj||| d| | tjdktjt%d |j|j }||}tj||| d| | tjdktjd|j|j }|r|}nd}|j||}tj ||}|| |j| |jdddd}|| | |j|jz}||}||fS) ad Args: hidden_q (`torch.Tensor`): Input of transformer block(self-attention block). It can be the raw embedding of a batch of sequences. hidden_kv (`torch.Tensor` of shape `(batch, len_k, dim_model)`)): Tensor *key_value* and *query* of shape `(batch, len_k, dim_model)` attention_mask (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Avoid invalid areas to participate in the calculation of self-attention. position_bias (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Provide positional information to self-attention block. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). rrr)rNr(Fz-inf)devicer.)r,rKrLrMviewrGrHpermuteupdaterIrmatmul transposemathsqrt masked_filltensor scalar_tensorfloatr^r.rPrS contiguousrN)r!rUrVrWrXrYrZr[kwargs batch_sizelen_qlen_kquerykeyvaluescore attn_weightss r#r6zCpmAntAttention.forwardRs:]]1%%  a  q!!x((nnY''y)) :udndmLLTTUVXY[\^_``hhz5$.$-HHPPQRTUWXZ[\\ :udndmLLTTUVXY[\^_``  &(//UDNKKJCHHRLLE UCMM"b$9$9::TYt}=U=UU %!     Aue < < U@S@S S  f el%+ V V V    U##!     Aue < < U@S@S S  %,ek J J J     LLL < #LL''E UE** :t~udmLLTTUVXY[\^_``  ""'' E4>DM;YZZ""5))l""r$r)FNN) r7r8r9rrrr; BoolTensorboolrr6r<r=s@r#r?r?=s  |      4*/(,!%M#M#,M#<M#( M# | M#  $; M#M#$;M#M#M#M#M#M#M#M#r$r?ceZdZd deffd Z d dejdejdejdzdedzd edzd edzf d Z xZ S)CpmAntSelfAttentionBlockNrctt||_t |||_|jr+tj |j|_ dSd|_ dSNrI) rrrlayernorm_before_attentionr?self_attentionrQrrrRrSrTs r#rz!CpmAntSelfAttentionBlock.__init__sr *9&*A*A'-f JJJ    8++F,<==DLLLDLLLr$Fr%rWrXrYrZr[c ||}||||||||\}} |j||}||z}|| fS)a Args: hidden_states (`torch.Tensor` of shape `(batch, len_seq, dim_model)`): Input of transformer block(self-attention block). It can be the raw embedding of a batch of sequences. attention_mask (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Avoid invalid areas to participate in the calculation of self-attention. position_bias (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Provide positional information to self-attention block. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )r{r|rS) r!r%rWrXrYrZr[rkoutputsrss r#r6z CpmAntSelfAttentionBlock.forwardsv411-@@ $ 3 3       ! !  < #ll7++G%/ l**r$rNFNN r7r8r9rrrr;rurr6r<r=s@r#rwrws  |      .2).(,!%)+)+|)+ )+|d* )+  $; )+  )+$;)+)+)+)+)+)+)+)+r$rwc:eZdZdeffd ZdejfdZxZS)CpmAntDenseGatedACTrc&ttj|j|jd|_tj|j|jd|_tj |_ dSNFrA) rrrrJrdim_ffw_0w_1rGELUactr s r#rzCpmAntDenseGatedACT.__init__sh 9V/UKKK9V/UKKK8==??r$r%c|||}||}||z}|S)zTransform an input tensor from one feature space to another via a nonlinear operation Args: hidden_states (`torch.Tensor` of shape `(batch, seq_len, dim_in)`) )rrr)r!r% gate_scores r#r6zCpmAntDenseGatedACT.forwardsB XXdhh}5566 // "]2 r$ r7r8r9rrrr;r6r<r=s@r#rrsa#|###### U\        r$rc:eZdZdeffd ZdejfdZxZS)CpmAntFeedForwardrc,tt||_|j*t j|j|_nd|_t j |j |j d|_ dSr) rrrw_inrQrrrRrSrJrrw_outr s r#rzCpmAntFeedForward.__init__su '//   ' 8++F,<==DLLDLYv}f.@uMMM r$r%c||}|j||}||}|S)r')rrSrr!r%s r#r6zCpmAntFeedForward.forwardsE  -00 < # LL77M =11 r$rr=s@r#rrshN|NNNNNN U\        r$rc:eZdZdeffd ZdejfdZxZS)CpmAntFFNBlockrc tt||_t ||_|jr+tj |j|_ dSd|_ dSr) rrrlayernorm_before_ffnrffnrQrrrRrSr s r#rzCpmAntFFNBlock.__init__sl $3F$;$;!$V,,    8++F,<==DLLLDLLLr$r%c||}||}|j||}||z}|S)z Args: hidden_states (`torch.Tensor` of shape `(batch, len_seq, dim_model)`): Hidden states before feed forward layer. )rrrS)r!r% ln_outputsr~s r#r6zCpmAntFFNBlock.forwardsQ..}== ((:&& < #ll7++G%/ r$rr=s@r#rrsb |      |r$rceZdZd deffd Z d dejdejdejdzdedzd edzd edzf d Z xZ S)CpmAntTransformerBlockNrctt|||_t ||_dSry)rrrwself_attrrrTs r#rzCpmAntTransformerBlock.__init__"s@ 09MMM !&))r$Fr%rWrXrYrZr[c p|||||||\}}||}||fS)a Args: hidden_states (`torch.Tensor`): Input to the layer of shape `(batch, seq_len, dim_model)` attention_mask (`torch.Tensor`): Avoid invalid areas to participate in the calculation of shape `(batch, seq_len, seq_len)` position_bias (`torch.Tensor`): Provides position information to attention mechanism of shape `(num_heads, seq_len, seq_len)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rWrXrYrZr[)rr) r!r%rWrXrYrZr[rkrss r#r6zCpmAntTransformerBlock.forward'sQ4'+mm )'/+ '4' ' # |// l**r$rrrr=s@r#rr!s**|******.2).(,!%$+$+|$+ $+|d* $+  $; $+  $+$;$+$+$+$+$+$+$+$+r$rceZdZdeffd Z d dejdejdejdedzdedzd edzd edzfd Z xZ S) CpmAntEncoderrctj|_t jfdt |jD|_t|_ dS)Nc2g|]}t|S)rz)r).0irs r# z*CpmAntEncoder.__init__..Rs($q$q$qUV%;Fa%P%P%P$q$q$qr$) rrnum_hidden_layers num_layersr ModuleListrangelayersroutput_layernormr s `r#rzCpmAntEncoder.__init__Osm  2m$q$q$q$qZ_`d`oZpZp$q$q$qrr / 7 7r$Nr%rWrXrYoutput_hidden_statesrZr[c |rdnd} |rdnd} t|jD]+\} } |r| |fz } | ||||||} | \}}|r| |fz } ,||}|r| |fz } || | fS)a Args: hidden_states (`torch.Tensor`): Input to the layer of shape `(batch, seq_len, dim_model)` attention_mask (`torch.Tensor`): Avoid invalid areas to participate in the calculation of shape `(batch, seq_len, seq_len)` position_bias (`torch.Tensor`): Provides position information to attention mechanism of shape `(num_heads, seq_len, seq_len)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). N)rYrZr[) enumeraterr)r!r%rWrXrYrrZr[rkall_hidden_statesall_self_attnsrlayer layer_outputsrss r#r6zCpmAntEncoder.forwardVs:#7@BBD0:d!$+.. 2 2HAu# 6!m%55!!E"3 /# M+8 'M<  2^^*  ( ( $$$r$key_pos query_pos key_segment query_segmentc tj5|d}|d}|d}|d|dkrkey_pos.size(0) should be equal to query_pos.size(0), but got z and !z7keylen should be equal to key_segment.size(1), but got z;querylen should be equal to query_segment.size(1), but got r(r.r^)rrrr))rno_gradr,r-r_!_segment_relative_position_bucketr_position_bucketarangeint32r^rwhereF embeddingrr`rj) r!rrrrbatchkeylenquerylenrelative_position_bucketabsolute_position_bucketembedss r#r6z&CpmAntSegmentPositionEmbedding.forwards8]__% % LLOOE\\!__F ~~a((H||A).."3"333$U\UaUabcUdUdktkykyz{k|k|))!,,,,M>r$ c.d}|dz}|dktj|z}tj|}|dz}||k}|tj||z t j||z z ||z ztjz}tj|tj||dz }|tj ||tj|z }|S)Nrr)r) r/rrabslogrirdmin full_liker)r!relative_positionrrrelative_buckets max_exactis_smallrelative_position_if_larges r#rz/CpmAntSegmentPositionEmbedding._position_buckets -155ekBB[P!I&7881$ $y0%. I'--//); < <h|i/00 1Y& ( "U[// &" &+Y & O6 a H H& & " EK2C2F2Fu{2S2SUopppr$)rr) r7r8r9rrrr;r6rrr<r=s@r#rrs  |       22<2\ 2 | 2222h???        r$rcPeZdZfdZdejdejdejfdZxZS) CpmAntOutputcttj|j|j|_tj|j|j|_tj |j |_ dS)N)r) rrrrJrrr LayerNormlayer_norm_epsrRhidden_dropout_probrSr s r#rzCpmAntOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r$r% input_tensorrc||}||}|||z}|Sr)rrSr)r!r%rs r#r6zCpmAntOutput.forwards@ =11  ]33 }|'CDD r$rr=s@r#rrsi>>>>> U\RWR^r$rcXeZdZUeed<dZejfdZxZ S)CpmAntPreTrainedModelrcpmantc$t|t|trt j|jdSt|tr(t j|j d|j j dSdS)zInitialize the weightsg)r2stdN) r _init_weightsrrinitones_rrnormal_rrinit_std)r!moduler"s r#rz#CpmAntPreTrainedModel._init_weights s f%%% fo . . ] Jv} % % % % %  > ? ? ] L7ct{G[ \ \ \ \ \ \ ] ]r$) r7r8r9r__annotations__base_model_prefixrrrr<r=s@r#rrsg  U]__]]]]_]]]]]r$rceZdZdeffd ZdZdZdZe dde j dzde dzd e dzd e dzd e dzd e dzd e e j ezfdZxZS) CpmAntModelrct|t||_t j|j|j|_t j|j |j |j zz|j|_ t||_|j |_ |j |_ |dSr)rrrencoderr Embeddingrrsegment_embedding vocab_size prompt_types prompt_lengthinput_embeddingrrX post_initr s r#rzCpmAntModel.__init__s    $V,, !#f.BFDV!W!W!|   3f6J J JFL^  z&CpmAntModel.forward..s1bcbobobobobor$)last_hidden_staterZr% attentions)rrYrr"r[r.rrr/r^rsumrrr rrr,zerosfullr get_seq_lengthrjr rr!rXrtupler )r!rrYrrZr[r"rkr.r^segmentrr seq_lengthrpositionr past_lengthr%segment_statesrWrXrall_attentionsnew_attentions attentionnew_hidden_states hidden_states r#r6zCpmAntModel.forward>s(2C1N--TXT_Tq$8$D $+Jj &1%>>O:I:Uo44666[\ ((** ,,Y77 //88 !  +AAArssAAAI6N%6 55iwPVWW**8XwPP ';<<(:;%aaaKLL!!!&;< %aaaqqq&89 ;?<<      < < 8 (. !  )!!!T-?-A-A111*DEM)!#!/eeI"yAAAt7I7K7KTM_MaMa1a'b&ddNN!/ ,$&!$5UUL%,qqq$:L:N:NPQPQPQ7Q*R)TT%%$5! )?>> from transformers import CPMAntTokenizer, CpmAntForCausalLM >>> texts = "今天天气不错，" >>> model = CpmAntForCausalLM.from_pretrained("openbmb/cpm-ant-10b") >>> tokenizer = CPMAntTokenizer.from_pretrained("openbmb/cpm-ant-10b") >>> input_ids = tokenizer(texts, return_tensors="pt") >>> outputs = model.generate(**input_ids) >>> output_texts = tokenizer.batch_decode(outputs) >>> print(output_texts) ['今天天气不错，阳光明媚，我和妈妈一起去超市买东西。\n在超市里，我看到了一个很好玩的玩具，它的名字叫“机器人”。它有一个圆圆的脑袋，两只圆圆的眼睛，还有一个圆圆的'] ``` Nrr(r)losslogitsrZr%r()rr"rr'rintslicer<rr_r,r rZr%r()r!rrZr[rYrr=r"rWr>rk model_outputr% slice_indicesrAr@ loss_funcoutputs r#r6zCpmAntForCausalLM.forwardsPR&1%SV8W8Wk~ot444]k mAAA}aaa,?@AA  (**I9V[[V[[__==v{{2OOD FYabb!11F)-)9TGf$$v E%(8&4#.     r$c|jjSrrr rs r#rz&CpmAntForCausalLM.get_input_embeddingss {**r$c||j_dSrrI)r!rs r#rz&CpmAntForCausalLM.set_input_embeddingss&0 ###r$) NNNNNNNNr)r7r8r9_tied_weights_keysrrrrr;rrurBr-r r6rrr<r=s@r#r:r:sZ +,KL|*.(,!%)-,0&*#'.2-.F F <$&F F $; F  $; F #Tk F  t#F D[F  t+F el*F  ' 'F F F ^F P+++1111111r$r:)r:rr)0r:rdrtorch.nn.functionalr functionalrtorch.nnrrr activationsr cache_utilsrr generationr modeling_outputsr r modeling_utilsr utilsrrconfiguration_cpmantr get_loggerr7loggerModulerr?rwrrrrrrrrrrr:__all__rr$r#r[s %%%%%%&&&&&&!!!!!!........))))))OOOOOOOO------,,,,,,,,......  H % %bi2b#b#b#b#b#bib#b#b#J3+3+3+3+3+ry3+3+3+l")( 4RY6*+*+*+*+*+RY*+*+*+Z<@<@<@<@<@BI<@<@<@@        Y Y Y Y Y RYY Y Y z     29    ] ] ] ] ]O ] ] ]N N N N N 'N N N b Z1Z1Z1Z1Z1-Z1Z1  Z1z H G Gr$