i jrr)torcharangeint64einsumfloatcatsincos)rrinv_freq sinusoid_inps d/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/codegen/modeling_codegen.pycreate_sinusoidal_positionsr+0se QQekk JS PQRH<< WEKK0X0^0^0`bjkqqsL 99eii -uyy/FGQ OOxcUSS2SS2SS2SSS24nUSS2SS2SS2SSS24n[R"U*U4SS9nURS5$)Nrrr)r stackflatten)r-x1x2s r*rotate_every_twor57sS 1aCaC<B 1aADqD= B bS"I2&A 99R=r,tensorr&r'c[R"USS2SS2SSS24SS5n[R"USS2SS2SSS24SS5nX-[U5U--$)Nrr)r repeat_interleaver5)r6r&r's r*apply_rotary_pos_embr9?s\  ! !#aD!m"4a ;C  ! !#aD!m"4a ;C L-f5; <[TU]5 URn[R"UR 5Ul[R"UR5UlX l Uc-[RSURRS35 URUlUR UlURUR -UlUR"UR -UR:wa&[%SURSUR S35e[&R("[&R*"UR"[&R,S95R/[&R0"55Ul[R4"URURS-SS 9Ul[R4"URURSS 9UlUR:UlUR:=(d URn[=X45Ulg) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.zEembed_dim must be divisible by num_attention_heads (got `embed_dim`: z and `num_attention_heads`: z).rrF)bias) super__init__max_position_embeddingsrDropout attn_pdrop attn_dropout resid_pdrop resid_dropout layer_idxlogger warning_once __class____name__ hidden_size embed_dimnum_attention_headshead_dim ValueErrorr sqrtr6float32toget_default_dtype scale_attnLinearqkv_projout_proj rotary_dimr+embed_positions)selfconfigrG max_positions pos_embd_dimrJs r*r@CodeGenAttention.__init__Fs 66 JJv'8'89ZZ(:(:;"      !8!8 9:,,   ++#)#=#= $*B*BB ==433 3t~~ EWX\XfXfWgh++/+C+C*DBH  **U\\$--u}}%UVYYZ_ZqZqZst $..$..12D5Q  $..$..uM  ++8$.. :=Wr,cURURSSX$-U4-5nURURSSS-URSS-5nU$)Nr/r0)r/)reshapeshape)r[r-n_headdim_headmp_numreshapeds r* _split_headsCodeGenAttention._split_headsds[99QWWSb\V-=x,HHI##AGGCRL5$88>>"#;N$NOr,c[UR5S:Xa$URSSSSS5R5nO][UR5S:Xa#URSSSS5R5nO![ S[UR535eUR 5SS X#-4-nUR U5$) z= Merges attn_head_size dim and num_attn_heads dim into n_ctx rrrrz3Input tensor rank should be one of [4, 5], but is: Nr0)lenrbpermute contiguousrPsizeview)r[r6rNattn_head_size new_shapes r* _merge_headsCodeGenAttention._merge_headsis v||  !^^Aq!Q2==?F  ! #^^Aq!Q/:: MMD$<$<<F !KK "Ec!!%)A)A4==Y_!`%=%=t}}U[\!!%)A)A4==Y_!` aAq)..  ! !\%8%8 8-001D1DEO#2D  .;;vv||B'71'<"ES ?? &1a!24??!223EAq$//"334F!Q#4T__#445E1aDOO$556F(S9E(S9E))UFO4CIIufo26E&sC5C(S9Ekk!Q1% aAq)  !)-"0 L $**366-2E2E+F~~_klJC%)JJu5R[$\! \'' T5M5Mt}}] mmK0 ((5 L((r,) rDrMrZrOrGrNrXrWrFrYrUN)NNNNNNFFN)rK __module__ __qualname____firstlineno__r@rgrsrrr FloatTensorr LongTensorboolrtupleTensorr__static_attributes__ __classcell__rJs@r*r;r;Es<X< &$ )D'+6:3715$),159G) 1 12G)UOG)!!2!23 G) u//0 G) E--. G)D>G)$D>G)!!1!12G)  ellE%,,//0u||U5<<%8% c@Q:RRST V G)G)r,r;ch^\rSrSrU4SjrS\\RS\R4SjrSr U=r $) CodeGenMLPc >[TU]5 URn[R"X15Ul[R"X5Ul[URUl [R"UR5Ul gr) r?r@n_embdrrVfc_infc_outractivation_functionactrBrEdropout)r[intermediate_sizer\rMrJs r*r@CodeGenMLP.__init__s` MM YYy< ii 1= &445zz&"4"45 r,rrcURU5nURU5nURU5nURU5nU$r)rrrr)r[rs r*rCodeGenMLP.forwards@ =1 /  M2  ]3 r,)rrrr) rKrrrr@rr rrrrrs@r*rrs16Xe.?.?%@UEVEVr,rcf^\rSrSrSU4SjjrSS\\RS\\S\\RS\\RS\\RS\\ S \\ S \\RS \ \ \R\\ \R\ \RS 4444S jjrSrU=r$) CodeGenBlockc>[TU]5 URb UROSUR-n[R "URUR S9Ul[X5Ul [X15Ul g)Nrkeps) r?r@n_innerrr LayerNormlayer_norm_epsilonln_1r;attnrmlp)r[r\rG inner_dimrJs r*r@CodeGenBlock.__init__s_ &,nn&@FNNa&--FW LLF4M4MN $V7 i0r,rrr|rr}rrrr.c Un URU5nURUUUUUUUUS9upURU5n X-U -nX4$)N)rrr|rr}rrr)rrr) r[rrr|rr}rrrresidual attn_outputsr~feed_forward_hidden_statess r*rCodeGenBlock.forwardsl! -0 %)YY'!)%/)&/ & " &*XXm%<"$AHL **r,)rrrrr)rKrrrr@rr rrrrrrrrrrrs@r*rrs1'+6:3715$),159+ 1 12+UO+!!2!23 + u//0 + E--. +D>+$D>+!!1!12+ uU\\"HU5<<uGXGXZ]G]A^3^-_$`` a++r,rcP^\rSrSr%\\S'SrSrS/rSr Sr U4Sjr Sr S r U=r$) CodeGenPreTrainedModelir\ transformerTrpast_key_valuesc&>[TU]"U0UD6 gr)r?r@)r[inputskwargsrJs r*r@CodeGenPreTrainedModel.__init__$s &+F+r,c[U[R45(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weights.)meanstdNr) isinstancerrVweightdatanormal_r\initializer_ranger>zero_ Embedding padding_idxrfill_)r[modules r* _init_weights$CodeGenPreTrainedModel._init_weights's fryyl + + MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r,)rKrrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraphr@rrrrs@r*rrs9 %&*#'("3!,**r,rcr^\rSrSrU4SjrSrSr\SS\\ RS\\ \ \ \ \ R4S\\ RS\\ RS \\ RS \\ RS \\ RS \\S \\S\\S\\S\\ RS\ \ \44Sjj5rSS\ \ RS4S\ RS\ RS\ S \4 Sjjr\S\ RS\S\S\ R,S\ RS\4 Sj5rSrU=r$) CodeGenModeli8c >[TU]U5 URUlURUl[ R "URUR5Ul[ R"UR5Ul [ R"[UR5Vs/sH n[XS9PM sn5Ul[ R "URUR"S9Ul['UR(UR*UR,-5UlSUlUR15 gs snf)N)rGrF)r?r@rrM vocab_sizerrwterB embd_pdropdrop ModuleListrangen_layerrhrrln_fminrYn_ctxrNgradient_checkpointing post_init)r[r\irJs r*r@CodeGenModel.__init__:s   ++<< 1 14>>BJJv001 5QWQ_Q_K`aK`a V AK`abLLV5N5NO f//A[A[1[\&+#  bs/EcUR$rr)r[s r*get_input_embeddings!CodeGenModel.get_input_embeddingsJs xxr,cXlgrr)r[new_embeddingss r*set_input_embeddings!CodeGenModel.set_input_embeddingsMs!r, input_idsrr|token_type_idsrr} inputs_embedsrroutput_hidden_states return_dictrrc U bU OURRn U bU OURRn UbUOURRnU bU OURRn USLUSL- (a [ S5eUR (a/UR(aU(a[RS5 SnUcURU5n[U[S5[45(d [ S5eU(a Uc [5nURSnU c7UbUR!5OSn["R$"XU-UR&S9n UcU R)S5nUR+X7XU 5nUR-X`RR.5nUnUb(UR1S U5nURU5nUU-nUR3U5nS UUR5S 54nU (aS OSnU (aS OSn[7UR85H:unnU (aUU4-nU"UUUUUUUU U S 9nUSnU (dM1UUS4-nM< UR;U5nUR1U5nU (aUU4-nU (d[=S UUUU455$[?UUUUS 9$)au inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzBThe `past_key_values` should be either a `Cache` object or `None`.rrrr/r)rr|rr}rrrc3,# UH ocMUv M g7frr).0vs r* 'CodeGenModel.forward..scacs )last_hidden_staterr attentions) r\rrruse_return_dictrPrtrainingrHrIrrtyperr rbget_seq_lengthr r!r unsqueeze_update_causal_mask get_head_maskrrprro enumeraterrrr )r[rrr|rrr}rrrrrrr seq_lengthpast_seen_tokensrrtoken_type_embeds output_shapeall_self_attentionsall_hidden_statesrblockoutputss r*rCodeGenModel.forwardPs.2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK??ab b 0*nO"((+  !CRC^==?de "\\*:zL] &&y++2E2EF %  %+00Z@N $ 8 ),==M -0 J (:(:2(>? $5b4"6BD!$&&)HAu#$58H$H!**)#A,#"3- G$AJM  &9WQZM&I##*& -0 %**<8  1]4D D )?T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr,r&r'rr(c UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nrk) fill_valuerrr)diagonalrr/r)rr r2rfullrtriur!raexpandclonerbrS masked_fill) r|r&r'rrr(rrr5 mask_length padding_masks r*r1BCodeGenModel._prepare_4d_causal_attention_mask_with_cache_position s}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r,)rrMrrrrYrr) NNNNNNNNNNNN)F)rKrrrr@rrrrr rrrrrrrr rr staticmethodintrr1rrrs@r*rr8s "15NR6:59371559$(,0/3&*59r E,,-r "%uU5<<5H/I(I"JKr !!2!23 r !!1!12 r u//0 r E--.r   1 12r D>r $D>r 'tnr d^r !!1!12r  u-- .r r v#( BellK78BllB B  B  BH4 444{{ 4  4  44r,rzM The CodeGen Model transformer with a language modeling head on top. ) custom_introc ^\rSrSrS/rU4Sjr\SS\\RS\\ \ \ \ \R4S\\RS\\RS\\RS \\RS \\RS \\RS \\S \\S\\S\\S\\RS\ \ \44Sjj5rSrU=r$)CodeGenForCausalLMiCzlm_head.weightc>[TU]U5 [U5Ul[R "UR UR5UlUR5 gr) r?r@rrrrVrrlm_headr)r[r\rJs r*r@CodeGenForCausalLM.__init__KsE  '/yy0A0AB  r,rrr|rrr}rlabelsrrrrrrc (U bU OURRn URUUUUUUUU U U U U S9 nUSnURU5R [ R 5nSnUb`UR UR5nUR"UU4SURR0UD6nUR UR5nU (dU4USS-nUbU4U-$U$[UUURURURS9$)a inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) rr|rrr}rrrrrrrrr)losslogitsrrr )r\rrrHrSr rRr loss_functionrrrrrr )r[rrr|rrr}rrJrrrrrrtransformer_outputsr lm_logitsrLoutputs r*rCodeGenForCausalLM.forwardSsD8&1%J $D>J 'tnJ d^J !!1!12J u,, -!J J r,rF)rFrr)1__doc__typingrrr torch.utils.checkpointr activationsr cache_utilsrr generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsr rmodeling_utilsrutilsrrrconfiguration_codegenr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerrKrHrCrr+r5r9Moduler;rrrrrF__all__rr,r*resn" !.)>9O- 1 !!;J   H %PP3P5<<P ==ELL=u||=X]XdXd= W)ryyW)v(#+-#+L*_**:G)GGT V /V  V r Kr,