g)3) dataclass)OptionalUnionN)check_model_inputs)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introc\rSrSr%SrSr\\R\ S'Sr \R\ S'Sr \\ \ \R\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'Sr\\R\ S 'Sr\R\ S 'Sr\\ \ \R\ S 'Sr\\ \RS4\ S 'Sr\\ \RS4\ S'Sr\\R\ S'Srg)CsmOutputWithPast1a loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____firstlineno____doc__r&rtorch FloatTensor__annotations__r'r(tupler)r*r+r,r-r.r/r0__static_attributes__r1[/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/csm/modular_csm.pyr$r$1s'<)-D(5$$ %, $FE  $AEOXeE%*;*;$<=>E=AM8E%"3"3S"89:A:>Ju00#567>6:!2!23:.2%++2OS!8E%8I8I2J,K#LSKO%0A0A30F*G!HOHLhuU->->-C'DEL15M8E--.5r<r$c\rSrSrSrg) CsmRMSNormdr1Nr2r3r4r5r;r1r<r=r?r?dr<r?c\rSrSrSrg)CsmRotaryEmbeddinghr1NrAr1r<r=rDrDhrBr<rDc\rSrSrSrg)CsmMLPlr1NrAr1r<r=rGrGlrBr<rGc\rSrSrSrg) CsmAttentionpr1NrAr1r<r=rJrJprBr<rJc\rSrSrSrg)CsmDecoderLayertr1NrAr1r<r=rMrMtrBr<rMz[ The bare Csm Model outputting raw hidden-states without any specific head on top. cb^\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr Sr \\S.rU4SjrS rU=r$) CsmPreTrainedModelxconfigmodelTrMr()r)r*c>[TU]U5 [U[5(a]URn[ US- 5H>nUR RURSURRS9 M@ gg)Nrg)meanstd) super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeweightdatanormal_rRinitializer_range)selfmoduler[i __class__s r=rX CsmPreTrainedModel._init_weightssn f% f. / /"00M=1,- ""1%--3DKK[TU]U5 [R"URUR -UR 5Ul[R"UR URSS9Ul gNF)bias) rW__init__nn Embeddingr[ vocab_sizebackbone_hidden_size embed_tokensLinear hidden_sizeinputs_embeds_projectorrarRrds r=rwCsmDepthDecoderModel.__init__s]  LL&*>*>ARAR*RU[UpUpq')yy1L1LfN`N`gl'm$r< input_idsbackbone_last_hidden_stateattention_mask position_idsr( inputs_embeds use_cachecache_positionkwargsreturnc (Ub:[RR5(d[R S5 SnUSLUSL- (a [ S5eU(a Uc [ 5nUciUbUR5OSn UbURSOURSn Ub URO URn [R"XU -U S9nUc[R"US- SS9n XR-nURX-5nUSS:HnUb X&SS2S4'O?[RR5(dU(a[RS5 URU5n[!UR"UUUUUS 9nUnUR%S5nUR'UU5nUR(SUR"R*HnU"U4UUUUUUS .U D6nM UR-U5n[/UU(aUS 9$SS 9$) a* backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.rrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)rR input_embedsrrr(r)rrpast_key_valuerrposition_embeddings)last_hidden_stater()r7compiler is_compilinglogger warning_once ValueErrorr get_seq_lengthshaperarangeclamprzr|warningrr rR unsqueeze rotary_emblayersnum_hidden_layersnormr )rarrrrr(rrrrpast_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr)r decoder_layers r=forwardCsmDepthDecoderModel.forwards7&  #ENN,G,G,I,I   M  L  -t"; <Z[ [ 0*nO  !CRC^==?de :G:S 3 3A 6YbYhYhijYk -:-F]))IL\L\F"\\*:O`<`iopN  !KK(:BM"__4F --i.@AM+9!+<+A ()5&@ad#~~22449UNNQ44]C (;;&))+%  & &//2 "oom\J![[)H4;;+H+HIM) *).#-$7  MJ -0 &+/8O  >B  r<)r|r)NNNNNNNN)r2r3r4r5r r9rwrrr7 LongTensorrr8Tensorrboolrrrr:r rr;rorps@r=rrrrs !!n '+BF1537+/59$(59R ##R %-U->->$?R !. R u//0 R "% R   1 12R D>R !!1!12R +,R  u-- .R R r<rrc2^\rSrSrU4SjrSSjrSrU=r$)rZc>[TU]5 X l[R"[ R "URS- X55Ulg)Nr)rWrwr[rx Parameterr7emptyr])rar~r[rzrds r=rwCsmCodebooksHead.__init__s: *ll5;;t/A/AA/E{#_` r<c Uc3URSnUR[R"U5nOUS- nURUn[ URS5Vs/sH9n[ R RUSS2USS24XFR5PM; nn[R"USS9nU$s snf)Nrrdim) rr]r7rr\rx functionallinearTstack)rar)r seq_lengthcodebook_weightr codebook_idxs r=rCsmCodebooksHead.forwards  !&,,Q/J"kk%,,z*BCO*Q.M"kk-8O!&o&;&;A&> ? ?  MM q,/A!BODaDcDc d ?   Mq9   s%?B=)r[r]Nr2r3r4r5rwrr;rorps@r=rZrZsa r<rZa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) c@^\rSrSrSrSrSrU4SjrSS\RS\ \ S\ \RS\ \RS\ \R4 U4S jjjr \\SS\RS \ \RS\ \R S \ \RS\ \\ \\R4S\ \RS \ \RS \ \S\ \RS\\\R 4S\\S\\\44Sjj55rSrU=r$)CsmDepthDecoderForCausalLMiNc>[TU]U5 U?[URUR UR 5Ul[U5Ul gr) rWrwlm_headrZr~r[rzcodebooks_headrrrSrs r=rw#CsmDepthDecoderForCausalLM.__init__sE  L.v/A/A6CWCWY_YjYjk)&1 r<rr(rrrc >[T U]"XX4U40UD6nUSSS:HnU(dURS5 URS5 U$)Nrrrr)rWprepare_inputs_for_generationpop) rarr(rrrr model_inputsis_first_generation_steprds r=r8CsmDepthDecoderForCausalLM.prepare_inputs_for_generation!scw< ~ Y_ $00@#A!#D#I '   9 : (r<rrlabelsrlogits_to_keeprrc UR"SUUUUUUUU S.U D6n U Sn [U [5(a!U S:Xa [SS5nO[U *S5nOU nUR U SS2USS24U bXOS5nUR 5nSnUbBUSSS24R 5nUR "SUSURRUS.U D6n[UUU RU RU RS9$) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )rrrrr(rrrrrN.)r'rrz shift_labels)r&r'r(r)r*r1) rSrYintslicer contiguous loss_functionrRrzr r(r)r*)rarrrrr(rrrrrroutputsr) slice_indicesr'r&rs r=r"CsmDepthDecoderForCausalLM.forward7s82**  'A)%+')      nc * *" %a %~ot < *M$$ !]A- .Q_Qk0Mqu ""$  !#qr'?557L%%dt{{7M7M\hlrD&#33!//))   r<)rrSNNNN) NNNNNNNNNr)r2r3r4r5_tied_weights_keys_tp_plan_pp_planrwr7rrrr8rrrrrlistrrrrr:r rr;rorps@r=rrsHH2,0595959 ##"%!!1!12    1 12  !!1!12 ,'+BF1537KO59-1$(5934@ ##@ %-U->->$?@ !. @ u//0 @ "%tE4E4E/F(F"GH @   1 12@ ))*@ D>@ !!1!12@ c5<</0@ +,@  u,, -@ @ r<rc.^\rSrSrU4SjrSrSrU=r$)CsmBackboneModelEmbeddingsi|c>[TU]5 [R"URUR -UR 5UlURS[R"UR5UR -SS9 g)Naudio_tokens_offsetsF) persistent) rWrwrxryr[rzr~embed_audio_tokensregister_bufferr7rrs r=rw#CsmBackboneModelEmbeddings.__init__}sn "$,,0D0DvGXGX0X[a[m[m"n  "ELL1E1E$FIZIZ$Zgl  r<c^URXR-5nURSS9nU$)Nrr)rrsum)rarrs r=r"CsmBackboneModelEmbeddings.forwards4..y;T;T/TU #''A'. r<)rrrps@r=rr|s r<rcH^\rSrSrU4Sjr\\U4Sj55rSrU=r $)CsmBackboneModelicD>[TU]U5 [U5Ulgr)rWrwrr|rs r=rwCsmBackboneModel.__init__s  6v>r<c $>[TU]"S0UD6$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r1)rWr)ra super_kwargsrds r=rCsmBackboneModel.forwardsw...r<)r|) r2r3r4r5rwrrrr;rorps@r=rrs$? / /r<rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c<^\rSrSrSS/rU4SjrSrSrSr\ U4Sj5r U4S jr SS \ \ RS \ \ RS \ \ RS \ \ RS\ \ R4 SjjrSS \ R S\ \S\ \ R S\ \ R$S\ \ R 4 U4Sjjjr\\SS \ R S \ \ RS\ \ RS \ \ RS\ \ R S\ \\\\ R$4S\ \ R$S \ \ R S\ \S\ \ R S\\\ R4S\\S\\\44Sjj55rSrU=r $)CsmForConditionalGenerationiz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightc>[TU]U5 URUl[R"UR URSS9Ul[R"URUR 5Ul [RU5Ul [RUR5Ul[ R""UR$5UlUR)5 gru)rWrwrzrxr}r~rrytext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initrs r=rw$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r<c.URR$rrr|ras r=get_input_embeddings0CsmForConditionalGeneration.get_input_embeddingss""///r<c$XRlgrr)ravalues r=set_input_embeddings0CsmForConditionalGeneration.set_input_embeddingss+0(r<cURR(aOURURRR UR RR5 ggr)rRtie_codebooks_embeddings_tie_or_clone_weightsrr|rrrSrs r= _tie_weights(CsmForConditionalGeneration._tie_weightssL ;; / /  & &##00CC""((55  0r<c>URSS5(a[T U]"U0UD6up4O[T U]"U0UD6nSn[U5n[ UR 5R 5VVs0sH"upxURU5(dMXvSU_M$ n nn[ URR 5RSS0U E5 U Hn[UR XW-5 M SU;aUW4$U$s snnf)Noutput_loading_infoFdepth_decoder__from_model_config) getrWfrom_pretrainedlenvarsgeneration_configitems startswithrupdatedelattr) clsargsrrS loading_infoprefix prefix_lenattrrdepth_decoder_attrsrds r=r+CsmForConditionalGeneration.from_pretraineds  ::+U 3 3"''"94"J6"J E<G+TSnURRR5nURSS5 UR 5HupV[ URX5-U5 M [ TU]"U0UD6 g)Nr transformers_version)rr to_diff_dictrrsetattrrWsave_pretrained)rarrrrrrrds r=r$+CsmForConditionalGeneration.save_pretrainedsq!"00BBOOQ 6=.446KD D**FM5 A7 00r<r input_valuesinput_values_cutoffsrrcURU5nUGbM[RRUS5nX3S:R 5nXfS:n[ R "UR5URS9R[U5S5nXvRS5:n[ R"5 /n[X#5HupXS:n [U RSS- 5Hpn Xn XS-n U SX24nUR R#URS55nUR$R'SS5nUR)US5 Mr M [SU55n[ R*"UVs/sH7n[RRUSSSUURS- 45PM9 sn5nUR R-U5nSSS5 UR.R0nUU:HnUR2R5W5nUWUU'[ R6"SSUR.R84UR[ R:S 9UR.R<-nUR2R5U5R?S5nXR.R@:HnURCURE5S5UU'UbgURS5RCSSUR.R85nUUUU'UUU'US :HRGS S 9nS UUSUSSS24'UnXTS.$s snf!,(df  GN=f)a8 Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retreive codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nrrrrr.c3<# UHoRSv M g7f)rN)r).0els r= QCsmForConditionalGeneration._merge_input_ids_with_input_values..s&O=Nrxx{=Ns)rdtypeiTas_tuple)rr)$rrxrpaddiffr7rmaxrexpandrrno_gradzipr\rrencode audio_codes transposeappendrget_audio_codes_maskrRaudio_token_idrr|onesr[longcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rarr&r'rr audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrc start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesr-batched_audio_token_idsaudio_codes_maskr?audio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss r="_merge_input_ids_with_input_values>CsmForConditionalGeneration._merge_input_ids_with_input_valuessu*..y9  ##%==#4#45I6#R 01JKPPRM)!*;$A "<U"C&8i>O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA LGn$'&O=N&O#O */++`qr`qZ\R]]&&rAq!5EQR 5S+TU`qr+'$(#3#3#H#HIZ#[ !!$"[[77N(N: ..;;[T U]"S UUUUUS.UD6nUb|URS:XalURS5cZUR UURS5URS5URS5S9nUR USUSSS.5 U$) N)rr(rrrrrr&r'r)rr&r'r)rrrr1)rWrndimrr[r) rarr(rrrrr merged_inputsrds r=r9CsmForConditionalGeneration.prepare_inputs_for_generation;sw< +)')      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw r<rrrrc  Ub.URS:XaURXXH5n U SnU SnSnUR"SUUUUUU U S.U D6nUSn[U [5(a [ U *S5OU nUR USS2USS245nSnSnSnSnUbUSS2SS2S4nUR"SUUURRS.U D6nUSS2SS2SS24S :HRS S 9)nUUS SURRS- 24n[RRUS SS9nURSS9nUUSUSS- SS24nUUnUR "SUUU SUS.U D6nUR"nUU-n[%UUUUUR&UR(UR*Ub UR,OSUb UR&OSUb UR(OSUbUR*S9 $SS9 $)a_ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be infered from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrr)rrrr(rrrr)r'rrzrr3r*r.r))rTr1)rrr return_dictr) r&r0r+r'r(r)r*r,r-r.r/r1)r^r[rrYrrrrrRrzallr[rxrr4rFrr&r$r(r)r*r')rarr&rr'rr(rrrrrrr_backbone_outputsbackbone_hidden_statesrbackbone_logitsr&r0r+depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r=r#CsmForConditionalGeneration.forwardZsf  Y^^q%8 CC)=M*/:M"8,FI..  )%+')    "2!!48B>SV8W8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO ..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $($6$6%1+F# + %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r<)rrrrrrzr) NNNNNNNNNNr)!r2r3r4r5rrwrrr  classmethodrr$rr7rr[rrr8rrrrrrrrrr:r$rr;rorps@r=rrs @1 0141-1/37;)- PBELL)PBu||,PB'u||4 PB & PB %,,  PBj,0595959 ##"%!!1!12    1 12  !!1!12 >'+/3157;37KO59-1$(5934[ ##[ u||,[ !. [ 'u||4 [ u//0 [ "%tE4E4E/F(F"GH[   1 12[ ))*[ D>[ !!1!12[ c5<</0[ +,[  u'' ([ [ r<r)rPrrrrr)? dataclassesrtypingrrr7torch.nnrxtransformers.utils.genericr cache_utilsrr generationr masking_utilsr modeling_outputsr r modeling_utilsrprocessing_utilsrutilsrrrrautorllama.modeling_llamarrrrrrrrconfiguration_csmrr generation_csmr! get_loggerr2rr$r?rDrGrJrMrPrrModulerZrrrr__all__r1r<r=rs "" 9.)/O-&KK   @.   H %  )6 )6  )6Z   -  X  >  '  [[  [4\ :'9\ \ ~ryy.c !1?c c L  /z//. P "46HP  P f  r<