tjFrdZddlZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/e'j0e1Z2Gddej3Z4dej5de6dej5fdZ7 d=dej3dej5d ej5d!ej5d"ej5dzd#e8d$e8fd%Z9Gd&d'ej3Z:Gd(d)ej3Z;Gd*d+ej3Z<Gd,d-ej3Z=Gd.d/eZ>Gd0d1eZ?e%Gd2d3e Z@e%Gd4d5e@ZAGd6d7e@eZBe%d89Gd:d;e@ZCgd<ZDdS)>zPyTorch Zamba model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)lazy_load_kernel)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)resolve_internal_import)capture_outputs) ZambaConfigcTeZdZd deddffd ZdejdejfdZdZxZ S) ZambaRMSNormư>epsreturnNcttjt j||_||_dS)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer# __class__s f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/zamba/modeling_zamba.pyr'zZambaRMSNorm.__init__2sD l5:k#:#:;; # hidden_statesc|j}|tj}|ddd}|tj||jzz}|j||zS)NT)keepdim) dtypetor)float32powmeanrsqrtr,r+)r-r2 input_dtypevariances r0forwardzZambaRMSNorm.forward:s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r1cHt|jjd|jS)Nz, eps=)tupler+shaper,)r-s r0 extra_reprzZambaRMSNorm.extra_reprAs& )**II$2GIIIr1)r") __name__ __module__ __qualname__floatr'r)Tensorr?rC __classcell__r/s@r0r!r!1s$$$$$$$$$$;U\;el;;;;JJJJJJJr1r!r2n_repr$c|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rBexpandreshape)r2rKbatchnum_key_value_headsslenhead_dims r0 repeat_kvrSFs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr1modulequerykeyvalueattention_maskscalingdropoutc t||j}t||j} tj||dd|z} || |z} t j| dtj |j } t j | ||j } tj| | } | dd } | | fS)Nr4rr5)dimr7)ptrainingr)rSnum_key_value_groupsr)matmul transposer functionalsoftmaxr9r8r7r[r_ contiguous) rUrVrWrXrYrZr[kwargs key_states value_states attn_weights attn_outputs r0eager_attention_forwardrkRs3 ;<&B\$\!'-'E$ )d2 !'!9i X[_[h>hotuuu i X[_[h>hotuuu i X[_[h>hotuuu i :T] JFL^ejkkk r1Nr2rYpast_key_valuesrfr$c |jdd}g|d|jR}|||dd}|||dd} |||dd} ||| | |\} } tj |j j t} | ||| | |f|j sdn|j|jd|\} } | jg|dR} || } | | fS)Nr5rr4rT)r[rZ)rBrRr{viewrbr|r}updater get_interfacern_attn_implementationrkr_ryrZrNrer~)r-r2rorYrrf input_shape hidden_shape query_statesrgrhattention_interfacerjris r0r?zZambaAttention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT  &'6'='=j,Xa'b'b $J (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r1N)rDrErF__doc__rintr'r)rHr rrrAr?rIrJs@r0rmrmks  l{lsllllll.)- #)#)|#)#) t+ #)  #) +, #) u|U\D0% 2E2LL M#)#)#)#)#)#)#)#)r1rmcteZdZdZdeffd Z d dejdedzfdZ d dedzfdZ d dedzfd Z xZ S) ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rnc t||_||_|j|_|j|_|j|_|j |jz|_ |j |_ |j |_ |j |j z|_|j|_|j|_t'j|j |j |j|j|j |jdz |_|j|_t0|j|_|j|_t'j|j|j dz|j|_t'jt?j |j |j |jdzz|j|_!t'jt?j |j |j|j dz dz|j dzz |_"t'jt?j |j |j|_#t?j$d|jdzt>j%dddf}|&|j d'}t'jt?j(|)|j |jd|_*t'jt?j+|j |j|_,t'j|j |j|j|_-t]da/tat^d da1tat^d da2t]d a3titfd a5tatfdda6tatfdda7tqtjtltdtbtnfa9trstt;d|j<||_=dS)Nr) in_channels out_channelsrs kernel_sizegroupspaddingr4rrg?r7r5z causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmz8ops.triton.selective_state_update.selective_state_update) chained_pathselective_scan_fnmamba_inner_fnaqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)>r&r'rnror. mamba_d_statessm_state_size mamba_d_convconv_kernel_size mamba_expandintermediate_size mamba_dt_ranktime_step_rank n_mamba_headsmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrzin_projr(r)zeros x_proj_weightdt_proj_weight dt_proj_biasaranger9rMrelogrNA_logr*Dout_projr causal_conv1dgetattrrr mamba_ssmrselective_state_updaterrallis_fast_path_availablelogger warning_once layer_types layer_type)r-rnroAr/s r0r'zZambaMambaMixer.__init__s  "!-$2 & 3!'!4v7I!I$2#1"48JJ#3. i./#-))A-     !1&12 & 8y!143IA3MTXTabbb  \ K"#d&9A&==#    !l [+T-@$BU V VY\ \ !3& '   LT5GI\)])]^^ LD/!35= I I I$PQPQPQ' R HHT+R 0 0 ; ; = =\%)A,,"6"6t7I4K^`b"c"cdd ej);T=PQQRR $"8$:JQUQ^___ )99 &}6LdSS"=2DdKK%[11 !8 $^" " " $I/BDII ,> o7a<<);;M4>ZZ$//111 !& *t{7I!!!QPQPQPQ'7R*RXZ [ [ [ %6!T[%55M $ 7 7 : :5 A A K KB O O !-$1N111}GZ[]G^F^F`F`C`4a4k4klm4n4n$nM]..}t?TWdWjkmWn?npq>rss );;JWWX[^b^s]s]u]uXuv $])C)CC'M)R S S !-$1N111}GZ[]G^F^F`F`C`4a4k4klm4n4n$nM) -0H0H0K0K K  HHT[[%?%?XgX %NOOM) -0H0H0K0K K &--b$2DdFY[bccmmnoqrss ,QQQaaa];mKVVWY[]^^+ T0$2EtGZ[ac    1a#1!!!T':Y=P=PQSUW=X=XX\`\m AAtQQQ ]   ]334FGGYtz'')) * * *YqD!!!T111!458J111aaaQRQRQRTUTUTUW[K[8\\]] '111aaaD(89AaaaD!!!QQQ>N ? E E G GG w 9 9A"111aaaAqqq=1;;AqAAIMPXYZYZYZ\]\]\]_`_`_`bcefefefYfPgPqPqrsuvPwPwwI,y':':1a'@'@'C'CE'J'JAaaaQRQRQRTUWXWXWXjMLcLcdfLgLghhK    AAAqqq!!!QJ 7 8 8 8 8k,B777 !]TVAAAtQQQ J J J!%  ! !!Q ' ' / / B H H R RSTVW X X! ! %$r1c tttttt f}|jr<|rd|jjj vrtd| |||S| |||S)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)rY) rrrrrrrrrtype ValueErrorrr)r-r2rrYrfrs r0r?zZambaMambaMixer.forwards!$ #%68HJ^`n o" "    i) V4;M;T;Y-Y-Y i ,,]LYg,hh h   ^ \\\r1)NN) rDrErFrrr'r)rHr rrr?rIrJs@r0rrs  O8{O8O8O8O8O8O8d^bc%c%"\c%9>c%c%c%c%JR%R%ut|R%R%R%R%h ] ]54< ] ] ] ] ] ] ] ]r1rc$eZdZfdZdZxZS)ZambaMLPct||_|j|_|j|_t j|j|jd|_t j|j|jd|_t j|j|jd|_ t|j |_ dSNFrr) r&r'rnr.rrrz gate_projup_proj down_projr hidden_actact_fnr-rnr/s r0r'zZambaMLP.__init__s  !-!'!94#3T5KRWXXXy!143IPUVVV 4#94;KRWXXXV./ r1c||||||z}|Sr)r r rr )r-xr s r0r?zZambaMLP.forwardsANN4;;t~~a/@/@#A#ADLLQROO#STT r1)rDrErFr'r?rIrJs@r0rrsG00000r1rceZdZddededzffd Z ddejdejdedejdzd edzd e dzd e e d e ej e ej ej fdzffd ZxZS)ZambaAttentionDecoderLayerNrnroctt|||_t ||_t |j|j|_ t |j |j|_ dS)Nr#) r&r'rm self_attnr feed_forwardr!rt rms_norm_epsinput_layernormr.pre_ff_layernormrs r0r'z#ZambaAttentionDecoderLayer.__init__sv ' ::$V,,+F,HfNabbb ,V-?VEX Y Y Yr1Fr2original_hidden_statesrYr use_cacherfr$c tj||gd}||}|jd|||||d|\}}||}||}|S)ac Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). r5r)r2rorYrr)r) concatenaterrrr) r-r2rrorYrrrfrs r0r?z"ZambaAttentionDecoderLayer.forwards2)=:P*QWYZZZ ,,];; )4> ')+       q--m<< ))-88 r1r)NNF)rDrErFrrr'r)rHr boolrrrA FloatTensorr?rIrJs@r0rrs ZZ{ZsTzZZZZZZ/3(,!&''|'!& ' '  t+ '  '$;'+,' u %(95;L(L"MPT"TT U''''''''r1rc*eZdZdedeffd Z ddejdejdzdedzdejdzd ejdzd edzd e dzd ej dzd ejdzde e de eje ejejfdzffdZxZS)ZambaMambaDecoderLayerrnroctt|||_t |j|j|_||_dS)N)rnror) r&r'rmambar!r.rrrors r0r'zZambaMambaDecoderLayer.__init__sS $FiHHH +F,>FDWXXX"r1NFr2rrY causal_maskrr position_idstransformer_hidden_statesrfr$c  r|} | || zn|}||}|jd|||d| }| |z}|S)aX Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). N)r2rrYr)rr#) r-r2rrorYr$rrr%r&rfresiduals r0r?zZambaMambaDecoderLayer.forward#s|0! :S9^M5 5 5dq ,,];; "  '()     !=0 r1)NNNNNFNN)rDrErFrrr'r)rHr r LongTensorrrrArr?rIrJs@r0r!r!sE#{#s######7; $.2+/(,!&049=**|*!& t 3*: *  t+ * \D( **$;*&-*$)<$#6*+,* u %(95;L(L"MPT"TT U********r1r!c eZdZdedejdeffd Z ddej dej dzd e dzd ej dzd ej dzd e dzd e dzde edeejeejejfdzffdZxZS)ZambaHybridLayer shared_transflinearr#crt||_||_||_dSr)r&r'r,r- mamba_decoder)r-r,r-r#r/s r0r'zZambaHybridLayer.__init__Qs6 * "r1NFr2rrorYr$rrrfr$c ~|j|f|||||d|} || } |j|f| |||d|}|S)aF Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rrorYrr)r&rYrr)r,r-r/) r-r2rrorYr$rrrfr&s r0r?zZambaHybridLayer.forwardWs2%7D$6 % #9&+ % % % % !%)KK0I$J$J!**  &?)+      r1)NNNNNF)rDrErFrrrzr!r'r)rHrr rrrrArr?rIrJs@r0r+r+Ps'#&@#")#\r######7; $.2+/(,!&--|-!& t 3-: -  t+ - \D( --$;-+,- u %(95;L(L"MPT"TT U--------r1r+ceZdZUeed<dZdZddgZdgZdZ dZ dZ e e dZejfdZxZS) ZambaPreTrainedModelrnmodelTr+r!r)r2 attentionsc|jj}t|t |t r$t j|jd||jj dz}t j |j | ||jj |jj z|jjz}tjtj|jj|t%j|jjt%j|jjz zt%j|jjz|jj}|tjtj|  z}t j|j|tjd|jdztjdddf}||jd }t j|j!tj|"|j|j#dt j$|j%dSdS)NrT)r;stdrq)minrrr5)&rninitializer_ranger& _init_weights isinstancerinitnormal_rruniform_rrr.rr)rrandmathr time_step_max time_step_minclamptime_step_floorexpm1copy_rrrr9rMrrerrNrones_r) r-rUr6 dt_init_stdrdtinv_dtrr/s r0r9z"ZambaPreTrainedModel._init_weightssk+ f%%% fo . . ! L-CS A A A A+3T9K M&/+{ K K K![5 8OOSWS^SllN 4;4nEE8DK566$+B[9\9\\^(4;4556e 3e44  %)U["%5%5$5666F Jv*F 3 3 3 Q 5 9OOOPTVWVWVWPWXA1266AACCA Jv|UYq\\%9%9&:NPVPegi%j%j k k k Jvx % ! !r1)rDrErFr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulr!rm_can_record_outputsr)no_gradr9rIrJs@r0r2r2s &*#+-EF#4"5NL/$ U]__!!!!_!!!!!r1r2ceZdZdZdeffd Zeee dde j dzde j dzde j dzde dzd e j dzd edzd eed eezfd ZxZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rnct||j|_|j|_t j|j|j|j|_|j |_ g}d|_ t|j D]\}}t||}|dkrut j |jj|jjd}|t!t#||||j d|dd|di|_ ||t j||_t)|j|j |_d|_|dS) N)rohybridFrrz layers.(?!z\.)\d+.shared_transfzlayers.z.shared_transfr)r&r' pad_token_id padding_idx vocab_sizer Embeddingr. embed_tokenslayers_block_type_tied_weights_keys enumerater!rzrnrr+r ModuleListrr!rfinal_layernormgradient_checkpointing post_init)r-rnrlayer_idrr#r-r/s r0r'zZambaModel.__init__s    !. +L):FFDWXXX&+# r1N input_idsrYr%r inputs_embedsrrfr$c H|du|duz rtd|||}|}tj|} |r|t |j}|V||nd} tj|jd|j | z}| d}t|j||||} t|j D]\} } | || | || f||d|}||}t||r|ndS) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either one)rnrrr)rnrfrYrr%)rr)last_hidden_stater)rr\r)rr rnget_seq_lengthrrBrrrr_rrar)r-rerYr%rrfrrfr2rpast_seen_tokensr$rolayers r0r?zZambaModel.forwards  -t"; < s    --i88M% !&]!;!;  ?0*$+>>>O  CRC^==???de  < (;A(>}G[\\\_ooL'11!44L(;')+%    !*$+ 6 6   Iu!E&  !0#    MM,,];; &+/8BOOd    r1)NNNNNN)rDrErFrrr'rrrr)r)rHr rrrrrArr?rIrJs@r0rUrUs{8 .2.204(,26!%8 8 #d*8  t+8 &- 8  8 (4/ 8 $;8 +,8  ( (8 8 8 ^_ 8 8 8 8 8 r1rUc$eZdZddiZdeffd Zee ddej dzdej dzd ej dzd e dzd ej dzd ej dzd e dzdeej zdeedeezfdZ dfd ZxZS)ZambaForCausalLMzlm_head.weightzmodel.embed_tokens.weightrnct|t||_|j|_t j|j|jd|_| dSr) r&r'rUr3rZrrzr.lm_headrcr s r0r'zZambaForCausalLM.__init__sj    ''  +y!3V5FUSSS  r1NrrerYr%rrflabelsrlogits_to_keeprfr$c D|jd||||||d| } | j} t|trt | dn|} || dd| ddf} d}||j| ||jfi| }t|| | j | j | j S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rerYr%rrfrNlosslogitsrr2r4r) r3rir:rslicerp loss_functionrZrrr2r4)r-rerYr%rrfrqrrrrfoutputsr2 slice_indicesrvrus r0r?zZambaForCausalLM.forwardsH,64:, )%+' , , , ,  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4% D&#3!/)     r1TFc h|jj|d<tj|f||||||d|} | S)Nrr)rrYrfr%ris_first_iteration)rnnum_logits_to_keepr&prepare_inputs_for_generation) r-rerrYrfr%rr|rf model_inputsr/s r0r~z.ZambaForCausalLM.prepare_inputs_for_generationcs^$(;#A  1` a classification loss is computed (Cross-Entropy). )rYr%rrfrNrrz=Cannot handle batch sizes > 1 if no padding token is defined.r5rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rh regressionsingle_label_classificationmulti_label_classificationrt)r3rirrBrnrXrr8rr)int32rargmaxrrr/rD problem_typerr7longrrrrrrrrr2r4)r-rerYr%rrfrqrrftransformer_outputsr2rvrlast_non_pad_token non_pad_mask token_indices pooled_logitsruloss_fcts r0r?z&ZambaForSequenceClassification.forwards&8Btz 8 )%+' 8 8 8 8 ,= M**  "+JJ&,Q/J ; # + a\]] ] ; # +!#    "%)AAEEfmUZU`aaL!L)!F!Fr!J!J  !#    >*ZZZ    u|Jv}MMMOaab   YYv}--F{'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8M$9$9$;$;V^^=M=MNNDD#8M6::DD)-JJJ+--x 2 22t G GUWYY)-III,..x v66/ /?-;*5     r1)NNNNNNN)rDrErFr'rrr)r)rHr rrrrrArr?rIrJs@r0rr}s$.2.204(,26*.!%R R #d*R  t+R &- R  R (4/ R  4'R $;R +,R  1 1R R R ^R R R R R r1r)rnrrUr2)rT)Err?collections.abcrr)rtorch.nnrrrr r; activationsr cache_utilsr r generationr integrations.hub_kernelsr masking_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.import_utilsrutils.output_capturingrconfiguration_zambar get_loggerrDrModuler!rHrrSrGrkrmrrrr!r+r2rUrnr__all__rr1r0rs& $$$$$$ AAAAAAAAAA&&&&&&!!!!!!........))))))888888//////999999qqqqqqqqqqFFFFFFFF&&&&&&RRRRRRRRRRRR777777999999555555,,,,,,  H % %JJJJJ29JJJ* UU\ U# U%, U U U U&%% I% <% % < % L4' %  %%%%%2C)C)C)C)C)RYC)C)C)Ld]d]d]d]d]bid]d]d]P     ry    00000000f111117111h444441444n$!$!$!$!$!?$!$!$!N_ _ _ _ _ %_ _ _ Fggggg+_gggT    ^ ^ ^ ^ ^ %9^ ^   ^ B g f fr1