o sht= @sddlZddlZddlZddlmZddlmZmZmZm Z ddl Z GdddeZ zddl m Z ddlmZmZmZdZWn eyId ZYnwdd d Z ddd dd d eeeeee eeegeffffddZeeZeGdddZdS)N)Enum)AnyCallableOptionalUnionc@s,eZdZdZdZdZdZdZdZdZ dZ d S) RequestStatusz5Status of a generation request through its lifecycle.pending prefillingprefilling_splitsplit_pending_remainderdecodingfinishedfailedN) __name__ __module__ __qualname____doc__PENDING PREFILLINGPREFILLING_SPLITSPLIT_PENDING_REMAINDERDECODINGFINISHEDFAILEDrrX/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/transformers/utils/metrics.pyr sr)metrics)Status StatusCode get_tracerTFcstsddSfdd}|S)a Decorator that attaches a tracer to a class. This decorator should be applied to classes that need OpenTelemetry tracing. It adds a tracer attribute to the class instance that can be used by the traced decorator. Args: tracer_name_template: Optional template string for the tracer name. If provided, it should contain {module} which will be replaced with the class's full module path and {class_name} for the class name. If None, a default naming scheme will be used where: - If the module already starts with "transformers.", it will use that directly - Otherwise, it will prepend "transformers." to the module name Returns: Class decorator function cSs|SNr)clsrrr2szattach_tracer..cs*jtfdd}|_S)Ncsr|g|Ri|j}j}dur+|dr"|d|}nd|d|}nj||d}t||_dS)N transformers..)module class_name)rr startswithformatrtracer)selfargskwargs module_namer& tracer_name)r! original_inittracer_name_templaterrinit_with_tracer7s z:attach_tracer..decorator..init_with_tracer)__init__ functoolswraps)r!r1r0)r!r/r decorator4s z attach_tracer..decorator)_has_opentelemetry)r0r6rr5r attach_tracers r8) span_name standaloneadditional_attributesr;cs$fdd}|dur|S||S)a Decorator to trace function calls with OpenTelemetry. Can be used as @traced or @traced(span_name="custom_name") Args: func: The function to trace span_name: Optional custom name for the span (defaults to function name) standalone: If True, creates a parentless span additional_attributes: Optional list of additional attributes to set on the span. Each item is a tuple of (instance_attribute_name, span_attribute_key, value_or_transform_function) where: - instance_attribute_name: Name of the attribute to get from the class instance - span_attribute_key: Key to use when setting the attribute on the span - value_or_transform_function: Either a raw value to use directly, or a function to transform the attribute value before setting it on the span Returns: Decorated function with tracing cs0tsSddl}|fdd}|S)Nrc s,|rtdrjdur|dnd}|du}|r!t|dr!|j}n tdjdj}p1j}r7|jn|j}||}|dj|dj|d||rt |D]*\}} t | t t t tfsj| durv|d |t | qY|d |t t| qY|r|D]*\} } t | t t t tfs| dur|d | t | q|d | t t| qr|rވD]"} | \} }}t|| rt|| }t|r||}n|}|||qz|i|}|WWdSty }z|ttj||d}~ww1swYdS) N__self__rr)r#r$z function.namezfunction.modulezfunction.is_methodzargs.zkwargs.)hasattrr<r)rrr start_spanstart_as_current_span set_attribute enumerate isinstancestrintfloatbooltypeitemsgetattrcallable Exception set_statusrrERRORrecord_exception)r+r,instance is_methodr)namespan_fnspaniargkeyvalue attr_configinstance_attribute_namespan_attribute_keyvalue_or_transform_functionattribute_valuetransformed_valueresulte)r;funcr9r:rrwrapperpsT$          z*traced..decorator..wrapper)r7r3r4)r`r3rar;r9r:)r`rr6js 0ztraced..decoratorNr)r`r9r:r;r6rrbrtracedNs9rcc@seZdZdZdefddZddZedede d d fd d Z ed e d d fddZ edddZ ededed d fddZedede d d fddZd S)ContinuousBatchProcessorMetricsz0Metrics collection for ContinuousBatchProcessor.max_batch_tokenscCs||_|dS)zInitialize metrics for continuous batch processor. Args: max_batch_tokens: Maximum number of tokens in a batch N)re_setup_metrics)r*rerrrr2s z(ContinuousBatchProcessorMetrics.__init__cCsts tddStd|_gd}|jjddd|d|_|jjd d d d |_ |jjd dd d |_ gd}|jjddd|d|_ |jjdddd |_ |jj dddd |_|jj dddd |_gd}|jjddd|d|_|jjddd d |_|jjd!d"d d |_dS)#zIInitialize OpenTelemetry metrics and tracing if the library is available.zIOpenTelemetry is not installed. Metrics and tracing will not be recorded.Nz2transformers.generation.continuous_batch_processor) 2Kdi,i'ttft_millisecondsz#Time to first token in millisecondsms)rQ descriptionunit#explicit_bucket_boundaries_advisoryactive_requests_countz3Number of active requests currently being processedrequests)rQrurvwaiting_requests_countz*Number of requests waiting to be processed) rirkrnrorprqrri Ni0ui`request_latency_millisecondsz9End-to-end latency for completed requests in millisecondsdecode_prefill_ratioz3Ratio of decode tokens to prefill tokens in a batchratioprefill_tokens_processedz"Number of prefill tokens processedtokensdecode_tokens_processedz!Number of decode tokens processed) rg(ri<FPZ_brkbatch_fill_percentagez5Percentage of max_batch_tokens utilized in each batchpercentkv_cache_free_memory_bytesz/Free memory of the PagedAttentionCache in bytesbyteskv_cache_memory_bytesz0Memory usage of the PagedAttentionCache in bytes)r7loggerinfor get_metermetercreate_histogramttft_histogram create_gaugeactive_requests_gaugewaiting_requests_gaugerequest_latency_histogramdecode_prefill_ratio_gaugecreate_counterprefill_tokens_counterdecode_tokens_counterbatch_fill_percentage_histogramkv_cache_free_memory_gaugekv_cache_memory_gauge)r* ttft_bucketslatency_bucketsbatch_fill_bucketsrrrrfsx   z.ContinuousBatchProcessorMetrics._setup_metrics created_time request_idreturnNc C|tsdSt|d}z|j|td|d|ddWdSty=}ztd|WYd}~dSd}~ww)zRecord Time to First Token (TTFT). Args: created_time: The time the request was created request_id: The ID of the request N@@zRecorded TTFT for request : .2frtzFailed to record TTFT metric: )r7timerrecordrdebugrKwarning)r*rrttft_msr_rrrrecord_ttft_metric s  z2ContinuousBatchProcessorMetrics.record_ttft_metricrequests_in_batchc Cs&tr|sdSd}d}|D]}|jtjkr|d7}q |jtjtjfvr)|t|j7}q ||}zI|dkr9|j ||dkrC|j ||dkrQ||}|j |||j d}|j|td|d|d|dd |d |j d WdSty}ztd |WYd}~dSd}~ww) zRecord metrics about the batch composition including decode/prefill ratio and batch fill percentage. Args: requests_in_batch: List of request states in the current batch NrgY@zBatch metrics: z decode tokens, z prefill tokens, batch fill: rz% (/)z Failed to record batch metrics: )r7statusrrrrlen prompt_idsraddrrsetrerrrrrKr) r*r decode_tokensprefill_tokensstatetotal_batch_tokensr~fill_percentager_rrrrecord_batch_metricssD       z4ContinuousBatchProcessorMetrics.record_batch_metricsc CstsdSzj|jt|j}t|j}|jtjtjfvrdnd}|||j |j |j d|}|t|j|j |j |j d|}|j ||j |td|ddd|d|jd ||jd d d WdSty}ztd |WYd}~dSd}~ww)a&Record memory usage of the PagedAttentionCache without GPU synchronization. This calculates the theoretical memory usage based on cache configuration and the number of blocks currently in use. Args: cache: The PagedAttentionCache object to measure NzKV Cache memory: irzMB, Used blocks: rz (rkz.1fz%)z*Failed to record KV cache memory metrics: )r7 num_blocksr _free_blocks key_cachedtypetorchfloat16bfloat16 block_sizenum_key_value_headshead_dimrrrrrrKr)r*cachenum_used_blocks num_layersbytes_per_parameter memory_bytesfree_memory_bytesr_rrrrecord_kv_cache_memory_metricsFsb     z>ContinuousBatchProcessorMetrics.record_kv_cache_memory_metricsactive_requestswaiting_requestsc CsvtsdSz|j||j|td|d|dWdSty:}ztd|WYd}~dSd}~ww)zRecord metrics about active and waiting requests. Args: active_requests: Number of active requests waiting_requests: Number of waiting requests NzQueue metrics: z active requests, z waiting requestsz Failed to record queue metrics: )r7rrrrrrKr)r*rrr_rrrrecord_queue_metrics|s  z4ContinuousBatchProcessorMetrics.record_queue_metricsc Cr)zRecord metrics about a completed request. Args: created_time: The time the request was created request_id: The ID of the request Nrz Recorded request completion for rrrtz,Failed to record request completion metric: )r7rrrrrrKr)r*rr latency_msr_rrrrecord_request_completions  z9ContinuousBatchProcessorMetrics.record_request_completion)rN)rrrrrDr2rfrcrErCrlistrrrrrrrrrds Q( 5rdr )r3loggingrenumrtypingrrrrrr opentelemetryropentelemetry.tracerrrr7 ImportErrorr8rtuplerCrc getLoggerrrrdrrrrs4    0& Z