J?r?J@r@ S SKAJBrB S SKCJDrDJErE S SKFJGrG S SKHJIrI S SKJJKrK S SKLJMrMJNrNJOrOJPrPJQrQ S SKRJSrS S SKTJUrUJVrVJWrWJXrXJYrYJZrZJ[r[J\r\J]r]J^r^J_r_J`r`JaraJbrbJcrc S S KdJere S S KfJgrgJhrhJiri " S S \eS S!9rj " S" S#\SS S!9rk " S$ S%\IS S!9rl\h" \j5 rm\h" \k5 rn\h" \l5 ro1 S&krp1 S'krq1 S(krr\4R " \t5 ruS)S*S+S,.0rv\w" \vR 5 5 ry " S- S.\R 5 r{S/\4S0 jr|S1\}S2S3S4S34S5 jr~ " S6 S75 r " S8 S95 r\ " S: S;5 5 r " S< S=\55 r\tS>:X a \" 5 r\GR 5 gg)? N)ArgumentParser Namespace) dataclassfield)BytesIO)Thread) GeneratorIterableOptionalUnion)
model_info)HF_HUB_OFFLINE)Image)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES)is_fastapi_availableis_librosa_availableis_openai_availableis_pydantic_availableis_uvicorn_available )
AutoConfigLogitsProcessorListPreTrainedTokenizerFastProcessorMixinTextIteratorStreamer)ContinuousBatchingManager
RequestStatus)is_torch_availablelogging )BaseTransformersCLICommand)
AutoProcessorBitsAndBytesConfigGenerationConfigPreTrainedModel)FastAPI
HTTPException)CORSMiddleware) JSONResponseStreamingResponse)
Transcription) TranscriptionCreateParamsBase)ChatCompletionMessageParam)ChatCompletionChunkChoice ChoiceDeltaChoiceDeltaToolCallChoiceDeltaToolCallFunction) CompletionCreateParamsStreaming)ResponseResponseCompletedEvent ResponseContentPartAddedEvent ResponseContentPartDoneEventResponseCreatedEvent
ResponseErrorResponseErrorEventResponseFailedEventResponseInProgressEvent ResponseOutputItemAddedEventResponseOutputItemDoneEventResponseOutputMessageResponseOutputTextResponseTextDeltaEventResponseTextDoneEvent) ResponseCreateParamsStreaming) BaseModel TypeAdapterValidationErrorc $ \ rS rSr% Sr\\S' Srg))TransformersResponseCreateParamsStreamingr zo
OpenAI's ResponseCreateParamsStreaming with an additional field for the generation config (as a json string).
generation_config N__name__
__module__ __qualname____firstlineno____doc__str__annotations____static_attributes__rL U/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/commands/serving.pyrI rI r rV rI F)totalc $ \ rS rSr% Sr\\S' Srg)+TransformersCompletionCreateParamsStreamingy zq
OpenAI's CompletionCreateParamsStreaming with an additional field for the generation config (as a json string).
rK rL NrM rL rV rW r[ r[ y rX rV r[ c B \ rS rSr% Sr\\S' \\S' Sr\ \
\S' Sr g) %TransformersTranscriptionCreateParams zo
OpenAI's TranscriptionCreateParamsBase with an additional field for the generation config (as a json string).
filerK FstreamrL N) rN rO rP rQ rR bytesrT rS ra r boolrU rL rV rW r^ r^ s" !&&rV r^ >
textuserstorepromptinclude reasoning
background
truncation tool_choice service_tier top_logprobsmax_tool_callsprevious_response_id> nstopre audiorf logprobsmetadata functions
modalities
predictionrl rm rn
function_callstream_optionsresponse_formatpresence_penaltyreasoning_effortweb_search_optionsparallel_tool_callsmax_completion_tokens> rg rh languager{ chunking_strategytimestamp_granularitiesqwenz z )startendc $ \ rS rSrSrSrSrSrSrg)Modality LLMVLMSTTTTSrL N) rN rO rP rQ r r r r rU rL rV rW r r s
C
C
C
CrV r argsc [ U 5 $ )zr
Factory function used to instantiate serving server from provided command line arguments.
Returns: ServeCommand
) ServeCommand)r s rW serve_command_factoryr s rV reqmodel_generation_configr% returnc ~ U R S5 b# [ S 0 [ R " U S 5 D6nO[ R
" U5 nUR " S 0 UD6nUR 5 H u pVUc M
[ X5U5 M U R S5 b [ U S 5 Ul
U R S5 b [ U S 5 Ul
U R S5 b [ U S 5 Ul U R S5 b
U S Ul
U R S5 b
U S Ul U R S5 b, [ U S 5 Ul [ U S 5 S:X a S Ul U R S
5 b [ U S
5 Ul U R S 5 b [$ R& " U S 5 U$ )
ax
Creates a generation config from the parameters of the request. If a generation config is passed in the request,
it will be used as a baseline for parameterization. Otherwise, we will use the model's default generation config.
Other parameters in the request will be applied on top of the baseline.
Args:
req (`dict`):
The request which may optionally contain generation parameters.
model_generation_config (`GenerationConfig`):
The model's default generation config.
kwargs (`dict`):
Additional parameters to set in the generation config.
Returns:
The prepared `GenerationConfig` object.
rK max_output_tokens
max_tokensfrequency_penalty
logit_biasrr temperatureg Ftop_pseedrL )getr% jsonloadscopydeepcopyupdateitemssetattrintmax_new_tokensfloatrepetition_penalty
sequence_bias stop_stringsr do_sampler torch manual_seed)r r kwargsrK non_standard_kwargskvs rW !create_generation_config_from_reqr s . ww"#/ , Ttzz#>Q:R/S T MM*A B+22Reset the tool call state (assumes we're outside a tool call).Fr N)inside_tool_callhas_tool_name_definedarg_nesting_levelbufferr s rW r ToolState.reset s! % %*"!" rV )r r r r N)rN rO rP rQ rR r r rU rL rV rW r r s ArV r c R \ rS rSrSr SSSS\S\\S 4S jjrS
r S r
S r S
r g)
TimedModeli' z
A class that holds a PreTrainedModel instance and its associated processor.
Automatically deletes the instances after a specified timeout.
Nmodelr& timeout_seconds processor)r r c Xl [ UR 5 U l X0l X l [ R " U R
U R 5 U l U R R 5 g r ) r rS name_or_path
_name_or_pathr r threadingTimer
_delete_model_timerr )r r r r s rW r TimedModel.__init__- sU
!3!3 4" . ood&:&:DDelete the wrapped model and processor and clean up resources.r Nz was removed from memory after z seconds of inactivity)
hasattrr r gccollectr cuda is_available empty_cacheloggerinfor r r s rW r TimedModel._delete_model@ s 4 ! !djj&<
DJ !DN JJL zz&&((
&&( KK%%&&EdFZFZE[[qr
'= !rV c N [ U S5 ( + =( d U R SL $ )z)Check if the instances have been deleted.r N)r r r s rW
is_deletedTimedModel.is_deletedQ s 4))?TZZ4-??rV )r r r r r r )
rN rO rP rQ rR r r r r r r r rU rL rV rW r r ' sJ SW E"M NO "@rV r c \ rS rSr% Sr\" SSS0S9r\\S' \" SS/ S QS
.S9r \
\ \S ' \" S SS
0S9r \ \S' \" SSS0S9r
\
\ \S' \" S SS0S9r\ \S' \" S SS0S9r\ \S' \" SSSS/S
.S9r\\S' \" S SS0S9r\ \S' \" S SS 0S9r\\S ' \" S SS 0S9r\\S!' \" S"SS#0S9r\\S$' \" S%SS&0S9r\\S'' \" SSS(0S9r\
\ \S)' \" S SS*0S9r\ \S+' \" S SS,0S9r\ \S-' \" SSS.0S9r\
\ \S/' S0rg)1ServeArgumentsiV z
Arguments for the serve CLI.
See the metadata arg for each argument's description -- the metadata will be printed with
`transformers serve --help`
autohelpzfDevice to use for inference; will default to `auto` andplace the model on an accelerator if available.)defaultru devicezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.)r bfloat16float16float32)r choices torch_dtypeFz2Whether to trust remote code when loading a model.trust_remote_codeNzWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz$Interface the server will listen to.hosti@ z Port the server will listen to.porti, z@Time in seconds after which a model will be removed from memory.
model_timeoutr z8Logging level as a string. Example: 'info' or 'warning'. log_levelz1The default seed for torch, should be an integer. default_seedztWhether to enable CORS. Some apps that make requests from external domains (e.g. Cursor) require CORS to be enabled. enable_corsz+Whether to turn on strict input validation.input_validationzName of the model to be forced on all requests. This is useful for testing Apps that don't allow changing models in the request. force_modelrL ) rN rO rP rQ rR r r rS rT r r r rc r r r r r r r r r r r r r r rU rL rV rW r r V s
>
FC "'PA
"K# $)] ^ t */ r
*# efL$ efL$ %UFZhmotgu=v ww!&uHm?n!o$o kV=c4deD#edf6W-XYD#Y \]M3 &*d!eIs #(([ \#L(3- &
K # B
d "' 2
"K# rV r c r \ rS rSr\S\4S j5 rS\4S jrS\ SSS S
S \
4S jr S\ 4S
jr S\ 4S jr
S\ 4S jr S2S\\ S\\ S\\ S\\ S\\ S\\S S\4S jjrSSS\4S jrS r\R, " SS 9S\\ \\4 4S j5 rS \ S\\SS4 4S! jr\S\S\ 4S" j5 r \S#\ 4S$ j5 r S \ S\\SS4 4S% jr S \ S\\SS4 4S& jr S \ S\\SS4 4S' jr!S \ S\"4S( jr#\S\S\S) 4S* j5 r$S+\S\4S, jr%S-\4S. jr&S-\S\'\\(4 4S/ jr)S-\S\'\\*4 4S0 jr+S1r,g)3r i parserc X [ 4nU R SUS9nUR [ S9 g)z
Register this command to argparse so it's available for the transformer-cli
Args:
parser: Root parser to register command-specific arguments
serve)dataclass_types)funcN)r
add_parser set_defaultsr )r r serve_parsers rW register_subcommand ServeCommand.register_subcommand s3 *+ ((/(R !!'<!=rV r c [ ( d [ S5 eXl U R R S:H U l U R R
U l U R R b* [ R " U R R 5 [ R " S5 nUR [ R U R R R 5 5 [ R " S5 nUR [ R U R R R 5 5 0 U l S U l S U l S U l S U l g )NzaMissing dependencies for the serving CLI. Please install with `pip install transformers[serving]`
sdpa_paged transformersz+transformers.generation.continuous_batching) serve_dependencies_available ImportErrorr r use_continuous_batchingr r r r r
get_loggersetLevel
log_levelsr lower
loaded_models#running_continuous_batching_manager
last_messages
last_kv_cache
last_model)r r transformers_logger cb_loggers rW r ServeCommand.__init__ s ++ s
'+yy'D'D 'T$ 9900 99 ! ! - dii 4 4 5 & 0 0 @$$W%7%7 8K8K8Q8Q8S%TU&&'TU 7--dii.A.A.G.G.IJK 57X\0 " ! rV requestschema_TypedDictMeta validatorrF
unused_fieldsc : [ R SU 35 [ UR 5 5 nUR nXV-
nU( a% [ R SU 35 [
SSU 3S9eU R R ( aC UR U5 XT- n U ( a% [ R SU 35 [
SSU 3S9egg! [ aC n[ R SUR 5 35 [
SUR 5 S9eSnAff = f)a&
Validates the request against the schema, and checks for unexpected keys.
Args:
request (`dict`):
The request to validate.
schema (`_TypedDictMeta`):
The schema of the request to validate. It is a `TypedDict` definition.
validator (`TypeAdapter`):
The validator to use to validate the request. Built from `schema`.
unused_fields (`set`):
Fields accepted by `schema`, but not used in `transformers serve`.
Raises:
HTTPException: If the request is invalid or contains unexpected or unused fields.
zValidating request: z Unexpected keys in the request: i ) status_codedetailzValidation error: Nz Unused fields in the request: ) r debugsetkeys__mutable_keys__errorr( r r validate_pythonrG errors)
r r r r r
input_keys
possible_keysunexpected_keyseunused_fields_in_requests
rW _validate_request ServeCommand._validate_request s . +G956 (
//
$4 LL;O;LM N C:Z[jZk8lm m 99 % %
H))'2 (2'A $' =>V=W XY# #.LMeLf,g ( & #
H 1!((* >?#AHHJGG
Hs
C
D>DDc @ U R U[ [ [ S9 g N)r r r r )r. rI response_validatorUNUSED_RESPONSE_FIELDSr r s rW validate_response_request&ServeCommand.validate_response_request s! <(0
rV c @ U R U[ [ [ S9 g r1 )r. r[ completion_validator UNUSED_CHAT_COMPLETION_FIELDSr4 s rW validate_chat_completion_request-ServeCommand.validate_chat_completion_request s! >*7
rV c @ U R U[ [ [ S9 g r1 )r. r^ transcription_validatorUNUSED_TRANSCRIPTION_FIELDSr4 s rW validate_transcription_request+ServeCommand.validate_transcription_request s! 8-5
rV N
request_idcontentr role
finish_reason
tool_callsr2 r c
[ U[ [ R " 5 5 U[ [ UUUS9SUS9/SSS9nSUR SS 9 S
3$ ) aO
Builds a chunk of a streaming OpenAI Chat Completion response.
IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps,
like Cursor, assume that when the field exists, it has data.
Args:
request_id (`str`):
The request ID.
content (`str`, *optional*):
Content of the response from the model.
model (`str`, *optional*):
The model that generated the content.
role (`str`, *optional*):
The role of the next content, until a new role is defined.
finish_reason (`str`, *optional*):
The reason the generation by the model has finished.
tool_calls (`list[ChoiceDeltaToolCall]`, *optional*):
Data about the tool calls, when they are triggered.
Returns:
`str`: The built chunk, a string containing a JSON string with the payload.
)rB rC rE r )deltaindexrD r zchat.completion.chunk)idcreatedr r system_fingerprintobjectdata: T exclude_none
)r/ r timer0 r1 model_dump_json)r rA rB r rC rD rE chunks rW build_chat_completion_chunk(ServeCommand.build_chat_completion_chunk' sq @ $ $% ' !#-
"/
"*!
$ --4-@AFFrV responserE c ( SUR SS9 S3$ )a
Builds a event of a streaming OpenAI Response response.
IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps,
like Cursor, assume that when the field exists, it has data.
Args:
response (`BaseModel`):
The response to build an event from. One of the multiple OpenAI Response output types
Returns:
`str`: The built chunk, a string containing a JSON string with the payload.
rM TrN rP )rR )r rV s rW build_response_event!ServeCommand.build_response_event[ s" 00d0CDDIIrV c B ^ [ 5 nT R ( a UR [ S/SS/S/S9 UR S5 S[
4U 4S jj5 nUR S5 S[
4U 4S jj5 nS S
KJn UR S 5 SU4U 4S jj5 nUR S
5 UR S
5 U 4S j5 5 n[ R " UT R R T R R T R R S9 g )N*T)
allow_originsallow_credentials
allow_methods
allow_headersz/v1/chat/completionsr c > TR U S9 TR ( a TR U 5 nOTR U 5 n[ USS9$ Nr text/event-stream
media_type)r: r
#continuous_batching_chat_completiongenerate_chat_completionr+ r outputr s rW chat_completion)ServeCommand.run..chat_completiony sK 1 1' 1 B++ AA'J 66w?$V8KL LrV z
/v1/responsesc X > TR U S9 TR U 5 n[ USS9$ ra )r5 generate_responser+ rh s rW responses#ServeCommand.run..responses s2 * *7 * ;++G4F$V8KL LrV r )Requestz/v1/audio/transcriptionsc
># U R 5 IS h vN n[ US R 5 I S h vN US S9n[ R SUS R
SUS R SUS R S- S S 35 S S S 5 IS h vN TR WS
9 TR U5 n[ US S 9$ N N N2! , IS h vN ( d f NG= f7f)
Nr` r )r` r zReceived file: z
; MIME type: z; size: z.2fz KiBrb rc rd ) formr^ readr r" filename content_typesizer? generate_transcriptionr+ )r rs parsed_requestri r s rW audio_transcriptions.ServeCommand.run..audio_transcriptions s ||~~!F #F| 0 0 22 w-"
%d6l&;&;% [ ST R 5 S.5 $ )Nlist)rL data)r* get_gen_modelsr s rW get_all_models(ServeCommand.run..get_all_models s 64;N;N;P QR RrV )r r r )r' r add_middlewarer) postdictfastapirp optionsr uvicornrunr r r r )r apprj rn rp rz r s ` rW r ServeCommand.runk s i "e"& "e "e
( ) MT M
* M
/ " Mt M
# M $ , - M M
. M"
\ " S
# S Cdiinn499>>TYYM`M`arV )maxsizec / SQn[ ( aS U Vs/ sHF nUS[ R R 5 R 5 UR S5 S S.PMH sn$ U Vs/ sH n[ U5 PM nnU Vs/ sH6 nUR SUR R 5 UR S.PM8 sn$ s snf s snf s snf )a
This is by no means a limit to which models may be instantiated with `transformers serve`: any chat-based
model working with generate can work.
This is a limited list of models to ensure we have a discoverable /v1/models endpoint for third-party
integrations.
) zMenlo/Jan-nanozMenlo/Jan-nano-128kzQwen/Qwen2.5-0.5B-InstructzQwen/Qwen2.5-3B-InstructzQwen/Qwen2.5-7B-InstructzQwen/Qwen2.5-14B-Instructz meta-llama/Llama-3.1-8B-Instructz meta-llama/Llama-3.2-1B-Instructz!meta-llama/Llama-3.3-70B-Instructz HuggingFaceTB/SmolVLM-Instructz!ibm-granite/granite-vision-3.2-2bzQwen/Qwen2.5-VL-7B-Instructr /r )rI rL rJ owned_by) r datetimenow timestampsplitr
rI
created_atauthor)r modelsr model_infoss rW r ServeCommand.get_gen_models s
> $ $E % ' 0 0 4 4 6 @ @ B % C 0 3 $
;AA&:e,&KA ) )E (( % $ / / 9 9 ; % )
Bs A C 'C# T R R U TR S5 TR S9nSnT R USTS9v T R H nUR
U:w a M TR S5 b) U( d" UR [ R :X a MN SnUR [ R :X a SOS nUR [ R :X a T R XTS9v g T R XR TS 9v M g ! [ a9 n[ R [ U5 5 S
[ U5 S 3v S nAg S nAff = f7f) NrA )rA r F assistantrC r Trr rD r )rA rB r data: {"error": ""})r add_requestr r rT rA statusr FINISHED
next_token Exceptionr r& rS )
_inputsrA queue_is_flushedresultrD r, rK model_id_and_revisionr r s
rW stream_chat_completionPServeCommand.continuous_batching_chat_completion..stream_chat_completion sV !
7 ! E E Q Q (=N_NnNn R
$) 66z [p6qq " F FF ((J6 ww|,8AQ!==M,B,BB $/3 ,.4mm}?U?U.UF[_M }}
(>(>> " > > &K` ? " > >'1;L;LTi ? ! G(
7 SV$*3q6(#66
7s; E%C8D =E%> D E%
E")/E E% E""E%r )process_model_namer r rr load_model_and_processorr r r rK r r init_continuous_batchingr logit_processorr apply_chat_templatetor )
r r must_discard_cacher r r inputsr rK r s
`` @@rW rf 0ServeCommand.continuous_batching_chat_completion si !% 7 7G E 2doo E/ 77C88==DRS=T;?8 889NO+29k+J+JI''PY = $)$;$;"//"//
3 3 ;7<7U7U"3t 8V 8D 4 H[G\D 4 4 D 4 4 : : < ..s:tko.pss LL
" 7 " 7H &fQi00rV c U R R nU[ R " 5 ; a [ R
nU$ U[ R " 5 ; a [ R nU$ [ SU 35 e)NzUnknown modality: ) __class__rN r valuesr r r r
ValueError)r model_classnamemodalitys rW get_model_modality ServeCommand.get_model_modality, sm //22 H O O Q Q ||H A H H J
J ||H 1/1B CD DrV r c > / nU GH nUS / S.nU[ R :X a+ [ US [ 5 ( a US OUS S nXTS' GO7U[ R :X Ga" [ US [ 5 ( a US R SUS S.5 OUS H nUS S:X a US R U5 M" US S:X d M- SUS S ; a [ R " S
S US S 5 n[ R " [ [ R " U5 5 5 n[ R " S S
S9nUR n UR! UR 5 OUS S n US R SU S.5 M UR U5 GM U$ )NrC rC rB rB rd )typerd r image_urlbase64urlz^data:image/.+;base64,r z.pngF)suffixdeleteimage)r r )r r
isinstancerS r appendresubr openr r b64decodetempfileNamedTemporaryFilenamesave)
r r processor_inputsmessageparsed_messagerB
image_datar r` r s
rW *get_processor_inputs_from_inbound_messages7ServeCommand.get_processor_inputs_from_inbound_messages8 s G&-fo" EN8<<'0:79;Ms0S0S'),Y`ajYklrYs,3y)X\\) gi 0#66"9-44fgV_N`5ab#*9#5"6?f4 *9 5 < rV c ^ ^^^^^ T R R b T R R US' US nUS S S:X a gT R US 5 mTT R :g nTT l T R T5 u mnT R T5 nT R
X%5 nSm[ H2 nUTR R S R 5 ; d M0 Um O UR USUR S S5 S
SSS 9nUR TR 5 nUR S S
5 mSn STR R S R 5 ; a Sn [ UU SS9n
[! UTR" S9n Sn T R% U5 ( a U( d T R& n 0 UEU
U SU S.EmUUUUU U4S jn
U
" U
T5 $ )z
Generates an OpenAI Chat Completion using `generate`.
Args:
req (`dict`): The request to generate an OpenAI Chat Completion for.
Returns:
`Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks.
Nr r rC r r Ttoolsr )r r r return_dicttokenizerA req_0gptossFskip_special_tokens skip_promptr )streamerrK return_dict_in_generatepast_key_valuesc 3 ># SnS nST
R R S R 5 ; a SnSnU
U4S jn[ UT S9nSn UR 5 [ 5 nTR
TS TS
9v U GHF nST
R R S R 5 ; a% UR S 5 ( a US [ S 5 * nXh-
nU( a X6; a SnMh Mj TGb UR 5 [ T S :X a SUl M UR 5 [ T S
:X a& UR 5 TR
US STS9v M UR ( Ga@ U=R U-
sl
UR ( dV [ R " SUR 5 n U c GM8 U R# S5 n SUl [% [' U S9SSUS- S9n
OUS:X a GMp SUR ; a GM U=R( UR+ S5 -
sl U=R( UR+ S5 - sl UR( S: a& SR- UR/ S5 S S 5 S- n[% [' US9SSS9n
TR
US U
/TS 9v GM+ US:w d GM4 TR
XTS 9v GMI TR
US TS 9v UR- 5 UR- 5 g ! [0 a9 n [2 R5 [7 U 5 5 S [7 U 5 S!3v S n A NOS n A ff = f! UR- 5 f = f7f)"NFr r T<|channel|>final<|message|>c L > TR " S0 U D6nUR Tl g NrL generater r r generate_outputr r s rW generate_with_cachebServeCommand.generate_chat_completion..stream_chat_completion..generate_with_cache " "'..":6":%4%D%D"rV targetr r r r
<|return|>r r rE )rA rC rD r z\"name\": \"(.*?)\"r! )r function
_tool_call)r rH r rI z"arguments": {{}) arguments)r rH r )rA rC rE r )rB r rr r r r ) config
architecturesr r r r rT endswithlenstrip_TOOL_CALL_TOKENSr r r r r searchgroupr2 r3 r countjoinr r r r& rS )r _request_id
filter_cot
cot_trace_endr threadresults
tool_stater tool_nametoolr, generation_kwargsr r rA r tool_model_familys rW r EServeCommand.generate_chat_completion..stream_chat_completion sW J M5<<55a8>>@@ !
=
E #6?PQFGh
&[
66z [p6qq &F 5<<#=#=a#@#F#F#HH!??<88%+,@s->?P-QRY-ZZ:>J 7 $ "<<>->?P-QRW-XX & , , ."&"B"B+6%).:&; #C # %%666 & - - 7 - $.#C#C,.II6LjN_N_,` #,#4$,090BICG
@':-Hi-X*+)3'2\'A (" $*R<$, $4:;L;L#L$, !+ < < S@Q Q < * < < S@Q Q <#-#?#?!#C-/WWV\\#5Fs5K-Ls-RF':-HSY-Z*+)3(" #'"B"B+6Ttf\q #C # % | " > > '?T ? k 'p 66{RX`u6vv
7 SV$*3q6(#66
7
sC AM I K9 .:K9 (M9
L</L72L? 7L<<L? ?MM)r r r r r r r _MODELS_WITH_TOOL_SUPPORTr r r r r r r r r rK is_continuationr )r r r r r r r supported_model_familiesr r generation_streamerrK r r r r r rA r s ` @@@@@rW rg %ServeCommand.generate_chat_completion] s 99 , 9900CL9c[`[r[r s
$ $-? . .M
+!2'+ ,
y y v &&9:FFrV c ^ ^^
^ ^ ^
T R TS 5 m T T R :g nT T l T R T 5 u m n[ TS [ 5 ( a) ST; a STS S./O/ nUR STS S.5 O[ TS [ 5 ( a; ST; a/ TS S S S:w a STS S./TS QnO`TS nTS US S ' OOTS nOI[ TS [ 5 ( a& ST; a STS S./O/ nUR TS 5 O [ S
5 eUR US S S
9nUR T R 5 nTR SS5 m
S nST R R S R 5 ; a Sn[! UUS S9n[# TT R$ S9nSnT R' T5 ( a U( d T R( nU[* R, " U5 UUS US.m
U
U U UU
U 4S jn U " UT
5 $ )z
Generates an OpenAI Response using `generate`.
Args:
req (`dict`): The request to generate an OpenAI Response for.
Returns:
`Generator[str, None, None]`: A generator that yields the OpenAI Response events.
r input instructionssystemr re r rC rB z%inputs should be a list, dict, or strTr )r r rp r r Fr r N)r attention_maskr rK r r c 3 ># SnS nSTR R S R 5 ; a SnSnUU 4S jn[ UTS9nSnSnSn UR 5 [
R
" 5 n [
SU[ S T 3U S
TTR S 5 S S
S00S/ / TR SS5 STR S5 S9 S9n
US-
nT R U
5 v [ SU[ S T 3U STTR S 5 S S
S00S/ / TR SS5 STR S5 S9 S9n US-
nT R U 5 v [ SUU[ ST 3SSS/ S 9S 9n US-
nT R U 5 v [ S ST 3UUU[ S S / S!9S"9n
US-
nT R U
5 v S nU H nSTR R S R 5 ; a% UR S#5 ( a US [! S#5 * nX-
nU( a
X>; a SnS nMi Mk [# S$ST 3UUUUS S%S&./S'9nUS-
nT R U5 v M [% S(ST 3UUSUS S%S&./S)9nUS-
nT R U5 v [' S*ST 3UUU[ S UR( / S!9S"9nUS-
nUS-
nT R U5 v [+ S+UU[ ST 3SS,SUR, // S-9S 9nUS-
nUS-
nT R U5 v [/ S.U[ S T 3U S,TTR S 5 S S
S00UR0 /S/ TR SS5 STR S5 S/9 S9nUS-
nT R U5 v UR3 5 UR3 5 g ! [4 a n[6 R9 S0[; U5 35 [= S1U[; U5 S29nUS-
nT R U5 v [? S3U[ S T 3W S4TTR S 5 S S
S00/ S/ SSTR S5 [A S5[; U5 S69S79
S9nUS-
nT R U5 v S nANS nAff = f! UR3 5 f = f7f)8NFr r Tr c L > TR " S0 U D6nUR Tl g r r r s rW r TServeCommand.generate_response..stream_response..generate_with_cachen r rV r zresponse.createdresp_queuedr# formatr rd rV r r ru ) rI r r r r# rd rL r ri r rl ru )r sequence_numberrV r! zresponse.in_progress in_progresszresponse.output_item.addedmsg_r r )rI r r rC rB )r r, output_indexitemzresponse.content_part.added output_textr )r rd annotations)r item_idr, r/
content_indexpartr zresponse.output_text.deltagX@)tokenlogprob)r r3 r, r/ r4 rG rt zresponse.output_text.done)r r3 r, r/ r4 rd rt zresponse.content_part.donezresponse.output_item.done completed)rI r r rC rB r2 zresponse.completed) rI r r r r# rd ri rL r r rl ru z"Exception in response generation: r& )r r, r zresponse.failedfailed server_error)coder )
rI r r r r# rd ri rL r r rl ru r& )!r r r r r rQ r9 r5 r rX r= r> r@ r7 rA r r
rB rC r8 rd r? r5 r6 r0 r r r r& rS r; r< r: ) r r r r r r r, r/ r4 r response_createdresponse_in_progressresponse_output_item_addedresponse_content_part_addedr r response_output_text_deltaresponse_output_text_doneresponse_content_part_doneresponse_output_item_doneresponse_completedr, error_eventresponse_failedr r r r rA r s rW stream_response7ServeCommand.generate_response..stream_responsed s J M5<<55a8>>@@ !
=
E #6?PQF O L MN
!YY[
$8+$3 % ":,/#- ' 3%(WW^%< &(8 9 ) !,/GG4I5,Q$*!$!4
$ $ 1$//0@AA'>/$3 % ":,/#- , 3%(WW^%< &(8 9 ) !,/GG4I5,Q$*!$!4
($$ 1$//0DEE .J5$3!-. !* .Y}[fpr .* 1$//0JKK /L6 ":, /$3!-"/+ RUWX
/+ 1$//0KLL &F 5<<#=#=a#@#F#F#HH!??<88%+,@s > ? ) ,/GG4I5,Q$*!$!4
&"$ 1$//0BCC
J
I !
A A#a& JK 0 $3 F
1$// <<"5*$3 % ":,/#- ' 3%(WW^%< &(8 9 ! ) ,1$*!$!4 +!/$'F #, 1$//@@C!
AH
s> A Q:L5N Q:
Q" B9Q Q% Q""Q% %Q77Q:)r r r r rS r r} r r r r r r r r r r r rK r r r ones_like)r r r r r r r rK r rG r r r rA s `` @@@@rW rm ServeCommand.generate_response% s= !% 7 7G E 2doo E/ 889NOy c'lC ( (M[_bMbxC4GHIhjF MM6c'lC D
G d
+
+ $w<?6*h6'/CF1Ii( W
G d
+
+M[_bMbxC4GHIhjF MM#g, ' DE E..vTbf.g5<<(WW 3W=
# u||11!4::< <"' 2 3
>c[`[r[r s
$ $-? . .M #oof 5+!2'+ ,
a a F 2J??rV c ^
^ ^ ^
[ 5 ( d [ S5 eU R US 5 nU R U5 u m m [ T R
SSS9n[
UT R S9nT R R n[ R " US 5 n[ R " XeSS9u pxT " XuSS 9R T R 5 m
T
S
R T R 5 T
S
' UUSS .m
U
U U U
4S jn U " 5 $ )
z
Generates an OpenAI Transcription using the audio file.
Args:
req (`dict`): The request containing the audio file and model information.
Returns:
`Generator[str, None, None]`: A generator that yields the transcription result.
z]Missing librosa dependency for audio transcription. Please install with `pip install librosa`r Tr r r` )srmonor )
sampling_rater input_features)r rK r c 3 ># TR " S0 TDTD6n TR U R SS9S n[ US9nUR SS9 v g 7f)NT)r r )rd rN rL )r batch_decode sequencesr, rR )
generated_idstranscription_text
transcription audio_inputs audio_modelaudio_processorr s rW _generate_transcriptionDServeCommand.generate_transcription.._generate_transcriptionq sd ' 0 0 U< UCT UM!0!=!=m>U>Uko!=!pqr!s )/A BM"222EF Gs AA)r r r load_audio_model_and_processorr r r rK feature_extractorrN ior librosaloadr r dtype)r r r r rK model_sampling_rate audio_bytes audio_array_rY rV rW rX r s @@@@rW rx #ServeCommand.generate_transcriptionI s# $%% o
!% 7 7G E '+'J'JK`'a$ _ 2 % %4T
> )F)F
. ? ? M MjjV- kPTU &{fjknn
*66F)G)J)J;K\K\)] %& ,!2'+
H H '((rV c N UR S5 =( d UR S5 nSnU R c SnOc[ U R 5 [ U5 : a SnO>[ [ U R 5 5 H nU R U X$ :w d M Sn O X l U$ )a
Determines whether the current request is a continuation of the last request. In other words, if it is the
same chat session.
Args:
req (`dict`): The request to check.
Returns:
`True` if the request is a continuation of the last request, `False` otherwise.
r r" TF)r r r
range)r r r req_continues_last_messagesis rW r ServeCommand.is_continuationy s 77:&:#'''*:&*# %*/ '
##
$H
5*/ ' 3t 1 123%%a(HK727/ 4
&**rV r$ c U R ( a7 [ SU R U R U R U R S9nU$ U R
( a [ SS9nU$ SnU$ )z
Returns the quantization config for the given CLI arguments.
Args:
args (`ServeArguments`): The serve arguments. May contain quantization settings, device, etc.
Returns:
`Optional[BitsAndBytesConfig]`: The quantization config.
T)r bnb_4bit_compute_dtyper bnb_4bit_use_double_quantbnb_4bit_quant_storage)r N)r r$ r r r r )r quantization_configs rW get_quantization_config$ServeCommand.get_quantization_config sv "4 !'+'7'7$($<$<*.*C*C'+'7'7
# #"
"4 !# #" #' ""rV model_idc v U R R b U R R nSU; a U$ U S3$ )a
Applies the `force_model` CLI argument and canonicalizes the model name to the format "model_id@revision".
If the model_id DOESN'T contain an @, it defaults to "model_id@main".
Args:
model_id (`str`): The model ID.
Returns:
`str`: The canonicalized model name to be used
@z@main)r r )r rr s rW r ServeCommand.process_model_name s< 99 ,yy,,H (?O5!!rV r c U R n[ R SU 35 SU; a UR SS5 u p4OUSpC[ R
" UUUR S9nUR S; a UR O[ [ UR 5 nU R U5 nUUR USUR S.nUb XxS
' [ R
" U40 UD6n [ [ U R S 5 n
U
R
" U40 UD6n [ U S S 5 c U R UR 5 n U R" R$ S L =( a U R" R& S
:H n U R" R$ S L=( a U R" R$ S: n
U ( d U
( a SU R" l [ R SU 35 X4$ )a
Generic method to load a model and a data processor from a model ID and revision, making use of the serve CLI
arguments.
Args:
model_id_and_revision (`str`):
The model ID and revision to load.
model_cls (`type[PreTrainedModel]`):
The model class to load.
Returns:
`tuple[PreTrainedModel, Union[ProcessorMixin, PreTrainedTokenizerFast]]`: The loaded model and
data processor (tokenizer, audio processor, etc.).
zLoading rt r! main)revisionr )r Nr )rx r r
device_mapr Nro r
hf_device_map rr z
Loaded model )r r r r r# from_pretrainedr r getattrr rp r r r
r r r rK r
max_length)r r r rr rx data_processorr ro model_kwargsr architecturer has_default_max_lengthhas_short_max_new_tokenss rW _load_model_and_data_processor+ServeCommand._load_model_and_data_processor s yy h 4 567 ' '!6! p5CZCZCiCilpCp ! "%=59E # # 2 m$9#:;<$$rV c XR ; d" U R U R 5 ( aB U R U5 u p#[ UU R R
US9U R U' X#4$ U R U R
5 U R U R nU R U R nX#4$ )a$
Loads the text model and processor from the given model ID and revision into the ServeCommand instance.
Args:
model_id_and_revision (`str`):
The model ID and revision to load.
Returns:
`tuple[PreTrainedModel, PreTrainedTokenizerFast]`: The loaded text model and processor.
r r r r r r r r r r r )r r r r s rW r %ServeCommand.load_model_and_processor s !(:(: :d>P>PQf>g>r>r>t>t # B BCX Y E8B $ 7 7#9D 4 5
4 5 A A C&&'<=CCE **+@AKKI rV c XR ; d" U R U R 5 ( aB U R U5 u p#[ UU R R
US9U R U' X#4$ U R U R
5 U R U R nU R U R nX#4$ )a
Loads the audio model and processor from the given model ID and revision into the ServeCommand instance.
Args:
model_id_and_revision (`str`):
The model ID and revision to load.
Returns:
`tuple[PreTrainedModel, ProcessorMixin]`: The loaded audio model and processor.
r r )r r rW rX s rW r[ +ServeCommand.load_audio_model_and_processor s !(:(: :d>P>PQf>g>r>r>t>t+/+N+NOd+e (K8B $ 7 7)9D 4 5 ++
4 5 A A C ,,-BCIIK " 0 01F G Q QO++rV )r r r r r r r r
)r NNNNN)-rN rO rP rQ staticmethodr r r r r r# r. r5 r: r? r rS r} rT rX r functools lru_cacheanyr r rf r& r r r rg rm rx rc r rp r r tupler r r r[ rU rL rV rW r r s >N > > ^ @// !/ ! /
/b
d
%'!% # "'+<@2G SM2G # 2G } 2G
sm 2G }
2G T"7 892G
2GhJ[ JS J 8bt &+T#s(^ 4 + '+ZW1t W1 #tUY/@Z W1r / h " x " " HFGD FGYsD$5O FGPb@T b@iT4.H b@H .)$ .)9S$_3M .)` +4 +D +< #n #BV9W # #8"3 "3 "";%C ;%z c eO]tLtFu 2,C ,ERacqRqLr ,rV r __main__)r r r enumr r r] r r r r rQ argparser r dataclassesr r r r typingr r
r r huggingface_hubr
huggingface_hub.constantsr PILr r
&transformers.models.auto.modeling_autor r transformers.utils.import_utilsr r r r r r r r r r r generation.continuous_batchingr r utilsr r r" r r# r$ r% r& r^ r r r r' r( fastapi.middleware.corsr) fastapi.responsesr* r+ openai.types.audio.transcriptionr, .openai.types.audio.transcription_create_paramsr- openai.types.chatr. 'openai.types.chat.chat_completion_chunkr/ r0 r1 r2 r3 *openai.types.chat.completion_create_paramsr4 openai.types.responsesr5 r6 r7 r8 r9 r: r; r< r= r> r? r@ rA rB rC -openai.types.responses.response_create_paramsrD pydanticrE rF rG rI r[ r^ r2 r8 r= r3 r9 r> r rN r r r} r$ r Enumr r r r r r r r r r rL rV rW r sY . ( 7 7 & 4 V / ( k 4 6k;O;QkViVk .6A>\< [ " \@@ 4QY^ 6U]b '0MUZ ' %%NO&'RS )*O P %!.#
H %
!!2!7!7!9 : tyy 8
8 /8 8 v ,@ ,@^ W W WtA,- A,H$ z NE IIK rV