tj@ddlmZddlZddlZddlZddlmZddlmZm Z m Z erddl m Z GddZ Gd d e ZGd d eZGd deZGddeZdS)) annotationsN)Queue) TYPE_CHECKINGAnycast)PreTrainedTokenizerBaseceZdZdZdZdZdS) BaseStreamerzG Base class from which `.generate()` streamers should inherit. ct)z;Function that is called by `.generate()` to push new tokensNotImplementedErrorselfvalues _/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/generation/streamers.pyputzBaseStreamer.put!!###ct)zHFunction that is called by `.generate()` to signal the end of generationr rs rendzBaseStreamer.end%rrN)__name__ __module__ __qualname____doc__rrrrr r s<$$$$$$$$rr c8eZdZdZddd Zd Zd ZdddZdZdS) TextStreamera Simple text streamer that prints the token(s) to stdout as soon as entire words are formed. Parameters: tokenizer (`AutoTokenizer`): The tokenizer used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer >>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt") >>> streamer = TextStreamer(tok) >>> # Despite returning the usual output, the streamer will also print the generated text to stdout. >>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20) An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, ``` F tokenizerr skip_promptbool decode_kwargsrc Z||_||_||_g|_d|_d|_dS)NrT)r r!r# token_cache print_lennext_tokens_are_prompt)rr r!r#s r__init__zTextStreamer.__init__Fs8"&*')&*###rct|jdkr |jddkrtdt|jdkr|d}|jr|jr d|_dS|j|tt|j j |jfi|j }| dr||jd}g|_d|_nt|dkrU|t!|dr-||jd}|xjt|z c_nB||j|ddz}|xjt|z c_||dS) zm Receives tokens, decodes them, and prints them to stdout as soon as they form entire words. rz'TextStreamer only supports batch size 1FN  )lenshape ValueErrorr!r'r%extendtolistrstrr decoder#endswithr&_is_chinese_charordrfindon_finalized_text)rrtextprintable_texts rrzTextStreamer.putPs u{  a  EKNQ$6$6FGG G    ! !!HE    ; */D ' F  ///C..t/?VV4CUVVWW ==   2!$."2"23N!D DNN YY]]t44Sb]]CC]!$."2"23N NNc.11 1NNN"$.4::c??Q3F"FGN NNc.11 1NN ~.....rct|jdkrNtt|jj|jfi|j}||jd}g|_d|_nd}d|_| |ddS)z;Flushes any remaining cache and prints a newline to stdout.rNT) stream_end) r.r%rr3r r4r#r&r'r9)rr:r;s rrzTextStreamer.endrs t 1 $ $2T^243CZZtGYZZ[[D!$."2"23N!D DNNN&*# ~$?????rr:r3r>c2t|d|sdnddS)zNPrints the new text to stdout. If the stream is ending, also prints a newline.Tr=N)flushr)printrr:r>s rr9zTextStreamer.on_finalized_texts& d$j$BBBdCCCCCCrc|dkr|dksT|dkr|dksH|dkr|dks<|dkr|dks0|d kr|d ks$|d kr|d ks|d kr|dks |dkr|dkrdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)rcps rr6zTextStreamer._is_chinese_chars6\\bFllf vg "--g "--g "--g "--f vg "--4urNF)r r r!r"r#rr:r3r>r") rrrrr(rrr9r6rrrrr*s6+++++ / / /D @ @ @DDDDDrrc@eZdZdZ ddfd ZdddZdZdZxZS)TextIteratorStreamerac Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is useful for applications that benefit from accessing the generated text in a non-blocking way (e.g. in an interactive Gradio demo). Parameters: tokenizer (`AutoTokenizer`): The tokenizer used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer >>> from threading import Thread >>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt") >>> streamer = TextIteratorStreamer(tok) >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20) >>> thread = Thread(target=model.generate, kwargs=generation_kwargs) >>> thread.start() >>> generated_text = "" >>> for new_text in streamer: ... generated_text += new_text >>> generated_text 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` FNr r r!r"timeout float | Noner#rc tj||fi|t|_d|_||_dSN)superr(r text_queue stop_signalrI)rr r!rIr# __class__s rr(zTextIteratorStreamer.__init__sD KAA=AAA'' rr:r3r>c|j||j|r(|j|j|jdSdS)\Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.rIN)rNrrIrOrBs rr9z&TextIteratorStreamer.on_finalized_textsZ D$,777  H O   0$,  G G G G G H Hrc|SrLrrs r__iter__zTextIteratorStreamer.__iter__ rcx|j|j}||jkrt |SNrS)rNgetrIrO StopIterationrs r__next__zTextIteratorStreamer.__next__s9##DL#99 D$ $ $// !LrFNr r r!r"rIrJr#rrErF) rrrrr(r9rUr[ __classcell__rPs@rrHrHs%%T" $       HHHHH rrHc@eZdZdZ ddfd ZdddZdZdZxZS)AsyncTextIteratorStreamera Streamer that stores print-ready text in a queue, to be used by a downstream application as an async iterator. This is useful for applications that benefit from accessing the generated text asynchronously (e.g. in an interactive Gradio demo). Parameters: tokenizer (`AutoTokenizer`): The tokenizer used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Raises: TimeoutError: If token generation time exceeds timeout value. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, AsyncTextIteratorStreamer >>> from threading import Thread >>> import asyncio >>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt") >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> async def main(): ... # Important: AsyncTextIteratorStreamer must be initialized inside a coroutine! ... streamer = AsyncTextIteratorStreamer(tok) ... generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20) ... thread = Thread(target=model.generate, kwargs=generation_kwargs) ... thread.start() ... generated_text = "" ... async for new_text in streamer: ... generated_text += new_text >>> print(generated_text) >>> asyncio.run(main()) An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, ``` FNr r r!r"rIrJr#rc Ntj||fi|tj|_d|_||_tj|_ttdd}tj dkot||_ |j r|nd|_dS)NrI) )rMr(asynciorrNrOrIget_running_looploopgetattrsys version_infocallablehas_asyncio_timeoutasyncio_timeout)rr r!rIr#timeout_contextrPs rr(z"AsyncTextIteratorStreamer.__init__s KAA=AAA!-// ,.. !'9d;;#w#>#\8OC\C\ 262JTPTrr:r3r>c|j|jj||r,|j|jj|jdSdS)rRN)rgcall_soon_threadsaferN put_nowaitrOrBs rr9z+AsyncTextIteratorStreamer.on_finalized_text!sZ &&t'A4HHH  Y I * *4?+EtGW X X X X X Y Yrc|SrLrrs r __aiter__z#AsyncTextIteratorStreamer.__aiter__'rVrcK |jrk|jd||j4d{V|jd{V}dddd{Vn#1d{VswxYwYn8t j|j|jd{V}||jkrt|S#t j $rtwxYwrX) rlrmrIrNrYrewait_forrOStopAsyncIteration TimeoutErrorrs r __anext__z#AsyncTextIteratorStreamer.__anext__*s ' \D,@,L// ==88888888"&/"5"5"7"7777777E888888888888888888888888888&.t/B/B/D/Ddl[[[[[[[[[((((*** # ! ! !..  !s..C A$C $ A..C 1A.2 If you're running on an environment like tmux, the draft text may fail to overwrite itself. Parameters: tokenizer (`AutoTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. sleep_time (`float`, *optional*): Time to sleep between diffusion drafts, which may be helpful to visualize intermediate outputs. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor, TextDiffusionStreamer >>> model = DiffusionGemmaForBlockDiffusion.from_pretrained( ... "google/diffusiongemma-26B-A4B-it", device_map="auto", ... ) >>> processor = AutoProcessor.from_pretrained("google/diffusiongemma-26B-A4B-it") >>> chat = [{"role": "user", "content": "Why is the sky blue?"},] >>> input_ids = processor.apply_chat_template( ... chat, tokenize=True, return_tensors="pt", add_generation_prompt=True ... ) >>> streamer = TextDiffusionStreamer(tokenizer=processor.tokenizer) >>> model.generate(input_ids.to(model.device), max_new_tokens=512, streamer=streamer) ``` FNr r r!r" sleep_timerJr#rc htj||fi|d|_d|_||_dS)NF)rMr( _has_draft _takes_logitsr{)rr r!r{r#rPs rr(zTextDiffusionStreamer.__init__csB KAA=AAA#$rcJ|jrtdddd|_dSdS)Nz8r=Trr@F)r}rArs r _clear_draftz"TextDiffusionStreamer._clear_draftrs6 ? $ -Rt 4 4 4 4#DOOO $ $rc |t|jdkr |jddkrtdt|jdkr|d}|jj|fi|j}tdddtd|d ddd|_|j tj |j d Sd S) z Receives the full sequence of draft tokens, decodes them, and prints them in yellow. Overwrites previous draft. r*rz0TextDiffusionStreamer only supports batch size 1z7r=TrzzN) rr.r/r0r r4r#rAr}r{timesleep)rrkwargsr:s r put_draftzTextDiffusionStreamer.put_draftxs  u{  a  EKNQ$6$6OPP P    ! !!HE$t~$UAAd.@AA g2T**** &&&&Bd;;;; ? & Jt ' ' ' ' ' ' &rcr|t|dS)zEReceives confirmed tokens, clears draft, and prints them permanently.N)rrMr)rrrPs rrzTextDiffusionStreamer.puts1   Ercp|tdS)z1Flushes any remaining cache and prints a newline.N)rrMr)rrPs rrzTextDiffusionStreamer.ends*   rr\)r r r!r"r{rJr#r) rrrrr(rrrrr^r_s@rrzrz:s&&V"#' % % % % % % %$$$ (((, rrz) __future__rrerirqueuertypingrrrtokenization_utils_baser r rrHrarzrrrrs#""""" ++++++++++BAAAAAA $ $ $ $ $ $ $ $ppppp<pppfBBBBB<BBBJUUUUU UUUp\\\\\L\\\\\r