tjQ ddlZddlZddlmZmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZmZGddZGd d ejjZd eeeezeeezffd ZGd dejjZGddejjZ d$de dejdzdejdzdedzdedzf dZGddejjZGddejjZ GddejjZ! d%de dejdzdejdzfdZ"dZ#d efd!Z$d"ej%j&j'fd#Z(dS)&N) DynamicCache DynamicLayerDynamicSlidingWindowLayerEncoderDecoderCache StaticCache StaticLayerStaticSlidingWindowLayer)GenerationConfig)PreTrainedModel)is_torch_greater_or_equal"is_torch_greater_or_equal_than_2_6cLeZdZdZddedefdZdZdZd Zd Z d Z ddZ d S)TorchExportableModuleForVLMa| A wrapper class for exporting Vision-Language Models (VLMs) like SmolVLM2 for ExecuTorch. This class handles the export of three main components: 1. Vision encoder (processes images to visual features) 2. Connector/projector (maps visual features to text embedding space) 3. Text decoder (generates text from combined visual and text tokens) max_batch_size max_cache_lenc||_||_||_|j|_|jj|_|jj|_|jj|_d|_ d|_ d|_ dS)a  Initialize the exportable VLM module. Args: model: The VLM (e.g. SmolVLM) model instance max_batch_size: Maximum batch size. Always 1 for ExecuTorch max_cache_len: Maximum cache length for text generation N) modelrrconfig vision_modelvision_encoder connector text_model text_decoderexported_vision_encoderexported_connectorexported_text_decoder)selfrrrs b/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/integrations/executorch.py__init__z$TorchExportableModuleForVLM.__init__,sl ,*l $k6.!K2(,$"&%)"""cB|jtjddddtj}dtjjjtjjjdi}tj|j|f|d|_|jS) z$Export the vision encoder component.ridtype pixel_values)rr%Fargsdynamic_shapesstrict) revaltorchrandnfloat32exportDimAUTOr)r r(r+s r!export_vision_encoderz1TorchExportableModuleForVLM.export_vision_encoderDs   """{1acGGG  <#(<#( (-|':':  ) (;( ( $++r#c|j|jjj}|jjj}|jjj}||z}||z}tjd||tj }ddtj j j ii}tj |j|f|d|_ |j S)zExport the connector component.rr&image_hidden_statesFr))rr-r vision_config hidden_size image_size patch_sizer.r/r0r1r2r3r)r vision_hidden_sizer9r:patches_per_dim num_patchesr6r+s r!export_connectorz,TorchExportableModuleForVLM.export_connector\s "[6B[.9 [.9 $ 2%7 #k![:LTYTabbb0!U\5E5J1KL#(,"5"5 N%') #6# # &&r#ct|j|_d}tjd|ftj}tj|tj}t|j|j j j }tj d|dz }d|id|id}|j |||d |_|jS) z"Export the text decoder component.)rr%rr&seq_length_dimmaxr input_idscache_positionF)rDrEr+r,)%TorchExportableModuleForDecoderOnlyLMrexportable_text_decoderr.zeroslongarangeminrr text_configmax_position_embeddingsr1r2r)r seq_lengthrDrEmax_seq_length seq_len_dimr+s r!export_text_decoderz/TorchExportableModuleForVLM.export_text_decoderus(MSWSd'e'e'e$ KJuzBBB j CCCT/1H1`aal&&'7^a=O&PP [) +.   &*%A%H%H)) &I& & "))r#c z|jdi||jdi||jdi||j|j|jdS)z'Export all components of the VLM model.)rrr)r4r>rQrrr)r kwargss r!r1z"TorchExportableModuleForVLM.exportsn"",,V,,,'''''  **6***":0 6   r#cdS)a Simplified forward pass for inference with guaranteed non-null input_ids and cache_position. Args: pixel_values: Input images [1, channels, height, width] (optional) input_ids: Text token IDs [1, seq_len] (required - won't be None) cache_position: Cache positions [seq_len] (required - won't be None) Returns: Output with logits for text generation NrS)r r(rDrEs r!forwardz#TorchExportableModuleForVLM.forwardr#N2F?c dS)a Simplified generate method with guaranteed non-null input_ids. Args: pixel_values: Input images [1, channels, height, width] (optional) input_ids: Initial text tokens [1, seq_len] (required - won't be None) max_new_tokens: Maximum number of tokens to generate do_sample: Whether to use sampling or greedy decoding temperature: Temperature for sampling Returns: Generated sequences NrS)r r(rDmax_new_tokens do_sample temperaturerTs r!generatez$TorchExportableModuleForVLM.generaterWr#)rr)NNrXFrY) __name__ __module__ __qualname____doc__intr"r4r>rQr1rVr^rSr#r!rr"s**c*c****0,,,0'''2***6        be      r#rceZdZdZ ddededzdedzdejdzddf fd Z dd ej dzd ej dzd ej dzdej fd Z dd ej dzd ej dzd ej dzd e dzde dzdej jf dZ e ddej jdedede dededededefdZxZS) rFa A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM with cache. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. Nr batch_sizerdevicereturnct|j}t |dr |jdurt dt |dr*t|ddt|||||_ dStj dt|||||_ dS)z Initializes the exportable module. Args: model (`PreTrainedModel`): The pretrained model to wrap. Raises: ValueError: If the model is configured with a unsupported cache implementation. use_cacheFz5The model must have caching enabled to be performant. layer_typessliding_windowNzmUsing `StaticCache` for export as `layer_types` is not specified or `sliding_window` is `null` in the config.) superr"rget_text_confighasattrri ValueErrorgetattr$TorchExportableModuleWithHybridCacherlogginginfo$TorchExportableModuleWithStaticCache)r rrerrfr __class__s r!r"z.TorchExportableModuleForDecoderOnlyLM.__init__s --//v{++ Vv/?5/H/HTUU U 6= ) ) hgf>NPT.U.U.a=eZQ^`fggDJJJ L   >eZQ^`fggDJJJr#rD inputs_embedsrEc:|j||S)a Forward pass of the module, which is compatible with the ExecuTorch llm runner. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. inputs_embeds (`torch.Tensor`): Tensor representing current input embeddings to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. )rDrv)rrV)r rDrvrEs r!rVz-TorchExportableModuleForDecoderOnlyLM.forwards"z!!I]!SSSr#r+r,c`|du|duz stdt|jdr-t|j|jj|j}|j}n=t|jdr|jjj}nd}t jd|4|||n+tj |j dtj |d }n3|||n+tj |j d tj |d }tj |jd ||||nd } | S)ar Export the wrapped module using `torch.export`. Args: input_ids (`Optional[torch.Tensor]`): Tensor representing current input token id to the module. Must specify either this or inputs_embeds. inputs_embeds (`Optional[torch.Tensor]`): Tensor representing current input embeddings to the module. Must specify either this or input_ids. cache_position (`Optional[torch.Tensor]`): Tensor representing current input position in the cache. If not provided, a default tensor will be used. dynamic_shapes (`Optional[dict]`): Dynamic shapes to use for export if specified. strict(`Optional[bool]`): Flag to instruct `torch.export` to use `dynamo`. Returns: torch.export.ExportedProgram: The exported program that can be used for inference. Examples: Export with input_ids: ```python # Prepare inputs input_ids = torch.tensor([[1, 2, 3]], dtype=torch.long, device=model.device) cache_position = torch.arange(input_ids.shape[-1], dtype=torch.long, device=model.device) # Export exported = exportable_module.export( input_ids=input_ids, cache_position=cache_position ) ``` Export with inputs_embeds: ```python # Prepare embeddings inputs_embeds = torch.randn(1, 3, 768, device=model.device) # batch_size=1, seq_len=3, hidden_size=768 cache_position = torch.arange(inputs_embeds.shape[1], dtype=torch.long, device=model.device) # Export exported = exportable_module.export( inputs_embeds=inputs_embeds, cache_position=cache_position ) ``` Nz2Need to specify either input_ids or inputs_embeds.base_model_prefixrcpuzfTorchExportableModuleForDecoderOnlyLM.export Can't infer device from the model. Set to CPU by default.r'rfrCr)rvrErSTr*rTr+r,) rornrrpryrfrrwarningr.rJshaperIr1) r rDrvrEr+r,base model_device input_kwargsexported_programs r!r1z,TorchExportableModuleForDecoderOnlyLM.exportsejT!mt&;< SQRR R 4:2 3 3 4:tz'CTZPPD;LL TZ ) ) :+2LL L Ox     &!-#1.\)/""5UZP\]]] LL"/!-#1.\-"5a"8 S_``` L!<.. J)#/66T /   r#FrYrXrzrpromptr[r\r]top_ktop_pc z|} ||dj|} | } d} t | jdD]G} | dd| | dzf}t j| gt j|}| ||}| dz } Ht |D]}| ddddf}t j| gt j|}| ||}|r|dkr||z }n|}|dkr7|t j ||dd k}td ||<|d krt j |d \}}t j t j |dd}||k}|dddf|dddf<d|d<|d||}td ||<t j |d}t j|d}n|dd }|dkr|d}t j| |gd} | dz } ||jkrn|| dd S)a Generate a sequence of tokens using an exported program. Args: exported_program (`torch.export.ExportedProgram`): The exported model being used for generate. tokenizer: The tokenizer to use. prompt (str): The input prompt. max_new_tokens (int): Maximum number of new tokens to generate. do_sample (bool): Whether to use sampling or greedy decoding. temperature (float): The temperature for sampling. top_k (int): The number of highest probability tokens to keep for top-k sampling. top_p (float): The cumulative probability for nucleus sampling. device (str): The device to use. Returns: str: The generated text. pt)return_tensorsrrNr|rCr{).r{Nz-infrYT) descendingdim.).r) num_samples)rkeepdimr)skip_special_tokens)modulerDtoclonerangerr.tensorrItopkfloatsortcumsumsoftmaxscatter multinomialargmaxrsqueezecatitem eos_token_iddecode)r tokenizerrr[r\r]rrrfexported_modulerD generated_ids curr_positionicurr_input_idscurr_cache_position_outputslogitsindices_to_remove sorted_logitssorted_indicescumulative_probssorted_indices_to_removeprobs next_token_ids r!r^z.TorchExportableModuleForDecoderOnlyLM.generateQs,<+1133IfT:::DGGOO "))  yq)**  A&qqq!a!e)|4N"', ejY_"`"`"`  .I\]]]A Q MM~&&5 5 A*111bcc62N"', ejY_"`"`"` &oObcccG E??${2FF$F199(.FE1J1J11Mm1\(\%05f F,-3;;49JvRV4W4W4W1M>',|EM-UW4X4X4X^`'a'a'a$0@%/G,8PQTVYWYVYQY8Z8`8`8b8b,S!""W578,V4)A(H(H^]u(v(v%05f F,- f"555 % 1%Q G G G !(2t D D   ""Q&& - 5 5b 9 9 "I}m&D"MMMM Q M!!##y'===> a 0dKKKr#NNN)NNNNN)rFrYrXrYrz)r_r`rarbr rcr.rfr"TensorrVdictboolr1ExportedProgram staticmethodstrrr^ __classcell__rus@r!rFrFsY"&$(&* hhh$JhTz h  t# h  hhhhhhF*.-1.2 TT<$&T|d*T t+ T  TTTT**.-1.2&*" Z Z <$&Z |d*Z  t+ Z t Z t Z   %Z Z Z Z x ! iLiL,6iLiL iL  iL  iLiLiLiL iLiLiL\iLiLiLiLiLr#rFrgctdrCfdjdj D}fdjdj D}n4tdjjz}tdj}||fS)zuReturns a tuple `(num_heads, head_dim)` containing either 2 ints, or a list of int with the value for each layer.global_head_dimc8g|]}|dkrjnjSfull_attention)rhead_dim.0layerrs r! z#get_head_shapes..s>   ',/?&?&?F " "V_   r#NcFg|]}|dkrjrjnjSr)attention_k_eq_vnum_global_key_value_headsnum_key_value_headsrs r!rz#get_head_shapes..sM   (((V-D(  - -+   r#rr)rnrjnum_kv_shared_layersrpr8num_attention_heads)rr num_headss` r!get_head_shapesrsv()) W    +,Jv/J.J,JK        +,Jv/J.J,JK    6:v/AVE_/_``F$96;UVV h r#c eZdZdZ ddededzdedzdejdzddf fd Z dd ej dzd ej dzd ej dzfd Z e d ej jdej dedej fdZxZS)rta A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM to `StaticCache`. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. Note: This class is specifically designed to support export process using `torch.export` in a way that ensures the model can be further lowered and run efficiently in `ExecuTorch`. NrrerrfrgcXt|j}|j}|t d|jst d|jdkrt d|jin|j}|'| dd}|td|'| dd}|td || d |j }||_ t|| |_t|jjD]6\}} t#| t$rt'||jj|<7t)|\} } |j j} |j|| | | |t|jjD]e\}} |d || jd |d|| jd |d|| jd fdS)a Initializes the wrapper module with the pretrained model. Args: model (`PreTrainedModel`): The pretrained model to wrap. The model must have caching enabled and use a 'static' caching implementation. batch_size (`Optional[int]`): The batch size of the model. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we raise a ValueError. max_cache_len (`Optional[int]`): The maximum cache length for generation. Same mechanism as `batch_size` if not provided. device (`Optional[torch.device]`): The device to use. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we use `model.device` (no error is raised). Raises: AssertionError: If the pretrained model does not have caching enabled or if it does not use a 'static' caching implementation in `model.generation_config`. ValueError: If `batch_size` or `max_cache_len` is not provided, either as an argument or in `cache_config`. NvThe model must have a generation config to be exported with static caching. Please set `generation_config` in `model`.zvThe model must have caching enabled to be exported with static caching. Please set `generation_config.use_cache=True`.staticzThe model must use a 'static' caching implementation to be exported with static caching. Please set `generation_config.cache_implementation='static'`.reFbatch_size must be provided, either as an argument or in cache_config.rImax_cache_len must be provided, either as an argument or in cache_config.rf)rr key_cache_F persistent value_cache_cumulative_length_)rlr"rrmgeneration_configAssertionErrorricache_implementation cache_configgetrorfrr static_cache enumeratelayers isinstancer r rr'early_initializationregister_bufferkeysvaluescumulative_lengthr rrerrfrrrrrrrr'rus r!r"z-TorchExportableModuleWithStaticCache.__init__s2 --//!3  $ = !*  A   1X = = P  /;CrrIZIg   %)),==J! !ijjj  (,,_dCCM$ !lmmm >!%%h ==F 'mFSSS"$"3":;; I IHAu%!9:: I.9-.H.H!(+-f55 8   ..z9hPUW]^^^"$"3":;; f fHAu  !1a!1!15:% P P P  !3!3!3U\e T T T  !9a!9!95;R_d e e e e f fr#rDrvrEc|jjD]"}|j|d#|j}|||d|d}t |dr|jS|jS)a8 Forward pass of the module, which is compatible with the ExecuTorch runtime. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. inputs_embeds (`torch.Tensor`): Tensor representing current input embeddings to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. This forward adapter serves two primary purposes: 1. **Making the Model `torch.export`-Compatible**: The adapter hides unsupported objects, such as the `Cache`, from the graph inputs and outputs, enabling the model to be exportable using `torch.export` without encountering issues. 2. **Ensuring Compatibility with `ExecuTorch` runtime**: The adapter matches the model's forward signature with that in `executorch/extension/llm/runner`, ensuring that the exported model can be executed in `ExecuTorch` out-of-the-box. rNTrDrvattention_maskpast_key_valuesrir)rrrcopy_rrnrlast_hidden_state)r rDrvrErroutss r!rVz,TorchExportableModuleWithStaticCache.forward0s<&- = =E  # ) ).*; < < < <+zz'+    4 " " *; ) )r#rprompt_token_idsr[c |j}|jd}||z}|D]9\}}|dr|jd}t ||}n:g} t t ||D]} ||dd| | dzftj | gtj |} | |d|  tj | dddddfd  } | | t| |kr|tj | ggtj |tj t| gtj |} tj | dddddfd  } | | t| |ktj | gtj |S) a Generate a sequence of tokens using an exported program. This util function is designed to test exported models by simulating the generation process. It processes the input prompt tokens sequentially (no parallel prefill). This generate function is not intended to replace the original `generate` method, and the support for leveraging the original `generate` is potentially planned! Args: exported_program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. prompt_token_ids (`torch.Tensor`): Tensor representing the input prompt token IDs. max_new_tokens (`int`): Maximum number of new tokens to generate. Note that the total generation length is limited by both `max_new_tokens` and the model's cache size. Returns: torch.Tensor: A tensor containing the generated sequence of token IDs, including the original prompt tokens. r{ key_cacherNrr|rCrr)rfr named_buffers startswithrKrrrVr.rrIappendrrlen) rrr[rfprompt_token_lenmax_generation_length buffer_namebufferrresponse_tokens input_posresult current_tokens r!r^z-TorchExportableModuleWithStaticCache.generateasr."(+1"5 0> A#3#A#A#C#C   K%%k22  & Q (+,A=(Q(Q%  s#8:JKKLL J JI%,,..66*111i)a-.G+GH$|YKuzRXYYY7F  " "#3A#6y#A#F#F#H#H I I I I VAAAr111H%52>>>CCEE }---/""%:::%,,..66,'8 SYZZZ$|S-A-A,B%*]cddd7F"L2qqq)9rBBBGGIIM  " "= 1 1 1 /""%:::|_-UZOOOOr#r)r_r`rarbr rcr.rfr" LongTensorrrVrr1rr^rrs@r!rtrts_"&$(&* OfOfOf$JOfTz Of  t# Of  OfOfOfOfOfOff.2-1.2 /*/*#d*/*|d*/* t+ /*/*/*/*b2P,62P,2P2P  2P2P2P\2P2P2P2P2Pr#rtc eZdZdZ d dededzdedzdejdzddf fd Z d d ej dzd ej dzd ej dzdej fd Z xZ S)rqa A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM to hybrid `StaticCache`. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. Nrrerrfrgc$t||_|j}|j}|t d|jst d|jin|j}|'| dd}|td|'| dd}|td|| d|j }t|| |_ t|j jD]6\}} t!| t"rt%||j j|<7t'|\} } |jj} |j || | | |t|j jD]e\}} |d || jd |d || jd |d|| jd fdS)a Initializes the exportable module. Args: model (`PreTrainedModel`): The pretrained model to wrap. batch_size (`Optional[int]`): The batch size of the model. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we raise a ValueError. max_cache_len (`Optional[int]`): The maximum cache length for generation. Same mechanism as `batch_size` if not provided. device (`Optional[torch.device]`): The device to use. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we use `model.device` (no error is raised). Raises: AssertionError: If the model doesn't have the expected configuration for hybrid StaticCache. ValueError: If `batch_size` or `max_cache_len` is not provided, either as an argument or in `cache_config`. Nrz Model must have caching enabled.rerrrrfrrrFrrr)rlr"rrrmrrrirrrorfrcacherrrr r rr'rrrrrrs r!r"z-TorchExportableModuleWithHybridCache.__init__sS,  --//!3  $ =  E !CDD D.;CrrIZIg  %)),==J! !ijjj  (,,_dCCM$ !lmmm >!%%h ==F!mLLL "$*"344 B BHAu%!9:: B'2='A'A !!$-f55 8   '' IxPVWWW"$*"344 f fHAu  !1a!1!15:% P P P  !3!3!3U\e T T T  !9a!9!95;R_d e e e e f fr#rDrvrEc|jjD]"}|j|d#|||d|jd}|jS)a Forward pass of the module, which is compatible with the ExecuTorch llm runner. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. inputs_embeds (`Optional[torch.Tensor]`): Tensor representing current input embeddings to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. rNTr)rrrrrr)r rDrvrErrs r!rVz,TorchExportableModuleWithHybridCache.forwardsl(Z& = =E  # ) ).*; < < < <**' J   ~r#r) r_r`rarbr rcr.rfr"rrrVrrs@r!rqrqs"&$(&* BfBfBf$JBfTz Bf  t# Bf  BfBfBfBfBfBfL.2-1.2 !!#d*!|d*! t+ !  !!!!!!!!r#rqrexample_input_idsexample_cache_positionr+r,c0ddl}|j5||n|jdgg|j|j}||n|jdg|j|j}t dr4|jt|d||d|||nd }n`|tj d |tj d |jj t|d||dd d }|cdddS#1swxYwYdS)a Convert a `PreTrainedModel` into an exportable module and export it using `torch.export`, ensuring the exported model is compatible with `ExecuTorch`. Args: model (`PreTrainedModel`): The pretrained model to be exported. example_input_ids (`Optional[torch.Tensor]`): Example input token id used by `torch.export`. example_cache_position (`Optional[torch.Tensor]`): Example current cache position used by `torch.export`. dynamic_shapes(`Optional[dict]`): Dynamic shapes used by `torch.export`. strict(`Optional[bool]`): Flag to instruct `torch.export` to use `dynamo`. Returns: Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. rNrr|z2.6.0rSrCTr}zWDynamic shapes spec will be ignored by convert_and_export_with_cache for torch < 2.6.0.zSThe strict flag will be ignored by convert_and_export_with_cache for torch < 2.6.0.F)r*rT pre_dispatchr,) torch.export._traceno_gradrrIrfr r1rtrrr~_trace_export)rrrr+r,r.rs r!convert_and_export_with_cacher s, ' ' !,  se5:elKKK &1 # "qcELIII  %W - - $|224U;;%6J`aa-!'!3vv 3    )m! uvvv %|2::4U;;%6J`aa" ;    O' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' sC*D  DDc(eZdZdZfdZdZxZS) Seq2SeqLMEncoderExportableModulez A wrapper module designed to make a Seq2Seq LM encoder exportable with `torch.export`. This module ensures that the exported encoder model is compatible with ExecuTorch. cVt||_dSN)rlr"encoder)r encoder_modelrus r!r"z)Seq2SeqLMEncoderExportableModule.__init__Ns$ $ r#c8||jS)N)rD)rr)r rDs r!rVz(Seq2SeqLMEncoderExportableModule.forwardRs||i|00BBr#r_r`rarbr"rVrrs@r!rrHsX %%%%%CCCCCCCr#rc(eZdZdZfdZdZxZS)/Seq2SeqLMDecoderExportableModuleWithStaticCachez A wrapper module designed to make a Seq2Seq LM decoder exportable with `torch.export`, specifically for use with static caching. This module ensures the exported decoder is compatible with ExecuTorch. ct||_|j|_|j|_t |j}t|j||_ t|j j D]6\}}t|trt||j j |<7t!|j\}}|j |||t$j|t)|j t+|j|_t/t|j j D]e\}}|d||jd|d||jd|d||jdfdS)NrrrFrrr)rlr" get_decoderdecoderlm_headrnext parametersrfrrrrrr r rrr.r0rrr%register_dynamic_cache_export_supportrrrr) r rmax_static_cache_lengthrerrrrrrus r!r"z8Seq2SeqLMDecoderExportableModuleWithStaticCache.__init__]s ((** } l E,,..//6 (t{Jabbb"$"3":;; S SHAu%!9:: S.9:Q.R.R!(+-dk:: 8 ..z9hPUP]_klll():LPTP[<\<\<\]] -///"$"3":;; f fHAu  !1a!1!15:% P P P  !3!3!3U\e T T T  !9a!9!95;R_d e e e e f fr#c|jjD]"}|j|d#||||jd}||d}|S)NrT)rDencoder_hidden_statesrri)rrrrrrr)r decoder_input_idsr"rErr lm_logitss r!rVz7Seq2SeqLMDecoderExportableModuleWithStaticCache.forward|s}&- = =E  # ) ).*; < < < <,,'"7 J   LL,, r#rrs@r!rrVsV fffff>r#rc<eZdZ d fd ZdZdZd d Zd ZxZS) Seq2SeqLMExportableModulerrrct||_||_|j|_||_td||||d|jj |_d|_ d|_ dS)NT)rer)ri max_lengthrrr) rlr" full_model get_encoderrrmax_hidden_seq_lengthr rrexported_encoderexported_decoder)r rrer,rmax_cache_lengthrus r!r"z"Seq2SeqLMExportableModule.__init__s ((** l %:"!1'!5(!10= " " " !% $r#c~t|j|jj}t jd|j }t j 5t j||fdd|iid}dddn #1swxYwY|S)Nencoder_seq_lengthrArDrTr+r,) rrrr*rfr-r.r1r2r,r )r encoder_input_idswrapped_encoderrPr-s r!_export_encoderz)Seq2SeqLMExportableModule._export_encoders:4<HHKKDOLbcchhjjl&&';A[&\\ ]__  $|22"3!5{UVXcTdFenr 3                    s=)B22B69B6c |jj}t|j|jjd|jjd|}||}||}||}tj d|j }tj 5tj ||||fdd|iddd }dddn #1swxYwY|S) Nrre)rr reencoder_hidden_seq_lengthrAr)r#r"rETr2) r*rfrrrrrr-r.r1r2r,r )r r#r"rE target_devicewrapped_decoderencoder_seq_len_dimr.s r!_export_decoderz)Seq2SeqLMExportableModule._export_decoders. ;o(,(>(K(O(OP_(`(`1>BB<PP    R   TVV .00?? 5 8 8 G G'**=99$l../JPTPj.kk]__  $|22"$9>J)-./1D-E&*    3                   s;-D44D8;D8Nc |jj}||n tjdtj|}||n"tjdggtj|}||n!tjdgtj|}||nJtj|jj dd|j j ftj |} | ||_||| ||_|S)N)r r|rrer=)r*rfr.onesrIrrHrrrrd_modelr0r5r-r;r.) r r3r#r"rErfexample_encoder_input_idsexample_decoder_input_idsrexample_encoder_hidden_statess r!r1z Seq2SeqLMExportableModule.exports&'!,  G5:fEEE "!,  se5:fEEE " -8NNelA3V[V`io>p>p>p  %0 " !'488FFDKL_`m &!% 4 45N O O $ 4 4 %'DF\! !   r#c tj5|jj}|j|kr||}|j|}tjdggtj|}dg}t|dz D]}|j ||tj|gtj|}tj |dddddfd } | | tj| ggtj|}| |jjkrn|cdddS#1swxYwYdS)Nrr|rr{r)r.r r*rfrr-rrrIrr.rrrrr) r rr[rencoder_outputr#rrr next_tokens r!r^z"Seq2SeqLMExportableModule.generates ]__ ! !?1L &,66#3#6#6|#D#D A-..  7.5577%~u|QCuzbn7o7o7o #\&B*:CCCHHJJ $$Z000%*L:,uzZf$g$g$g!!7!DDDEE!A ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !sEE//E36E3)rr'rrNNNN) r_r`rar"r5r;r1r^rrs@r!r&r&sos%%%%%%,    ! ! ! F@!!!!!!!!!!!!!!r#r&example_attention_maskc ttj5tj|d||t |jddd}|cdddS#1swxYwYdS)a Export a model with DynamicCache using `torch.export`, ensuring the exported model is compatible with `ExecuTorch`. Args: model (`PreTrainedModel`): The pretrained model to be exported. example_input_ids (`Optional[torch.Tensor]`): Example input token id used by `torch.export`. example_attention_mask (`Optional[torch.Tensor]`): Example attention mask used by `torch.export`. Returns: Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. rSrT)rDrrriF)r,N)rr.r r1rr)rrrGrs r!export_with_dynamic_cacherIs"*+++     <..  ."8#/u|#D#D#D!    /                       s=A,,A03A0cN tjjtdt tjdtjdtjj tddS#t$r}dt|vrYd}~dSd}~wwxYw)z> Utilities for `DynamicCache` <> torch.export support cdtjjt |Sr)r.utils_pytree _dict_flatten_get_cache_dict dynamic_caches r!z7register_dynamic_cache_export_support..Gs"%+"5"C"COTaDbDb"c"cr#.cdtjjt |Sr)r.rLrM_dict_flatten_with_keysrOrPs r!rRz7register_dynamic_cache_export_support..Js&u{7J7b7b ..88r#)serialized_type_nameflatten_with_keys_fncftjjt ||Sr)r.fxrM_dict_flatten_specrO)rspecs r!rRz7register_dynamic_cache_export_support..Qs% 0 C COTYDZDZ\` a ar#z!already registered as pytree nodeN) r.rLrMregister_pytree_noder_unflatten_dynamic_cacher`r_rYregister_pytree_flatten_specror)es r!rr?s  00  c c $$0$;!U!Ul>S!U!U"" 1    55  a a      .c!ff < <  = < < < < <sA9A== B$BB$rctd|jDrtdtst jdd|jDd|jDdS)z9Convert cache to dictionary format for pytree operations.c3PK|]!}t|ttf V"dSr)rrrrrs r! z"_get_cache_dict..[s6 f fPUz%,0I!JKK K f f f f f fr#zFThis pytree flattening function should only be applied to DynamicCachez[DynamicCache + torch.export is tested on torch 2.6.0+ and may not work on earlier versions.c*g|]}|j |jSr)rrbs r!rz#_get_cache_dict..bs!UUUUej>Tej>T>T>Tr#c*g|]}|j |jSr)rrbs r!rz#_get_cache_dict..cs![[[%,BZ BZBZBZr#)r value_cache)anyr RuntimeErrorrrrr~)rs r!rOrOYs f fY^Ye f f fffecddd -wuvvvVUelUUU[[%,[[[  r#contextctjj||}t }|dg}|dg}t tt|t|D]S}|t|kr||nd}|t|kr||nd}| |||T|S)Nrrf) r.rLrM_dict_unflattenrrrrBrupdate) rri dictionaryrkey_list value_listidxkeyvalues r!r]r]gs$44VWEEJ NNE~~k2..H r22JSXJ8899&&"S]]22hsmm#&Z#8#8 3d S%%%%% Lr#rF)NN))rrr. cache_utilsrrrrrr r generation.configuration_utilsr modeling_utilsr pytorch_utilsr rrnnModulerFtuplerclistrrtrqrrrr rrr&rIrrOrLrMContextr]rSr#r!r|s >=====,,,,,, U U U U U U U U pALALALALALEHOALALALHuS49_cDIo%EF,@P@P@P@P@P58?@P@P@PFlllll58?lllb.226"& ? ? ? |d*? "L4/? 4K ? 4K ? ? ? ? D C C C C Cux C C C88888eho888vI!I!I!I!I!I!I!I!\.226   |d* "L4/    D4 <     ek.A.I      r#