dh"vSSKrSSKJrJrJrJr SSKJr SSKJ r SSK J r SSK J r SrSrS r"S S \ 5rg) N)AnyListMappingOptional)CallbackManagerForLLMRun)LLM) ConfigDict)enforce_stop_tokenszgoogle/flan-t5-largetext2text-generation)r text-generation summarizationc^\rSrSr%SrSr\\S'\r \ \S'Sr \ \ \S'Sr\ \ \S'\"SS 9r\SS\ S \ S \ \S \ \ S\ \ S\ \ S \ \S\ \S\ \S\S\4Sjj5r\S\\ \44Sj5r\S\ 4Sj5rSS\ S\ \\ S\ \S\S\ 4 SjjrSrg)WeightOnlyQuantPipelineaqWeight only quantized model. To use, you should have the `intel-extension-for-transformers` packabge and `transformers` package installed. intel-extension-for-transformers: https://github.com/intel/intel-extension-for-transformers Example using from_model_id: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) config = WeightOnlyQuantConfig hf = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation" pipeline_kwargs={"max_new_tokens": 10}, quantization_config=config, ) Example passing pipeline in directly: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( AutoModelForSeq2SeqLM ) from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) from transformers import AutoTokenizer, pipeline model_id = "google/flan-t5-large" tokenizer = AutoTokenizer.from_pretrained(model_id) config = WeightOnlyQuantConfig model = AutoModelForSeq2SeqLM.from_pretrained( model_id, quantization_config=config, ) pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10, ) hf = WeightOnlyQuantPipeline(pipeline=pipe) Npipelinemodel_id model_kwargspipeline_kwargsallow)extrataskdevice device_map load_in_4bit load_in_8bitquantization_configkwargsreturnc 2Ub&[U[5(aUS:a [S5e[RR S5c [S5eSSKJn Jn SSK J n SS K J n SS K J n [U[5(a,US:a&U "5(d [S 5eS [U5-nO[U[5(aUS:aSnUcUcSnU=(d 0nUR "U40UD6nUS:XaU R "U4UUU SUS.UD6nO6US;aU R "U4UUU SUS.UD6nO[SUS["S35eSU;a-UR%5VVs0sHunnUS:wdMUU_M nnnU=(d 0nU"SUUUUUS.UD6nUR&[";a [SUR&S["S35eU"SUUUUS.U D6$![a [S 5ef=f![an[SUS35UeSnAff=fs snnf)z5Construct the pipeline object from model_id and task.Nz7`Device` and `device_map` cannot be set simultaneously!torchz;Weight only quantization pipeline only support PyTorch now!r)AutoModelForCausalLMAutoModelForSeq2SeqLM)is_ipex_available) AutoTokenizer)rzCould not import transformers python package. Please install it with `pip install transformers` and `pip install intel-extension-for-transformers`.z)Don't find out Intel GPU on this machine!zxpu:cpur F)rrruse_llm_runtimer)r r Got invalid task , currently only  are supportedzCould not load the z# model due to missing dependencies.trust_remote_code)rmodel tokenizerrr)rrrr) isinstanceint ValueError importlibutil find_spec-intel_extension_for_transformers.transformersr"r#,intel_extension_for_transformers.utils.utilsr$ transformersr%r ImportErrorstrfrom_pretrained VALID_TASKSitemsr)clsrrrrrrrrrrr"r#r$r% hf_pipeline _model_kwargsr-r,ekv_pipeline_kwargsrs i/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_community/llms/weight_only_quantization.py from_model_id%WeightOnlyQuantPipeline.from_model_idOs  !z&#'>'>6B;VW W >> # #G , 4M    W 2 < fc " "v{$&& !LMM#f+-J  $ $!F >!" $* !11(LmL  ((,<<!-!-(;$)) $BB-==!-!-(;$)) $!'v.&&1].B - /!.!4!4!6!6A!?R:R1!6 +0b &     == +#HMM?3""-n>  &,     K F  \ %dV+NO   s<G5G1G15G1$ H4HG.1 H;H  HcJURURURS.$)zGet the identifying parameters.rrrrHselfs rD_identifying_params+WeightOnlyQuantPipeline._identifying_paramss'  --#33  cg)zReturn type of llm.weight_only_quantizationr.rIs rD _llm_type!WeightOnlyQuantPipeline._llm_types*rMpromptstop run_managerc URU5nURRS:XaUSS[U5SnOpURRS:Xa USSnOMURRS:Xa USSnO*[SURRS [S 35eU(a [ Xb5nU$) aCall the HuggingFace model and return the output. Args: prompt: The prompt to use for generation. stop: A list of strings to stop generation when encountered. Returns: The generated text. Example: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline llm = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation", ) llm.invoke("This is a prompt.") r rgenerated_textNr r summary_textr(r)r*)rrlenr1r;r )rJrRrSrTrresponsetexts rD_callWeightOnlyQuantPipeline._calls4==( ==  !2 2A;/0V?D ]]  #9 9A;/0D ]]  ? 2A;~.D#DMM$6$6#78""-n>  't2D rMr.)r NNNFFN)NN)__name__ __module__ __qualname____firstlineno____doc__rr__annotations__DEFAULT_MODEL_IDrr9rrdictrr model_config classmethodr0boolrrEpropertyrrKrPrrr[__static_attributes__r.rMrDrrs/bHc$Hc$*#'L(4.'1&*OXd^*4L !#$('+*.',',-1h h h  h SM h tn h "$h tnh tnh &c]h h  h h T WS#X%6  *3**%):> ++tCy!+67 +  + ++rMr)r2typingrrrr langchain_core.callbacks.managerr#langchain_core.language_models.llmsrpydanticr langchain_community.llms.utilsr rc DEFAULT_TASKr;rr.rMrDrps8//E3>)% J dcdrM