}tj<DddlZddlZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z m Z ddlmZddlmZddlmZmZmZejeZhd Zhd Ze Gd d Zd ejezdeeeffdZd eejzdee eeeze!zfddfdZ"d eejzdeejzddfdZ#dej$dedeejze!zddfdZ%deeze!zde!fdZ&dedefdZ'dedeeddfdZ(dej$dej)de*fd Z+dS)!N) GeneratorIterable)contextmanager) dataclassfield)Path)Any)DDUFCorruptedFileErrorDDUFExportErrorDDUFInvalidEntryNameError>.txt.json.model .safetensors> config.jsonscheduler_config.jsontokenizer_config.jsonpreprocessor_config.jsonceZdZUdZeed<eed<eed<edZe ed<e de e d d ffd Z dd edefd Zd S) DDUFEntryaObject representing a file entry in a DDUF file. See [`read_dduf_file`] for how to read a DDUF file. Attributes: filename (str): The name of the file in the DDUF archive. offset (int): The offset of the file in the DDUF archive. length (int): The length of the file in the DDUF archive. dduf_path (str): The path to the DDUF archive (for internal use). filenamelengthoffsetF)repr dduf_pathreturnNc#BK|jd5}tj|dtj5}||j|j|jzVdddn #1swxYwYddddS#1swxYwYdS)a-Open the file as a memory-mapped file. Useful to load safetensors directly from the file. Example: ```py >>> import safetensors.torch >>> with entry.as_mmap() as mm: ... tensors = safetensors.torch.load(mm) ``` rbr)raccessN)ropenmmapfileno ACCESS_READrr)selffmms a/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/huggingface_hub/serialization/_dduf.pyas_mmapzDDUFEntry.as_mmap:s7^  & & B!188::a8HIII BRt{T['@@AAAA B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B Bs54BA<0 B<B BB BBButf-8encodingc|jd5}||j||j|cdddS#1swxYwYdS)zRead the file as text. Useful for '.txt' and '.json' entries. Example: ```py >>> import json >>> index = json.loads(entry.read_text()) ``` r)r+N)rr!seekrreadrdecode)r%r+r&s r( read_textzDDUFEntry.read_textKs^  & & A! FF4;   66$+&&--x-@@ A A A A A A A A A A A A A A A A A AsAA00A47A4)r*)__name__ __module__ __qualname____doc__str__annotations__intrrrrrbytesr)r0r(rr#s  MMM KKK KKKe'''It'''B5$#45BBB^B A A# AC A A A A A Ar:rrrci}t|}td|tjt |d5}|D]}td|j|j tj krtd t|jn*#t$r}td|j|d}~wwxYwt||}t|j||j|||j< dddn #1swxYwYd|vrtd t#j|d}t)||td |d t-|d |S) a Read a DDUF file and return a dictionary of entries. Only the metadata is read, the data is not loaded in memory. Args: dduf_path (`str` or `os.PathLike`): The path to the DDUF file to read. Returns: `dict[str, DDUFEntry]`: A dictionary of [`DDUFEntry`] indexed by filename. Raises: - [`DDUFCorruptedFileError`]: If the DDUF file is corrupted (i.e. doesn't follow the DDUF format). Example: ```python >>> import json >>> import safetensors.torch >>> from huggingface_hub import read_dduf_file # Read DDUF metadata >>> dduf_entries = read_dduf_file("FLUX.1-dev.dduf") # Returns a mapping filename <> DDUFEntry >>> dduf_entries["model_index.json"] DDUFEntry(filename='model_index.json', offset=66, length=587) # Load model index as JSON >>> json.loads(dduf_entries["model_index.json"].read_text()) {'_class_name': 'FluxPipeline', '_diffusers_version': '0.32.0.dev0', '_name_or_path': 'black-forest-labs/FLUX.1-dev', ... # Load VAE weights using safetensors >>> with dduf_entries["vae/diffusion_pytorch_model.safetensors"].as_mmap() as mm: ... state_dict = safetensors.torch.load(mm) ``` zReading DDUF file rzReading entry z)Data must not be compressed in DDUF file.z!Invalid entry name in DDUF file: N)rrrrmodel_index.json7Missing required 'model_index.json' entry in DDUF file.zDone reading DDUF file z. Found z entries)rloggerinfozipfileZipFiler5infolistdebugr compress_type ZIP_STOREDr _validate_dduf_entry_namer _get_data_offsetr file_sizejsonloadsr0_validate_dduf_structurekeyslen)rentrieszfr@erindexs r(read_dduf_filerS[sNGYI KK0Y00111 Y - -KKMM  D LL9$-99 : : :!W%777,-XYYY i)$-8888, i i i,-`QUQ^-`-`aaghh i&b$//F%.vdnXa&&&GDM " " $(($%^___ Jw12<<>> ? ?EUGLLNN333 KKS)SSS\\SSSTTT Ns7AD1/CD1 C+C&&C++9D11D58D5rOctd|dt}d}tjt |dtj5}|D]\}}||vrtd||||dkr\ tj t| }n'#tj $r}td|d}~wwxYw t|}n%#t$r}td||d}~wwxYwtd |d t#||| dddn #1swxYwY|td  t%||n"#t&$r}td |d}~wwxYwtd |dS)a Write a DDUF file from an iterable of entries. This is a lower-level helper than [`export_folder_as_dduf`] that allows more flexibility when serializing data. In particular, you don't need to save the data on disk before exporting it in the DDUF file. Args: dduf_path (`str` or `os.PathLike`): The path to the DDUF file to write. entries (`Iterable[tuple[str, Union[str, Path, bytes]]]`): An iterable of entries to write in the DDUF file. Each entry is a tuple with the filename and the content. The filename should be the path to the file in the DDUF archive. The content can be a string or a pathlib.Path representing a path to a file on the local disk or directly the content as bytes. Raises: - [`DDUFExportError`]: If anything goes wrong during the export (e.g. invalid entry name, missing 'model_index.json', etc.). Example: ```python # Export specific files from the local disk. >>> from huggingface_hub import export_entries_as_dduf >>> export_entries_as_dduf( ... dduf_path="stable-diffusion-v1-4-FP16.dduf", ... entries=[ # List entries to add to the DDUF file (here, only FP16 weights) ... ("model_index.json", "path/to/model_index.json"), ... ("vae/config.json", "path/to/vae/config.json"), ... ("vae/diffusion_pytorch_model.fp16.safetensors", "path/to/vae/diffusion_pytorch_model.fp16.safetensors"), ... ("text_encoder/config.json", "path/to/text_encoder/config.json"), ... ("text_encoder/model.fp16.safetensors", "path/to/text_encoder/model.fp16.safetensors"), ... # ... add more entries here ... ] ... ) ``` ```python # Export state_dicts one by one from a loaded pipeline >>> from diffusers import DiffusionPipeline >>> from typing import Generator, Tuple >>> import safetensors.torch >>> from huggingface_hub import export_entries_as_dduf >>> pipe = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") ... # ... do some work with the pipeline >>> def as_entries(pipe: DiffusionPipeline) -> Generator[tuple[str, bytes], None, None]: ... # Build a generator that yields the entries to add to the DDUF file. ... # The first element of the tuple is the filename in the DDUF archive (must use UNIX separator!). The second element is the content of the file. ... # Entries will be evaluated lazily when the DDUF file is created (only 1 entry is loaded in memory at a time) ... yield "vae/config.json", pipe.vae.to_json_string().encode() ... yield "vae/diffusion_pytorch_model.safetensors", safetensors.torch.save(pipe.vae.state_dict()) ... yield "text_encoder/config.json", pipe.text_encoder.config.to_json_string().encode() ... yield "text_encoder/model.safetensors", safetensors.torch.save(pipe.text_encoder.state_dict()) ... # ... add more entries here >>> export_entries_as_dduf(dduf_path="stable-diffusion-v1-4.dduf", entries=as_entries(pipe)) ``` zExporting DDUF file ''NwzCan't add duplicate entry: r=z#Failed to parse 'model_index.json'.zInvalid entry name: zAdding entry 'z' to DDUF filer>zInvalid DDUF file structure.zDone writing DDUF file )r?r@setrArBr5rFr addrJrK _load_contentr/JSONDecodeErrorrGr rD_dump_content_in_archiverLr )rrO filenamesrRarchivercontentrQs r(export_entries_as_ddufr_syp KK4 444555I E Yg.@ A AAW!( A A Hg9$$%&NH&N&NOOO MM( # # #---X J}W'='='D'D'F'FGGEE+XXX)*OPPVWWX P4X>>, P P P%&GX&G&GHHaO P LLB(BBB C C C $Wh @ @ @ @! AAAAAAAAAAAAAAAA( }WXXXE  2222 !EEE<==1DE KK5)5566666sr9E$3C  E$ C.C))C..E$2DE$ D$ DD$$3E$$E(+E(F F1F,,F1 folder_pathctdttttfffd }t ||dS)a Export a folder as a DDUF file. AUses [`export_entries_as_dduf`] under the hood. Args: dduf_path (`str` or `os.PathLike`): The path to the DDUF file to write. folder_path (`str` or `os.PathLike`): The path to the folder containing the diffusion model. Example: ```python >>> from huggingface_hub import export_folder_as_dduf >>> export_folder_as_dduf(dduf_path="FLUX.1-dev.dduf", folder_path="path/to/FLUX.1-dev") ``` rc3KtdD]}|s|jtvrt d|dD|}t|j dkrt d|d| |fVdS)Nz**/*zSkipping file 'z' (file type not allowed)z"' (nested directories not allowed)) rglobis_filesuffixDDUF_ALLOWED_ENTRIESr?rD relative_torNpartsas_posix)pathpath_in_archiver`s r(_iterate_over_folderz3export_folder_as_dduf.._iterate_over_folder s%%**622 3 3D<<>> {"666 NtNNNOOO"..{;;O?())Q.. WtWWWXXX!**,,d2 2 2 2 2 3 3r:N)rrtupler5r_)rr`rms ` r(export_folder_as_ddufrosg${##K 3(5d+;"< 3 3 3 3 3 39&:&:&<&<=====r:r]rr^c||dd5}t|ttfrSt|}|d5}t j||ddddn #1swxYwYn>t|t r||ntd|dddddS#1swxYwYdS)NrVT) force_zip64rizInvalid content type for z. Must be str, Path or bytes.) r! isinstancer5rshutil copyfileobjr8writer )r]rr^ archive_fh content_path content_fhs r(r[r[s h 6 6g* gT{ + + g==L""4(( LJ":z?KKK L L L L L L L L L L L L L L L  ' ' g   W % % % %!"eh"e"e"eff fggggggggggggggggggs7ACA=1 C=B CB ACCCct|ttfr!t|St|tr|St dt |d)zoLoad the content of an entry as bytes. Used only for small checks (not to dump content into archive). z6Invalid content type. Must be str, Path or bytes. Got .)rrr5r read_bytesr8r type)r^s r(rYrY)so 'C;''iG}}''))) GU # #igW[\cWdWdggghhhr: entry_namec$d|ddztvrtd|d|vrtd|d|d}|ddkrtd|d|S) NrzzFile type not allowed: \z0Entry names must use UNIX separators ('/'). Got /z-DDUF only supports 1 level of directory. Got )splitrgr stripcount)r}s r(rGrG6s Z  c " "2 &&.BBB'(N*(N(NOOO z'(h[e(h(h(hiii!!#&&Jq  '(eXb(e(e(efff r:rR entry_namesc<t|ts tdt|ddD}|D]U|vrtddt fdt Dstddt dVd S) a Consistency checks on the DDUF file structure. Rules: - The 'model_index.json' entry is required and must contain a dictionary. - Each folder name must correspond to an entry in 'model_index.json'. - Each folder must contain at least a config file ('config.json', 'tokenizer_config.json', 'preprocessor_config.json', 'scheduler_config.json'). Args: index (Any): The content of the 'model_index.json' entry. entry_names (Iterable[str]): The list of entry names in the DDUF file. Raises: - [`DDUFCorruptedFileError`]: If the DDUF file is corrupted (i.e. doesn't follow the DDUF format). z>Invalid 'model_index.json' content. Must be a dictionary. Got rzcLh|]!}d|v|dd"S)rr)r).0entrys r( z+_validate_dduf_structure..Vs.QQQEC5LLEKK$$Q'LLLr:zMissing required entry 'z' in 'model_index.json'.c3*K|] }d|vVdS)rNr9)rrequired_entryrfolders r( z+_validate_dduf_structure..Zs6rr>f//~//;>rrrrrrr:z!Missing required file in folder 'z!'. Must contains at least one of N)rrdictr r|anyDDUF_FOLDER_REQUIRED_ENTRIES)rRr dduf_foldersrs ` @r(rLrLAs$ eT " "v$%tfjkpfqfq%t%t%tuuuQQ[QQQL   ()dF)d)d)dee errrrrUqrrrrr (|F||]y|||  r:rPr@c|jtd|j}|j||jd}t |dkrtdt |ddd}t |ddd}|dz|z|z}|S)a1 Calculate the data offset for a file in a ZIP archive. Args: zf (`zipfile.ZipFile`): The opened ZIP file. Must be opened in read mode. info (`zipfile.ZipInfo`): The file info. Returns: int: The offset of the file data in the ZIP archive. Nz+ZipFile object must be opened in read mode.zIncomplete local file header.little)fpr header_offsetr-r.rNr7 from_bytes)rPr@rlocal_file_header filename_lenextra_field_len data_offsets r(rHrH`s u}$%RSSS&MEJJ} 2 ""$%DEEE>>"3BrE":HEELnn%6r"u%=xHHO "$|3oEK r:),rJloggingr"osrsrAcollections.abcrr contextlibr dataclassesrrpathlibrtypingr errorsr r r getLoggerr1r?rgrrPathLiker5rrSrnr8r_rorBr[rYrGrLZipInfor7rHr9r:r(rs  ////////%%%%%%((((((((WWWWWWWWWW  8 $ $    4A4A4A4A4A4A4A 4AnBbkC/BDi4HBBBBJV7cBK&7V7(5QTVY\`V`chVhQhKiBjV7osV7V7V7V7r!>S2;%6!>S2;EV!>[_!>!>!>!>H ggo g gsUWU`O`chOh gmq g g g g i3:- i% i i i i##Chsm>"""C""""""r: