}tj)NdZddlZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZdd lmZmZmZmZmZmZmZmZmZmZdd lmZdd lmZm Z dd l!m"Z"m#Z#e$eeZ%ee Z&ej'd de&De(Z)ee(dze dd*e%dee%fZ+edZ,e,-dgddddddeddddddf dee(dzedfdedededee)dze d fd!ed"e+d#ee.e d$d%d&fd'ee.e d(d)fd*ee.e d+d,d-fd.ed/ed0dfd1Z/e,-d2gd3 dSd5ee(ed6fd!ed/ed0dfd7Z0e,-d8d9d:g dTd5ee(ed;fd.ed"e+d/ed0df d<Z1e,-d=gd> dTd5ee(ed;fd?ee(dze d@dAfdBee(dze dCfd/ed0df dDZ2e,-dEdFdGg dUdEee(edHfd/ed0dfdIZ3e,-dJgdK dVd5ee(ed;fdLee.e dMdNfdOee.e dPdQfd/ed0df dRZ4dS)WzDContains commands to interact with datasets on the Hugging Face Hub.N) Annotatedget_args)execute_raw_sql_query)CLIErrorRepositoryNotFoundErrorRevisionNotFoundError) DatasetSort_TExpandDatasetProperty_T) DatasetCard) REPO_LIST_DEFAULT_LIMIT AuthorOpt FilterOptLimitOpt RevisionOpt SearchOptTokenOpt get_hf_apimake_expand_properties_parser typer_factory)list_repo_files_cmd)ArgumentOption)_dataclass_to_dictoutDatasetSortEnumci|]}||Sr).0ss Z/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/huggingface_hub/cli/datasets.py r"-s/L/L/L1/L/L/L)typezComma-separated properties to return. When used, only the listed properties (and id) are returned. Example: '--expand=downloads,likes,tags'. Valid: z, .)helpcallbackz"Interact with datasets on the Hub.)r&z list | ls)zhf datasets lsz*hf datasets ls --sort downloads --limit 10zhf datasets ls --search "code"z*hf datasets ls --filter benchmark:officialz$hf datasets ls HuggingFaceFW/finewebz'hf datasets ls HuggingFaceFW/fineweb -Rz.hf datasets ls HuggingFaceFW/fineweb --tree -h)examplesFrepo_idzVDataset ID (e.g. `username/repo-name`) to list files from. If omitted, lists datasets.searchauthorfiltersortz Sort results.limitexpandhuman_readablez--human-readablez-hz=Show sizes in human readable format (only for listing files).as_treez--treez3List files in tree format (only for listing files). recursivez --recursivez-Rz0List files recursively (only for listing files).revisiontokenreturnc ||tjd|tjd|tjd|tjd|tkrtjd|tjdt|d||| | | S|rtjd | rtjd |rtjd | tjd t | } |r|jnd} d| |||| ||D}tj|dS)zList datasets on the Hub, or files in a dataset repo. When called with no argument, lists datasets on the Hub. When called with a dataset ID, lists files in that dataset repo. Nz'Cannot use --search when listing files.z'Cannot use --author when listing files.z'Cannot use --filter when listing files.z%Cannot use --sort when listing files.z&Cannot use --limit when listing files.z'Cannot use --expand when listing files.dataset)r) repo_typer0r1r2r3r4z(Cannot use --tree when listing datasets.z-Cannot use --recursive when listing datasets.z2Cannot use --human-readable when listing datasets.z,Cannot use --revision when listing datasets.r4c,g|]}t|Srr)r dataset_infos r! zdatasets_ls..s0     <((   r#)r,r+r*r-r.r/) click BadParameterr rrvalue list_datasetsrtable)r)r*r+r,r-r.r/r0r1r2r3r4apisort_keyresultss r! datasets_lsrF<s\  $%NOO O  $%NOO O  $%NOO O  $%LMM M + + +$%MNN N  $%NOO O")    M !KLLLR !PQQQW !UVVV !OPPP 5 ! ! !C!+tzztH  -- .     GIgr# leaderboard)z4hf datasets leaderboard SWE-bench/SWE-bench_VerifiedzLhf datasets leaderboard SWE-bench/SWE-bench_Verified --limit 5 --format jsonzIhf datasets ls --filter benchmark:official # list available leaderboards dataset_idz?The benchmark dataset ID (e.g. `SWE-bench/SWE-bench_Verified`).c"t|}||}d|d|D}tj|gddtjd|r%tjd |d jd dSdS) zList model scores from a dataset leaderboard. This command helps find the best models for a task or compare models by benchmark scores. Use 'hf datasets ls --filter benchmark:official' to list available leaderboards.r9)r)c,g|]}t|Srr;rentrys r!r=z(datasets_leaderboard..s!JJJU!%((JJJr#N)rankmodel_idr@sourcerOheadersid_keyzPUse 'hf datasets ls --filter benchmark:official' to list available leaderboards.zUse 'hf models info rz' to get details about a model.)rget_dataset_leaderboardrrBhintrO)rIr.r4rCrGrEs r!datasets_leaderboardrVs 5 ! ! !C--j-AAKJJk&5&6IJJJGI777 H _```b ` A(?```aaaaabbr#infoz&hf datasets info HuggingFaceFW/finewebz9hf datasets info my-dataset --expand downloads,likes,tagsz+The dataset ID (e.g. `username/repo-name`).ct|} ||||}nJ#t$r}td|d|d}~wt$r}td|d|d|d}~wwxYwt j|dS) z$Get info about a dataset on the Hub.r9)r)r3r/z Dataset 'z ' not found.Nz Revision 'z' not found on 'z'.)rr<rrrrdict)rIr3r/r4rCrWes r! datasets_infor[s 5 ! ! !CU XfUU "DDD;:;;;<.s)TTT%emu{e?S?S?S?S?Sr#cDg|]}|j|j|j|jdS)r]r^urlsize)r`r^rerfrLs r!r=z$datasets_parquet..s;af5<%+eiQVQ[\\r#rdrerQN)rlist_dataset_parquet_filesrrB)rIr]r^r4rCentriesfilteredrEs ` r!datasets_parquetrjs 5 ! ! !C,,Z,OOGTTTT7TTTHjrGIgAAA%PPPPPPr#sqlzhf datasets sql "SELECT COUNT(*) AS rows FROM read_parquet('https://huggingface.co/api/datasets/cfahlgren1/hub-stats/parquet/models/train/0.parquet')"zhf datasets sql "SELECT * FROM read_parquet('https://huggingface.co/api/datasets/cfahlgren1/hub-stats/parquet/models/train/0.parquet') LIMIT 5" --format jsonzRaw SQL query to execute.c t||}n/#t$r"}tt||d}~wwxYwt j|dS)zAExecute a raw SQL query with DuckDB against dataset parquet URLs.) sql_queryr4N)r ImportErrorrstrrrB)rkr4resultrZs r! datasets_sqlrqsf&&EBBB &&&s1vvA%&Ifs A;Acard)z&hf datasets card HuggingFaceFW/finewebz1hf datasets card HuggingFaceFW/fineweb --metadataz?hf datasets card HuggingFaceFW/fineweb --metadata --format jsonz-hf datasets card HuggingFaceFW/fineweb --textmetadataz --metadataz'Output only the metadata from the card.textz--textz(Output only the text body (no metadata).cR|r|rtdtj||}|r-tj|jdS|rtj|jdStj|jtj d|ddS)z7Get the dataset card (README) for a dataset on the Hub.z---metadata and --text are mutually exclusive.r9zUse `hf datasets card z/ --metadata` to extract only the card metadata.N) rr loadrrYdatato_dictrtcontentrU)rIrsrtr4rrs r! datasets_cardrzs HDHFGGG  Je 4 4 4Dg ""$$%%%%% g   e*eeefffffr#)rHN)NNNrb)FFN)5__doc__enumtypingrrr>huggingface_hub._dataset_viewerrhuggingface_hub.errorsrrrhuggingface_hub.hf_apir r huggingface_hub.repocardr _cli_utilsr rrrrrrrrr _file_listingr _frameworkrr_outputrrsorted_EXPAND_PROPERTIES _SORT_OPTIONSEnumrorjoin ExpandOpt datasets_clicommandboolrFrVr[rjrqrzrr#r!rsKJ &&&&&&&& AAAAAA[[[[[[[[[[IIIIIIII000000                        /.....((((((((,,,,,,,,VHH%<==>>'' $)-/L/Lm/L/L/LSVWWW $J FEeienenoAeBeBEEE../ABB  }"FGGG     -    7OO  d nooo qO  O  O O $O$$$ & O O O !4.mnnn pO$ xSTTT V%O, }d)[\\\ ^-O45O6 7O8 9OOO  Odbb#xx-noooopb b b bbbb& 0C! #xx-Z[[[[\     " jnTX Q Q#xx-Z[[[[\ Q cDj&&:d"e"e"ee f Q S4Z-O!P!P!PP Q Q  Q  Q Q Q Q   c j   3&ABBBB C         gl_d gg#xx-Z[[[[\gff\8abbbbcg D&&0Z[[[[ \g  g  ggggggr#