tjXdZddlmZddlZddlZddlmZddlmZGddZ dS)zJSON text splitter.) annotationsN)Any)DocumentceZdZUdZdZded< dZded< d)d*fd Zed+dZ ed,dZ d-dZ d.d/dZ d0d1d Z d2d3d#Z d4d5d(ZxZS)6RecursiveJsonSplitteraSplits JSON data into smaller, structured chunks while preserving hierarchy. This class provides methods to split JSON data into smaller dictionaries or JSON-formatted strings based on configurable maximum and minimum chunk sizes. It supports nested JSON structures, optionally converts lists into dictionaries for better chunking, and allows the creation of document objects for further use. intmax_chunk_sizeimin_chunk_sizeN int | NonereturnNonect||_||nt|dz d|_dS)aInitialize the chunk size configuration for text processing. This constructor sets up the maximum and minimum chunk sizes, ensuring that the `min_chunk_size` defaults to a value slightly smaller than the `max_chunk_size` if not explicitly provided. Args: max_chunk_size: The maximum size for a chunk. min_chunk_size: The minimum size for a chunk. If `None`, defaults to the maximum chunk size minus 200, with a lower bound of 50. N2)super__init__r maxr )selfr r __class__s [/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_text_splitters/json.pyrzRecursiveJsonSplitter.__init__sR ,) N^c)2.. datadict[str, Any]cDttj|S)z1Calculate the size of the serialized JSON object.)lenjsondumps)rs r _json_sizez RecursiveJsonSplitter._json_size5s4:d##$$$rdpath list[str]valuercb|ddD]}||i}|||d<dS)z;Set a value in a nested dictionary based on the given path.N) setdefault)r r!r#keys r_set_nested_dictz&RecursiveJsonSplitter._set_nested_dict:sA9 & &C S"%%AA$r( rct|tr fd|DSt|trfdt |DS|S)NcBi|]\}}||S)_list_to_dict_preprocessing).0kvrs r zERecursiveJsonSplitter._list_to_dict_preprocessing..Ks-TTTtq!At77::TTTrc\i|](\}}t||)Sr+)strr,)r-iitemrs rr0zERecursiveJsonSplitter._list_to_dict_preprocessing..NsCAtA88>>r) isinstancedictitemslist enumerate)rrs` rr,z1RecursiveJsonSplitter._list_to_dict_preprocessingEs dD ! ! UTTTTtzz||TTT T dD ! ! (   r current_pathlist[str] | Nonechunkslist[dict[str, Any]] | Nonelist[dict[str, Any]]c|pg}||nig}t|tr|r|D]\}}g||}||d}|||i}|j|z } || kr||d||k||jkr|i||||n|r||d|||S)zESplit json into maximum size dictionaries while preserving structure.Nr%) r5r6r7rr r(r append _json_split) rrr:r<r'r#new_path chunk_sizesize remainings rrAz!RecursiveJsonSplitter._json_splitUs2$)r !-B4 dD ! ! Bd B"jjll > > U/\/3/!__VBZ88 U|44 /*< )##))&*hFFFF!T%888 b)))$$UHf==== >" B  ! !&*lD A A A rF json_data convert_listsboolc|r)|||}n||}|ds||S)aSplits JSON into a list of JSON chunks. Args: json_data: The JSON data to be split. convert_lists: Whether to convert lists in the JSON to dictionaries before splitting. Returns: A list of JSON chunks. r%)rAr,pop)rrFrGr<s r split_jsonz RecursiveJsonSplitter.split_jsontsb  1%%d&F&Fy&Q&QRRFF%%i00Fbz  JJLLL rT ensure_asciicN|||}fd|DS)a~Splits JSON into a list of JSON formatted strings. Args: json_data: The JSON data to be split. convert_lists: Whether to convert lists in the JSON to dictionaries before splitting. ensure_ascii: Whether to ensure ASCII encoding in the JSON strings. Returns: A list of JSON formatted strings. )rFrGc<g|]}tj|S))rL)rr)r-chunkrLs r z4RecursiveJsonSplitter.split_text..s(QQQ 5|<<<QQQr)rK)rrFrGrLr<s ` r split_textz RecursiveJsonSplitter.split_texts7"9MRRRQQQ&QQQQrtexts metadataslist[dict[Any, Any]] | Nonelist[Document]c|pigt|z}g}t|D]`\}}||||D]B} tj||} t | | } || Ca|S)aCreate a list of `Document` objects from a list of json objects (`dict`). Args: texts: A list of JSON data to be split and converted into documents. convert_lists: Whether to convert lists to dictionaries before splitting. ensure_ascii: Whether to ensure ASCII encoding in the JSON strings. metadatas: Optional list of metadata to associate with each document. Returns: A list of `Document` objects. )rFrGrL) page_contentmetadata)rr9rQcopydeepcopyrr@) rrRrGrLrS metadatas_ documentsr3textrOrXnew_docs rcreate_documentsz&RecursiveJsonSplitter.create_documentss$32$U"3   '' * *GAtm,) * * =A77"III  ))))  * r)rN)r r r r r r)rrr r )r rr!r"r#rr r)rrr r)NN)rrr:r;r<r=r r>)F)rFrrGrHr r>)FT)rFrrGrHrLrHr r")FTN) rRr>rGrHrLrHrSrTr rU)__name__ __module__ __qualname____doc__r __annotations__r r staticmethodrr(r,rArKrQr_ __classcell__)rs@rrr s_N*N HL       0%%%\%\&*..2 D$8$! RRRRR2$!15 rr) rc __future__rrYrtypingrlangchain_core.documentsrrr+rrrjs"""""" ------rrrrrrrrrrr