tjj"zddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZGddeZdS) N)PathLike)Path)AnyCallableDictIteratorOptionalUnion)Document) BaseLoaderceZdZdZ ddeeefdedeedeed ee e e ge fd ed efd Z d e e fdZdeded e e fdZded efdZde eefded e eeffdZded dfdZdS) JSONLoadera Load a `JSON` file using a `jq` schema. Setup: .. code-block:: bash pip install -U jq Instantiate: .. code-block:: python from langchain_community.document_loaders import JSONLoader import json from pathlib import Path file_path='./sample_quiz.json' data = json.loads(Path(file_path).read_text()) loader = JSONLoader( file_path=file_path, jq_schema='.quiz', text_content=False) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quizg .json', 'seq_num': 1} Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} NFT file_path jq_schema content_keyis_content_key_jq_parsable metadata_func text_content json_linesc  ddl}||_n#t$rtdwxYwt||_|||_||_||_||_ ||_ ||_ dS)a~Initialize the JSONLoader. Args: file_path (Union[str, PathLike]): The path to the JSON or JSON Lines file. jq_schema (str): The jq schema to use to extract the data or text from the JSON. content_key (str): The key to use to extract the content from the JSON if the jq_schema results to a list of objects (dict). If is_content_key_jq_parsable is True, this has to be a jq compatible schema. If is_content_key_jq_parsable is False, this should be a simple string key. is_content_key_jq_parsable (bool): A flag to determine if content_key is parsable by jq or not. If True, content_key is treated as a jq schema and compiled accordingly. If False or if content_key is None, content_key is used as a simple string. Default is False. metadata_func (Callable[Dict, Dict]): A function that takes in the JSON object extracted by the jq_schema and the default metadata and returns a dict of the updated metadata. text_content (bool): Boolean flag to indicate whether the content is in string format, default to True. json_lines (bool): Boolean flag to indicate whether the input is in JSON Lines format. rNz=jq package not found, please install it with `pip install jq`) jq ImportErrorrresolvercompile _jq_schema_is_content_key_jq_parsable _content_key_metadata_func _text_content _json_lines) selfrrrrrrrrs n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_community/document_loaders/json_loader.py__init__zJSONLoader.__init__TsD  IIIDGG   O   i0022**Y//+E('+)%s (returnc#zKd}|jrt|jd5}|D]:}|}|r"|||D] }|V|dz } ; ddddS#1swxYwYdS||jd|D] }|V|dz } dS)z-Load and return documents from the JSON file.rz utf-8-sig)encodingN)r ropenstrip_parse read_text)r!indexflinedocs r" lazy_loadzJSONLoader.lazy_loadsB   $$k$:: 'a''D::<>       s>A33A7:A7contentr,c#rK|jtj|}|j||t ||dzD]X\}}||}||t|j |}t||VYdS)z#Convert given content to documents.Nr')sample)r3sourceseq_num) page_contentmetadata) rinputjsonloadsr_validate_content_key enumerate _get_text _get_metadatastrrr )r!r1r,datair3textr7s r"r*zJSONLoader._parses$$TZ%8%899   (  & &t , , ,"433 A AIAv>>>00D))c$.&9&91*Hx@@@ @ @ @ @  A Ar3c|j\|jrG|j|j}||}n||j}n|}|jr7t|ts"| tdt|dt|tr|St|ttfr|rtj|ndS|t|ndS)zConvert sample to string formatNz%Expected page_content is string, got z instead. Set `text_content=False` if the desired input for `page_content` is not a string)rrrrr8firstr isinstancer? ValueErrortypedictlistr9dumps)r!r3compiled_content_keyr1s r"r=zJSONLoader._get_texts   (/ 4'+wt7H'I'I$.44V<<BBDD !23G   ?j#&>&> ?7CV4W 444  % % ?N $ . . ?*194:g&&&r 9#*#63w<<rCadditional_fieldsc |jM|||}t|ts tdt |d|S|S)z Return a metadata dictionary base on the existence of metadata_func :param sample: single data payload :param additional_fields: key-word arguments to be added as metadata values :return: NzUExpected the metadata_func to return a dict but got ``)rrGrJrHrI)r!r3rNresults r"r>zJSONLoader._get_metadatasq   *((1BCCFfd++  2"&v,,222M$ $rCr@c|}t|ts tdt |d|js2||jtd|jd|jr\|j |j | td|jddSdS)zCheck if a content key is validztExpected the jq schema to result in a list of objects (dict), so sample must be a dict but got `rPNz_Expected the jq schema to result in a list of objects (dict) with the key `z ` which should be parsable by jq) rFrGrJrHrIrgetrrrr8rB)r!r@r3s r"r;z JSONLoader._validate_content_keys"&$'' G7;F||GGG  0  4,--58#'#4888   ,  12288@@EEGGOW#'#4WWW    OOrC)NFNTF)__name__ __module__ __qualname____doc__r r?rr boolrrr#rr r0intr*rr=r>r;rCr"rr sFFX&*5:@D! 1&1&h'1&1&c] 1& %-TN 1&  $t); <= 1&1&1&1&1&1&f8H-$AcA#A(82DAAAA"??????4%38n%;>% c3h%%%%(#$rCr)r9osrpathlibrtypingrrrrr r langchain_core.documentsr )langchain_community.document_loaders.baser rrZrCr"r`s AAAAAAAAAAAAAAAA------@@@@@@ffffffffffrC