tj<dZddlmZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZdd lmZmZe rdd lmZmZmZmZdd lmZ ddlZd Zn #e $rd ZYnwxYw ddl!m"Z"d Z#n #e $rd Z#YnwxYwej$e%Z&eddZ'GddeeZ(Gdde(Z)Gdde*e Z+e d GddZ,d dZ-dS)!zText splitter base interface.) annotationsN)ABCabstractmethod) dataclass)Enum) TYPE_CHECKINGAnyLiteralTypeVar)BaseDocumentTransformerDocument)Selfoverride)Callable CollectionIterableSequence)SetTF)PreTrainedTokenizerBaseTS TextSplitter)boundceZdZdZddedddfd8dZed9dZ d:d;dZdd&Z e d?d+Z e d@dAd5ZedBd7ZdS)Crz)Interface for splitting text into chunks.iFT chunk_sizeint chunk_overlaplength_functionCallable[[str], int]keep_separatorbool | Literal['start', 'end']add_start_indexboolstrip_whitespacereturnNonec|dkrd|}t||dkrd|}t|||krd|d|d}t|||_||_||_||_||_||_dS)a$Create a new `TextSplitter`. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk `(True='start')` add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document Raises: ValueError: If `chunk_size` is less than or equal to 0 ValueError: If `chunk_overlap` is less than 0 ValueError: If `chunk_overlap` is greater than `chunk_size` rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)selfrrrr r"r$msgs [/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_text_splitters/base.py__init__zTextSplitter.__init__/s4 ??===CS// ! 1  D]DDCS// ! : % %6}66666 S// !%+ /- /!1textstr list[str]cdS)zSplit text into multiple components. Args: text: The text to split. Returns: A list of text chunks. N)r/r4s r1 split_textzTextSplitter.split_text\sr3Ntexts metadataslist[dict[Any, Any]] | Nonelist[Document]c |pigt|z}g}t|D]\}}d}d}||D]} tj||} |jrE||z|jz } || td| }|| d<t| }t| | } | | |S)a,Create a list of `Document` objects from a list of texts. Args: texts: A list of texts to be split and converted into documents. metadatas: Optional list of metadata to associate with each document. Returns: A list of `Document` objects. r start_index) page_contentmetadata) len enumerater9copydeepcopyr-r*findmaxr append) r/r:r; metadatas_ documentsir4indexprevious_chunk_lenchunkrAoffsetnew_docs r1create_documentszTextSplitter.create_documentsgs32$U"3   '' * *GAtE!" .. * *=A77(4"%77$:MMF IIeSF^^<--i88 !#  K KA((++D [1A1AA1E1E 1M"##4+++NN'(  {##a''//+yAAC C((( $"555 [9I9IA9M9M STU*++!AII!6!6{1~!F!F-0-=-=-A-AMMq"'2!""o  $"555 [9I9IA9M9M STU*++!AII   q ! ! ! Tc+.>.>.B.B]]J JEEook955 ? KK    r3 tokenizerrkwargsr c tsd}t|ttsd}t|d fd }|d d|i|S) aText splitter that uses Hugging Face tokenizer to count length. Args: tokenizer: The Hugging Face tokenizer to use. Returns: An instance of `TextSplitter` using the Hugging Face tokenizer for length calculation. z`Could not import transformers python package. Please install it with `pip install transformers`.zATokenizer received was not an instance of PreTrainedTokenizerBaser4r5r%rcHt|SrZ)rBtokenizer4rjs r1_huggingface_tokenizer_lengthzNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengthsy))$//00 0r3rr4r5r%rr8)_HAS_TRANSFORMERSr( isinstancer)clsrjrkr0rps ` r1from_huggingface_tokenizerz'TextSplitter.from_huggingface_tokenizers! "E S// !)%<== "T S// ! 1 1 1 1 1 1sKK#@KFKKKr3gpt2all encoding_name model_nameallowed_special(Literal['all'] | AbstractSet[str] | Nonedisallowed_special Literal['all'] | Collection[str]rc  ttsd}t||tj| ntj| d  fd }t |tr ||d}i||}|d d |i|S) amText splitter that uses `tiktoken` encoder to count length. Args: encoding_name: The name of the tiktoken encoding to use. model_name: The name of the model to use. If provided, this will override the `encoding_name`. allowed_special: Special tokens that are allowed during encoding. disallowed_special: Special tokens that are disallowed during encoding. Returns: An instance of `TextSplitter` using tiktoken for length calculation. Raises: ImportError: If the tiktoken package is not installed. NzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.r4r5r%rcNt|SN)rzr|)rBencode)r4rzr|encs r1_tiktoken_encoderz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoder s4 $3'9 r3)rxryrzr|rrqr8)set _HAS_TIKTOKEN ImportErrortiktokenencoding_for_model get_encoding issubclassTokenTextSplitter) rtrxryrzr|rkr0r extra_kwargsrs `` @r1from_tiktoken_encoderz"TextSplitter.from_tiktoken_encoders2  "!eeO #A  c"" "  !-j99CC' 66C         c, - - 0!.(#2&8 L 0/,/Fs??#4????r3Sequence[Document]c F|t|S)zTransform sequence of documents by splitting them. Args: documents: The sequence of documents to split. Returns: A list of split documents. )rUlist)r/rJrks r1transform_documentsz TextSplitter.transform_documents!s##DOO444r3)rrrrrrr r!r"r#r$r#r%r&r4r5r%r6rZ)r:r6r;r<r%r=)rJrRr%r=)rVr6rWr5r%rX)r^r_rWr5r%r6)rjrrkr r%rrvNNrw) rxr5ryrXrzr{r|r}rkr r%r)rJrrkr r%r)__name__ __module__ __qualname____doc__rBr2rr9rQrUr]ri classmethodrurrrr8r3r1rr,sG33 039> %!%+2+2+2+2+2Z   ^ JN8 A A A A ****XLLL[L@$!%DH?D 9@9@9@9@[9@v 5 5 5X 5 5 5r3c6eZdZdZ ddfd ZddZxZS)rz/Splitting text to tokens using model tokenizer.rvNrwrxr5ryrXrzr{r|r}rkr r%r&c  |t}tjdi|tsd}t ||t j|}nt j|}||_||_ ||_ dS)aCreate a new `TextSplitter`. Args: encoding_name: The name of the tiktoken encoding to use. model_name: The name of the model to use. If provided, this will override the `encoding_name`. allowed_special: Special tokens that are allowed during encoding. disallowed_special: Special tokens that are disallowed during encoding. Raises: ImportError: If the tiktoken package is not installed. NzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.r8) rsuperr2rrrrr _tokenizer_allowed_special_disallowed_special) r/rxryrzr|rkr0r __class__s r1r2zTokenTextSplitter.__init__3s*  "!eeO""6""" #A  c"" "  !-j99CC' 66C /#5   r3r4r6cdfd }tjjjj|}t ||S) arSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text: The input text to be split into smaller chunks. Returns: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. _textr5r% list[int]cRj|jjSr)rrrr)rr/s r1_encodez-TokenTextSplitter.split_text.._encodeks1?)) $ 5#'#;* r3)rtokens_per_chunkdecoderro)rr5r%r) Tokenizerr*r)rrsplit_text_on_tokens)r/r4rrjs` r1r9zTokenTextSplitter.split_text[sc       -!-?)    $CCCCr3r) rxr5ryrXrzr{r|r}rkr r%r&r)rrrrr2r9 __classcell__)rs@r1rr0ss99$!%DH?D &6&6&6&6&6&6&6PDDDDDDDDr3rceZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdS)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6N) rrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6r8r3r1rr|s,, C B D F B B C E F A C D D E EH E D C F E A C DG FJ!LLLr3r)frozencBeZdZUdZded< ded< ded< ded<d S) rzTokenizer data class.rrrzCallable[[list[int]], str]rzCallable[[str], list[int]]rN)rrrr__annotations__r8r3r1rrsQ*,&&&&=&&&&==r3rr4r5rjr%r6cg}||}d}|j|jkrd}t||t |krt ||jzt |}|||}|sne||}|r|||t |krn%||j|jz z }|t |k|S)zSplit incoming text and return chunks using tokenizer. Args: text: The input text to be split. tokenizer: The tokenizer to use for splitting. Returns: A list of text chunks. rz3tokens_per_chunk must be greater than chunk_overlap)rrrr(rBminrrH) r4rjr^ input_ids start_idxr0cur_idx chunk_idsdecodeds r1rrsF  &&II!Y%<<<Coo c)nn $ $i)"<rs##"""""" ########!!!!!!GFFFFFFF,,,,,,,,3HHHHHHHHHHHH222222OOOMMMMMLLLLLL  8 $ $ WT(((A5A5A5A5A5*CA5A5A5HIDIDIDIDID IDIDIDX"""""sD"""B $ > > > > > > > > s$AA'&A'+A44A>=A>