@h1@SSKJr SSKrSSKrSSKJrJr SSKJrJ r J r SSKJ r SSK Jr SSKJr SSKJrJrJrJrJrJr SS KJrJr \R6"\5r\"S S S 9r"S S \\5r"SS\5r "SS\!\5r"\"SS9"SS55r#SSjr$g)) annotationsN)ABCabstractmethod) CollectionIterableSequence)Set) dataclass)Enum)AnyCallableLiteralOptionalTypeVarUnion)BaseDocumentTransformerDocumentTS TextSplitter)boundc\rSrSrSrSS\SSS4SSjjr\SSj5rSSS jjr SS jr SS jr SS jr \ SSj5r\ SS \"5S4SSjj5rSSjrSrg )rz)Interface for splitting text into chunks.iFTcUS::aSU3n[U5eUS:aSU3n[U5eX!:aSUSUS3n[U5eXlX lX0lX@lXPlX`lg)aCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)self chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespacemsgs U/var/www/html/shao/venv/lib/python3.13/site-packages/langchain_text_splitters/base.py__init__TextSplitter.__init__s* ?0 =CS/ ! 1 4]ODCS/ !  %.}o><46 S/ !%+ /- /!1cg)z$Split text into multiple components.N)r"texts r* split_textTextSplitter.split_textEsr-Nc U=(d 0/[U5-n/n[U5HupVSnSnURU5Hn [R"X55n UR (a<Xx-UR - n URU [SU 55nXzS'[U 5n[XS9n URU 5 M M U$)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater1copydeepcopyr rfindmaxrappend) r"texts metadatas _metadatas documentsir0indexprevious_chunk_lenchunkr6offsetnew_docs r*create_documentsTextSplitter.create_documentsIs32$U"3   'GAE!" .==7(("7$:M:MMF IIeSF^r?docs r*split_documentsTextSplitter.split_documents]sMryC LL)) *   S\\ *$$U$@@r-cxURU5nUR(aUR5nUS:XagU$)N)joinr!strip)r"docs separatorr0s r* _join_docsTextSplitter._join_docses3~~d#  ! !::>B>N>N=OQ{#a'//+AC C( "5"55 [9IA9M STU**+!AI!6!6{1~!F-0-=-AMq"'2!"o  "5"55 [9IA9M STU**+!AI   q ! c+.>.B]J JE9:ook5 ? KK  r-c ^SSKJn [TU5(d Sn[U5eSU4SjjnU"SSU0UD6$![a Sn[U5ef=f) z>Text splitter that uses HuggingFace tokenizer to count length.r)PreTrainedTokenizerBasezATokenizer received was not an instance of PreTrainedTokenizerBasec8>[TRU55$N)r7tokenizer0 tokenizers r*_huggingface_tokenizer_lengthNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengths9--d344r-z`Could not import transformers python package. Please install it with `pip install transformers`.r%r0strreturnintr/)$transformers.tokenization_utils_baserc isinstancer ImportError)clsrhkwargsrcr)ris ` r*from_huggingface_tokenizer'TextSplitter.from_huggingface_tokenizersp " Ti)@AAW!o% 5K#@KFKK  "E S/ !  "s ,9Agpt2allc ^^^ SSKnUbURU5m OURU5m SUUU 4Sjjn[ U[ 5(a UUTTS.n 0UEU EnU"SSU0UD6$![a Sn[U5ef=f) z9Text splitter that uses tiktoken encoder to count length.rNzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.c8>[TRUTTS95$N)allowed_specialdisallowed_special)r7encode)r0r{r|encs r*_tiktoken_encoder=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders* $3'9 r-) encoding_name model_namer{r|r%rkr/)tiktokenrqencoding_for_model get_encoding issubclassTokenTextSplitter) rrrrr{r|rsrr)r extra_kwargsr~s `` @r*from_tiktoken_encoder"TextSplitter.from_tiktoken_encoders #   !--j9C'' 6C   c, - -!.(#2&8 L 0/,/F?#4???? #A  c" "  #s A%%A=c 6UR[U55$)z2Transform sequence of documents by splitting them.)rLlist)r"rArss r*transform_documents TextSplitter.transform_documentss##DO44r-)r rrrrr!)r#rnr$rnr%zCallable[[str], int]r&z$Union[bool, Literal['start', 'end']]r'boolr(rrmNoner0rlrm list[str]re)r>rr?zOptional[list[dict[Any, Any]]]rmlist[Document])rAzIterable[Document]rmr)rRrrSrlrm Optional[str])rZz Iterable[str]rSrlrmr)rhr rsr rmr)rrztype[TS]rrlrrr{'Union[Literal['all'], AbstractSet[str]]r|&Union[Literal['all'], Collection[str]]rsr rmr)rASequence[Document]rsr rmr)__name__ __module__ __qualname____firstlineno____doc__r7r+rr1rHrLrTr` classmethodrtsetrr__static_attributes__r/r-r*rrsQ3 03?D %!%&2&2&2. &2 = &2  &2&2 &2P33MQ+I (A(TLL,$$(CF5EJ *@ *@*@"*@A *@ C *@  *@ *@*@X5+57:5 5r-cf^\rSrSrSrSS\"5S4S U4SjjjrS SjrSrU=r $) rz/Splitting text to tokens using model tokenizer.rvNrwc >[T U]"S0UD6 SSKnUbUR U5nOUR U5nXlX0lX@lg![a Sn[U5ef=f)zCreate a new TextSplitter.rNzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.r/) superr+rrqrr _tokenizer_allowed_special_disallowed_special) r"rrr{r|rsrr)r~ __class__s r*r+TokenTextSplitter.__init__s "6" #   !--j9C'' 6C /#5  #A  c" "  #s AA(c^SU4Sjjn[TRTRTRRUS9n[ XS9$)a/Splits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text (str): The input text to be split into smaller chunks. Returns: List[str]: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. cb>TRRUTRTRS9$rz)rr}rr)_textr"s r*_encode-TokenTextSplitter.split_text.._encodes4??)) $ 5 5#'#;#;* r-)r$tokens_per_chunkdecoder}rg)rrlrmz list[int]) Tokenizerrrrrsplit_text_on_tokens)r"r0rrhs` r*r1TokenTextSplitter.split_textsC  --!--??))  $CCr-)rrr) rrlrrr{rr|rrsr rmrr) rrrrrrr+r1r __classcell__)rs@r*rrsh9$$(CF5EJ 66"6A 6 C 6  6 668DDr-rc\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSr g) Languagei"z"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6r/N)!rrrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6rr/r-r*rr"s, C B D F B B C E F C D D E EH E D C F E A C DG FJ!Lr-rT)frozencH\rSrSr%SrS\S'S\S'S\S'S\S 'S rg ) riBzTokenizer data class.rnr$rzCallable[[list[int]], str]rzCallable[[str], list[int]]r}r/N)rrrrr__annotations__rr/r-r*rrBs)*, &&= &&=r-rc/nURU5nSn[XAR-[U55nX4UnU[U5:aUR UR U55 U[U5:XaU$XARUR - - n[XAR-[U55nX4UnU[U5:aMU$)z6Split incoming text and return chunks using tokenizer.r)r}minrr7r=rr$)r0rhrZ input_ids start_idxcur_idx chunk_idss r*rrPsF  &II)888#i.IGG,I c)n $ i&&y12 c)n $  M //)2I2III i"<"<rs" #::.!G   8 $ T(E5*CE5P=D =D@"sD"@ $ > > >r-