tjHf~dZddlmZddlZddlmZmZddlmZm Z Gdde Z ddZ Gdde Z dS)zCharacter text splitters.) annotationsN)AnyLiteral)Language TextSplitterc2eZdZdZ ddfd ZddZxZS)CharacterTextSplitterz(Splitting text that looks at characters. F separatorstris_separator_regexboolkwargsrreturnNonec Vtjdi|||_||_dS)Create a new TextSplitter.N)super__init__ _separator_is_separator_regex)selfr r r __class__s `/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_text_splitters/character.pyrzCharacterTextSplitter.__init__s7 ""6"""##5   text list[str]cjrjntjj}t ||j}d}jot fd|D}d}js |sj}||S)zSplit into chunks without re-inserting lookaround separators. Args: text: The text to split. Returns: A list of text chunks. keep_separator)z(?=z(?z3CharacterTextSplitter.split_text...sD9 9 ./DO & &q ) )9 9 9 9 9 9 r)rrreescape_split_text_with_regex_keep_separatorany _merge_splits)rr sep_patternsplitslookaround_prefixes is_lookaround merge_seps` r split_textz CharacterTextSplitter.split_texts $7 WDOORYt=W=W  ( +d.B    =0 S9 9 9 9 3F9 9 9 6 6  $ (  (I!!&)444r)r F)r r r rrrrrrr rr)__name__ __module__ __qualname____doc__rr3 __classcell__rs@rr r sd22 #( 6 6 6 6 6 6 6!5!5!5!5!5!5!5!5rr rr r r!bool | Literal['start', 'end']rrc|r|rtjd|d||dkr-fdtdtdz dDn)fdtdtdD}tdzdkr |d dz }|dkr g|d n dg|}n%tj||}nt |}d |DS) N()endc8g|]}||dzzSrr$isplits_s r z*_split_text_with_regex..Fs*UUU!'!*wq1u~-UUUrrrBc8g|]}||dzzSrArrCs rrFz*_split_text_with_regex..Hs*VVVqwqzGAEN2VVVrcg|]}||Srr)r$ss rrFz*_split_text_with_regex..Us # # #! #A # # #r)r(splitrangelenlist)rr r!r/rEs @rr*r*=s:  /h/9///66G"U**VUUUuQG q@PRS7T7TUUUUVVVVE!S\\STRRY  4 4A$$Q''$*:::""1%%%%%"&"4"4[*"M"MK '' 444"$K%4 ''****!%!1!1!^!D!DJ '' 3333  -,,[*EEK    , , ,rc8|||jS)zSplit the input text into smaller chunks based on predefined separators. Args: text: The input text to be split. Returns: A list of text chunks obtained after splitting. )r_rW)rrs rr3z)RecursiveCharacterTextSplitter.split_textsd&6777rlanguagerc B||}|d|dd|S)aReturn an instance of this class based on a specific language. This method initializes the text splitter with language-specific separators. Args: language: The language to configure the text splitter for. **kwargs: Additional keyword arguments to customize the splitter. Returns: An instance of the text splitter configured for the specified language. T)rRr r)get_separators_for_language)clsrhrrRs r from_languagez,RecursiveCharacterTextSplitter.from_languages444X>> sLjTLLVLLLrc |tjtjhvrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tj krgdS|tj krgd S|tj krgd S|tj krgd S|tj krgd S|tjkrgd S|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkr*d}d|dd|dd|dd|dd|d d!d"d#d$d%d&d'd(d)d*d+d,d-gS|tjvrd.|d/}t;|d.|d0t=t}t;|)1aNRetrieve a list of separators specific to the given language. Args: language: The language for which to get the separators. Returns: A list of separators appropriate for the specified language. Raises: ValueError: If the language is not implemented or supported. ) class z void z int z float z double  if  for  while  switch  case r rUrVr')  func  var  const  type rorprrrsr rUrVr')rn public protected private  static rorprqrrrsr rUrVr')rnrxryrzz internal z companion z fun  val rurorprqz when rs else r rUrVr') function rv let rurnrorprqrrrs default r rUrVr') enum interface z namespace rwrnr~rvrrurorprqrrrsrr rUrVr') r~rnro foreach rq do rrrsr rUrVr') z message z service rz option  import z syntax r rUrVr')rn def z def r rUrVr')r~z setClass\(z setMethod\(z setGeneric\(ror}rprq repeat z library\(z require\(r rUrVr')z =+ z -+ z \*+ z .. * r rUrVr') rrnro unless rqrprz begin z rescue r rUrVr')rz defp z defmodule z defprotocol z defmacro z defmacrop rorrqrsz cond z with rprr rUrVr') z fn rvrrorqrpz loop  match rvr rUrVr')rnz object rr|rurorprqrrsr rUrVr')rtrn struct rrorprqrrrrsr rUrVr') z #{1,6} z``` z \*\*\*+ z ---+ z ___+ r rUrVr')z \\chapter{z \\section{z \\subsection{z \\subsubsection{z \\begin{enumerate}z \\begin{itemize}z \\begin{description}z \\begin{list}z \\begin{quote}z \\begin{quotation}z \\begin{verse}z \\begin{verbatim}z \\begin{align}z$$$rVr')z x & &#### H x| # # : x~ % % > x| # #  x' ' '%%%% L x* * * . x, , ,?C,s+++0s000BsBBB,s,,,,s,,,#%+ 0 x2 2 2@h@@@CS// ! X X XX X X oor)NTF) rRrSr!r;r rrrrr)rr rRrrrr4)rhrrrrrQ)rhrrr) r5r6r7r8rr_r3 classmethodrl staticmethodrjr9r:s@rrQrQXs(,9=#( 6 6 6 6 6 6 6((((T 8 8 8 8MMM[M"p p p \p p p p p rrQ)rr r r r!r;rr) r8 __future__rr(typingrrlangchain_text_splitters.baserrr r*rQrrrrs"""""" @@@@@@@@/5/5/5/5/5L/5/5/5d$$$$6K K K K K \K K K K K r