j>ddlmZddlmZddlmZGddeZy)) annotations)Any) TextSplittercJeZdZdZ ddd dfdZddZxZS) NLTKTextSplitterz"Splitting text using NLTK package.F)use_span_tokenizec t|di|||_||_||_|jr|jdk7r d}t | ddl}|jr+|jj|j|_ y|jj|_ y#t$r}d}t||d}~wwxYw)zInitialize the NLTK splitter.z6When use_span_tokenize is True, separator should be ''rNzANLTK is not installed, please install it with `pip install nltk`.) super__init__ _separator _language_use_span_tokenize ValueErrornltktokenize_get_punkt_tokenizer _tokenizer sent_tokenize ImportError) self separatorlanguagerkwargsmsgrerr __class__s g/var/www/html/Fitness-lenito-AI-main/venv/lib/python3.12/site-packages/langchain_text_splitters/nltk.pyr zNLTKTextSplitter.__init__ s "6"#!"3  " "t"'<JCS/ ! , &&"&--"D"DT^^"T"&--"="= ,UCc" + ,s:B& B&& C/B==Ccj|jrot|jj|}g}t |D]:\}\}}|dkDr||dz d}||||||z}n|||}|j |<n|j||j }|j||jS)z&Split incoming text and return chunks.r)r) rlistr span_tokenize enumerateappendr _merge_splitsr) rtextspanssplitsistartendprev_endsentences r split_textzNLTKTextSplitter.split_text&s  " "66t<=EF#,U#3r>s"6-;|-;r0