tjudZddlmZddlmZmZddlmZmZm Z  ddl m Z dZ n #e $rdZ YnwxYwGdd eZd S) z$Sentence transformers text splitter.) annotations)Anycast) TextSplitter Tokenizersplit_text_on_tokens)SentenceTransformerTFc^eZdZUdZ ddfd ZddZddZddZdZde d<d dZ xZ S)!%SentenceTransformersTokenTextSplitterz8Splitting text to tokens using sentence model tokenizer.2'sentence-transformers/all-mpnet-base-v2N chunk_overlapint model_namestrtokens_per_chunk int | None model_kwargsdict[str, Any] | NonekwargsrreturnNonec tjdi|d|itsd}t|||_t |jfi|pi|_|jj|_||dS)aTCreate a new `TextSplitter`. Args: chunk_overlap: The number of tokens to overlap between chunks. model_name: The name of the sentence transformer model to use. tokens_per_chunk: The number of tokens per chunk. If `None`, uses the maximum tokens allowed by the model. model_kwargs: Additional parameters for model initialization. Parameters of sentence_transformers.SentenceTransformer can be used. Raises: ImportError: If the `sentence_transformers` package is not installed. rzCould not import sentence_transformers python package. This is needed in order to use SentenceTransformersTokenTextSplitter. Please install it with `pip install sentence-transformers`.)rN) super__init___HAS_SENTENCE_TRANSFORMERS ImportErrorrr _model tokenizer_initialize_chunk_configuration)selfrrrrrmsg __class__s l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/langchain_text_splitters/sentence_transformers.pyrz.SentenceTransformersTokenTextSplitter.__init__s, ??6??????) #N  c"" "$)$/RRl>PbRR . ,,>N,OOOOOc|jj|_| |j|_n||_|j|jkr*d|jd|jd|jd}t |dS)NzThe token limit of the models 'z' is: z. Argument tokens_per_chunk=z > maximum token limit.)rmax_seq_lengthmaximum_tokens_per_chunkrr ValueError)r"rr#s r%r!zESentenceTransformersTokenTextSplitter._initialize_chunk_configuration<s(, (B%  #$($AD ! !$4D !  4#@ @ @+$/++5++.2.C+++  S// ! A @r&text list[str]cdfd }tjjjj|}t ||S) aSplits the input text into smaller components by splitting text on tokens. This method encodes the input text using a private `_encode` method, then strips the start and stop token IDs from the encoded result. It returns the processed segments as a list of strings. Args: text: The input text to be split. Returns: A list of string components derived from the input text after encoding and processing. r+rr list[int]c>|ddS)N)_encode)r+r"s r%%encode_strip_start_and_stop_token_idsz_SentenceTransformersTokenTextSplitter.split_text..encode_strip_start_and_stop_token_ids\s<<%%ad+ +r&)rrdecodeencode)r+r r+rrr.)r_chunk_overlaprr r4r)r"r+r3r s` r% split_textz0SentenceTransformersTokenTextSplitter.split_textMsc , , , , , ,-!2>(8    $CCCCr&cFt||S)anCounts the number of tokens in the given text. This method encodes the input text using a private `_encode` method and calculates the total number of tokens in the encoded result. Args: text: The input text for which the token count is calculated. Returns: The number of tokens in the encoded text. )lenr2)r"r+s r% count_tokensz2SentenceTransformersTokenTextSplitter.count_tokenshs4<<%%&&&r&l _max_length_equal_32_bit_integerr.cf|j||jd}td|S)Ndo_not_truncate) max_length truncationr.)r r5r<r)r"r+&token_ids_with_start_and_end_token_idss r%r2z-SentenceTransformersTokenTextSplitter._encodexs?151F1F <(2G2 2 . K!GHHHr&)r r NN) rrrrrrrrrrrr)rrrr)r+rrr,)r+rrrr6) __name__ __module__ __qualname____doc__rr!r8r;r<__annotations__r2 __classcell__)r$s@r%r r sBB C'+.2 #P#P#P#P#P#P#PJ"""""DDDD6 ' ' ' '-2$1111IIIIIIIIr&r N)rE __future__rtypingrrlangchain_text_splitters.baserrrsentence_transformersr rrr rr&r%rLs**""""""WWWWWWWWWW'"&'''!&'jIjIjIjIjILjIjIjIjIjIs %//