Text splitter integrations

pip install -U langchain-text-splitters

uv add langchain-text-splitters

Text splitters break large docs into smaller chunks that will be retrievable individually and fit within model context window limit.

There are several strategies for splitting documents, each with its own advantages.

Integration components

• Typical retrieval flow: Document loaders → Text splitters (this page) → Embedding models → Vector stores → Retrievers → Chat models
• Agent capabilities: Tools, Middleware
• Execution and state: Sandboxes, Backends, Checkpointers, Stores

Text structure-based

Text is naturally organized into hierarchical units such as paragraphs, sentences, and words. We can leverage this inherent structure to inform our splitting strategy, creating split that maintain natural language flow, maintain semantic coherence within split, and adapts to varying levels of text granularity. LangChain's RecursiveCharacterTextSplitter implements this concept:

• The RecursiveCharacterTextSplitter attempts to keep larger units (e.g., paragraphs) intact.
• If a unit exceeds the chunk size, it moves to the next level (e.g., sentences).
• This process continues down to the word level if necessary.

Example usage:

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=0)
texts = text_splitter.split_text(document)

Available text splitters:

• Recursively split text

Length-based

An intuitive strategy is to split documents based on their length. This simple yet effective approach ensures that each chunk doesn't exceed a specified size limit. Key benefits of length-based splitting:

• Straightforward implementation
• Consistent chunk sizes
• Easily adaptable to different model requirements

Types of length-based splitting:

• Token-based: Splits text based on the number of tokens, which is useful when working with language models.
• Character-based: Splits text based on the number of characters, which can be more consistent across different types of text.

Example implementation using LangChain's CharacterTextSplitter with token-based splitting:

from langchain_text_splitters import CharacterTextSplitter

text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base", chunk_size=100, chunk_overlap=0
)
texts = text_splitter.split_text(document)

Available text splitters:

• Split by tokens
• Split by characters

Document structure-based

Some documents have an inherent structure, such as HTML, Markdown, or JSON files. In these cases, it's beneficial to split the document based on its structure, as it often naturally groups semantically related text. Key benefits of structure-based splitting:

• Preserves the logical organization of the document
• Maintains context within each chunk
• Can be more effective for downstream tasks like retrieval or summarization

Examples of structure-based splitting:

• Markdown: Split based on headers (e.g., #, ##, ###)
• HTML: Split using tags
• JSON: Split by object or array elements
• Code: Split by functions, classes, or logical blocks

Available text splitters:

• Split Markdown
• Split JSON
• Split code
• Split HTML