ScaleDown middleware integration

Middleware integrations for ScaleDown Constrained Decoding Models. ScaleDown Constrained Decoding Models are trained for one narrow task each, so they handle the high-volume steps around an agent, such as compacting history or reading a long tool result, faster and more cheaply than a second call to a general-purpose model. The agent keeps its own model for planning and generation. Learn more about middleware.

All ScaleDown middleware fails open. If ScaleDown is unreachable, out of credits, or rejects a request, the agent runs as if the middleware were not there, and a warning is logged.

Setup

To use the ScaleDown middleware, create a ScaleDown account, get an API key from the ScaleDown dashboard, and install the langchain-scaledown package. New accounts include 50 million free tokens.

Credentials

import getpass
import os

if "SCALEDOWN_API_KEY" not in os.environ:
    os.environ["SCALEDOWN_API_KEY"] = getpass.getpass("Enter your ScaleDown API key: ")

Every middleware reads SCALEDOWN_API_KEY. To point at a different API host, set SCALEDOWN_BASE_URL. To share one connection across middleware, create a ScaledownClient and pass it as client= to each one.

It is also helpful (but not required) to set up LangSmith for observability and tracing. To enable automated tracing, set your LangSmith API key:

os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")
os.environ["LANGSMITH_TRACING"] = "true"

Installation

The ScaleDown middleware lives in the langchain-scaledown package:

pip install -U langchain-scaledown

uv add langchain-scaledown

Summarization

Long-running agents run out of context. Summarizing older history with another call to a general-purpose model adds cost and latency every time the history fills up. ScaledownSummarizer writes the summary with ScaleDown instead, and plugs into the built-in SummarizationMiddleware through its summarizer argument. The built-in middleware still decides when to summarize and which recent messages to keep.

The default summary has the same sections as the LangChain summary prompt: session intent, summary, artifacts, and next steps.

from langchain.agents import create_agent
from langchain.agents.middleware import SummarizationMiddleware
from langchain_scaledown import ScaledownSummarizer

agent = create_agent(
    model="claude-sonnet-4-6",
    tools=[read_file, write_file, run_tests],
    middleware=[
        SummarizationMiddleware(
            model="claude-sonnet-4-6",  # Used for its context window and token counting
            trigger=("fraction", 0.8),
            keep=("fraction", 0.1),
            trim_tokens_to_summarize=None,  # Send all older history to ScaleDown
            summarizer=ScaledownSummarizer(),  # [!code highlight]
        ),
    ],
)

ScaledownSummarizationMiddleware is a shortcut for the same setup, with one difference: if the ScaleDown call fails, it leaves the history as is and the model is called normally.

from langchain.agents import create_agent
from langchain_scaledown import ScaledownSummarizationMiddleware

agent = create_agent(
    model="claude-sonnet-4-6",
    tools=[read_file, write_file, run_tests],
    middleware=[
        ScaledownSummarizationMiddleware(  # [!code highlight]
            "claude-sonnet-4-6",
            trigger=("fraction", 0.8),
            keep=("fraction", 0.1),
        ),
    ],
)

The Deep Agents summarization middleware accepts the same summarizer argument. It also saves the evicted history to a file the agent can read back.

Context compression

ScaledownCompressionMiddleware compresses a large system prompt relative to the user's latest question before each model call. The system message is the context that gets compressed, and the last user message is the query that decides what to keep. The user message and the conversation history pass through unchanged.

Use it for needle-in-a-haystack workloads, such as retrieval-augmented generation (RAG), where a big block of retrieved documents holds a few relevant facts. Do not use it for short prompts or creative generation, because compression can drop instructions, tone, or detail the model needs. With Deep Agents, the system message also holds the harness's own instructions, which compression would condense too.

from langchain.agents import create_agent
from langchain_scaledown import ScaledownCompressionMiddleware

agent = create_agent(
    model="claude-sonnet-4-6",
    system_prompt=f"Answer using these documents:\n\n{retrieved_docs}",
    middleware=[ScaledownCompressionMiddleware(min_context_chars=2000)],  # [!code highlight]
)

The request passes through unchanged when the system message is shorter than min_context_chars, there is no user message, or the ScaleDown call fails.

Tool result compression

Ask a support agent about the refund policy for EU orders, and it can pull the store's entire policy page into context. Every later step pays for those tokens again. ScaledownToolCompressionMiddleware adds an optional compression_query parameter to your tools. When the model needs only part of a result, it fills in compression_query, for example "refund policy for EU orders". The middleware removes the parameter before the tool runs, then compresses the output to the relevant parts before the model sees it.

from langchain.agents import create_agent
from langchain_scaledown import ScaledownToolCompressionMiddleware

agent = create_agent(
    model="claude-sonnet-4-6",
    tools=[search_docs, fetch_url],
    middleware=[ScaledownToolCompressionMiddleware(tools=["fetch_url"])],  # [!code highlight]
)

Your tools do not change, so this works with any tool, including MCP tools. The compressed result starts with a note so the model can call the tool again without compression_query if it needs the full output:

[Compressed by ScaleDown for "refund policy for EU orders": 18,402 -> 3,110 chars. Call this tool again without `compression_query` for the full output.]

The result passes through unchanged when the model omits compression_query, the output is shorter than min_result_chars, the result is an error or contains non-text content, or the ScaleDown call fails.

Extraction

Agents often need a few facts from the user before they can act, such as an order number, the error they saw, or an email address. ScaledownExtractionMiddleware extracts them with ScaleDown instead of asking the agent's model on every turn. Once per run, before the first model call, it sends the user's messages and your schema to ScaleDown and stores the result in agent state under scaledown_extraction. Tools can read it during the run, and it is in the final output of agent.invoke().

Only the user's messages are used, never the agent's own replies, so every extracted value is something the user said. Describe each field in plain language:

from langchain.agents import create_agent
from langchain.tools import ToolRuntime, tool
from langchain_scaledown import ScaledownExtractionMiddleware

ticket = ScaledownExtractionMiddleware(
    entities={
        "order_id": "Order number the customer mentions",
        "error_message": "Exact error message or code the customer saw",
        "steps_tried": "What the customer says they already tried",
        "customer_email": "Customer's email address",
    },
    context_chars=0,  # Skip evidence snippets to keep state small
)


@tool
def file_ticket(runtime: ToolRuntime) -> str:
    """File a support ticket with the details gathered so far."""
    fields = runtime.state.get("scaledown_extraction", {}).get("fields", {})
    return crm.create_ticket(**fields)  # Your ticketing system


agent = create_agent(
    model="claude-sonnet-4-6",
    tools=[file_ticket],
    middleware=[ticket],  # [!code highlight]
)

The record has one entry per field. Fields that were not found are None:

result["scaledown_extraction"]["fields"]
# {
#   "order_id": "A-1042",
#   "error_message": "ERR_PAYMENT_DECLINED",
#   "steps_tried": "tried a different card",
#   "customer_email": None,
# }

With a checkpointer, extraction re-runs every turn over the whole conversation, so the record fills in as the user gives more detail. Extracted values are user text. If you set inject_into_prompt=True so the model can see the fields, that text is added to the system prompt, so only enable it for fields where that is acceptable.

API reference

For the source and full parameter documentation of every ScaleDown class, see the langchain-scaledown repository.