Skip to main content

NVIDIA middleware integration

Middleware integrations and model-specific harness profiles for NVIDIA services and models. Use them to route calls across LangChain chat models or optimize Deep Agents behavior for NVIDIA Nemotron 3 Ultra. Learn more about middleware.

To use Nemotron 3.5 Super VL with the default Deep Agents harness, see the ChatNVIDIA examples.

Overview

IntegrationDescription
Deep Agents profile for Nemotron 3 UltraApply model-specific prompts and middleware for more reliable agentic behavior
Model routing with NeMo SwitchyardRoute each model call to the target selected by a configured Switchyard algorithm

Optimize Deep Agents for Nemotron 3 Ultra

Deep Agents includes a built-in harness profile for NVIDIA Nemotron 3 Ultra. The profile adds model-specific prompt guidance, tool descriptions, and middleware for tool calling, filesystem operations, retries, context management, and final answers. It was developed through evaluation-driven harness tuning.

Note

The built-in Nemotron 3 Ultra harness profile requires deepagents>=0.7.0.

What the profile changes

  • Repairs common filesystem tool arguments and normalizes empty tool results.
  • Adds continuation guidance when read_file returns a full page of results.
  • Retries selected filesystem failures and model rate limits.
  • Normalizes ChatNVIDIA messages, reasoning tags, and text-formatted tool calls.
  • Adds progress, follow-up, entity-resolution, and final-answer safeguards.

Setup

Install Deep Agents and the NVIDIA chat model integration:

pip install -U "deepagents>=0.7.0" langchain-nvidia-ai-endpoints

Configure NVIDIA_API_KEY as described in the ChatNVIDIA setup guide.

Use the built-in profile

Create a deep agent with the Nemotron 3 Ultra model. Deep Agents recognizes the model and applies the profile automatically, so you do not need to instantiate or pass its middleware:

from deepagents import create_deep_agent
from langchain_nvidia_ai_endpoints import ChatNVIDIA

model = ChatNVIDIA(model="nvidia/nemotron-3-ultra-550b-a55b")
agent = create_deep_agent(model=model)

result = agent.invoke({
    "messages": [
        {
            "role": "user",
            "content": "Inspect this project and summarize its architecture.",
        }
    ]
})
print(result["messages"][-1].content_blocks)

The profile is registered for Nemotron 3 Ultra model identifiers served through NVIDIA, Baseten, Fireworks, OpenRouter, Nebius, and Together. Registration is model-specific, so other models from these providers remain unchanged.

API reference

Model routing with NeMo Switchyard

The experimental SwitchyardRoutingMiddleware routes each deep agent model call through a configured NeMo Switchyard libsy algorithm. Use it to combine LangChain chat models with different cost and performance profiles while Deep Agents continues to manage the agent loop, tools, state, and middleware composition.

Warning

This integration is experimental. Its APIs and behavior are subject to breaking changes without notice. Review the current limitations before using it in an application.

Features

  • Adapt any BaseChatModel as a Switchyard target with LangChainLlmClient.
  • Use any algorithm exposed by the installed nemo-switchyard Python bindings.
  • Preserve Deep Agents tool binding, callbacks, LangChain tracing, and structured output.
  • Inspect the selected model and complete routing trace on each returned AIMessage.
  • Route synchronous and asynchronous agent calls through the same algorithm.

Setup

The integration requires Python 3.12 or newer, a source checkout of NeMo Switchyard, and a checkout of the langchain-nvidia repository. Configure credentials for every chat model that the router can select. The middleware itself does not require a provider API key.

Installation

Build Switchyard's libsy Python bindings from source, then install the integration with its Deep Agents and OpenRouter dependencies:

git clone https://github.com/NVIDIA-NeMo/Switchyard.git
python -m pip install -e ./Switchyard

git clone https://github.com/langchain-ai/langchain-nvidia.git
python -m pip install -e "./langchain-nvidia/libs/switchyard[openrouter]"

The OpenRouter example requires OPENROUTER_API_KEY. Confirm that your account can access both configured models before running it.

Note

SwitchyardRoutingMiddleware requires deepagents>=0.7.4.

Instantiation

Create two LangChain chat models, adapt them as Switchyard targets, and construct the routing middleware. The order of the targets passed to stage_router matters: pass the capable target first and the efficient target second.

from langchain_openrouter import ChatOpenRouter

from switchyard.libsy import LlmTarget, algorithms
from langchain_nvidia_switchyard import LangChainLlmClient, SwitchyardRoutingMiddleware

efficient_model = ChatOpenRouter(model="nvidia/nemotron-3-ultra-550b-a55b")
capable_model = ChatOpenRouter(model="anthropic/claude-sonnet-4.6")

router = algorithms.stage_router(
    LlmTarget("capable", LangChainLlmClient(capable_model)),
    LlmTarget("efficient", LangChainLlmClient(efficient_model)),
    picker="efficient_first",
    confidence_threshold=0.5,
    recent_window=3,
)

middleware = SwitchyardRoutingMiddleware(router)

Stage routing is signal-driven. It can route ordinary turns to the efficient target and escalate turns with critical failed-tool signals to the capable target without making a separate judge-model call.

Use with a deep agent

Pass the middleware to create_deep_agent. Deep Agents requires a base model, but the middleware replaces it for each routed call. Reuse one configured target to avoid constructing an unused model.

from deepagents import create_deep_agent

agent = create_deep_agent(
    model=efficient_model,
    middleware=[middleware],
)

result = await agent.ainvoke({
    "messages": [
        {
            "role": "user",
            "content": "Summarize the important files in this project.",
        }
    ]
})

The asynchronous ainvoke path is canonical. Use agent.invoke(...) in ordinary synchronous applications, but use await agent.ainvoke(...) in notebooks, async web handlers, and async tests.

Choose a routing algorithm

SwitchyardRoutingMiddleware accepts an opaque switchyard.libsy.Algorithm, so the middleware does not depend on a concrete routing strategy. Choose from the algorithms exposed by the installed bindings:

  • Stage routing: Route from message-history signals, including assistant tool calls and tool results, without a judge by default.
  • LLM task classifier: Call a judge model before selecting the efficient or capable target.
  • Random routing: Select among weighted targets, with an optional seed for a reproducible process-local selection sequence.
  • No-op: Check the middleware boundary without a provider call.

Inspect routing decisions

Every routed AIMessage contains the complete ordered decision trace in response_metadata["switchyard"]:

from langchain.messages import AIMessage

message = next(
    message
    for message in reversed(result["messages"])
    if isinstance(message, AIMessage)
)

routing = message.response_metadata["switchyard"]
print(routing["selected_model"])
print(routing["decisions"])

Some algorithms make more than one decision. Use decisions for the full trace; selected_model contains only the final selection.

API reference

See also