ElevenLabs

The ElevenLabs adapter is voice-focused. It exposes five capabilities:

• Realtime voice agents (elevenlabsRealtime / elevenlabsRealtimeToken) — full-duplex voice-to-voice conversations powered by ElevenLabs Conversational AI agents.
• Text-to-speech (elevenlabsSpeech) — one-shot speech generation via generateSpeech().
• Music & sound effects (elevenlabsAudio) — one-shot audio generation via generateAudio().
• Transcription (elevenlabsTranscription) — speech-to-text via generateTranscription().
• Voice design (elevenlabsVoiceDesign): create a new voice via generateVoice().

It does not support text chat() or summarize() — use OpenAI, Anthropic, or Gemini for those.

The realtime adapter uses an agent-based architecture where you configure your conversational AI agent in the ElevenLabs dashboard (voice, personality, knowledge base, tools) and then connect to it at runtime. The adapter wraps the @elevenlabs/client SDK for seamless integration with useRealtimeChat and RealtimeClient.

Installation

npm i @tanstack/ai-elevenlabs

pnpm add @tanstack/ai-elevenlabs

yarn add @tanstack/ai-elevenlabs

bun add @tanstack/ai-elevenlabs

Peer dependencies:

npm i @tanstack/ai @tanstack/ai-client

pnpm add @tanstack/ai @tanstack/ai-client

yarn add @tanstack/ai @tanstack/ai-client

bun add @tanstack/ai @tanstack/ai-client

Server Setup

The server generates a signed WebSocket URL so your API key never reaches the client. The signed URL is valid for 30 minutes.

import { realtimeToken } from '@tanstack/ai'
import { elevenlabsRealtimeToken } from '@tanstack/ai-elevenlabs'

// In your API route (Express, Hono, TanStack Start, etc.)
export async function POST() {
  const token = await realtimeToken({
    adapter: elevenlabsRealtimeToken({
      agentId: process.env.ELEVENLABS_AGENT_ID!,
    }),
  })

  return Response.json(token)
}

With Overrides

You can override agent settings at token generation time without changing your dashboard configuration:

const token = await realtimeToken({
  adapter: elevenlabsRealtimeToken({
    agentId: process.env.ELEVENLABS_AGENT_ID!,
    overrides: {
      voiceId: 'custom-voice-id',
      systemPrompt: 'You are a helpful voice assistant.',
      firstMessage: 'Hello! How can I help you today?',
      language: 'en',
    },
  }),
})

Client Setup

React (useRealtimeChat)

import { useRealtimeChat } from '@tanstack/ai-react'
import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'

function VoiceChat() {
  const {
    status,
    mode,
    messages,
    connect,
    disconnect,
    pendingUserTranscript,
    pendingAssistantTranscript,
    inputLevel,
    outputLevel,
  } = useRealtimeChat({
    getToken: () =>
      fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()),
    adapter: elevenlabsRealtime(),
  })

  return (
    <div>
      <p>Status: {status}</p>
      <p>Mode: {mode}</p>
      <button onClick={status === 'idle' ? connect : disconnect}>
        {status === 'idle' ? 'Start Conversation' : 'End Conversation'}
      </button>
      {pendingUserTranscript && <p>You: {pendingUserTranscript}...</p>}
      {pendingAssistantTranscript && (
        <p>AI: {pendingAssistantTranscript}...</p>
      )}
      {messages.map((msg) => (
        <div key={msg.id}>
          <strong>{msg.role}:</strong>
          {msg.parts.map((part, i) => (
            <span key={i}>
              {part.type === 'text' ? part.content : null}
              {part.type === 'audio' ? part.transcript : null}
            </span>
          ))}
        </div>
      ))}
    </div>
  )
}

Non-React (RealtimeClient)

import { RealtimeClient } from '@tanstack/ai-client'
import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'

const client = new RealtimeClient({
  getToken: () =>
    fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()),
  adapter: elevenlabsRealtime(),
  onMessage: (message) => {
    console.log(`${message.role}:`, message.parts)
  },
  onStatusChange: (status) => {
    console.log('Status:', status)
  },
  onModeChange: (mode) => {
    console.log('Mode:', mode)
  },
})

await client.connect()

Client Tools

ElevenLabs supports client-side tools that execute in the browser. Define tools using the standard toolDefinition() API:

import { toolDefinition } from '@tanstack/ai'
import { useRealtimeChat } from '@tanstack/ai-react'
import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'
import { z } from 'zod'

const getWeatherDef = toolDefinition({
  name: 'getWeather',
  description: 'Get weather for a location',
  inputSchema: z.object({
    location: z.string(),
  }),
  outputSchema: z.object({
    temperature: z.number(),
    conditions: z.string(),
  }),
})

const getWeather = getWeatherDef.client(async ({ location }) => {
  const res = await fetch(`/api/weather?location=${location}`)
  return res.json()
})

// Pass tools to the hook
const chat = useRealtimeChat({
  getToken: () =>
    fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()),
  adapter: elevenlabsRealtime(),
  tools: [getWeather],
})

Tool results are automatically serialized to strings and returned to the ElevenLabs agent. The adapter converts TanStack tool definitions into the @elevenlabs/client clientTools format internally.

Configuration

elevenlabsRealtimeToken Options

Used on the server to generate a signed WebSocket URL.

elevenlabsRealtime Options

Used on the client to establish the connection.

Differences from OpenAI Realtime

ElevenLabs and OpenAI take different approaches to realtime voice:

Audio Visualization

The ElevenLabs adapter provides audio visualization data through the same interface as other realtime adapters:

import { useRealtimeChat } from '@tanstack/ai-react'
import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'

const {
  inputLevel, // 0-1 normalized microphone volume
  outputLevel, // 0-1 normalized speaker volume
  getInputFrequencyData, // Uint8Array frequency spectrum
  getOutputFrequencyData,
} = useRealtimeChat({
  getToken: () =>
    fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()),
  adapter: elevenlabsRealtime(),
})

Note: ElevenLabs provides volume levels and frequency data but does not expose time-domain data. The getInputTimeDomainData() and getOutputTimeDomainData() methods return static placeholder arrays. The default audio sample rate is 16kHz.

Environment Variables

Set these in your server environment:

ELEVENLABS_API_KEY=your-elevenlabs-api-key
ELEVENLABS_AGENT_ID=your-agent-id

Get your API key from the ElevenLabs dashboard. Create and configure agents in the Conversational AI section of the dashboard.

Text-to-Speech

For one-shot speech generation (not realtime), use elevenlabsSpeech with generateSpeech():

The format option supports mp3 (default), pcm, opus, and wav. WAV output contains 44.1 kHz, 16-bit mono PCM with a RIFF header. Requests for aac or flac throw before the API call. An explicit modelOptions.outputFormat overrides format and returns the selected provider format without WAV wrapping.

import { generateSpeech } from "@tanstack/ai";
import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";

const result = await generateSpeech({
  adapter: elevenlabsSpeech("eleven_v4"),
  text: "Hello from ElevenLabs!",
  voice: "Rachel",
  format: "mp3",
});

console.log(result.audio); // Base64-encoded audio

Dialogue and timings

ElevenLabs has a separate dialogue endpoint that takes up to 10 distinct voices, and a timestamped twin of each endpoint. turns and timestamps pick between them, so you never choose an endpoint by hand:

import { generateSpeech } from "@tanstack/ai";
import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";

const result = await generateSpeech({
  adapter: elevenlabsSpeech("eleven_v4"),
  turns: [
    { text: "Knock knock.", voice: "bYTqZQo3Jz7LQtmGTgwi" },
    { text: "Who is there?", voice: "6lCwbsX1yVjD49QmpkTR" },
  ],
  timestamps: true,
});

// Character timings for the whole clip.
console.log(result.alignment?.unit); // 'character'

// One entry per turn, with the voice that spoke it.
for (const segment of result.segments ?? []) {
  console.log(segment.turnIndex, segment.voice, segment.text);
}

segments only comes back from the dialogue endpoint. A single-voice request with timestamps: true returns alignment alone.

Music & Sound Effects

elevenlabsAudio covers both music generation and sound effects depending on the model:

import { generateAudio } from "@tanstack/ai";
import { elevenlabsAudio } from "@tanstack/ai-elevenlabs";

// Music generation
const music = await generateAudio({
  adapter: elevenlabsAudio("music_v2_5"),
  prompt: "An upbeat synthwave track for a product launch",
});

// Sound effects
const sfx = await generateAudio({
  adapter: elevenlabsAudio("eleven_text_to_sound_v2"),
  prompt: "A glass shattering on concrete",
});

Listing Voices

elevenlabsSpeech can read the account's voice catalog, which includes anything generateVoice() has saved:

import { listVoices } from "@tanstack/ai";
import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";

const { voices } = await listVoices({
  adapter: elevenlabsSpeech("eleven_v4"),
  origins: ["generated", "cloned"],
});

ElevenLabs has no origin filter on GET /v1/voices, so the adapter fetches the catalog and narrows in memory. Its famous and high_quality categories both read as professional.

Voice Design

elevenlabsVoiceDesign creates a new voice from a description. The voice IDs it returns go straight into generateSpeech():

import { generateVoice } from "@tanstack/ai";
import { elevenlabsVoiceDesign } from "@tanstack/ai-elevenlabs";

const result = await generateVoice({
  adapter: elevenlabsVoiceDesign("eleven_ttv_v3"),
  prompt: "A warm, gravelly narrator in his sixties with a slight Irish lilt",
  name: "Irish Narrator",
});

const [voice] = result.voices;
if (!voice) throw new Error("The provider returned no voices.");

console.log(voice.voiceId);
console.log(voice.saved); // true, because a name was given

Without a name, you get preview voices to audition. With a name, the best candidate is kept in your ElevenLabs voice library.

eleven_ttv_v3 also accepts referenceAudio, a clip of a real speaker used as a design reference — it still needs a prompt, and modelOptions.promptStrength balances the two. eleven_multilingual_ttv_v2 takes no reference audio. See Voice Creation for the full guide.

Models

ElevenLabs deprecated eleven_turbo_v2, eleven_turbo_v2_5, eleven_monolingual_v1, scribe_v1, and music_v1. The adapter still accepts them, and each is commented as deprecated in model-meta.ts. Move to the current model in the same row.

Transcription

Transcribe audio with elevenlabsTranscription:

import { generateTranscription } from "@tanstack/ai";
import { elevenlabsTranscription } from "@tanstack/ai-elevenlabs";
import { audioFile } from "./audio";

const result = await generateTranscription({
  adapter: elevenlabsTranscription("scribe_v2"),
  audio: audioFile,
});

console.log(result.text);

Behind a proxy

Route every request through a gateway, such as Cloudflare AI Gateway or a corporate proxy, with baseURL and defaultHeaders. These two option names are the same on every TanStack AI adapter, so one gateway config works for all of them.

import { createElevenLabsSpeech } from "@tanstack/ai-elevenlabs";

const adapter = createElevenLabsSpeech("eleven_v4", process.env.ELEVENLABS_API_KEY!, {
  baseURL: "https://gateway.example.com/elevenlabs",
  defaultHeaders: { "cf-aig-authorization": `Bearer ${process.env.GATEWAY_TOKEN}` },
});

This applies to the speech, audio, and transcription adapters. baseUrl and headers are aliases of the same two options. If you set both forms, baseURL and defaultHeaders win.

API Reference

elevenlabsRealtimeToken(options)

Creates an ElevenLabs realtime token adapter for server-side use with realtimeToken().

Parameters:

• options.agentId - Agent ID from the ElevenLabs dashboard
• options.overrides?.voiceId - Custom voice ID
• options.overrides?.systemPrompt - Custom system prompt
• options.overrides?.firstMessage - First message the agent speaks
• options.overrides?.language - Language code

Returns: A RealtimeTokenAdapter for use with realtimeToken().

elevenlabsRealtime(options?)

Creates an ElevenLabs realtime client adapter for use with useRealtimeChat or RealtimeClient.

Parameters:

• options.connectionMode? - 'websocket' or 'webrtc' (default: auto-detect)
• options.debug? - Enable debug logging

Returns: A RealtimeAdapter for use with useRealtimeChat() or RealtimeClient.

elevenlabsSpeech(model, config?) / createElevenLabsSpeech(model, apiKey, config?)

Creates an ElevenLabs text-to-speech adapter for use with generateSpeech().

elevenlabsAudio(model, config?) / createElevenLabsAudio(model, apiKey, config?)

Creates an ElevenLabs audio adapter that covers both music generation and sound effects (selected via the model id) for use with generateAudio().

elevenlabsTranscription(model, config?) / createElevenLabsTranscription(model, apiKey, config?)

Creates an ElevenLabs transcription adapter for use with generateTranscription().

elevenlabsVoiceDesign(model, config?) / createElevenLabsVoiceDesign(model, apiKey, config?)

Creates an ElevenLabs voice-design adapter for use with generateVoice().

elevenlabsSpeech(...).listVoices(options?)

Reads the account's voice catalog via GET /v1/voices, for use with listVoices().

Limitations

• No text chat support -- Use OpenAI, Anthropic, Gemini, or another text adapter for chat().
• No summarization -- Use a text adapter for summarize().
• No image input (realtime) -- ElevenLabs realtime does not support sending images during a conversation.
• No runtime session updates (realtime) -- Session configuration is fixed at connection time.
• No time-domain audio data (realtime) -- Frequency data and volume levels are available, but waveform data is not.
• Agent required (realtime) -- You must create and configure an agent in the ElevenLabs dashboard before using the realtime adapter.

Next Steps

• Realtime Voice Chat Guide - Complete guide to building realtime voice applications
• OpenAI Adapter - Alternative realtime voice provider with WebRTC
• Tools Guide - Learn about the isomorphic tool system