Skip to main content

LLMs

Overview

CrewAI integrates with multiple LLM providers through providers native sdks, giving you the flexibility to choose the right model for your specific use case. This guide will help you understand how to configure and use different LLM providers in your CrewAI projects.

What are LLMs?

Large Language Models (LLMs) are the core intelligence behind CrewAI agents. They enable agents to understand context, make decisions, and generate human-like responses. Here's what you need to know:

LLM Basics

Large Language Models are AI systems trained on vast amounts of text data. They power the intelligence of your CrewAI agents, enabling them to understand and generate human-like text.

Context Window

The context window determines how much text an LLM can process at once. Larger windows (e.g., 128K tokens) allow for more context but may be more expensive and slower.

Temperature

Temperature is a sampling control supported by some models. Lower values generally make sampling more focused, while higher values increase variability. Some newer reasoning models ignore, deprecate, or reject this parameter, so check the selected model's documentation before setting it.

Provider Selection

Each LLM provider (e.g., OpenAI, Anthropic, Google) offers different models with varying capabilities, pricing, and features. Choose based on your needs for accuracy, speed, and cost.

Setting up your LLM

There are different places in CrewAI code where you can specify the model to use. Once you specify the model you are using, you will need to provide the configuration (like an API key) for each of the model providers you use. See the provider configuration examples section for your provider.

The simplest way to get started. Set the model in your environment directly, through an .env file or in your app code. If you used crewai create to bootstrap your project, it will be set already.

MODEL=provider/model-id  # e.g. openai/gpt-5.6-terra

# Be sure to set your API keys here too. See the Provider
# section below.

Warning

Never commit API keys to version control. Use environment files (.env) or your system's secret management.

Create a YAML file to define your agent configurations. This method is great for version control and team collaboration:

researcher:
    role: Research Specialist
    goal: Conduct comprehensive research and analysis
    backstory: A dedicated research professional with years of experience
    verbose: true
    llm: provider/model-id  # e.g. anthropic/claude-sonnet-4-6
    # (see provider configuration examples below for more)

Info

The YAML configuration allows you to:

  • Version control your agent settings
  • Easily switch between different models
  • Share configurations across team members
  • Document model choices and their purposes

For maximum flexibility, configure LLMs directly in your Python code:

from crewai import LLM

# Basic configuration
llm = LLM(model="provider/model-id")  # e.g. gemini/gemini-3.6-flash

# Advanced configuration with detailed parameters
llm = LLM(
    model="provider/model-id",
    timeout=120,
    max_tokens=4000,
    response_format={"type": "json"},  # For structured outputs
)

Info

Parameter explanations:

  • timeout: Maximum wait time for response
  • max_tokens: Limits response length
  • response_format: Specifies output structure

Sampling controls such as temperature and top_p, penalty parameters, token-limit names, and reasoning controls are model-specific. Add them only when the selected provider and model support them. See the provider examples below and the provider's model documentation.

Info

CrewAI provides native SDK integrations for OpenAI, Anthropic, Google (Gemini API), Azure, AWS Bedrock, and Snowflake Cortex — no extra install needed beyond the provider-specific extras (e.g. uv add "crewai[openai]").

All other providers are powered by LiteLLM. If you plan to use any of them, add it as a dependency to your project:

uv add 'crewai[litellm]'

Provider Configuration Examples

CrewAI supports a multitude of LLM providers, each offering unique features, authentication methods, and model capabilities. In this section, you'll find detailed examples that help you select, configure, and optimize the LLM that best fits your project's needs.

Warning

Model availability changes frequently and can vary by account, region, and cloud platform. The examples below use models that are current at the time of writing, but they are not exhaustive support lists. Before deploying, verify the model ID and lifecycle status in the provider's linked model catalog.

Streaming Responses

CrewAI supports streaming responses from LLMs, allowing your application to receive and process outputs in real-time as they're generated.

Enable streaming by setting the stream parameter to True when initializing your LLM:

from crewai import LLM

# Create an LLM with streaming enabled
llm = LLM(
    model="openai/gpt-5.6-terra",
    stream=True  # Enable streaming
)

When streaming is enabled, responses are delivered in chunks as they're generated, creating a more responsive user experience.

CrewAI emits events for each chunk received during streaming:

from crewai.events import (
  LLMStreamChunkEvent
)
from crewai.events import BaseEventListener

class MyCustomListener(BaseEventListener):
    def setup_listeners(self, crewai_event_bus):
        @crewai_event_bus.on(LLMStreamChunkEvent)
        def on_llm_stream_chunk(self, event: LLMStreamChunkEvent):
          # Process each chunk as it arrives
          print(f"Received chunk: {event.chunk}")

my_listener = MyCustomListener()

Tip

Click here for more details

All LLM events in CrewAI include agent and task information, allowing you to track and filter LLM interactions by specific agents or tasks:

from crewai import LLM, Agent, Task, Crew
from crewai.events import LLMStreamChunkEvent
from crewai.events import BaseEventListener

class MyCustomListener(BaseEventListener):
    def setup_listeners(self, crewai_event_bus):
        @crewai_event_bus.on(LLMStreamChunkEvent)
        def on_llm_stream_chunk(source, event):
            if researcher.id == event.agent_id:
                print("\n==============\n Got event:", event, "\n==============\n")


my_listener = MyCustomListener()

llm = LLM(model="openai/gpt-5.6-terra", stream=True)

researcher = Agent(
    role="About User",
    goal="You know everything about the user.",
    backstory="""You are a master at understanding people and their preferences.""",
    llm=llm,
)

search = Task(
    description="Answer the following questions about the user: {question}",
    expected_output="An answer to the question.",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[search])

result = crew.kickoff(
    inputs={"question": "..."}
)

Info

This feature is particularly useful for:

  • Debugging specific agent behaviors
  • Logging LLM usage by task type
  • Auditing which agents are making what types of LLM calls
  • Performance monitoring of specific tasks

Async LLM Calls

CrewAI supports asynchronous LLM calls for improved performance and concurrency in your AI workflows. Async calls allow you to run multiple LLM requests concurrently without blocking, making them ideal for high-throughput applications and parallel agent operations.

Use the acall method for asynchronous LLM requests:

import asyncio
from crewai import LLM

async def main():
    llm = LLM(model="openai/gpt-4o")

    # Single async call
    response = await llm.acall("What is the capital of France?")
    print(response)

asyncio.run(main())

The acall method supports all the same parameters as the synchronous call method, including messages, tools, and callbacks.

Combine async calls with streaming for real-time concurrent responses:

import asyncio
from crewai import LLM

async def stream_async():
    llm = LLM(model="openai/gpt-4o", stream=True)

    response = await llm.acall("Write a short story about AI")

    print(response)

asyncio.run(stream_async())

Structured LLM Calls

CrewAI supports structured responses from LLM calls by allowing you to define a response_format using a Pydantic model. This enables the framework to automatically parse and validate the output, making it easier to integrate the response into your application without manual post-processing.

Structured output support varies by provider and model. Test your chosen model before relying on structured responses in production.

For example, you can define a Pydantic model to represent the expected response structure and pass it as the response_format when instantiating the LLM. The model will then be used to convert the LLM output into a structured Python object.

from crewai import LLM

class Dog(BaseModel):
    name: str
    age: int
    breed: str


llm = LLM(model="openai/gpt-5.6-terra", response_format=Dog)

response = llm.call(
    "Analyze the following messages and return the name, age, and breed. "
    "Meet Kona! She is 3 years old and is a black german shepherd."
)
print(response)

# Output:
# Dog(name='Kona', age=3, breed='black german shepherd')

Advanced Features and Optimization

Learn how to get the most out of your LLM configuration:

Common Issues and Solutions

Warning

Most authentication issues can be resolved by checking API key format and environment variable names.

# OpenAI
OPENAI_API_KEY=sk-...

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

Check

Always include the provider prefix in model names

# Correct
llm = LLM(model="openai/gpt-4")

# Incorrect
llm = LLM(model="gpt-4")

Tip

Gateways such as OpenRouter return 200 OK as soon as the upstream provider accepts the request, so a provider timeout arrives in the response body instead of the status code.

CrewAI raises the same exception the upstream code would have produced as a real HTTP status, so a masked failure is caught by the retry handling you already have:

import openai
from pydantic import BaseModel

from crewai import LLM


class Report(BaseModel):
    summary: str


llm = LLM(model="openrouter/z-ai/glm-5.3", response_format=Report)

try:
    result = llm.call("Summarize the incident", response_model=Report)
except openai.InternalServerError as e:
    # "z-ai/glm-5.3 via openrouter.ai returned HTTP 200 with an upstream error
    #  and no choices: The operation was aborted (upstream code 504)"
    print(f"Upstream provider failed, safe to retry: {e}")

Warning

A large or deeply nested response_model makes upstream timeouts more likely. Treat these as transient provider failures, not as the model producing malformed structured output.

Tip

Use larger context models for extensive tasks

# Large context model
llm = LLM(model="openai/gpt-4o")  # 128K tokens