Databricks and Langfuse

What is Databricks? > Databricks is a unified analytics platform founded by the creators of Apache Spark. It provides an interactive workspace for collaborative data engineering, machine learning, and data analytics. With Databricks, teams can build, train, and deploy models at scale, efficiently harnessing big data and advanced analytics tools.

What is Langfuse? > Langfuse is a comprehensive platform designed to help developers monitor, trace, and evaluate their language models in production. It offers powerful insights through detailed logging and event tracing, ensuring robust performance monitoring and easier debugging of AI applications. Use Langfuse Cloud (hosted by Langfuse, free tier, no infrastructure to run) or self-host it.

Tracing and Observability

Databricks serving endpoints expose an OpenAI-compatible API, so you can trace them with Langfuse in three ways: via the OpenAI SDK, via LangChain, or via LlamaIndex.

For all three approaches, configure your Langfuse and Databricks credentials as environment variables:

import os

# Get keys for your project from the project settings page: https://cloud.langfuse.com
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_BASE_URL"] = "https://cloud.langfuse.com"  # 🇪🇺 EU region
# Other Langfuse data regions include 🇺🇸 US: https://us.cloud.langfuse.com, 🇯🇵 Japan: https://jp.cloud.langfuse.com and ⚕️ HIPAA: https://hipaa.cloud.langfuse.com

os.environ["DATABRICKS_TOKEN"] = "dapi-..."  # Databricks personal access token
os.environ["DATABRICKS_HOST"] = "https://dbc-XXXXX-XXXX.cloud.databricks.com"  # Databricks workspace URL

pip install langfuse openai

import os
from langfuse.openai import OpenAI

# Create an OpenAI-like client pointing to Databricks
client = OpenAI(
    api_key=os.environ.get("DATABRICKS_TOKEN"),
    base_url=f"{os.environ.get('DATABRICKS_HOST')}/serving-endpoints",
)

response = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are an AI assistant."},
        {"role": "user", "content": "What is Databricks?"},
    ],
    model="mistral-7b",  # Adjust based on your Databricks serving endpoint name
    max_tokens=256,
)
print(response.choices[0].message.content)

pip install langfuse databricks-langchain

from databricks_langchain import ChatDatabricks
from langfuse.langchain import CallbackHandler

# Initialize Langfuse CallbackHandler for LangChain (tracing)
langfuse_handler = CallbackHandler()

chat_model = ChatDatabricks(
    endpoint="mistral-7b",  # Your Databricks Model Serving endpoint name
    temperature=0.1,
    max_tokens=256,
)

messages = [
    ("system", "You are a chatbot that can answer questions about Databricks."),
    ("user", "What is Databricks Model Serving?"),
]

chat_model.invoke(messages, config={"callbacks": [langfuse_handler]})

pip install langfuse llama-index llama-index-llms-databricks openinference-instrumentation-llama-index

import os
from langfuse import get_client
from llama_index.core.llms import ChatMessage
from llama_index.llms.databricks import Databricks
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor

langfuse = get_client()

# Initialize LlamaIndex instrumentation
LlamaIndexInstrumentor().instrument()

# Create a Databricks LLM instance
llm = Databricks(
    model="mistral-7b",  # Your Databricks serving endpoint name
    api_key=os.environ.get("DATABRICKS_TOKEN"),
    api_base=f"{os.environ.get('DATABRICKS_HOST')}/serving-endpoints/",
)

messages = [
    ChatMessage(role="system", content="You are a helpful assistant."),
    ChatMessage(role="user", content="What is Databricks?"),
]

response = llm.chat(messages)
print(response)

Playground & Evaluations

This guide walks you through integrating Databricks language model endpoints with Langfuse. By doing so, you can quickly experiment with prompts and debug interactions using the Langfuse Playground, as well as benchmark your models systematically with Evaluations.

With Langfuse, you can:

• Experiment in the Playground: The interactive Playground lets you test your language models in real-time. You can send custom prompts, review detailed responses, and add prompts to your Prompt Library.
• Benchmark with Evaluations: LLM-as-a-Judge evaluations provide a way to benchmark your application's performance. You can run pre-defined test templates, analyze metrics like latency and accuracy, and refine your models based on measurable outcomes.

Set Up a Serving Endpoint in Databricks

Begin by setting up a serving endpoint in Databricks. This lets you query custom fine-tuned models or models served via a gateway such as OpenAI or Anthropic. For advanced configuration options, refer to the Databricks docs.

Add the Model in your Project Settings

Next, add your Databricks model endpoint to your Langfuse project settings.

Make sure you've entered the correct endpoint URL and authentication details. The model name is the name of the serving endpoint you created in Databricks.

Use the Model in the Playground

The Langfuse Playground offers an interactive interface where you can:

• Send prompts and view quick results.
• Add prompts to your Prompt Library.

Select Databricks as your LLM provider and choose the endpoint you configured earlier.

Use the Model for Evaluations

LLM-as-a-judge is a technique to evaluate the quality of LLM applications by using an LLM as a judge. The LLM is given a trace or a dataset entry and asked to score and reason about the output. The scores and reasoning are stored as scores in Langfuse.

If you want to learn more about LLM Evals, check out our blog post:

LLM Evaluation 101: Best Practices and Challenges