Programmatic access to traces

Search, read, and analyze traces programmatically. Use mlflow.search_traces() to query traces stored in Unity Catalog tables, the MLflow tracking server, or inference tables — and, for traces in Unity Catalog, query the Delta tables directly with SQL. Once you have a trace, read its object model — metadata, spans, assessments, and token usage — to inspect what happened. You can select subsets of traces to analyze or to create evaluation datasets.

mlflow.search_traces() API

def mlflow.search_traces(
    experiment_ids: list[str] | None = None,
    filter_string: str | None = None,
    max_results: int | None = None,
    order_by: list[str] | None = None,
    extract_fields: list[str] | None = None,
    run_id: str | None = None,
    return_type: Literal['pandas', 'list'] | None = None,
    model_id: str | None = None,
    sql_warehouse_id: str | None = None,
    include_spans: bool = True,
    locations: list[str] | None = None,
) -> pandas.DataFrame | list[Trace]

mlflow.search_traces() lets you filter and select data along a few dimensions:

• Filter by a query string

• Filter by locations: experiment, run, model, or Unity Catalog schema

• Limit data: max results, include or exclude spans

• Adjust return value format: data format, data order

search_traces() returns either a pandas DataFrame or a list of Trace objects, which can then be analyzed further or reshaped into evaluation datasets. See the schema details of these return types.

See the mlflow.search_traces() API docs for full details.

mlflow.search_traces() parameters

Search query syntax

The filter_string argument uses a SQL-like query language to filter traces. String values must be wrapped in single quotes (for example, trace.status = 'OK'), and numeric values must not be quoted (for example, trace.execution_time_ms > 1000). Combine conditions with AND. The OR operator is not supported.

Supported filters and comparators

The following fields and comparators are supported on Databricks-managed MLflow.

Differences from OSS MLflow

The search query syntax on Databricks-managed MLflow closely tracks OSS MLflow, with the following differences:

Search for third-party OpenTelemetry spans

To search traces ingested from third-party OpenTelemetry tools such as Langfuse, use the span.attributes.* prefix instead. See Search for traces by OTel span attributes .

Query trace tables with SQL

When traces are stored in Unity Catalog, you can query them with Databricks SQL in addition to the SDK. The MLflow service stores span data in OpenTelemetry-compliant tables and automatically creates Databricks SQL views that transform that data into the MLflow format.

Azure Databricks recommends querying the views (or using the SDK) rather than the underlying OpenTelemetry tables, whose schemas can change over time. For large trace volumes, view performance can degrade: create a materialized view over the views and update it incrementally, or use the SDK for best performance on recent data.

{table_prefix}_trace_unified

A unified view of all trace data, grouped by trace ID. Each row holds the raw span data plus the trace metadata (MLflow tags, metadata, and assessments). Top-level columns:

trace_id: STRING
client_request_id: STRING
request_time: TIMESTAMP
state: STRING
execution_duration_ms: DOUBLE
request: STRING
response: STRING
trace_metadata: VARIANT
tags: MAP<STRING, STRING>
spans: LIST<STRUCT>          # per-span records: name, kind, timing, attributes, status, events, links
assessments: LIST<STRUCT>    # feedback and expectation records with source, value, rationale, metadata

The trace_metadata column and span attributes fields are VARIANT. Read them with colon-path syntax and cast to the type you need, rather than with map key lookup:

SELECT spans[0].attributes:`mlflow.spanInputs`::STRING FROM my_catalog.my_schema.my_prefix_trace_unified

{table_prefix}_trace_metadata

Contains only the MLflow tags, metadata, and assessments grouped by trace ID. It is more performant than the unified view when you need MLflow annotation data but not span data. Columns: trace_id, client_request_id, tags, trace_metadata, and assessments (same structure as in the unified view).

Annotation data formats

MLflow annotation entities (metadata, tags, assessments, and run links) are also stored in the {table_prefix}_otel_annotations table, one row per entity with a typed annotation_type (METADATA, TAG, FEEDBACK, EXPECTATION, or RUN_LINK). The table is append-only with soft deletes, so de-duplicate on retrieval: take the latest row per annotation_id (order by updated_at descending) and drop rows where deleted_at is set. The value and metadata columns are VARIANT (JSON). For assessments, user-supplied metadata sits alongside internal MLflow keys (prefixed mlflow.); ignore the internal keys when reading user metadata.

Analyze query performance

To diagnose slow queries, inspect query profiles in the SQL warehouse query history: open the SQL warehouses page, select your warehouse, and click Query history. Filter for queries with MLflow as the source, then open a query to view its profile and check:

• Scheduling time: High scheduling time means queries are queued behind heavy warehouse load. Switch to a different warehouse in the MLflow UI, or configure a different warehouse in your client.

• Overall query performance: For consistently slow queries, use a larger SQL warehouse, tighten the bounds on trace.timestamp_ms, and remove other filter predicates where possible.

Read trace data

An MLflow Trace has two components:

• TraceInfo, metadata about the trace:

  • ID and status
  • Preview data
  • Timing
  • Tags
  • Token usage
  • Assessments

• TraceData, the execution data:

  • Spans
  • Full request and response data

Basic metadata properties

# Primary identifiers
print(f"Trace ID: {trace.info.trace_id}")
print(f"Client Request ID: {trace.info.client_request_id}")

# Status information
print(f"State: {trace.info.state}")  # OK, ERROR, IN_PROGRESS

# Request/response previews (truncated)
print(f"Request preview: {trace.info.request_preview}")
print(f"Response preview: {trace.info.response_preview}")

Storage location and experiment

location = trace.info.trace_location
print(f"Location type: {location.type}")

# Stored in Unity Catalog (recommended)
if location.uc_table_prefix:
    print(f"UC location: {location.uc_table_prefix.full_table_prefix}")

# Stored in an MLflow experiment
if location.mlflow_experiment:
    print(f"Experiment ID: {trace.info.experiment_id}")

# Stored in a Databricks inference table
if location.inference_table:
    print(f"Table: {location.inference_table.full_table_name}")

The experiment is the UI entry point regardless of backend. Use trace.info.experiment_id to open the trace in the MLflow UI even when it is stored in Unity Catalog.

Request and response previews

The request_preview and response_preview properties provide truncated summaries of the full request and response data, so you can understand what happened without loading the complete payloads.

request_preview = trace.info.request_preview
response_preview = trace.info.response_preview

# Full request/response data (see below)
full_request = trace.data.request
full_response = trace.data.response

Time-related properties

# Timestamps (milliseconds since epoch)
print(f"Start time (ms): {trace.info.request_time}")
print(f"Timestamp (ms): {trace.info.timestamp_ms}")  # Alias for request_time

# Duration
print(f"Execution duration (ms): {trace.info.execution_duration}")

# Convert to human-readable format
import datetime
start_time = datetime.datetime.fromtimestamp(trace.info.request_time / 1000)
print(f"Started at: {start_time}")

Tags and metadata

# Tags (mutable, can be updated after creation)
for key, value in trace.info.tags.items():
    print(f"  {key}: {value}")

print(f"Environment: {trace.info.tags.get('environment')}")

# Trace metadata (immutable, set at creation)
for key, value in trace.info.trace_metadata.items():
    print(f"  {key}: {value}")

Token usage information

MLflow Tracing can track token usage of LLM calls, using token counts returned by LLM provider APIs.

# Get aggregated token usage (if available)
token_usage = trace.info.token_usage
if token_usage:
    print(f"Input tokens: {token_usage.get('input_tokens')}")
    print(f"Output tokens: {token_usage.get('output_tokens')}")
    print(f"Total tokens: {token_usage.get('total_tokens')}")

Assessments

Find assessments with search_assessments():

# Get all assessments
all_assessments = trace.search_assessments()

# Search by name
helpfulness = trace.search_assessments(name="helpfulness")
if helpfulness:
    assessment = helpfulness[0]
    print(f"Helpfulness: {assessment.value}")
    print(f"Source: {assessment.source.source_type} - {assessment.source.source_id}")
    print(f"Rationale: {assessment.rationale}")

# Search by type
feedback_only = trace.search_assessments(type="feedback")
expectations_only = trace.search_assessments(type="expectation")

# Search by span ID
span_assessments = trace.search_assessments(span_id=retriever_span.span_id)

# Include overridden assessments
all_including_invalid = trace.search_assessments(all=True)

Access assessment details:

for assessment in trace.info.assessments:
    print(f"Assessment: {assessment.name}")
    print(f"  Type: {type(assessment).__name__}")
    print(f"  Value: {assessment.value}")
    print(f"  Source: {assessment.source.source_type.value}")

    if assessment.rationale:
        print(f"  Rationale: {assessment.rationale}")
    if assessment.metadata:
        print(f"  Metadata: {assessment.metadata}")
    if assessment.error:
        print(f"  Error: {assessment.error}")

Work with spans

Spans are the building blocks of traces, representing individual operations or units of work. The Span class represents immutable, completed spans retrieved from traces.

Access span properties

# Access all spans from a trace
spans = trace.data.spans
print(f"Total spans: {len(spans)}")

span = spans[0]

# Basic properties
print(f"Span ID: {span.span_id}")
print(f"Name: {span.name}")
print(f"Type: {span.span_type}")
print(f"Parent ID: {span.parent_id}")  # None for root spans

# Timing (nanoseconds)
duration_ms = (span.end_time_ns - span.start_time_ns) / 1_000_000
print(f"Duration: {duration_ms:.2f}ms")

# Status
print(f"Status code: {span.status.status_code}")

# Inputs and outputs
print(f"Inputs: {span.inputs}")
print(f"Outputs: {span.outputs}")

Find specific spans

Use search_spans() to find spans matching specific criteria:

import re
from mlflow.entities import SpanType

# Search by exact name
retriever_spans = trace.search_spans(name="retrieve_documents")

# Search by regex pattern
tool_spans = trace.search_spans(name=re.compile(r".*_tool$"))

# Search by span type
chat_spans = trace.search_spans(span_type=SpanType.CHAT_MODEL)
llm_spans = trace.search_spans(span_type="CHAT_MODEL")  # String also works

# Search by span ID
specific_span = trace.search_spans(span_id=retriever_spans[0].span_id)

# Combine criteria
tool_fact_check = trace.search_spans(
    name="fact_check_tool",
    span_type=SpanType.TOOL,
)

Span attributes

from mlflow.tracing.constant import SpanAttributeKey

chat_span = trace.search_spans(span_type=SpanType.CHAT_MODEL)[0]

# Get all attributes
for key, value in chat_span.attributes.items():
    print(f"  {key}: {value}")

# Get a specific attribute
specific_attr = chat_span.get_attribute("custom_attribute")

# Access chat-specific attributes using SpanAttributeKey
messages = chat_span.get_attribute(SpanAttributeKey.CHAT_MESSAGES)
tools = chat_span.get_attribute(SpanAttributeKey.CHAT_TOOLS)

# Access per-span token usage
input_tokens = chat_span.get_attribute("llm.token_usage.input_tokens")
output_tokens = chat_span.get_attribute("llm.token_usage.output_tokens")

Request and response data

import json

# Get root span request/response
request_json = trace.data.request
response_json = trace.data.response

# Parse JSON strings
if request_json:
    request_data = json.loads(request_json)
if response_json:
    response_data = json.loads(response_json)

Best practices

Keyword arguments

Always use keyword (named) arguments with mlflow.search_traces(). It allows positional arguments, but the function arguments are evolving.

Good practice: mlflow.search_traces(filter_string="trace.status = 'OK'")

Bad practice: mlflow.search_traces([], "trace.status = 'OK'")

filter_string gotchas

When searching using the filter_string argument to mlflow.search_traces(), remember to:

• Use prefixes: trace., tag., or metadata.

• Use backticks if tag or attribute names have dots: tag.`mlflow.traceName`

• Use single quotes only: 'value' not "value"

• Use Unix timestamp (milliseconds) for time: 1749006880539 not dates

• Use AND only: No OR support

See Search query syntax for the full list of supported fields and operators.

SQL warehouse integration

A Databricks SQL warehouse is required to read traces stored in Unity Catalog experiments. Set MLFLOW_TRACING_SQL_WAREHOUSE_ID before calling mlflow.search_traces() or mlflow.get_trace() on a Unity Catalog-backed experiment. Without this env var set, the read fails with SQL warehouse ID is required for accessing traces in UC tables.

For large non-Unity Catalog datasets such as inference tables, a SQL warehouse is optional and improves query performance.

import os

os.environ['MLFLOW_TRACING_SQL_WAREHOUSE_ID'] = 'fa92bea7022e81fb'

# Required for UC-backed experiments. Improves performance for large non-UC datasets.
traces = mlflow.search_traces(
    filter_string="trace.status = 'OK'",
    locations=['my_catalog.my_schema'],
)

Pagination

mlflow.search_traces() returns results in memory, which works well for smaller result sets. To handle large result sets, use MlflowClient.search_traces() since it supports pagination.

Additional resources

• Detect and monitor issues across MLflow traces - Let MLflow detect issues across your traces.

• Build evaluation datasets - Convert queried traces into test datasets.

Next step: Collect feedback and build datasets