Manage production scorers

After you set up production monitoring, you can manage your scorers throughout their lifecycle. This page covers how to list, update, stop, restart, and delete scorers.

For the full API parameter reference, see Manage production scorers.

Scorer lifecycle

Scorer lifecycles are centered around MLflow experiments. Scorers are immutable: each lifecycle operation returns a new scorer instance rather than modifying the original.

Lifecycle example

The following example demonstrates a scorer moving through all lifecycle states:

from mlflow.genai.scorers import Safety, scorer, ScorerSamplingConfig, delete_scorer

# Register → Start → Update → Stop → Delete
safety_judge = Safety().register(name="safety_check")
safety_judge = safety_judge.start(
    sampling_config=ScorerSamplingConfig(sample_rate=1.0),
)
safety_judge = safety_judge.update(
    sampling_config=ScorerSamplingConfig(sample_rate=0.8),
)
safety_judge = safety_judge.stop()
delete_scorer(name="safety_check")

Manage scorers

The following APIs are available to manage scorers.

List scorers

To view all registered scorers for your experiment:

from mlflow.genai.scorers import list_scorers

# List all registered scorers
scorers = list_scorers()
for scorer in scorers:
    print(f"Name: {scorer.name}")
    print(f"Sample rate: {scorer.sample_rate}")
    print(f"Filter: {scorer.filter_string}")
    print("---")

Get and update a scorer

Use get_scorer() to retrieve a scorer by name, then update() to modify its configuration. Because scorers are immutable, update() returns a new instance.

from mlflow.genai.scorers import get_scorer, ScorerSamplingConfig

# Get existing scorer and update its configuration (immutable operation)
safety_judge = get_scorer(name="safety_monitor")
updated_judge = safety_judge.update(sampling_config=ScorerSamplingConfig(sample_rate=0.8))

# The original scorer remains unchanged; update() returns a new scorer instance
print(f"Original sample rate: {safety_judge.sample_rate}")  # Original rate
print(f"Updated sample rate: {updated_judge.sample_rate}")   # New rate

Stop and delete scorers

Stopping a scorer sets its sample rate to 0 but keeps it registered. Deleting a scorer removes it from the server entirely.

from mlflow.genai.scorers import get_scorer, delete_scorer, ScorerSamplingConfig

# Get existing scorer
databricks_scorer = get_scorer(name="databricks_mentions")

# Stop monitoring (sets sample_rate to 0, keeps scorer registered)
stopped_scorer = databricks_scorer.stop()
print(f"Sample rate after stop: {stopped_scorer.sample_rate}")  # 0

# Restart monitoring from a stopped scorer
restarted_scorer = stopped_scorer.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))

# Or remove scorer entirely from the server
delete_scorer(name=databricks_scorer.name)

Immutable updates

Scorers, including LLM Judges, are immutable objects. When you update a scorer, an updated copy is created rather than modifying the original. This immutability helps ensure that scorers meant for production are not accidentally modified.

from mlflow.genai.scorers import Safety, ScorerSamplingConfig

original_judge = Safety().register(name="safety")
original_judge = original_judge.start(
   sampling_config=ScorerSamplingConfig(sample_rate=0.3),
)

# Update returns new instance
updated_judge = original_judge.update(
    sampling_config=ScorerSamplingConfig(sample_rate=0.8),
)

# Original remains unchanged
print(f"Original: {original_judge.sample_rate}")  # 0.3
print(f"Updated: {updated_judge.sample_rate}")    # 0.8

Best practices

• Check the scorer state before operations using sample_rate.

• Use the immutable pattern. Assign the results of .start(), .update(), .stop() to variables.

• Understand the difference between .stop() (preserves registration) and delete_scorer() (removes entirely).

Scorer lifecycle API reference

Scorer instance methods

Scorer.register()

API Reference: Scorer.register

Register a custom scorer function with the server. Used for scorers created with the @scorer decorator.

@scorer
def custom_scorer(outputs):
    return len(str(outputs.get("response", "")))

# Register the custom scorer
my_scorer = custom_scorer.register(name="response_length")

Parameters:

• name (str): Unique name for the scorer within the experiment. Defaults to the existing name of the scorer.

Returns: New Scorer instance with server registration

Scorer.start()

API Reference: Scorer.start

Begin online evaluation with the specified sampling configuration.

from mlflow.genai.scorers import ScorerSamplingConfig

# Start monitoring with sampling
active_scorer = registered_scorer.start(
    sampling_config=ScorerSamplingConfig(
        sample_rate=0.5,
        filter_string="trace.status = 'OK'"
    ),
)

Parameters:

• name (str): Name of the scorer. If not provided, defaults to the current name of the scorer.

• sampling_config (ScorerSamplingConfig): Trace sampling configuration

  • sample_rate (float): Fraction of traces to evaluate (0.0-1.0). Default: 1.0
  • filter_string (str, optional): MLflow-compatible filter for trace selection

Returns: New Scorer instance in active state

Scorer.update()

API Reference: Scorer.update

Modify the sampling configuration of an active scorer. This is an immutable operation.

# Update sampling rate (returns new scorer instance)
updated_scorer = active_scorer.update(
    sampling_config=ScorerSamplingConfig(
        sample_rate=0.8,
    ),
)

# Original scorer remains unchanged
print(f"Original: {active_scorer.sample_rate}")  # 0.5
print(f"Updated: {updated_scorer.sample_rate}")   # 0.8

Parameters:

• name (str): Name of the scorer. If not provided, defaults to the current name of the scorer.

• sampling_config (ScorerSamplingConfig): Trace sampling configuration

  • sample_rate (float): Fraction of traces to evaluate (0.0-1.0). Default: 1.0
  • filter_string (str, optional): MLflow-compatible filter for trace selection

Returns: New Scorer instance with updated configuration

Scorer.stop()

API Reference: Scorer.stop

Stop online evaluation by setting sample rate to 0. Keeps the scorer registered.

# Stop monitoring but keep scorer registered
stopped_scorer = active_scorer.stop()
print(f"Sample rate: {stopped_scorer.sample_rate}")  # 0

Parameters:

• name (str): Name of the scorer. If not provided, defaults to the current name of the scorer.

Returns: New Scorer instance with sample_rate=0

Scorer registry functions

mlflow.genai.scorers.get_scorer()

API Reference: get_scorer

Retrieve a registered scorer by name.

from mlflow.genai.scorers import get_scorer

# Get existing scorer by name
existing_scorer = get_scorer(name="safety_monitor")
print(f"Current sample rate: {existing_scorer.sample_rate}")

Parameters:

• name (str): Name of the registered scorer

Returns: Scorer instance

mlflow.genai.scorers.list_scorers()

API Reference: list_scorers

List all registered scorers for the current experiment.

from mlflow.genai.scorers import list_scorers

# List all registered scorers
all_scorers = list_scorers()
for scorer in all_scorers:
    print(f"Name: {scorer._server_name}")
    print(f"Sample rate: {scorer.sample_rate}")
    print(f"Filter: {scorer.filter_string}")

Returns: List of Scorer instances

mlflow.genai.scorers.delete_scorer()

API Reference: delete_scorer

Delete a registered scorer by name.

from mlflow.genai.scorers import delete_scorer

# Delete existing scorer by name
delete_scorer(name="safety_monitor")

Parameters:

• name (str): Name of the registered scorer

Returns: None

Scorer properties

Scorer.sample_rate

Current sampling rate (0.0-1.0). Returns 0 for stopped scorers.

print(f"Sampling {scorer.sample_rate * 100}% of traces")

Scorer.filter_string

Current trace filter string for MLflow trace selection.

print(f"Filter: {scorer.filter_string}")

Configuration classes

ScorerSamplingConfig

API Reference: ScorerSamplingConfig

Data class that holds sampling configuration for a scorer.

from mlflow.genai.scorers import ScorerSamplingConfig

config = ScorerSamplingConfig(
    sample_rate=0.5,
    filter_string="trace.status = 'OK'"
)

Attributes:

• sample_rate (float, optional): Sampling rate between 0.0 and 1.0

• filter_string (str, optional): MLflow trace filter

Metric backfill

backfill_scorers()

from databricks.agents.scorers import backfill_scorers, BackfillScorerConfig

job_id = backfill_scorers(
    experiment_id="your-experiment-id",
    scorers=[
        BackfillScorerConfig(scorer=safety_scorer, sample_rate=0.8),
        BackfillScorerConfig(scorer=response_length, sample_rate=0.9)
    ],
    start_time=datetime(2024, 1, 1),
    end_time=datetime(2024, 1, 31)
)

Parameters:

All parameters are keyword-only.

• experiment_id (str, optional): The ID of the experiment to backfill. If not provided, uses the current experiment context

• scorers (Union[List[BackfillScorerConfig], List[str]], required): List of BackfillScorerConfig objects with custom sample rates (if sample_rate is not provided in BackfillScorerConfig, defaults to the registered scorer's sample rate), OR list of scorer names (strings) to use current sample rates from the experiment's scheduled scorers. Cannot be empty.

• start_time (datetime, optional): Start time for backfill evaluation. If you omit this but pass end_time, the backfill starts one day before end_time

• end_time (datetime, optional): End time for backfill evaluation. If you omit this but pass start_time, the backfill runs through the current time

Returns: Job ID of the created backfill job for status tracking (str)