Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Tip
Optionally, install mlflow/skills for a deeper integration with MLflow.
Human judgment — ratings, comments, and ground-truth expectations from developers, users, and domain experts — is what keeps evaluation aligned with what "good" actually means for your app. In MLflow you record that judgment directly on a trace, and a trace that carries a ground-truth expectation becomes a ready-made test case for automated evaluation.
Quick overview: turn a trace into a test case
In the Traces tab, click a trace to open it.
Click Assess, then Add expectation to record the correct answer (ground truth). The expectation is stored on the trace, alongside its inputs, outputs, and any other feedback.
Click Add to dataset to add the trace — with its expectation — to an evaluation dataset.
That trace is now a test case: you captured the expected answer on a real interaction and promoted it to a dataset you can evaluate future versions against. Because the feedback lives on the trace, anyone who reviews traces — not just the developer — can do this without leaving the trace UI.
Who leaves feedback
Feedback comes in two forms — feedback evaluates the app's actual output (was the response good?), and an expectation defines the correct, ground-truth output. Each source of judgment has its own mechanism.
Developers
Annotate traces as you build, to track quality notes and mark examples for regression testing. See Label traces as a developer.
Domain experts
Engage subject-matter experts through the Review App to define what a high-quality response looks like and to align LLM judges with your business requirements. There are two approaches: Test an agent with the Chat UI or Enable experts and collaborators to label traces.
End users
Capture feedback from users of your deployed app to surface problematic queries and preserve successful interactions. See Enrich traces: tags, context, and feedback .
Next steps
Enable experts and collaborators to label traces - Bring in domain experts and collaborators to label traces.
Building MLflow evaluation datasets - Package labeled traces and expectations into a dataset.
Evaluate and improve - Evaluate and improve your app against the datasets you build.