MLflow 3 GenAI Evaluation

Before Writing Any Code

1. Read GOTCHAS.md - 15+ common mistakes that cause failures
2. Read CRITICAL-interfaces.md - Exact API signatures and data schemas

End-to-End Workflows

Follow these workflows based on your goal. Each step indicates which reference files to read.

Workflow 1: First-Time Evaluation Setup

For users new to MLflow GenAI evaluation or setting up evaluation for a new agent.

Workflow 2: Production Trace -> Evaluation Dataset

For building evaluation datasets from production traces.

Workflow 3: Performance Optimization

For debugging slow or expensive agent execution.

Workflow 4: Regression Detection

For comparing agent versions and finding regressions.

Workflow 5: Custom Scorer Development

For creating project-specific evaluation metrics.

Workflow 6: Unity Catalog Trace Ingestion & Production Monitoring

For storing traces in Unity Catalog, instrumenting applications, and enabling continuous production monitoring.

Workflow 7: Judge Alignment with MemAlign

For aligning an LLM judge to match domain expert preferences. A well-aligned judge improves every downstream use: evaluation accuracy, production monitoring signal, and prompt optimization quality. This workflow is valuable on its own, independent of prompt optimization.

Workflow 8: Automated Prompt Optimization with GEPA

For automatically improving a registered system prompt using optimize_prompts(). Works with any scorer, but paired with an aligned judge (Workflow 7) gives the most domain-accurate signal. For the full end-to-end loop combining alignment and optimization, see user-journeys.md Journey 10.

Reference Files Quick Lookup

Critical API Facts

• Use: mlflow.genai.evaluate() (NOT mlflow.evaluate())
• Data format: {"inputs": {"query": "..."}} (nested structure required)
• predict_fn: Receives **unpacked kwargs (not a dict)
• MemAlign: Scorer-agnostic (works with any feedback_value_type -- float, bool, categorical); token-heavy on the embedding model so set embedding_model explicitly
• Label schema name matching: The label schema name in the labeling session MUST match the judge name used in evaluate() for align() to pair scores
• Aligned judge scores: May be lower than unaligned judge scores -- this is expected and means the judge is now more accurate, not that the agent regressed
• GEPA optimization dataset: Must have both inputs AND expectations per record (different from eval dataset)
• Episodic memory: Lazily loaded -- get_scorer() results won't show episodic memory on print until the judge is first used
• optimize_prompts: Requires MLflow >= 3.5.0

See GOTCHAS.md for complete list.

Related Skills

• databricks-docs - General Databricks documentation reference
• databricks-model-serving - Deploying models and agents to serving endpoints
• databricks-agent-bricks - Building agents that can be evaluated with this skill
• databricks-python-sdk - SDK patterns used alongside MLflow APIs
• databricks-unity-catalog - Unity Catalog tables for managed evaluation datasets