How Do I Design Evals for an Agent That Takes Multiple Steps?

As AI product builders at companies like PayPal, H2O.ai, and through tools like PM Toolkit, we constantly face the challenge of evaluating multi-step agents. These agents don’t just produce a one-off AI kill switch answer; they perform a sequence of actions, call external tools, and make decisions at every step. Designing evaluations (evals) for such agents goes far beyond typical single-response model benchmarks. In this post, I’ll walk you through a structured approach to agent eval design that emphasizes step-level evals and tool calling, grounded in real product workflows.

The Model Is Not the Product: A Fundamental Mindset

Before launching into eval specifics, it’s critical to remember that the model is not the product. Your multi-step agent might use an LLM, knowledge graph retrieval, rule-based systems, and external APIs in tandem. The final product experience depends on all these parts working together smoothly, not just isolated model outputs.

Leading companies understand this well. For example, PayPal’s AI-driven fraud tools don’t just look at a single risk score but combine signals across steps in a workflow to make real-time decisions. H2O.ai’s AutoML agents balance automation and human augmentation carefully to maximize user trust and impact. The PM Toolkit helps product managers quantify and prioritize these workflows with calculators like TAM/SAM/SOM and RICE, proving you need a holistic lens beyond “model accuracy.”

Define an Eval Before Release: What Are You Measuring and Why?

Too often, AI teams start collecting data or running benchmarks before clearly defining the evaluation goals. With multi-step agents, defining your eval upfront pays even greater dividends. Ask yourself:

What success looks like: What exact behaviors and outcomes should the agent demonstrate at each step?
What failure modes matter most: Are incomplete workflows, tool-call errors, or hallucinations your biggest risks?
What user pain points exist today: What workflows are you automating or augmenting? Mapping current manual steps reveals eval targets.
Which KPIs align with product impact: Consider PM Toolkit calculators like retention, churn, NPS, or PMF score as part of your evaluation plan.

For example, if you build an agent to assist PayPal’s fraud analysts, a critical step-level success metric could be “correctly flags suspicious transactions with at least 95% precision in tool calls to external credit bureaus.” This is a far more actionable and business-relevant metric than just “overall accuracy.”

Step-Level Evals: The Heart of Multi-Step Agent Validation

Multi-step agents perform a sequence of decisions and tool calls. Assessing only the final outcome obscures where failures occur and hinders iterative improvement. Let’s break down how to design step-level evals:

Decompose the workflow: Document every step logically, including user inputs, model predictions, tool calls, and downstream decisions.
Create step-specific test cases: For each step, design test inputs and expected outputs. For example, “Given transaction data X, the model should call credit API Y with parameters Z.”
Measure step accuracy and consistency: Include metrics for tool call success rates, API latency, confidence calibration, and error rates at each step.
Track error propagation: Understand how errors in early steps amplify downstream impacts — a failed identity verification step may doom the entire workflow.
Leverage spreadsheets and automated pipelines: Before fully automated evals, use spreadsheets to log step results, error types, and aggregate metrics. This approach was a staple in PM Toolkit’s early AI product research phases.

This step-level approach makes failure modes transparent and fixes more targeted. For instance, if your agent from H2O.ai calls a feature extraction service that sometimes times out, your step-level eval identifies how often and under what conditions these timeouts occur—something impossible if you only look at the final output.

Example Step-Level Eval Table

Step Input Expected Output Metric Observed Value Pass/Fail Threshold Entity Extraction Customer email text Extract email, name F1 score 0.92 > 0.90 Tool Call: Credit API Customer ID, Transaction ID Credit score > threshold API success rate 98% > 95% Decision Step All inputs Approve/Reject Precision 0.94 > 0.90

Automation vs Augmentation vs Innovation: Patterns in Multi-Step AI Agents

When designing evals, consider what pattern your agent follows:

Automation: The agent fully replaces human steps. Evals focus on precision, recall, reliability, and kill switches. For instance, PayPal’s fraud detection agent aims for high precision to avoid false positives.
Augmentation: The agent supports humans but leaves final control with them. Evals measure augmentation effectiveness, workflow velocity, DAU/MAU changes, NPS scores, and error reduction.
Innovation: The agent introduces fundamentally new capabilities, like new tool calls or user experiences. Evals incorporate impact/effort matrices from PM Toolkit and user interviews to validate adoption risks.

Align eval design with these patterns to ensure you’re measuring what matters most to users and business. For example, an augmented workflow at H2O.ai might require capturing cycle time improvements via PM Toolkit’s calculators and comparing manual vs assisted scenarios.

Practical Tips for Agent Eval Design In Your Product Cycle

Start with user stories: Write eval examples as user stories, not abstract prompts. What is the user trying to accomplish at each step?
Use PM Toolkit calculators for business context: Evaluate potential TAM/SAM/SOM, churn impact, CAC, retention, and MRR/ARR implications to prioritize your eval efforts.
Set ship/no-ship thresholds upfront: Define clear quantitative criteria for model or agent versions to pass before rollout — for example, API error rate < 3%, step accuracy > 90%.
Build lightweight eval spreadsheets early: Track inputs, expected outputs, and metrics manually before automating pipelines.
Design kill switches for safety: Shipping without a kill switch for faulty multi-step agents risks irreversible user harm.
Run A/B tests on real workflows: Use PM Toolkit A/B test calculators and sample size estimators to validate that step-level improvements translate into end metrics like conversion rate or NPS.

Conclusion

Designing evals for multi-step agents requires a disciplined, workflow-first approach. By decomposing agent behavior into step-level evaluations and contextualizing metrics within product and business goals, you avoid the pitfalls of “model wars” and hand-wavy accuracy claims. Remember, the model is not the product—your evals must capture the full agent experience, from tool calling to augmentation impact, backed by data-driven thresholds crafted using tools like PM Toolkit and proven practices from PayPal, H2O.ai, and beyond.

Start small with spreadsheets, iterate quickly, and always tie evaluation to your user’s real work today. This mindset will help you build safer, more reliable, and more impactful multi-step AI agents.