Skip to main content

​
Simulations

Simulations stress-test your agent the way real users would — before you put it in front of customers. An AI persona caller role-plays a realistic (and sometimes difficult) user, holds a full multi-turn conversation with your real agent, and an AI judge scores how the agent did. Generate a diverse test suite in one click, run it across chat or voice, and see exactly where your agent passes, wobbles, or breaks.
Simulations test your agent’s actual configuration — its system prompt, model, tools, knowledge bases, and voice workflow. What you test is what ships.

​
How It Works

  1. A persona-driver LLM plays a caller with a goal, personality, and (optionally) facts it knows.
  2. It runs a multi-turn conversation against your real agent.
  3. A judge model reads the transcript and returns a verdict: Goal Status, Conversation Quality, and a pass/fail per success criterion, with reasoning.
  4. Results — including the full transcript — are saved per scenario and aggregated per run.

​
Test Scenarios

A scenario is a single test case: a caller persona plus what they’re trying to do and how you’ll know the agent handled it well.

​
Scenario Types

A good suite has a realistic mix of all three.

​
Auto-Generating Scenarios

You don’t have to write scenarios by hand. Click Generate, and the platform reads your agent’s configuration and produces a diverse, guardrail-probing set — cooperative callers, edge-case callers, and adversarial callers — grounded in your agent’s own rules.
1

Open the Simulation tab

Go to your agent in Agent Studio and open the Simulation tab.
2

Generate

Click Generate, choose how many scenarios you want, and optionally add guidance (e.g. “focus on angry customers disputing charges”).
3

Review & edit

The generated scenarios appear in your scenario list. Edit, add, or remove any of them before running.

​
Running a Simulation

1

Select scenarios

In the Simulation tab, select the scenarios to run (or run all enabled ones).
2

Choose a mode

Pick Chat only or Voice + Chat (see Modes below).
3

Pick a judge model

The default judge is GPT-4o. You can change it to any supported model — the same model picker as the agent’s Behavior tab.
4

Set max turns

Cap how long each conversation can run before it’s cut off.
5

Run

Click Run. Each scenario runs as its own conversation, and results stream into Simulation Results as they complete.

​
Modes

Chat

A text conversation between the persona caller and your agent. Available on all plans. Fast and inexpensive — ideal for iterating on prompts and guardrails.

Voice + Chat

A real voice call: an AI caller actually speaks to your agent through the live voice pipeline (Vega speech-to-text + Aero text-to-speech), so you test the agent exactly as a phone caller would hear it. Available on Pay-as-you-go and Enterprise plans.
Voice simulations place real voice calls and deduct from your wallet. The caller side (speech-to-text, the persona model, text-to-speech) and the judge are billed to your account; your agent’s side is metered exactly like any other voice call. Each scenario is capped at a maximum call duration for safety.

​
Workflow Agents

If your agent uses a Voice Workflow, simulations drive the workflow runtime — nodes, transitions, and variable extraction — so the test reflects the agent’s real branching flow, not just a flat prompt.
During a simulation, side-effecting steps (API requests, tools, call transfers) are simulated, not executed — the agent is recorded as having attempted them, but no real external call is made. This keeps tests safe and repeatable.

​
Reading Results

Each scenario produces a result you can open to see the full transcript and verdict. A run aggregates its scenarios into pass / review / fail counts and a quality breakdown, so you can see overall health at a glance and drill into any failure.

​
Judge Models

The default judge is GPT-4o for strong, consistent verdicts. Use a lighter model for quick iteration, or a stronger one for high-stakes reviews.

​
Billing

Simulations consume tokens (and, for voice, voice usage) billed to your account:
  • Chat — the agent’s tokens, the persona caller’s tokens, and the judge’s tokens are each metered.
  • Voice — the caller side’s voice usage (speech-to-text, persona model, text-to-speech) plus the judge are billed to your wallet; your agent’s side is billed as a normal voice call. Billing is idempotent, so a retried call is never charged twice.
Auto-generating scenarios also consumes a small number of tokens for the generation model.

​
Best Practices

  • Generate first, then curate — start from an auto-generated suite, then tighten the scenarios that matter most to your business.
  • Always include red-team scenarios — the cheapest place to discover a jailbreak is in a simulation, not production.
  • Test chat before voice — iterate quickly and cheaply in chat, then validate the final flow in voice.
  • Re-run after every change — treat your scenario suite as a regression test for prompt, tool, and workflow edits.
  • Write sharp success criteria — specific yes/no questions (“Did the agent verify the caller’s identity before sharing account details?”) give the judge a clear, actionable target.
  • Act on review and fail — read the judge’s reasoning and the transcript, then update the agent’s instructions, guardrails, or workflow.

Evaluations

Score individual agent responses across custom quality criteria.

Voice Workflows

Build branching, multi-step voice flows — fully covered by simulations.