Product · Evals & sandbox
Where agent quality gets measured and gated
Evals is the surface where agent output quality, regressions and drift are scored — and where a release can be gated before it ships. The framework, the scorecards and the console exist and are wired, and so is the part that makes it count on real traffic: the session source that backs sampling, and ordered session replay in the sandbox, are wired in-process with no operator configuration. What replay does NOT do is invent input — a session with no reconstructable timeline is reported degraded, never fabricated.
What it does
A framework for agent quality
Output-quality monitoring, regression testing and an isolated sandbox — the place where agent behavior is scored before and after a change.
Scorecards and output-quality monitoring
Score agent output against the checks you define and watch quality over time. The framework, the scorecards and the console are wired; what they measure becomes real once a session source is connected.
Regression testing and prompt A/B
Re-run a suite against a change to catch regressions before they ship, and compare prompt variants A against B on the same inputs — so a change is judged on evidence, not intuition.
Drift detection
Detect when agent output drifts from its expected baseline over time, so quality erosion is surfaced rather than discovered in production.
Isolated sandbox
An isolated test environment for pre- and post-deploy comparison, with session replay. Both are wired: the environment, and the ordered history source replay reconstructs a session from.
What’s real
The framework, the console, live sampling and ordered replay are all wired
This surface is the most seam-heavy in the product, so we are blunt about it — the honesty is the feature, not an apology:
- Live: the evals framework, the scorecards, the console, regression runs, prompt A/B and drift detection are built and wired, and the sandbox is an isolated environment for pre/post-deploy comparison.
- Live, with a stated limit: eval sampling reads real sessions through the wired session source, within a configurable recency window, and sandbox replay reconstructs a session’s ordered action sequence from its history. The limit is what happens when there is nothing to read — a session with no reconstructable timeline yields a degraded replay of zero steps, and a timeline longer than the replayable bound is refused outright rather than replayed in part. Neither case is filled in with invented input.
- Posture: the adaptive red-teaming engine is post-v1. For v1 we document the posture with compensating controls rather than overstate an engine that is not here yet.
Evals & sandbox — questions
Can I run evals against my real agent traffic today?
Yes. The session source that backs sampling is wired in-process with no operator configuration, so monitoring runs sample real sessions rather than seeded data — within a configurable recency window, which is what keeps samples fresh and bounded. Screenshots on this page still show seeded example data, because they are captures, not a live tenant.
Does session replay in the sandbox work?
Yes, and it is deterministic: replay reconstructs the session’s ordered tool and MCP action sequence from its history and re-executes it against the mocks you supply, so the same session and mocks always yield the same outputs. Two limits are stated rather than hidden: a session with no reconstructable timeline is reported degraded with zero steps, and a timeline beyond the replayable bound is refused rather than replayed in part.
Is there an automated red-teaming engine?
Not in v1. The adaptive red-teaming engine is post-v1. For v1 we document the security posture with compensating controls rather than imply an adaptive engine that is not built yet.
So what is actually usable right now?
The evals framework and console — scorecards, regression runs, prompt A/B and drift detection — plus the isolated sandbox for pre/post-deploy comparison, the wired session source that samples real sessions, and ordered replay reconstructed from session history. This is where agent quality and regressions are measured and gated. What is still post-v1 is the adaptive red-teaming engine, and this page says so where it belongs.
See where agent quality gets gated
Deploy Olivares on your own infrastructure and explore the evals framework and sandbox — scorecards, regression testing, pre- and post-deploy comparison, live session sampling and ordered replay reconstructed from session history.