ublo
bogdan's (micro)blog

bogdan » A minimum viable experimentation platform for AI agents

09:19 pm on Sep 12, 2026 | read the article | tags: ideas, medium

If you don’t already know, I’ve been toying lately with enterprise AI agents. Here’s the article A minimum viable platform for enterprise AI agents. So, it made me think about how I would decide whether a new version of an AI agent is actually better than the one already running in production.

During development, the process is usually manageable: keep a set of test cases, run both versions, inspect a few traces, ask another model to grade the results and decide whether the change looks promising enough to continue. This is far more difficult when the thing being evaluated is not just a prompt or a model call, but an entire system made of components that change independently and influence each other in ways that are not always obvious.

Imagine a customer-support agent built from:

agent runtime
memory and retrieval
system prompt
model
tools
guardrails
application logic

The retrieval strategy changes and task completion improves on the test set. Maybe the agent is better, yet it may also make more model calls, cost 20% more, behave differently on real users, or simply benefit from another component that changed during the same period. At that point the problem starts looking more like experimentation and less like model evaluation.

So here’s my attempt to design the smallest platform I would be comfortable using to answer a fairly simple question:

should version B replace version A?

There is obviously a scale assumption here: if there’s only one agent, twenty manually reviewed test cases and a few internal users, most of the platform is unnecessary. It starts becoming useful when several teams change models, prompts, memory, tools and agent implementations independently, experiments run continuously, and somebody eventually needs to make a production decision based on evidence that should remain understandable a few months later. Thus, a perfect fit for my enterprise AI agents platform scenario.

Defining what B actually is

The first problem is basic. Let’s assume an experiment says:

A = agent-v17
B = agent-v18

What exactly is agent-v18?

If the prompt changes during the experiment, the retrieval index is rebuilt and the model gateway changes its default model, the label itself tells me very little about what was actually evaluated. This applies especially to agents because the behavior seen at the end is the result of the entire pipeline, not of one isolated model invocation.

Hence, the immutable pipeline definition is the treatment being evaluated. Something like:

pipeline:
  name: support-agent
  version: 18

components:

  runtime:
    artifact: agent-runtime@sha256:...

  memory:
    artifact: memory-service@sha256:...
    configuration:
      strategy: hybrid
      top_k: 8

  model:
    route: support
    model: provider/model-x

  prompt:
    artifact: sha256:...

  tools:
    - artifact: customer-lookup@sha256:...
    - artifact: refund-tool@sha256:...

The exact artifact format is not particularly important. What matters is that an experimental variant can be reconstructed as a system configuration and that, months later, someone can answer what a user assigned to B was supposed to execute. This also keeps the experimentation platform independent of what is changing.

One experiment may compare:

model X vs model Y

another:

memory top_k=5 vs top_k=8

and another may compare two completely different agent implementations. From the point of view of the experiment, all of them are pipeline variants. Once an experiment starts, freeze those definitions. If the treatment changes, create a new experiment iteration rather than silently rewriting what B meant historically.

Assignment is not (another) production dependency

The next problem is deciding who gets A and who gets B. There is a tempting implementation in which every request asks a central experimentation service for its treatment (seen it!), but that also means that the experimentation platform becomes (a critical) part of the availability and latency budget of the application itself, which seems like a poor trade for something whose primary purpose is measurement.

The first version uses deterministic hashing. The experiment control plane publishes the current configuration to an SDK embedded in the application, while the actual decision happens locally, giving the platform a useful failure property: if the experiment service is unavailable, the platform temporarily loses the ability to start a new experiment or change an allocation, but the application is unaffected, with the SDK continuing to use its last valid configuration.

For the MVP, support a few useful randomization units:

request
user
session
workflow_instance

The last one exists mostly because agents do not necessarily share the lifetime of a web request. An agent may start a repository migration today, wait for an external approval and continue tomorrow. If the assignment is recalculated from tomorrow’s experiment configuration, the same workflow may start on one implementation and finish on another, which makes both operational behavior and analysis unnecessarily difficult.

The assignment therefore remains pinned:

workflow_instance = migration-8291
variant           = B
pipeline          = agent-v18

and is carried every time the workflow resumes.

Assignment is not exposure

There is another distinction that becomes important as soon as the treatment is a component somewhere inside an agent pipeline: being assigned to a treatment does not mean that treatment actually participated in the execution.

Suppose I am testing two memory implementations and a user is assigned to B: memory-v2, but the agent answers the request without performing a memory lookup. The assignment exists, but the component under test never had an opportunity to influence the outcome.

Let’s keep those as separate events:

ASSIGNMENT
user-18271 → B

EXPOSURE
memory-v2 retrieval started

This gives useful experiment-health information as well. Instead of seeing only the number of users randomized to each arm, I can also inspect:

assigned          10,428
exposed            8,912
exposure rate      85.46%

If A and B have unexpectedly different exposure rates, investigate that difference first. A related problem appears when the treatment that was assigned is not the one that eventually executes. Suppose B selects model Y, model Y fails (with 503) and the gateway falls back to X. The record will show:

assigned_variant = B
realized_model   = X
fallback_reason  = provider_unavailable

the execution stays in B.

This is the usual Intent-to-Treat, or ITT, principle: analyze the subject according to the treatment it was randomized to, even when the realized execution deviates from that treatment. In this example the fallback behavior is itself part of what users experience when they are assigned to B, and removing those requests after the fact would change the population based on something that happened after randomization.

Exposure and realized-treatment information are still valuable, but primarily as experiment-health and diagnostic signals unless the analysis explicitly defines something else.

Traces explain what happened

The experiment result tells me whether an outcome changed. It does not tell me why.

An agent execution may look like:

Use OpenTelemetry for this rather than inventing another tracing protocol. The experimentation layer only needs to attach a small amount of additional context:

experiment.id
experiment.version
experiment.variant
experiment.assignment_id

pipeline.id
pipeline.version

execution.purpose

Now, move from an aggregate result such as:

B has a lower task completion rate

to a much more useful question:

show failed B traces

and inspect whether the new pipeline retrieves worse context, retries tools, falls back to another model, enters loops, or simply takes longer paths through the same task.

The trace itself is not the experiment record though. A trace is very good at explaining how one execution unfolded, but experimental outcomes do not necessarily share its lifetime. A customer may interact with an agent today and renew a subscription a week later, at which point the original trace has long finished.

So keep a smaller experiment event stream alongside tracing:

assignment
exposure
outcome
feedback
evaluation

The trace records execution. The event stream provides the bookkeeping needed to connect randomization to outcomes that may arrive later.

Offline evaluation uses the same pipeline

Before putting B in front of production users, test it against known scenarios. For agents specifically, a scenario shouldn’t be reduced to:

prompt
expected answer

because the interesting result may not be the text returned by the model at all. It may be what happened to the environment. A customer-support scenario could look like:

scenario:
  id: duplicate-charge-001

input:
  message: >
    I was charged twice for order 9811.

environment:
  fixture: duplicate-charge-customer

expected:
  refund_count: 1
  ticket_status: resolved

limits:
  max_turns: 15
  max_tool_calls: 20
  max_cost: 0.50

Then run the same scenario for both pipeline A and pipeline B, possibly several times, because agent executions are stochastic and two runs of the same configuration may follow different trajectories even when they start from equivalent state. The MVP uses a resettable sandbox containing fixture-backed services or test databases. Don’t try to build general multi-turn trace replay yet.

Replay works while the new candidate asks for interactions that were already recorded. If candidate B takes a different decision on turn three and calls a tool that the original trajectory never called, there is no recorded world in which that branch exists.

At that point the useful result is:

REPLAY_DIVERGED

rather than inventing a mock response and pretending the execution is still a replay.

A stateful sandbox lets both agents branch naturally while keeping their starting conditions controlled, which seems like a better first approximation of how agent evaluation should work.

Shadow execution bridges the gap between offline and production

There is one useful step between a controlled sandbox and exposing users to B: run B asynchronously against a copy of real production inputs without serving its output.

Variant B executes on realistic traffic, while writes and external side effects are either disabled or redirected to an isolated environment.

This provides production-like inputs and allows observation of:

latency
cost
tool usage
model fallbacks
guardrail failures
judge scores
trajectory differences

before the treatment reaches users.

Shadow execution still doesn’t tell me whether B improves the outcome I ultimately care about, because users never experience B and therefore cannot react to it. What it gives is an intermediate deployment step: realistic input distributions and runtime behavior without accepting the full product risk of a live experiment.

Not everything needs an LLM judge

Once the agent finishes a scenario, I still need to evaluate the result. That’s why the first version uses two kinds of evaluator.

The first is deterministic. If the task was to issue exactly one refund and resolve a ticket, inspect the environment:

refund_count == 1
ticket_status == resolved

There is little benefit in asking another model whether it thinks the agent probably completed a task when I can verify the resulting state directly.

The second evaluator is model-based. Some properties are difficult to express as assertions:

helpfulness
clarity
tone
quality of explanation

For those, an LLM judge comes in handy, but the judge becomes another versioned component rather than an anonymous source of truth:

evaluator
evaluator version
judge model
judge prompt
rubric

Every generated score keeps that provenance.

I also need to maintain a small human-labelled calibration dataset and periodically compare the judge against it. An LLM judge is a measurement instrument; if the instrument changes, I want to know whether its measurements have changed as well.

Agentic judges that browse, call tools or investigate additional evidence are useful and probably unavoidable for some domains, but they don’t have a place in the first version.

Oh! About the statistics

There is a lot of sophistication available in mature experimentation systems, but my go-to choice is to implement a small set of methods correctly rather than start with an impressive statistical menu that few users understand.

Here’s what the first version supports:

A/B allocation
binary metrics
continuous metrics
95% confidence intervals
sample ratio mismatch detection
fixed-horizon analysis

That is enough to answer many useful questions.

For example:

Experiment
support-agent-v18

Randomization
user

Primary metric
ticket resolution

A       81.4%
B       85.1%

Difference
+3.7 percentage points

95% CI
[...]

Experiment health
SRM PASS

Sample ratio mismatch is particularly important because a result should not look authoritative when the randomization itself appears broken. If I intended to assign users 50/50 and observed something sufficiently inconsistent with that expectation, I would rather mark the experiment unhealthy and stop there than display a winner with a warning icon next to it.

The same health layer should enforce operational guardrails. An agent variant can fail economically without failing technically. A prompt loop that turns:

4 model calls

into:

47 model calls

may still return HTTP 200 and even produce a good answer.

So hard experiment limits become necessary, like:

max cost / execution
max tokens / execution
max agent turns
max tool calls
max variant spend / hour

Crossing one of those limits can stop an execution or pause further exposure to that variant.

For example:

Variant B

SRM                PASS
Error rate         PASS
Cost / request     +312%

Circuit breaker    TRIPPED
New exposure       PAUSED

The circuit breaker is not a statistical conclusion, but simply an operational safety mechanism that prevents a pathological treatment from consuming traffic and budget while the experiment is still gathering enough observations to say anything interesting statistically.

More sophisticated capabilities can come later: CUPED, sequential testing, holdouts, experiment layers, cluster randomization, adaptive allocation and so on.

The data model should leave room for them. The first implementation only needs to get ordinary A/B experiments right.

What to build first

After removing most of the things that can wait, the platform is manageable. It just needs a control service containing:

• Experiment Registry
• Pipeline Registry
• Dataset Registry
• Evaluator Registry
• Metric definitions

On top of this sits the SDK, which downloads experiment configuration and performs deterministic assignment locally. The platform also needs an ingestion path for:

assignment
exposure
outcome
feedback
evaluation

together with OpenTelemetry traces.

For storage, PostgreSQL is a solid choice for control-plane state, NATS JetStream or Kafka for event ingestion, ClickHouse for experiment and trace analytics, and S3-compatible object storage for larger artifacts.

The platform also needs an offline runner that executes a dataset against A and B inside resettable environments.

The resulting architecture is roughly:

There are plenty of things missing.

The MVP doesn’t solve arbitrary stateful environment virtualization, sophisticated experiment interaction, evaluator drift, automated root-cause analysis, persistent treatment effects or multi-agent causal attribution, and I don’t think the MVP needs to solve them.

What matters from the beginning are the parts that become difficult to repair after historical data already depends on them:

what was randomized
what treatment was assigned
what treatment actually ran
what pipeline version produced the execution
what outcome was measured
how that outcome was evaluated

If those identities survive the first implementation, the system can become considerably more sophisticated without changing the meaning of its old experiments.

The useful connection for me is that offline evaluation, shadow execution and online experimentation can share pipeline definitions, traces and evaluator infrastructure while still answering different questions.

Offline evaluation shows how A and B behave in controlled scenarios. Shadow execution shows how B behaves on production inputs without affecting users. Production experimentation shows what changes when real users are randomly assigned to A or B. Tracing gives evidence about what happened inside each execution and helps explain the result without pretending that a trace-level diagnosis is itself a causal estimate. And that should be enough for an MVP that will be able to answer three questions reliably:

What exactly did we run?
What happened when we ran it?
Did assigning users to B improve the outcome we care about?

Everything else can grow around that.

I’ve placed a more detailed, full-fledged version of the architecture here and an attempt for an MVP here. Feel free to check them out and send feedback.