A probabilistic acceptance criterion pairs a measurable metric, a defined sample or window, a numeric threshold, and a stated fallback — never a vague quality judgment. "The AI should give good recommendations" fails because nobody can test it. "Precision@3 ≥80% on a 200-session labeled set, reviewed weekly, with automatic fallback below 75%" can ship.
Quick answer: A probabilistic acceptance criterion = metric + sample/population + threshold + measurement method + fallback behavior. Drop any one of the five and an engineer has nothing to build a test against.
Why "The AI Should Give Good Recommendations" Isn't an Acceptance Criterion
Traditional acceptance criteria assume a deterministic system: given input X, output Y either happens or it doesn't. AI features break that assumption — the same input can produce different outputs across runs, model versions, or even random seeds, so "correct" has to be defined statistically, not categorically.
That single shift breaks most PRD templates. A criterion like "the system displays an error message when the form is invalid" is binary and trivially testable. "The AI should give good recommendations" isn't binary, has no sample size, no metric, and no threshold — an engineer reading it has three equally valid, mutually incompatible ways to build it, and QA has no pass/fail rule to run.
The Testability Test
Before a criterion goes into a spec, run it through one question: could two different engineers write two different automated tests that both technically satisfy this sentence? If yes, it's not done yet. Apply this to "good," "accurate," "helpful," "relevant," and "high-quality" — they all fail immediately, because each hides an undefined metric and an undefined threshold.
This isn't pedantry. Vague acceptance criteria don't disappear — they just get resolved later, by whoever writes the code, usually under deadline pressure and without the product context to make the call well. Writing the threshold into the spec is how a PM keeps that decision in product's hands instead of defaulting to an engineer's best guess at 11pm before a release.
The fix isn't to demand certainty AI can't offer. It's to specify the acceptable band of uncertainty explicitly — which is exactly what probabilistic acceptance criteria do, and it's a discipline that belongs alongside the rest of how specs are written for AI-era products; our PRD writing masterclass for the AI era covers the broader shift this fits inside.
The Anatomy of a Probabilistic Acceptance Criterion
Every defensible probabilistic acceptance criterion has five components, and a criterion missing any one of them isn't finished — it's a placeholder wearing the shape of a requirement. Think of it as a checklist you run every criterion through before it leaves the PRD.
- Population / sample — what set of inputs, sessions, or requests does this criterion apply to? ("All product-detail-page sessions with at least one prior purchase.")
- Metric — the specific, named, computable measure. Not "accuracy" in general —
precision@3,F1,hallucination rate,task completion rate. - Threshold — the numeric bar, stated as a range or floor, ideally with a baseline it's measured against (a prior system, a human benchmark, or a null model).
- Measurement method — how and how often the metric is computed: offline against a labeled eval set, online via live telemetry, human-rated sample, or
LLM-as-judge. - Fallback / degradation behavior — what the system does when the threshold isn't met: revert to a rule-based path, suppress the AI surface, escalate to a human, or alert the on-call PM.
Leave out the population and a passing eval on easy inputs hides failure on the hard 10%. Leave out the fallback and a real production dip has no defined system response — someone has to improvise one during an incident. All five components exist precisely so nobody has to improvise.
A useful shorthand: "[Metric] on [population], measured [how/how often], must stay [threshold]; below that, [fallback]." If you can't fill in all five blanks, the criterion isn't ready to leave the PRD.
A Worked Spec Fragment: Acceptance Criteria for an AI Recommendation Feature
Here's what this looks like as an actual artifact, not a description of one — a fragment from the acceptance-criteria table for an AI-powered "you may also want" recommendation feature on a product page. The specific numbers are illustrative; set your own from your baseline, not from this table.
| ID | Criterion | Metric | Threshold | Measured By | If Threshold Is Missed |
|---|---|---|---|---|---|
| AC-1 | Relevance | Precision@3 (top-3 recs judged relevant) | ≥80% | Weekly human-rated sample, n=50 sessions | 2 consecutive weeks below 80% → revert to rule-based fallback, alert PM |
| AC-2 | Latency | P95 response time | ≤400ms | Continuous production telemetry | P95 >400ms for 15 min → feature flag disables AI path, serves cached popular items |
| AC-3 | Catalog safety | Rate of recommending out-of-stock/discontinued items | ≤1% | Hourly automated check vs. live inventory | Any single recalled-item recommendation → immediate rollback, incident opened |
| AC-4 | Diversity | Sessions with ≥2 distinct categories in top 5 | ≥90% | Weekly batch analysis | <90% for 1 week → adjust re-ranking weight; threshold change requires PM sign-off |
| AC-5 | User trust | % of sessions where user dismisses all AI recs | ≤15% | Weekly product analytics | >20% for 2 weeks → qualitative research review (not automatic rollback) |
Notice what each row does that "give good recommendations" couldn't: it names an owner-checkable metric, a cadence, and a concrete system response. AC-3 is the one most PMs skip — a safety or hard-constraint criterion with a near-zero tolerance, distinct from the quality criteria around it. Recommendation quality can degrade gracefully; recommending a recalled product can't.
If a criterion like AC-1 or AC-4 depends on matching against specific product attributes — category, compatibility, prior purchase — those relationships have to already exist and be queryable in your schema. Acceptance criteria can't fix a data model that doesn't capture the relationship you're testing for; see our PM's guide to reviewing a data model if that mapping isn't solid yet.
Choosing Thresholds You Can Actually Defend
Pick thresholds from a baseline, not a gut feeling — the prior system's performance, a human-expert benchmark, or, at minimum, a documented business tolerance for error. A threshold with no stated reference is really just a number someone liked the sound of, and it won't survive the first "why 80% and not 75%?" question in review.
Three sources of a defensible baseline, in order of strength:
- The system you're replacing. If a rules engine currently gets 65% precision, "must beat 65%" is a real bar; "must hit 95%" might be an unfunded fantasy for v1.
- A human benchmark. Have two trained raters independently label the same 100 cases; their agreement rate is a ceiling — an AI system rarely should be held to a bar humans themselves can't consistently clear.
- A stated business tolerance. For some criteria there's no natural baseline (a brand-new feature), so the threshold becomes a genuine product bet — document it as one, not as a measured fact.
Different feature types warrant structurally different metrics, and picking the wrong metric shape for the feature type is a common spec mistake:
| AI Feature Type | Primary Metric(s) | Typical Threshold Shape | Measurement Method |
|---|---|---|---|
| Recommendation / ranking | Precision@k, nDCG, diversity | Single percentage vs. baseline | Periodic human-rated sample + online click-through |
| Classification / extraction | Precision, recall, F1 per class | Per-class floor, not one aggregate number | Held-out labeled set, confusion matrix by segment |
| Generative / conversational | Task success rate, hallucination rate, human preference win-rate | Pass rate on a curated eval prompt set | LLM-as-judge scoring + periodic human audit |
| Agentic / multi-step | Task completion rate, steps-to-completion, unsafe-action rate | Completion % floor + hard ceiling on unsafe actions | Simulated-environment runs + production trace review |
The per-segment row for classification isn't a nuance — it's the whole point. In Designing Machine Learning Systems, Chip Huyen argues for slice-based evaluation: a model can post a healthy aggregate score while quietly failing an entire subgroup, and an aggregate-only threshold will never catch it. Write acceptance criteria against the slices that matter (new users, a specific locale, low-data sessions), not only the overall average.
Google's internal Rules of Machine Learning guide, authored by Martin Zinkevich, makes a related point from the infrastructure side: it devotes an entire section to monitoring, treating a model's live behavior as a product requirement to be watched continuously — not a data-science deliverable that's "done" at launch. Your AC table's measurement method column is where that discipline actually lands in the spec.
Testing Non-Deterministic Behavior: From Golden Sets to Continuous Monitoring
Non-deterministic systems need two testing layers, not one: an offline eval set that runs before every release, and online production monitoring that runs continuously after. Neither substitutes for the other — offline evals catch regressions before users see them; online monitoring catches drift that no fixed eval set could anticipate.
Offline (pre-release):
- Build a golden set — a fixed, versioned sample of representative inputs with agreed-correct (or agreed-acceptable) outputs, ideally including the hard cases and edge cases, not just the easy majority.
- Run every model or prompt change against the golden set before it ships; block the release if any AC threshold regresses.
- Version the golden set itself — when it changes, note why, so a metric shift can be traced to "the model got worse" versus "we changed what we're measuring."
Online (post-release):
- Sample a slice of live traffic on a fixed cadence (the AC-1 table above samples 50 sessions weekly) rather than eyeballing dashboards ad hoc.
- Alert on threshold breaches automatically — a criterion with a defined fallback but no automated trigger for it is a criterion nobody actually enforces.
- Track the escalation path for the model's least confident or lowest-scoring outputs — where a low-confidence case routes to a human reviewer instead of shipping directly to the user matters for trust as much as the model's raw accuracy does; that path deserves the same prototyping attention as the main flow, and it's worth wireframing the override and review UI with the same rigor covered in our PM's guide to responsive wireframing.
The NIST AI Risk Management Framework formalizes this into its Measure function, which explicitly calls for context-relevant, ongoing measurement rather than a one-time certification. The framework's core argument is that AI risk isn't resolved at launch — it's managed continuously.
Microsoft's Guidelines for Human-AI Interaction (Amershi et al.) push the same idea toward the interface: several guidelines call for making the system's confidence and failure modes visible to the user, rather than presenting every AI output with uniform authority. A well-specified fallback (AC-1's revert-to-rule-based, AC-3's rollback) is the acceptance-criteria version of that same guidance.
Common mistake: treating a passing offline eval as proof the feature is done. An offline golden set is necessarily a sample of a distribution that keeps shifting — new products, new user behavior, a model provider's silent update. Acceptance criteria that only run pre-release will miss exactly the failures that matter most: the ones that show up after ship.
Keeping Acceptance Criteria Alive as the Model Changes
Acceptance criteria for AI features aren't a one-time gate — they need to be revisited on a cadence, because the model, the data, and the fallback logic all keep moving after launch. A criterion written for one model version can silently stop making sense after a provider update, a prompt change, or three months of catalog drift.
Three triggers should always prompt a re-check of every AC in the table, not just the criterion that seems related:
- Underlying model change — a new model version, a switched provider, or a re-trained internal model can shift every metric at once, even ones that seemed unrelated to the change.
- Population shift — a new market, a new user segment, or a changed catalog changes what "the population" in your criteria even means.
- Threshold review date — set one explicitly (quarterly is a reasonable default) so a threshold doesn't quietly go stale just because nobody revisited it.
This is exactly why acceptance criteria belong in a living document rather than a static PRD that gets written once and archived. Prodinja's Spec Studio keeps requirements, acceptance criteria, and open questions in one structured document you can edit as the spec evolves — so when a threshold needs revisiting after a model update, it's a version on the same artifact engineers already build from, not a scattered comment thread nobody reads.
That structural pattern — the spec as a living artifact instead of a point-in-time snapshot — is the same one covered in our piece on keeping a living PRD always current.
Key Takeaways
- A probabilistic acceptance criterion needs five parts: population, metric, threshold, measurement method, and fallback behavior — missing any one leaves it untestable.
- "Good," "accurate," and "relevant" are not metrics — replace them with a named, computable measure like
precision@3,F1, ortask completion rate. - Set thresholds from a baseline — the system you're replacing, a human-rater benchmark, or a stated business tolerance — never from an unanchored gut number.
- Evaluate by slice, not just aggregate — a healthy overall score can hide a subgroup that's failing entirely, a point evaluation researchers like Chip Huyen emphasize directly.
- Non-deterministic features need two testing layers: a versioned offline golden set before release, and continuous online monitoring with automated alerts after.
- Every threshold needs a defined fallback and an owner — a criterion without an enforced consequence for missing it is a suggestion, not a requirement.
- Acceptance criteria for AI features expire — revisit them on model changes, population shifts, and a fixed review cadence, not just once at launch.
Frequently Asked Questions
What's the difference between AI acceptance criteria and normal acceptance criteria?
Normal acceptance criteria describe a deterministic pass/fail behavior for a given input. AI acceptance criteria describe a statistical bar — a metric measured over a sample, with a threshold and a fallback — because the same input can legitimately produce different outputs across runs or model versions.
How do you write an acceptance criterion when you don't have a baseline yet?
Use the best available proxy — a competitor's known behavior, a manual/rules-based process it's replacing, or a small human-rated pilot sample — and label the threshold explicitly as a hypothesis to validate, not a measured fact. Revisit it after the first few weeks of real data rather than treating the launch number as permanent.
What sample size is enough for a probabilistic acceptance criterion?
There's no universal number, but a rough floor is enough cases that a single outlier can't flip the metric — commonly 50-200 labeled examples for a weekly human-rated sample, scaled up for rarer failure modes (like safety violations) where you need enough volume to actually observe the rare event. Statistical significance calculators exist for exactly this judgment call.
Should every AI feature have a human-in-the-loop fallback?
Not every feature needs a human reviewer in the live path, but every AI feature needs a defined degradation behavior for when its threshold is missed — a rule-based fallback, a cached result, or an explicit "we show nothing" state. Whether that fallback also routes to a human reviewer depends on the cost of a bad output: high-stakes or hard-to-reverse actions warrant it more than a low-stakes recommendation does.
Can acceptance criteria written for one model version carry over to a new model?
Sometimes, but never assume it without checking — a model swap or provider update can shift multiple metrics simultaneously, including ones that seem unrelated to what changed. Treat any underlying model change as a trigger to re-run the full acceptance-criteria table against the new model before it ships, not just the criteria that look obviously affected.