SRM in A/B Testing: The Silent Bug to Catch

Sample ratio mismatch (SRM) is a statistically significant divergence between the traffic split you configured (say, 50/50) and the split your logs actually show (say, 52/48). It signals a broken data pipeline, not noise, common causes are faulty randomization, differential logging loss, or bot traffic. Treat any SRM-flagged experiment as invalid until the root cause is found and fixed.

Quick Answer: If a chi-square test on your arm counts returns p < 0.01, your split isn't random anymore, something upstream (redirect logic, event logging, bot filtering) is broken, and every metric downstream of it is unreliable until you find and fix the cause.

What Exactly Is Sample Ratio Mismatch

Sample ratio mismatch is what happens when the number of users landing in each arm of your experiment deviates from the ratio you configured, by more than chance would explain. A 50/50 test that logs 10,412 control users against 9,588 treatment users isn't automatically broken, that's within normal variance. One that logs 11,200 against 8,800 almost certainly is.

The distinction matters because every random split has some noise. Flip a fair coin 20,000 times and you won't get exactly 10,000 heads. The question SRM answers isn't "is the split perfectly even," it's "is the observed imbalance more than random chance would produce." That's a statistical test, not an eyeball check, and it's one most experimentation programs skip.

Why "Close Enough" Is the Wrong Instinct

A PM staring at a dashboard showing 51.3% vs 48.7% will often shrug and move on. That instinct is understandable and wrong. Ronny Kohavi, who ran experimentation platforms at Microsoft and Airbnb and co-authored Trustworthy Online Controlled Experiments, has written extensively about SRM as one of the most common silent killers of experiment validity, precisely because the imbalance looks small enough to ignore.

The problem isn't the size of the gap in isolation. It's what a statistically real gap implies: something is systematically routing, dropping, or filtering users differently between arms. That "something" almost never affects only the assignment counter. It affects who ends up in each bucket, which corrupts the comparison your entire experiment exists to make.

The Chi-Square Gut Check, Explained Simply

The standard test for SRM is a one-sample chi-square goodness-of-fit test comparing observed arm counts to expected counts under your configured ratio. Run it on raw assignment counts before you look at any outcome metric, it takes seconds to compute and should be a non-negotiable pre-analysis gate, not an optional diagnostic you reach for after a result looks strange.

Here's the mechanic without the notation overload:

State your expected split. If you configured 50/50 across two arms with N total users, expected count per arm is N/2 each.
Compare to observed counts. Take the actual logged users per arm from your experiment platform.
Compute the chi-square statistic. Sum of (observed - expected)^2 / expected across arms.
Check the p-value. With one degree of freedom (for a two-arm test), a p-value below roughly 0.001 is the conventional trigger threshold, this is intentionally stricter than the 0.05 threshold you'd use for a metric read-out, because you're running this check on every single experiment and want to control false alarms tightly.
If it trips, stop. Don't proceed to metric analysis. Investigate allocation integrity first.

Most modern experimentation platforms (Optimizely, Statsig, GrowthBook, and internal tools built on similar logic) can compute this automatically as a pre-flight check. If yours doesn't, a spreadsheet with CHISQ.TEST or a five-line script does the job. The point isn't tooling sophistication, it's making the check mandatory rather than optional.

Georgi Georgiev, author of Statistical Methods in Online A/B Testing, frames SRM detection as a precondition for trusting any experiment output, not an advanced technique reserved for mature programs. If you're only running significance tests on your primary metric, you're missing the check that would tell you whether that metric is even measuring what you think it's measuring.

A Worked Example

Arm	Expected (50/50 of 20,000)	Observed	Deviation
Control	10,000	10,412	+412
Treatment	10,000	9,588	-412

Plugging these into the chi-square formula: (412^2/10000) + (412^2/10000) ≈ 33.9. With one degree of freedom, that statistic corresponds to a p-value far below 0.001, this is a clear SRM flag, not noise. A gap that size, on a 20,000-user experiment, is not something a fair random assignment mechanism would produce by chance.

Why SRM Is a Hard Stop, Not a Rounding Quirk

SRM invalidates comparisons because it's usually evidence that the two arms differ in composition, not just in count, and a composition difference is exactly the confound randomization exists to prevent. Once assignment isn't clean, you can no longer attribute an observed metric difference to your treatment; it might just be who ended up in each bucket.

This is the shift data-oriented PMs need to internalize: a passed SRM check is a precondition for reading results, not a nice-to-have footnote. If you wouldn't publish a result with an obviously broken randomizer, you shouldn't publish one with a statistically confirmed allocation skew either, the mechanism of harm is identical, it's just less visible.

Consider what a mismatch typically means in practice:

If bots hit one arm's redirect path more than the other's, treatment metrics get diluted or inflated by non-human behavior that has nothing to do with your feature.
If a slow-loading treatment variant causes users to bounce before an assignment event fires, you've silently filtered out exactly the impatient users most likely to convert poorly, and treatment now looks artificially better.
If mobile app version gating routes older app versions only to control, you're not testing your feature, you're testing app-version cohorts that also differ in tenure, device, and behavior.

Every one of those scenarios produces a metric delta that looks like a real product effect but is actually a sampling artifact. This is why an SRM failure poisons every downstream number, not just the primary metric, guardrail metrics, secondary metrics, and segment cuts all inherit the same corrupted assignment.

Comparing SRM to Other Data Quality Failures

Failure mode	What breaks	Detectable via	Fixable post-hoc?
Sample ratio mismatch	Arm composition/comparability	Chi-square test on assignment counts	Rarely, usually requires re-run
Novelty/primacy effect	Metric trend over time	Time-sliced metric plots	Partially, with longer run duration
Metric definition drift	Metric meaning across time	Instrumentation audit	Yes, with a corrected re-pull
Underpowered sample	Ability to detect true effect	Power calculation pre-launch	Yes, extend run duration

The distinguishing feature of SRM is that it's not a power or duration problem you can wait out. Running a mismatched experiment longer just accumulates more mismatched data. The only fix is finding what's breaking assignment and correcting it, which usually means a re-launch.

Common Root Causes of SRM

Most SRM cases trace back to one of a handful of recurring failure patterns: broken randomization logic, differential activity logging, bot or crawler contamination, or a redirect/caching layer that treats arms unevenly. Diagnosing which one you're facing usually starts with segmenting the mismatch by platform, geography, and time.

Broken or non-uniform randomization. A hashing function seeded on something correlated with user attributes (like account creation timestamp) instead of a uniform random ID can produce a skew that looks random but isn't. Test your randomization unit independent of any real experiment before trusting it.
Differential logging or event loss. If the treatment experience is slower, crashes more, or triggers a client-side event that fails silently on certain browsers, you lose treatment users disproportionately at the logging stage, even though assignment itself was fine.
Bot and crawler traffic. Automated traffic often behaves deterministically (same IP ranges, same user agents, hitting the same endpoints repeatedly) which can concentrate in one arm if your bot filter runs after assignment instead of before it.
Caching and CDN interference. A cached page response served to a user who should have gotten a fresh assignment can lock users into whichever variant got cached first, especially on first-visit experiments.
Redirect-based experiment implementations. Server-side redirects that fail, retry, or time out differently for one variant's destination URL will drop users unevenly, and those drops correlate with exactly the users experiencing friction.
Pre-existing exposure logging bugs. If your platform logs "exposure" only after a page fully renders, and one variant renders slower, you systematically under-log the slower variant's true traffic.

None of these are exotic. They're the ordinary failure modes of any system that routes traffic differently for two versions of an experience. The reason SRM catches them is that it's agnostic to which specific bug caused the skew, it just tells you a skew of statistical significance exists, and that's the trigger to go looking.

Where to Look First When SRM Fires

Segment the mismatch before you dig into infrastructure logs. Break the skew down by device type, browser, geography, and time-of-day, an SRM that's uniform across every segment points toward assignment logic; an SRM concentrated in one browser or one region points toward logging or rendering differences specific to that segment. This segmentation step usually cuts investigation time from days to hours.

If you're still building the muscle of specifying what "good" looks like before you launch, it helps to have already written down your assumptions and success criteria clearly. A well-structured testable product hypothesis makes it much easier to notice when the population you're actually measuring doesn't match the population you intended to test.

Making SRM Checks a Standing Habit, Not a One-Off

The reason SRM keeps recurring across otherwise mature experimentation programs is that it's a check people run when something already looks wrong, not a check baked into every experiment's readout by default. Fixing that requires making the check structurally unavoidable rather than dependent on someone remembering to run it.

Practical ways to build the habit:

Automate the chi-square test as a required gate before any experiment dashboard displays metric results, not an optional tab someone might not click.
Log assignment counts continuously, not just at analysis time, so a skew is visible within days of launch instead of discovered at the end of a four-week run.
Alert on skew during the ramp-up period, when a broken randomizer or logging bug is cheapest to catch, before most of your sample has already been spent.
Document root causes when found, so recurring infrastructure issues (a specific CDN behavior, a specific bot pattern) get fixed once instead of re-diagnosed every quarter.

This is also where the discipline connects back to earlier stages of experiment design. If you haven't yet nailed down the basics of how to set up your first A/B test correctly, including exposure logging and randomization unit choice, an SRM gate is a safety net, not a substitute for getting those fundamentals right at design time. And once a test does pass its SRM check, you still owe yourself the discipline of reading the handful of numbers that actually determine statistical significance before declaring a winner.

How This Fits the Bigger Experimentation Picture

SRM is one gate among several that a rigorous experimentation practice runs before trusting a result, alongside power calculations, guardrail metric checks, and novelty-effect monitoring. If you're building out a fuller picture of what a trustworthy testing program looks like end to end, it's worth grounding SRM checks inside a broader experimentation program rather than treating it as an isolated statistical trick.

It's also worth remembering that experiments don't exist in a vacuum, they're meant to validate hypotheses that came from somewhere. If your hypotheses are grounded in real user needs rather than guesses, understanding the underlying jobs customers are hiring your product to do or mapping friction across the customer journey gives you a stronger starting hypothesis, which means fewer wasted experiment cycles even before SRM enters the picture.

Where Prodinja Fits Into This

Prodinja's Spec Studio treats an experiment's readiness gates as a living checklist attached to the spec itself, not a separate document someone has to remember to consult. You can log a sample ratio mismatch check as one of those standing pre-analysis gates, so a spec's readout section is designed to stay locked until that check, alongside your other data-quality gates, has been marked passed. The goal is structural: no one reads out a result before the sanity checks are on record, not because a person double-checked, but because the workflow is built to ask.

This doesn't replace your statistics, you still need to run the chi-square test against your actual assignment data, wherever that lives. What it changes is whether that check is a step baked into how every experiment gets reviewed, versus a best practice that quietly erodes under deadline pressure.

Key Takeaways

Sample ratio mismatch is a statistically significant gap between your configured split and your observed split, detected with a chi-square goodness-of-fit test, not an eyeball comparison.
A p-value below roughly 0.001 on the assignment chi-square test is the conventional trigger to stop and investigate before reading any metric.
SRM is a hard stop, not a rounding quirk, because it usually signals that arm composition differs, which is exactly the confound random assignment is supposed to prevent.
Common root causes include broken randomization, differential event logging, bot traffic, caching interference, and uneven redirect failures.
Segmenting the mismatch by device, browser, and geography before digging into logs typically narrows the root cause much faster than reading raw infrastructure logs first.
Automating the SRM check as a mandatory pre-analysis gate, rather than an occasional manual step, is what actually makes it stick across an experimentation program.
A failed SRM check invalidates every downstream metric, not just the primary one, because the sampling artifact affects the whole population comparison, not a single number.

Frequently Asked Questions

What causes sample ratio mismatch in A/B testing?

Sample ratio mismatch is most often caused by broken or non-uniform randomization logic, differential event logging between variants, bot or crawler traffic concentrating in one arm, caching layers serving stale assignments, or redirect failures that drop users unevenly. The common thread is a pipeline stage that treats the two arms differently, even if the original randomization was sound.

How do you test for sample ratio mismatch?

Run a one-sample chi-square goodness-of-fit test comparing your observed arm counts to the counts expected under your configured split. A p-value below roughly 0.001 is the conventional threshold to flag a real mismatch worth investigating, rather than normal random variance in the split.

Is a small split difference like 51/49 a problem?

It depends entirely on sample size, not the raw percentage gap. On a small sample, 51/49 can be well within random noise; on a sample of hundreds of thousands, that same ratio can produce a chi-square statistic far past the significance threshold. Always run the actual test rather than judging by eye.

Can you fix an experiment after detecting SRM, or do you have to restart it?

In most cases, you need to find and fix the root cause, then re-run the experiment from a clean launch, because the corrupted assignment period can't be cleanly separated from valid data after the fact. Occasionally a mismatch traced to a clearly isolated logging bug on a known date range can be excluded, but that requires strong evidence the bug didn't affect user behavior itself, not just the log.

Does sample ratio mismatch affect segment-level analysis too?

Yes, an SRM failure poisons every metric derived from that experiment's population, including guardrail metrics and any segment cut, not only the primary metric you were tracking. The underlying problem is that the two populations being compared aren't the ones you intended to compare, and that distortion propagates through every downstream calculation.