At a glance
- Category
- Skillx Valley
- Difficulty
- Hard
- Play time
- 3–5 min
- Price
- Free
What is A/B Test Oracle?
Ten experiments from made-up startups, each with its dashboard: conversion for A and B, the sample, the chance B beats A, how far into the plan it is and a note. Ship B, kill it, or keep it running, and on Expert say why: a small sample, noise, a novelty effect, peeking, a guardrail, or a real result.
A/B Test Oracle is a free experimentation game in Skillx Valley. Made-up startups bring their A/B tests with a dashboard: conversion for A and B, the sample, the chance B beats A, the day of the plan, and a note.
Call each one: ship B, kill it, or keep it running. The traps are the classic ones: a sample too small, a gap inside the noise, a lift that fades (novelty), stopping early because it crossed 95% (peeking), and a guardrail metric that got worse. On Expert, also name the deciding check.
How to play
- Call1 Ship B · 2 Kill it · 3 Keep running. +100 if right, +10 per test.
- WhyOn Expert, name the reason too: +50.
- WinMake the right call on 70% of the experiments.
- Three levelsBeginner, Intermediate or Expert, picked before you play: the cases and the clock change, and each level keeps its own best score.
- Made upEvery startup, fund, product and person in the game is made up. Coins are fictional: not financial advice.
Scoring
+100 for the right call, +10 per test, +50 for naming the deciding check on Expert. A test never scores below zero.
Win with the right call on at least 70% of the experiments.
What it trains
- Experimentation
- Sample size, significance, novelty and peeking.
- Data judgement
- When to ship, kill or wait.
Strategy and tips
- Four checksA big enough sample, B clearly ahead, the planned end reached, and a steady lift with guardrails fine.
- Too early is not a lossA small sample or a test cut short means keep running, not kill.
- Read the noteRefunds doubling or a lift shrinking each week can overturn a great-looking average.
Why play A/B Test Oracle?
Teams ship false winners when they stop tests early or ignore side effects. A few simple checks make experiment results trustworthy.
Frequently asked questions
Is A/B Test Oracle free?
Yes. It is free and runs in your browser, with no download. Signing up is optional.
What is an A/B test?
An experiment that shows two versions to similar groups of people and compares a metric, such as sign-ups.
What is peeking?
Checking a test repeatedly and stopping the first time it looks significant, which produces many false winners.
Are the startups real?
No. Every startup, fund, product and person in the game is made up. The skills it teaches are real.
What is Skillx Valley?
Skillx Valley is the SkillXGames zone of games about startup and founder skills: fundraising, investing, customer interviews, pricing, churn, cold email, hiring, unit economics, experiments, team habits and the decision to pivot.
Are there difficulty levels?
Yes: Beginner, Intermediate and Expert. The cases and the clock change, and each level keeps its own best score.
