Why AI developers should report confidence intervals in error bars, not standard errors

Look at this chart. What do the error bars convey?

A normal person would say it shows GPT-5.5 clearly outperforms Claude Opus 4.7 on the BiglyBenchmark. But that’s not accurate. Why?

Lately, I’ve been seeing more error bars on my social media feed from developers. Anthropic seems to have remembered them. DeepMind reports results of their multi-agent math system with Accuracy ± standard errors. This is certainly a big improvement in the Wild West of AI evaluation, where raw means are reported without any uncertainty estimates. Even among AI research, only 16% of 445 benchmark papers conducted any statistical tests, let alone the right ones.

However, in many cases, these “error bars” don’t have a clear label. ScaleAI reported uncertainty ± without labels, and the full leaderboard labels them only a “range.” Anthropic labels BioMysteryBench benchmark results with “Error bars computed by bootstrap sampling within problems.” But what exactly was computed? A standard error—or a confidence interval? And if the latter, at what confidence level? Similarly, EpochAI’s interactive widget labels error bars as standard errors, but only in two views of the data.

“So what?,” you might think. “Reporting uncertainty is better than none!”

Sure. But standard errors can mislead in an important way.

How standard errors as error bars mislead

Consider this chart that uses standard errors as error bars:

You might imagine that GPT-5.5 is the clear winner with certainty, since its error bar doesn’t overlap the others. Yet that intuition is not accurate: standard errors do not indicate where the “true” value might lie—they instead describe how much the sample mean would vary across repeated experiments. And a standard error bar plotted as ±1 SE only covers about 68% of the probability distribution for a normal distribution.

That means whenever you see a “standard error” error bar, you should effectively double the width if you want to visually compare performance in terms of classical notions of statistical significance:

So when AI labs plot ±1 SE without labeling it, they are visually shrinking their uncertainty by about half. This can make the performance gap between two AI models look highly significant and separate, when a proper 95% confidence interval might show a complete overlap.

Confidence intervals

Confidence intervals (CIs), by contrast, do provide an interval that is designed to contain the true population parameter with a specified level of probability (typically 95%). And the visual interpretation of CIs doesn’t depend on the data following a normal distribution—unlike SEs—but rather the calibration of the method used to compute them, which we can verify with simulations on evals-like data.

Confidence intervals are the recommended method by scientists when reporting point estimates that may be compared. To be clear, standard errors are useful too, but they lead people to be misleadingly overconfident when used as error bars.

To bring maturity to AI evaluations, this is the simple ask: when you plot benchmark results, use 95% CIs, and write “Error bars: 95% CI” in the caption. Bootstrap CIs require no distributional assumptions and work at the small sample sizes typical of developer evals. The evalstats library computes them with one line of code.

We’re at an inflection point in AI evaluation culture. Error bars are appearing. Now we need to make sure they’re the right ones, labeled correctly. Standard errors look precise but quietly mislead. Confidence intervals tell the honest story.

Maybe we’re displaying uncertainty wrong?

Now, confidence intervals are certainly the best estimate we have of uncertainty, and reporting them is critical. I think this is as much as we can ask current developers.

However, there is known evidence that, though CI-based error bars lead to lower false positives compared to standard errors, the way we visualize these errors can still mislead.

Should we change our plots, too?

That is the argument of Error Bars Considered Harmful…, Correll and Gleicher (2014). They argue that bar charts with error bars suffer from two systematic biases:

• Within-the-bar bias: the glyph of a bar provides a false metaphor of containment, where values within the bar are seen as more likely than values outside it.
• Binary interpretation: values are within the margins of error, or they are not. This makes it difficult for viewers to make nuanced inferences about outcomes, and causes viewers to overestimate effect sizes when comparing bars.

They show in a series of experiments that gradient plots—where the 95% CI region is shown as a solid block of color and opacity decays continuously beyond it—lead to more accurate inference by humans.

The original paper derived the outer decay from a t-distribution. A distribution-free adaptation is to compute CIs at 90%, 95%, 99%, and 99.9% using the same method as before (bootstrap, t-interval, Bayesian—any will do), then map each boundary to an opacity. Here we make the 90% CI the fully opaque band, with 95% slightly faded, and steeper decay beyond. The reason to anchor on 90% rather than 95% comes from Belia et al. (2005): when two 95% CIs on independent means just touch, p ≈ .006—not .05 as many assume. The 90% CI is a closer visual proxy for the p ≈ .05 boundary, so solid-band overlap gives viewers a better-calibrated intuition for classical significance:

The visual result is a solid, readable bar with soft, fuzzy edges. It communicates two things a capped error bar cannot: that uncertainty is not binary (there is no sharp inside/outside), and that within-the-bar bias is weakened because the bar no longer looks like a filled container. Viewers shown this design make more calibrated judgments about whether two point estimates truly differ.