This roadmap covers shipped work, planned investigations, and library features still ahead. Items are ordered roughly by priority. The current focus is finishing the paper behind evalstats' method recommendations and polishing the package for a 1.0 release.

Shipped

The core comparison workflow — comparing models, prompts, and model×prompt combinations, across single-run and multi-run (seed variance) settings — is done and simulation-backed. Every single-run CI/p-value method has a multi-run counterpart, chosen automatically. So is a second major pillar that wasn't yet started when this roadmap was first written: prediction-powered inference (PPI) for mixed human-AI judge designs. Concretely:

  • PPI-corrected hypothesis tests: bias-corrected versions of nine standard tests (t-test, Welch's, Mann-Whitney U, paired t-test, Wilcoxon signed-rank, repeated-measures and one-way ANOVA, Friedman, Kruskal-Wallis) via evalstats.tests, including what we believe are the first PPI corrections for rank-based nonparametric tests.
  • Auditing your LLM judge: judge_alignment() reports the judge-human correlation appropriate to your test and the resulting effective sample size, with rule-of-thumb thresholds for when a judge is (or isn't) worth using.
  • Family-wise error control for multiple comparisons: Šidák for simultaneous CIs, Shaffer's or Romano–Wolf for p-value correction depending on N — see Which Method? for the full decision tree.
  • An evalstats label CLI command to draw a random, MCAR-compliant sample of items for human labeling, respecting PPI's sample-size floors.

Investigation notebooks published so far:

  • Model A vs. Model B: Is the Gap Real?: CIs on the score difference, single-run and multi-run nested-bootstrap variants, minimum N to detect a given gap, and honest reporting of statistical ties.
  • Finding Your Best Prompt: Per-variant CIs with multiple-comparison correction, single-run and multi-run modes, and how to report “Prompt C wins” with appropriate hedging.

Current Focus

The technical validation behind every default above — the Monte Carlo simulations, the real-data cross-checks, and the PPI calibration study — is being written up as a paper. Until it's available, Which Method? summarizes the recommendations directly. Alongside that: polishing the package and this guide for a 1.0 release, and closing the gaps below.

  • Between-subjects (unpaired) comparisons: compare()/analyze() currently assume a within-subjects design (every compared entity scores the same items). Genuinely unpaired designs — common in HCI, e.g. independent reviewer groups — have PPI-corrected test functions available directly (evalstats.tests), but no first-class compare()-style entry point yet.
  • Multi-run PPI correction: combining repeated-run data with LLM-judge bias correction in a single pass is not yet supported — currently you pick one or the other.
  • Finding the Best Model×Prompt Combo: factorial (two-way) analysis to detect interaction effects, score-matrix heatmaps with CI width, and multi-run variants.
  • Comparing Models Across Multiple Metrics: Pareto dominance as a weight-free criterion, composite score uncertainty, and visualizations that surface tradeoffs rather than a pre-digested verdict.

Near-Future Planned Foci

More nuanced investigations for practitioners who need to answer harder questions about seed variance, fine-tune validation, pipeline optimization, and eval set sizing.

  • Checking Response Consistency: Measuring run-to-run variance and related factors.
  • Before / After: Did My Fine-Tune Actually Help?: Simple before/after comparison methods.
  • How Many Eval Items Do I Actually Need?: Estimating required sample sizes, including with a mixed human-AI judge design.
  • Prompt Sensitivity Analysis: Exploring prompt effects and stability.
  • Stats for RAG Pipelines: Methods for analyzing RAG pipeline setups.
  • Regression Guard: Tools for regression detection and CI integration.
  • When Model Rankings Are Just Noise: Assessing ranking reliability.
  • Distilling a Benchmark: Reducing and validating benchmark sets.

Far-Out Explorations

I intend to explore alternative statistical paradigms, like Bayesian methods and e-values, which could provide more nuanced insights and better support for the iterative, noisy nature of LLM evals. However, I want to first provide some sound, "best effort" defaults within a more familiar frequentist framework, which many practitioners are already using and which is still widely requested by stakeholders, reviewers, and auditors. One idea here is for evalstats to add a Bayesian mode, where users can toggle between the two paradigms and get outputs that are consistent with the underlying philosophy without needed to change top-level API calls (e.g., compare_models() will report either frequentist or Bayesian outputs depending on the mode). This guide would then have an "alternative universe" where the same website and investigation notebooks are re-framed through a Bayesian lens, and unique Bayesian investigations are added (e.g., sequential testing and evidence combination across tests).