Which Method? — Stats for LLM Evals

Recommended Methods

These are evalstats' current default recommendations, chosen automatically under method="auto" based on your data's estimand, data type, and sample size. They summarize the results of an extensive Monte Carlo simulation study, validated against real LLM eval data. Boxed methods are the default; gray notes give the multi-run variant and conservative alternatives. Paired data is assumed throughout the p-value / FWER tree below.

95% Confidence Interval

What are you estimating a CI for?

Mean point estimate
- Binary (0 / 1)
  
  Wilson score interval Multi-run: Wilson flat
- Numeric (continuous or Likert)
  
  Logit-t interval Multi-run: on run means. Conservative: NIG (continuous), smooth bootstrap (Likert)
Mean of paired differences
- Binary (0 / 1)
  
  Bonett–Price adjusted Wald Multi-run: B–P shrunk
- Continuous
  
  Logit-t interval Multi-run: on run mean differences. Conservative: smooth bootstrap (Gaussian KDE)
- Likert / discrete
  
  Bayesian NIG Multi-run: on run mean differences

p-value / FWER Correction (paired data)

How many comparisons are you running?

Single pairwise comparison

Wilcoxon signed-rank test Conservative: smooth bootstrap (n≥30, Gaussian KDE); sign test (ultra-conservative)
Family of comparisons (k≥3)
- Need simultaneous CIs
  
  Šidák procedure Every n and eval type
- Need p-value correction
  - n < 30
    
    Shaffer's procedure Corrects the Wilcoxon p-value directly. Alternative: Benjamini–Hochberg if FDR suffices
  - n ≥ 30
    
    Romano–Wolf step-down Replaces Wilcoxon p-values with its own joint statistic

More detail coming soon

These recommendations are backed by comprehensive Monte Carlo simulations across eval data types, sample sizes, and comparison setups, cross-checked against real LLM eval data. A paper with the full methodology, simulation results, and justification behind each recommendation will be available soon.