Quick Start (CLI)
This guide walks you through running your first analysis using the command-line interface (CLI).
All you need is a CSV file of your eval scores in an expected format, and a few seconds to run the analysis and read the report.
The evalstats workflow can be described as:
0. Install evalstats
You can install evalstats via pip or uv:
pip install evalstats1. Prepare your CSV
evalstats expects a long-format (tidy) CSV: one row per
observation. At minimum you need input and score, plus either
prompt/template or model.
The input column identifies which individual test case each
row belongs to (e.g., questions in a Q&A dataset), which this lets evalstats use paired statistics that account
for per-input difficulty differences.
If you evaluated each prompt multiple times per input (e.g. at different random seeds),
add a run column. With 3 or more runs per cell, evalstats
automatically switches to an appropriate method like nested bootstrap
that accounts for run-level variance on top of input-level variance,
giving you more honest uncertainty estimates.
| prompt | input | run | score |
|---|---|---|---|
| prompt_a | q_001 | 0 | 0.68 |
| prompt_a | q_001 | 1 | 0.71 |
| prompt_a | q_001 | 2 | 0.74 |
| prompt_b | q_001 | 0 | 0.82 |
| prompt_b | q_001 | 1 | 0.79 |
| prompt_b | q_001 | 2 | 0.85 |
| prompt_c | q_001 | 0 | 0.89 |
| prompt_c | q_001 | 1 | 0.93 |
| prompt_c | q_001 | 2 | 0.91 |
| … | … | … | … |
Column names are matched case-insensitively. Accepted aliases:
prompt / template / prompt_template —
input / example / item / id —
score / value / result / metric —
run / seed / repeat / run_id / trial.
Add a model column for multi-model data.
2. Run the analysis
Point evalstats analyze at your file. It auto-detects the data type
(binary, continuous, Likert, etc.), recognizes the multi-run structure, and picks an
appropriate CI method automatically.
evalstats analyze results.csv3. Read the output
The terminal report breaks the analysis into several sections. Here's what you'd see for the CSV above (20 inputs × 3 prompts × 3 runs, continuous scores):
--- Mean Performance (marginal logit-t CIs) --- axis: [0.518, 0.975] (· ±1σ, ░▒▓█ CI gradient [99%/95%/90%/68%], │ grand mean) Prompt Interval Plot Mean CI Low CI High prompt_a ···░▒▓█████▓▒░░···│ 0.636 0.589 0.680 prompt_b ···░░▒▓██│███▓▒░···· 0.753 0.700 0.799 prompt_c │ ··░░▒▓█████▓▒░··· 0.864 0.816 0.901 --- Pairwise Comparisons (Paired logit-t CIs) --- legend: (· ±1σ, ░▒▓█ CI gradient [99%/95%/90%/68%], │ zero) axis: [-0.369, +0.369] effect: Left - Right Left Right Interval Plot Mean CI Low CI High ES prompt_c prompt_b ··│░▒▓█████▓▒░··· +0.1109 +0.0203 +0.1997 0.667 prompt_c prompt_a │ ··░░▓▓████▓▒░··· +0.2282 +0.1444 +0.3086 0.981 prompt_b prompt_a │·░░▓████▓░·· +0.1172 +0.0513 +0.1821 0.857 ES = Effect Size (r_rb) = rank biserial correlation (small≈0.1, medium≈0.3, large≈0.5) CI method: Paired logit-t | α=0.05 Simultaneous CI method: Šidák Statistically indistinguishable rank bands (95% CI): none --- Per-input Variance Across Runs (R=3 runs) --- key: ▁–█ = per-input noise (globally scaled; █ = 0.0860) Prompt Per-input noise run_std input_std total_std instability Consistency (ICC) Verdict prompt_a ▄▂▆▄▆▄▂▄▇▇▅▃▄▃▄▄▅▅▄▅ 0.0459 0.0968 0.1071 0.0428 0.82 (good) very stable across runs prompt_b ▃█▂▃▃▄▃▅▆▇▄█▁▅▃▆▄▂▃▇ 0.0485 0.1062 0.1168 0.0425 0.83 (good) very stable across runs prompt_c ▄▂▁▆▃▁▇▁▇▆▆▆▃▄▆▄▁▄▆▄ 0.0453 0.0901 0.1009 0.0376 0.80 (good) very stable across runs instability = how many points a score typically moves between repeated runs Consistency (ICC) = how much of the difference between inputs is real signal, rather than run-to-run noise --- Executive Summary (Prompt leaderboard) --- Prompt Grp Mean CI Per-run noise Stability Verdict ───────────────────────────────────────────────────────────────────────────────────────── prompt_c #1 0.864 [0.816, 0.901] ▂▃▅▆▃▅▂▅ Stable Likely best prompt_b #2 0.753 [0.700, 0.799] ▅▃▅▆▃▄▃▅ Stable Significant drop-off prompt_a #3 0.636 [0.589, 0.680] ▄▅▅▅▃▄▅▅ Stable Significant drop-off ─────────────────────────────────────────────────────────────────────────────────────────
Start at the Executive Summary at the bottom. Entities in the same rank
group (#1, #2, …) are statistically indistinguishable
after multiple-comparisons correction — even if one has a higher mean. Here, all three
prompts land in their own group: prompt_c is the clear best (#1),
prompt_b is a clear step down (#2), and prompt_a
trails behind that (#3).
The Pairwise Comparisons section shows a visual CI plot and effect-size
(ES) for each pair. When the CI interval plot spans the zero line (│), the
difference is not significant. The Statistically indistinguishable rank
bands line summarises which adjacent groups cannot be distinguished.
Saving output
# Save a Markdown report and plot
evalstats analyze results.csv --out report.md plot.png
# Save structured JSON for programmatic use
evalstats analyze results.csv --out analysis.json
# Print just the executive leaderboard (fastest read)
evalstats analyze results.csv --brief
See evalstats analyze --help for the full option list.
Python API
All functions accept NumPy arrays or plain Python lists and return a result object
with a .summary() method that prints the same terminal report as the CLI.
Comparing prompts
Pass a dict mapping prompt name → per-input score array. Each array must be the same length and correspond to the same inputs (paired data).
import numpy as np
import evalstats as estats
scores = {
"prompt_a": np.array([0.72, 0.60, 0.88, 0.75, ...]),
"prompt_b": np.array([0.85, 0.78, 0.90, 0.82, ...]),
"prompt_c": np.array([0.91, 0.83, 0.87, 0.95, ...]),
}
result = estats.compare_prompts(scores)
result.summary() # full terminal reportComparing models
Same interface as compare_prompts, keyed by model name instead.
scores = {
"gpt-4o-mini": np.array([0.82, 0.91, 0.78, ...]),
"gemma-3-4b-it": np.array([0.75, 0.84, 0.71, ...]),
"qwen3-8b": np.array([0.88, 0.93, 0.85, ...]),
}
result = estats.compare_models(scores)
result.summary()Comparing models × prompts
Pass a nested dict — outer keys are models, inner keys are prompt templates.
evalstats reports statistics for every model–prompt combination,
letting you see whether a prompt improvement generalizes across models.
scores = {
"gpt-4o-mini": {
"5-shot": np.array([0.79, 0.85, 0.83, ...]),
"0-shot": np.array([0.61, 0.73, 0.69, ...]),
},
"gemma-3-4b-it": {
"5-shot": np.array([0.68, 0.77, 0.72, ...]),
"0-shot": np.array([0.60, 0.72, 0.65, ...]),
},
}
result = estats.compare_models(scores, show_rank_probabilities=True)
result.summary()================================================
CROSS-MODEL RANKING (ALL MODEL/TEMPLATE PAIRS)
================================================
5-shot 0-shot
─────────────────────────────────
gpt-4o-mini 0.797 █* 0.676 ░
gemma-3-4b-it 0.708 ▒ 0.630 ·
─────────────────────────────────
* = statistically tied for best (95% CI, not significantly beaten) | heat: · (low) → █ (high), range [0.630, 0.797]
--- Rank Probabilities: All 4 by P(Best) (bootstrap, n=10000, ranked by mean) ---
Model Template P(Best) E[Rank]
gpt-4o-mini 5-shot 99.9% ██████████████ 1.00 █▆▃───────────
gemma-3-4b-it 5-shot 0.1% ░░░░░░░░░░░░░░ 2.15 ───▃▆█▆▃──────
gpt-4o-mini 0-shot 0.0% ░░░░░░░░░░░░░░ 2.91 ──────▃▆█▆▃───
gemma-3-4b-it 0-shot 0.0% ░░░░░░░░░░░░░░ 3.93 ───────────▃▆█
--- Mean Performance: All 4 (marginal CIs) ---
axis: [0.471, 0.935] (· ±1σ, ░▒▓█ CI gradient [99%/95%/90%/68%], │ grand mean)
Model Template Interval Plot Mean CI Low CI High
gpt-4o-mini 5-shot ··│··░░▓█████▓░······ 0.797 0.757 0.832
gemma-3-4b-it 5-shot ········░▒▓██│███▓░░········ 0.708 0.658 0.754
gpt-4o-mini 0-shot ······░▒▓████│▒░······ 0.676 0.636 0.714
gemma-3-4b-it 0-shot ······░░▓▓█████▓░░│······ 0.630 0.585 0.672
--- Executive Summary (Cross-model pair leaderboard) ---
Model Template Grp Mean CI Verdict
─────────────────────────────────────────────────────────────────────
gpt-4o-mini 5-shot #1 0.797 [0.757, 0.832] Likely best
gemma-3-4b-it 5-shot #2 0.708 [0.658, 0.754] Significant drop-off
gpt-4o-mini 0-shot #2 0.676 [0.636, 0.714] Significant drop-off
gemma-3-4b-it 0-shot #2 0.630 [0.585, 0.672] Significant drop-off
─────────────────────────────────────────────────────────────────────
In the crossed leaderboard, gpt-4o-mini / 5-shot ranks alone at the top.
The 5-shot setting is stronger for both models, but only
gpt-4o-mini / 5-shot is statistically distinguishable from the rest — the
other three rows (gemma-3-4b-it / 5-shot, gpt-4o-mini / 0-shot,
and gemma-3-4b-it / 0-shot) all land in the same group (#2),
meaning they aren't cleanly separated from each other at this confidence level.
Looking for p-values?
Pass p_values=True to print corrected p-values in the pairwise table.
You can optionally set pairwise_test="wilcoxon" to force Wilcoxon
signed-rank p-values (instead of auto-selection). With omnibus=True,
evalstats also prepends a Friedman omnibus test line before the
pairwise rows.
scores = {
"gpt-4o-mini": np.array([0.82, 0.91, 0.78, 0.86, 0.74, ...]),
"gemma-3-4b-it": np.array([0.75, 0.84, 0.71, 0.79, 0.68, ...]),
"qwen3-8b": np.array([0.88, 0.93, 0.85, 0.91, 0.83, ...]),
}
result = estats.compare_models(
scores,
omnibus=True,
p_values=True,
pairwise_test="wilcoxon",
)
result.summary()--- Mean Performance (marginal logit-t CIs) ---
axis: [0.566, 0.999] (· ±1σ, ░▒▓█ CI gradient [99%/95%/90%/68%], │ grand mean)
Model Interval Plot Mean CI Low CI High
gpt-4o-mini ······░░▒▓█│███▓▒░········ 0.808 0.761 0.848
gemma-3-4b-it ·······░░▓▓█████▓▒░│······· 0.729 0.681 0.772
qwen3-8b ·····░│▒▓▓████▓▒░······· 0.854 0.809 0.890
--- Pairwise Comparisons (Paired logit-t CIs) ---
Friedman omnibus: χ²(2) = 13.371, p = 0.001249*
legend: (· ±1σ, ░▒▓█ CI gradient [99%/95%/90%/68%], │ zero) axis: [-0.329, +0.329] effect: Left - Right
Left Right Interval Plot Mean CI Low CI High ES p (wsr)
qwen3-8b gpt-4o-mini ·····░░│████▓▒░······ +0.0456 -0.0198 +0.1106 0.326 0.07635
qwen3-8b gemma-3-4b-it ·····│·░░▓▓█████▓▒░······· +0.1248 +0.0435 +0.2044 0.639 0.0007445**
gpt-4o-mini gemma-3-4b-it ·······░│▓▓█████▓▒░······· +0.0792 -0.0025 +0.1598 0.480 0.007264*
ES = Effect Size (r_rb) = rank biserial correlation (small≈0.1, medium≈0.3, large≈0.5)
CI method: Paired logit-t | p-value method: Wilcoxon signed-rank | α=0.05
Simultaneous CI method: Šidák | FWER correction for p-values: Romano-Wolf
p (wsr) = Wilcoxon signed-rank (romano_wolf-corrected)
stars: * p<0.01, ** p<0.001, *** p<0.0001
Statistically indistinguishable rank bands (similar to critical difference diagrams) computed from 95% CI:
#1–#2: [qwen3-8b ─ gpt-4o-mini]
#2–#3: [gpt-4o-mini ─ gemma-3-4b-it]
--- Executive Summary (Model leaderboard) ---
Model Grp Mean CI Verdict
────────────────────────────────────────────────────────
qwen3-8b #1 0.854 [0.809, 0.890] Tied with gpt-4o-mini as best
gpt-4o-mini #1 0.808 [0.761, 0.848] Tied with qwen3-8b as best
gemma-3-4b-it #2 0.729 [0.681, 0.772] Significant drop-off
────────────────────────────────────────────────────────
The Friedman omnibus test (χ²(2) = 13.37, p = 0.0012) confirms there is overall
variation across models. In the pairwise comparisons, qwen3-8b vs
gpt-4o-mini has a CI that crosses zero ([-0.020, +0.111] — not
significant), so both land in rank group #1. The
p (wsr) column reports Romano–Wolf-corrected Wilcoxon p-values for each pair.
Only the gaps to gemma-3-4b-it are conclusive.
Loading from a CSV or Excel file
estats.analyze() takes a BenchmarkResult, not a file path directly.
Load your file into a DataFrame, parse it with estats.from_dataframe(), then
analyze the result. Useful when you want to run the analysis from a script or notebook.
import pandas as pd
import evalstats as estats
df = pd.read_csv("results.csv")
benchmark, load_report = estats.from_dataframe(df, return_report=True)
result = estats.analyze(benchmark)
estats.print_analysis_summary(result)
# With options
df = pd.read_excel("results.xlsx", sheet_name="Eval Results")
benchmark = estats.from_dataframe(df)
result = estats.analyze(
benchmark,
ci=0.99,
method="lmm", # mixed-effects model
n_bootstrap=20000,
)
estats.print_analysis_summary(result)Common Options
These parameters are accepted by all API functions (compare_prompts,
compare_models, analyze) and their CLI equivalents.
| Parameter | Default | Notes |
|---|---|---|
| ci | 0.95 | Confidence level. Use 0.99 for a more conservative interval. |
| method | "auto" | Auto-selects based on data type and sample size — see Which Method? for the full decision tree. Options: "lmm", "bootstrap", "smooth_bootstrap", "bootstrap_t", "bca", "bayes_bootstrap", "bayes_binary", "wilson", "wald", "clopper_pearson", "newcombe", "tango", "mj_floor", "bonett_price", "logit_t", "nig", "t_interval", "permutation", "sign_test". |
| n_bootstrap | 10000 | Bootstrap resamples. Increase for smoother CI estimates. |
| simultaneous_ci | True | Family-wise CI correction (Šidák by default; opt into max-T resampling via prefer="max_t"). Set to False for marginal CIs. |
| statistic | "mean" | Central tendency. Also supports "median". |
| reference | "grand_mean" | Comparison baseline for the advantage plot. Pass a template label to compare against a specific prompt or model. |
| omnibus | False | Run a Friedman omnibus test before pairwise comparisons (k ≥ 3). |
| p_values | False | Show pairwise p-values in the report. CLI: --p-values. |
| pairwise_test | "auto" | Pairwise p-value test. Options: "auto", "bootstrap", "wilcoxon", "nemenyi". CLI: --pairwise-test TEST (setting this explicitly also enables p-values). |
| correction | "auto" | FWER/multiple-comparison correction. "auto" resolves to Shaffer's (n<30) or Romano–Wolf (n≥30) for p-values, and Šidák for simultaneous CIs — see Which Method?. Other options: "holm", "bonferroni", "fdr_bh", "hochberg", "shaffer", "romano_wolf", "none". |
| failure_threshold | None | Flag inputs scoring below this value in the robustness table. |
Full runnable examples are in the
examples/ folder on GitHub.