Skip to content

Understanding Benchmark Results

Learn how to interpret BenchBox output and what the metrics mean.

Result Overview

After running a benchmark, BenchBox produces:

  • Console summary - Quick overview of timing and validation
  • JSON results - Detailed metrics for each query
  • Manifest files - Data generation metadata

Viewing Results

Terminal window
benchbox results --limit 1
benchbox export --last --format json --output-dir ./results

Key Metrics

Run Timing

The run object carries end-to-end numbers:

FieldMeaning
run.total_duration_msEnd-to-end benchmark duration in milliseconds
run.query_time_msTotal query execution time in milliseconds
summary.timing.total_msSum of measured query times in milliseconds
summary.timing.geometric_mean_msGeometric mean query time (drives Power@Size)
summary.validationpassed, failed, or partial
summary.queriestotal, passed, and failed execution counts

Per-Query Metrics

Each entry in queries looks like this:

{
"id": "Q1",
"ms": 14.6,
"rows": 4,
"status": "SUCCESS",
"stream": 0,
"iter": 2,
"run_type": "measurement",
"test_type": "power"
}
FieldMeaning
msQuery runtime in milliseconds
rowsNumber of result rows
statusSUCCESS, ERROR, or TIMEOUT
stream / iter / run_typeWhich stream, iteration, and phase produced the row

Understanding Validation

BenchBox validates query correctness by comparing row counts. The run-level verdict is summary.validation:

ValueMeaningAction
passedEvery check matched expectedNo action needed
failedA check differed from expectedCheck query translation
partialSome queries failed to runInspect the ERROR entries

A failed query carries an ERROR status and an error message instead of timing data.

TPC Metrics

For TPC-H/TPC-DS, BenchBox calculates Power@Size and Throughput@Size. At scale factor 1 and above a run that is not --official records summary.tpc_metrics as suppressed.

They are computed as:

Power@Size = (SF × 3600) / geometric_mean(query times, final iteration)
Throughput@Size = (Q × SF × 3600) / T_throughput

Where:

  • SF = Scale Factor (0.01, 0.1, 1, 10, etc.)
  • Q = queries scored in the throughput test: 22 × streams for TPC-H, 99 × streams for TPC-DS
  • T_throughput = wall-clock duration of the throughput test, from the first stream’s first query to the last stream’s last query (connection setup is excluded)

Higher is better. Compare these metrics only at the same scale factor.

BenchBox exports QphH@Size only for an official TPC-H run with all 22 power queries, successful RF1 and RF2 in Power, at least two completed throughput query streams, and one successful refresh pair per throughput stream using consecutive update sets. Failed queries, incomplete phases, query subsets, missing refresh evidence, or a nonofficial compliance class withhold the composite; qphh_unavailable lists the reasons. For TPC-DS, an official run of the load, power, throughput and maintenance phases with an even --streams of at least 4 exports qphds_at_size, and qphds_unavailable lists the reasons otherwise (see the TPC-DS official guide). A failed query or throughput phase also withholds Throughput@Size.

Price/Performance

Not calculated by BenchBox (requires cost data), but you can derive it:

Price/Performance = (Platform Cost) / Throughput@Size

Comparing Results

Terminal window
benchbox run --platform duckdb --benchmark tpch --output ./benchmark_runs
benchbox run --platform sqlite --benchmark tpch --output ./benchmark_runs

Find the two result files with benchbox results --limit 2 and pass them to benchbox compare.

The comparison shows:

  • Per-query timing differences
  • Relative speedup/slowdown
  • Validation status alignment

For the facts that must match before two results are comparable, see Benchmark Methodology.

Result File Locations

BenchBox stores results in the runs root: BENCHBOX_OUTPUT_DIR when set, otherwise benchmark_runs/ next to your checkout:

benchmark_runs/
├── results/ # JSON result files
│ └── tpch_sf001_duckdb_sql_<timestamp>_<id>.json
├── databases/ # Platform database files
│ └── tpch_sf001/
│ └── tpch_sf001_notuning_noconstraints.duckdb
└── datagen/ # Generated data (default location)
└── tpch_sf001/
├── lineitem.tbl.zst
├── orders.tbl.zst
└── _datagen_manifest.json

--output DIR redirects generated data to DIR/<benchmark>_<scale>/ (for example /tmp/bb-truth/tpch_sf001/ with --output /tmp/bb-truth); result files and databases stay in the runs root.

Interpreting Slow Queries

If a query is unexpectedly slow:

  1. Check scale factor - Larger data takes longer
  2. Review query plan - Use platform’s EXPLAIN
  3. Compare baselines - Run on DuckDB for reference
  4. Check validation - Ensure correct results
Terminal window
benchbox run --dry-run ./analysis --platform duckdb --benchmark tpch

This exports the query SQL as query_<id>.sql files under ./analysis/<benchmark>_<platform>_queries_<timestamp>/, alongside the dry-run JSON/YAML, schema, and DDL preview.

Next Steps