Contributing Benchmark Results

Thank you for contributing to the BenchBox community results dataset! Community-submitted results help everyone compare platforms on real workloads and grow the benchmark coverage.

Prerequisites

  1. Install BenchBox - follow the Getting Started guide

  2. Run a benchmark - you need a complete benchmark result to submit

Step-by-Step Submission Flow

1. Run a benchmark

Run a complete benchmark suite (not a cherry-picked subset of queries):

Install the platform extra first in your project environment (for example, uv add benchbox --extra duckdb).

uv run -- benchbox run --platform duckdb --benchmark tpch --scale 0.01

The result JSON is written to benchmark_runs/results/.

Optional companion files may be written next to the result JSON. Add --capture-plans on supported platforms to create <result>.plans.json. The --tuning flag accepts notuning (default), tuned, auto, or a path to a YAML config; use tuned, auto, or a YAML path only for intentionally tuned submissions, which create <result>.tuning.json when tuning clauses are applied. benchbox submit packages either companion automatically when it sits next to the result JSON, so missing companion files do not make a default submission incomplete.

2. Package the submission

Use uv run -- benchbox submit to create a submission package:

# Package the most recent result
uv run -- benchbox submit --last --output ./submission

# Show exact result paths if you need to choose a specific run
# (--paths writes one path per line to stdout, safe to pipe to xargs)
uv run -- benchbox results --paths --limit 25

# Or specify a result file directly
uv run -- benchbox submit benchmark_runs/results/tpch_sf001_duckdb_20260401_120000.json --output ./submission

# Preview what would be packaged (no files written)
uv run -- benchbox submit --last --dry-run

This creates a submission/ directory containing:

File

Description

bundle/<result>.json

The canonical schema-v2 result bundle

bundle/<result>.plans.json

Optional query execution plans from --capture-plans

bundle/<result>.tuning.json

Optional tuning details from an intentionally tuned run

<result>.manifest.json

Metadata: hash, benchmark, platform, contributor

CONTRIBUTING.md

PR instructions (for reference)

The manifest’s submitted_by field defaults to your git config user.name. If that is empty (e.g. a fresh sandbox or CI runner), benchbox submit warns and writes the field as "". Override with --submitted-by "Your Name" if you prefer not to set git config or want a different attribution.

3. Fork and open a PR

  1. Fork the BenchBox repository on GitHub (or use your existing fork)

  2. Copy the contents of submission/bundle/ into results-data/bundles/ in your fork

  3. Copy submission/<result>.manifest.json alongside the bundle files (the per-bundle manifest filename inherits the bundle stem so two contributors submitting the same week cannot collide)

  4. Regenerate the inventory before you commit:

    uv run -- python scripts/generate_corpus_inventory.py --write

  5. Commit and open a pull request against the published-results branch of joeharris76/BenchBox (the public repository)

Use this PR title format:

results: <benchmark> <platform> sf<scale>

Example: results: tpch DuckDB sf1.0

4. CI validation

When your PR is opened, the Validate Submission workflow runs automatically. It checks:

  • Schema compliance - the bundle is valid schema-v2 JSON with all required fields

  • Hash verification - the SHA-256 hash in the manifest matches the bundle contents

  • Sanity checks - no all-zero timings, no negative durations, valid platform/benchmark names

  • Metadata extraction - a summary comment is posted on the PR showing what the submission adds

If validation fails, the PR comment will explain what to fix. The workflow also checks that results-data/corpus-inventory.json matches the submitted bundles. If that check fails, rerun:

uv run -- python scripts/generate_corpus_inventory.py --write

5. Review and merge

A maintainer reviews the submission for quality and environment consistency. Once approved and merged into published-results, the bundle enters the public corpus. The results explorer is designed to build and deploy from main (not from published-results directly) via the docs CI workflow, so once that deploy path is live, a merged submission would appear on the site at the next release rather than immediately. Pre-launch caveat: release-cut currently strips results-explorer/ and results-data/ out of every release branch, so the main-deployed site is not live yet — see docs/operations/results-phase-2-runbook.md for the current state. Until that changes, treat benchbox.dev/results/ as a develop-built preview rather than a guarantee that your merged submission will appear there.

What Makes a Good Submission

  • Complete benchmark suite - run the full query set, not a cherry-picked subset

  • Stable environment - run on a dedicated machine or instance, not a shared laptop under load

  • Default configuration - unless you’re specifically benchmarking a tuned configuration, use defaults

  • Reproducible - include enough environment metadata that someone else could replicate the run

  • Honest results - don’t hand-optimize queries or cherry-pick favorable runs

Trust Labels

Results in the explorer carry trust labels:

Label

Meaning

Maintainer Run

Generated by BenchBox CI or project maintainers

Community Submission

Contributed via PR from an arms-length community member

Vendor Supplied

Produced by the platform vendor itself; applied by maintainers, never self-asserted

CI

Generated by automated CI pipelines

Local

Local/development runs

Community submissions are labeled “Community” in the explorer to distinguish them from maintainer-curated results. Vendor Supplied results appear in ranked tables (unlike community submissions) but always carry the distinct vendor badge, because the platform vendor has a direct interest in the outcome. You cannot self-apply the vendor label through a community PR. The label is derived from a bundle living under results-data/bundles/vendor/, and the submission CI (validate-submission.yml) rejects any pull request from a non-maintainer that adds files under that path. Community submissions belong in results-data/bundles/ (not vendor/); maintainers place and review vendor results.

Funding Disclosure

Independent of who produced a result, you may disclose how the run was funded. This is optional and orthogonal to the trust label (a community run can be on a free trial; a vendor result can be employer-funded).

Funding value

Meaning

employer

Paid for by your employer

personal

Paid for out of pocket

free-trial

Free trial or promotional credits

vendor-sponsored

The platform vendor paid for or sponsored the run

grant

Research or institutional grant

unspecified

Not disclosed (the default)

Quality Expectations

Submissions that don’t meet these criteria may be asked for revisions:

  1. Full query coverage - all queries in the benchmark must be included

  2. No synthetic data - results must come from actual benchmark execution

  3. Reasonable timings - query durations should be plausible for the platform and scale factor

  4. Valid metadata - benchmark ID, platform name, and scale factor must match known values

  5. Schema v2 format - BenchBox currently writes top-level version "2.1". Public submission validation accepts numeric 2.x versions for forward-compatible schema-v2 bundles, but rejects missing or malformed values. To check your bundle: uv run -- python -c "import json; print(json.load(open('bundle.json'))['version'])"

Running Validation Locally

You can validate your bundle before opening a PR. Run validation through uv run -- python so it uses the same environment as BenchBox.

# Validate a specific bundle
uv run -- python scripts/validate_submission.py path/to/result.json

# Validate all bundles in a directory
uv run -- python scripts/validate_submission.py results-data/bundles/

# Verify the inventory is current before you open the PR
uv run -- python scripts/generate_corpus_inventory.py --check

If you use pre-commit locally, install the hooks once so inventory drift is checked automatically:

pre-commit install

If this clone installed hooks before BenchBox added its pre-push timing-policy stage, re-run pre-commit install once so the pre-push hook is installed too.

Questions?

Open an issue or start a discussion.

Maintainers: see Phase 2 Results Operations Runbook.