BenchBox v0.4.0: new org, DuckLake, and result provenance

TL;DR: BenchBox now lives at BenchBox-dev/BenchBox. Published results can record who ran them and who paid, including a new vendor-supplied trust label, and the Results Explorer preview at benchbox.dev/results/ displays those fields. DuckLake lands as a beta platform. The bare clickhouse alias and the databricks-connect extra are gone; Throughput@Size figures from earlier releases need a rerun, not a re-export.

BenchBox v0.4.0 was released on August 28, 2026.

The headline change is the move to its own GitHub organization. The repository name, the PyPI project benchbox, and benchbox.dev are unchanged, and old remotes keep redirecting. The project now lives in an organization account.

The second change is a vocabulary for published results: who produced a run, its trust label, and disclosed funding. The Results Explorer preview at benchbox.dev/results/ first ships in this tagged release and shows those labels. An organization account and labels for who produced a result are both prerequisites for results from other people.

The third is DuckLake as --platform ducklake, still beta: Parquet table data with catalog metadata in a SQL database, and catalog backend and data path chosen independently.

At a glance

A new home: BenchBox-dev

The repository, issue and release tooling, CI, and package metadata now point at github.com/BenchBox-dev/BenchBox. Existing remotes that still use github.com/joeharris76/BenchBox.git keep redirecting, so updating a local remote is useful but not urgent:

git remote set-url origin https://github.com/BenchBox-dev/BenchBox.git

The repository name, PyPI project, domain, and uv add benchbox command stay the same. This is an ownership and namespace move.

Provenance labels, and where to see them

v0.4.0 adds a canonical vocabulary for result source, trust label, and funding, plus a benchbox run --funding flag and an optional provenance block in result bundles. The vendor-supplied trust label is new. The preview at benchbox.dev/results/ displays those fields in rankings, comparisons, and result details.

Ranked tables include maintainer-run, CI, and vendor-supplied results. Community submissions stay visible and are currently excluded from ranked tables. The vendor label cannot be self-applied: it is derived from bundles under results-data/bundles/vendor/, and submission CI rejects non-maintainer PRs that touch that path. In the August 28 preview snapshot, all 138 rows are maintainer-run with funding unspecified.

Accepted --funding values are employer, personal, free-trial, vendor-sponsored, grant, and unspecified (the default). Eligibility gates, the comparability receipt, and corpus curation belong in the companion post.

DuckLake beta

DuckLake stores table data as Parquet while keeping catalog metadata in a SQL database. BenchBox now runs it with --platform ducklake. Catalog backend and data path are independent choices, so catalog=duckdb|sqlite|postgres composes with a local or s3:// data_path. That is six possible combinations.

Four of those six passed TPC-H scale-factor-1 correctness validation: local, local_catalog_s3, postgres_catalog, and postgres_catalog_s3. SQLite catalogs are a supported option, not one of those four validated modes. We are keeping the beta label rather than extrapolating those four runs to every combination, scale, and benchmark.

DuckLake requires DuckDB 1.3 or later. The first run also needs network access to INSTALL the DuckLake extension. Install and try the local path with:

uv add "benchbox[ducklake]"
uv run -- benchbox run --platform ducklake --benchmark tpch --scale 0.01

A PostgreSQL catalog with S3-backed Parquet data uses the same command with extra platform options. Postgres plus S3 is one of the four SF1-validated modes; this snippet is the flag shape:

uv run -- benchbox run --platform ducklake --benchmark tpch --scale 0.01 \
  --platform-option catalog=postgres \
  --platform-option data_path=s3://bucket/prefix/

BenchBox reuses an existing catalog by default. --force rebuilds catalog state and local data, but it does not recursively delete an S3 data prefix.

Corrections worth knowing

TPC throughput was understated. Throughput@Size now includes every executed query, correcting 22x (TPC-H) and 99x (TPC-DS) understatements in earlier result versions. Historical bundles are left unchanged as records of what those versions produced. Re-exporting an old bundle does not fix the number, because the exporter reuses a stored throughput_at_size when one is present. If you published a Throughput@Size figure from an earlier release, rerun the benchmark on v0.4.0.

BENCHBOX_TUNING_ENABLED never worked. It set a config key nothing read at runtime, and the docs claimed it activated tuned runs in CI. Use --tuning tuned or --tuning auto; BENCHBOX_TUNING_CONFIG still works. If you set the old variable, delete it.

--tuning auto on SQL platforms is constraints-only. Primary-key, foreign-key, unique, and check constraints, and nothing else. DataFrame platforms keep their smart defaults. This documentation update describes the constraints --tuning auto already applied on SQL platforms.

Other notable changes

• Local MCP clients can use benchbox-mcp --transport streamable-http; existing stdio integrations are unchanged. Authenticated non-local deployments support persistent jobs, but shared production deployment remains deferred and unsupported.
• BenchBox now probes a bundled TPC generator before selecting it and compiles from source when the loader refuses it. That unblocks TPC-H data generation on macOS 15 and earlier Apple Silicon. The shipped darwin-arm64 binaries still target a newer OS (minos 26.0); the fallback is a slower first run, not a rebuilt binary.
• Secret redaction now covers more DuckLake, MotherDuck, export, and MCP paths. Backend-provided exception text can still repeat values, so credentials should not be placed in values a backend may echo.
• Write and Transaction Primitives now detect staged data from the wrong scale factor and rebuild it automatically.

Changed behavior to be aware of

The ClickHouse alias was added in v0.2.1 as a deprecation shim and removed on schedule. Passing bare clickhouse now raises a ValueError that names the replacements, rather than silently picking a deployment mode. The Databricks change renames only the BenchBox extra; it still installs the upstream databricks-connect package.

Try it yourself

After upgrading to v0.4.0:

1. Confirm the installed version:

benchbox --version

2. Run a smoke benchmark with an explicit funding disclosure:

uv add "benchbox[duckdb]"
uv run -- benchbox run --platform duckdb --benchmark tpch --scale 0.01 \
  --funding personal --non-interactive

3. If you used the bare ClickHouse selector, confirm a named deployment mode:

uv run -- benchbox run --platform clickhouse-local --benchmark tpch --scale 0.01 \
  --dry-run ./preview

4. Try the local DuckLake path:

uv add "benchbox[ducklake]"
uv run -- benchbox run --platform ducklake --benchmark tpch --scale 0.01

5. Browse the Results Explorer preview at benchbox.dev/results/.

v0.4.0 was a maintainer-driven cycle. If a result, label, or migration note needs correction, start a discussion.

References

• Changelog entry: CHANGELOG.md ([0.4.0] - 2026-08-27)
• Release tag: v0.4.0
• DuckLake platform documentation: docs/platforms/ducklake.md
• Provenance vocabulary: benchbox/core/results/provenance.py
• MCP server reference: docs/reference/mcp.md