Cutting a BenchBox release

BenchBox releases follow a version-branch flow on a single repo (BenchBox-dev/BenchBox) with two long-lived branches: develop (dev work) and release (release-only). This guide is the maintainer runbook.

The flow (2 commands)

git checkout develop && git pull
make release-cut VERSION=X.Y.Z
# review the PR; wait for validate-base and release-required-result
make release-finalize VERSION=X.Y.Z

That’s the entire flow. The two Make targets do the rest. Wheel install, release canary, and correctness remain the blocking gates; UAT is a non-blocking matrix campaign.

Pre-merge release-required contract

Release PRs target release and must be opened from branches accepted by .github/workflows/validate-release-pr.yml (vX.Y.Z, optionally with a suffix). Before a release PR can merge, the release-only ruleset must require:

  • validate-base

  • release-required-result

release-required-result is the stable umbrella check in .github/workflows/test.yml. A green result means the release PR branch passed:

  • the required fast lane, test (ubuntu-latest, 3.12);

  • the bounded real-result correctness gate, make test-correctness-gate (DuckDB x TPC-H at SF=1 with the pinned reference qgen seed, through generate/load/execute with EXACT stored answer-set row-count validation of the 18 answer-stable TPC-H queries; Q11/Q16/Q18/Q20 are excluded for answer-set boundary sensitivity, and validation is cardinality-level, not value-level). The stored digests are Linux-generated, so on Apple silicon with macOS 26 or newer run make ci-linux for parity rather than reading a local mismatch as a defect;

  • the credential-free integration-not-slow suite: tests/integration -m "integration and not (slow or stress or resource_heavy or live_integration)";

  • isolated exact-one-wheel package build/install smoke;

  • dependency upper-bound checks;

  • release-branch curation checks that confirm dev-only paths are absent.

It does not rerun the full stress matrix, live cloud integrations, or long-running UAT on the release PR itself. Slow/resource-heavy coverage is enforced through the freshness-based release canary below. Wheel install, canary, and correctness gates remain blocking; long-running UAT is a non-blocking matrix campaign.

Release canary and ruleset drift

Release PRs also depend on the validate-base workflow’s release-readiness steps. That workflow queries the latest completed release-canary.yml run, reads the canary summary artifact, and fails the release PR when the canary is missing, red, older than 48 hours, or when the tested develop SHA recorded in the artifact is not an ancestor of the release PR head. The only bootstrap exception is the first release that introduces release-canary.yml before the workflow exists on the default branch; in that case validate-base runs the same non-fast canary suite and ruleset drift check inline, then later releases return to the scheduled/manual canary evidence path.

release-canary.yml runs daily and on manual dispatch. Scheduled runs execute from the default branch, but the workflow checks out develop before running release evidence and records that checked SHA in release-canary-summary.json. Its blocking canary suite is the credential-free non-fast family: (slow or resource_heavy) and not (stress or live_integration). The same workflow also runs scripts/ruleset_drift_check.py against docs/operations/repo-admin-settings.md, so ruleset drift makes the canary red instead of silently invalidating release assumptions. The ruleset drift check must use RULESET_DRIFT_TOKEN, a repository secret with enough ruleset visibility to expose bypass actors; the default GITHUB_TOKEN is insufficient for that part of the contract.

Stress tests, live cloud integrations, and long-running UAT remain advisory until their credential, cost, and flake policies are stable enough to make them release-blocking.

UAT matrix campaign evidence (advisory)

scripts/release_readiness_check.py reports committed UAT evidence when it exists. A green verdict, clean source tree, ancestor source SHA, and recent completion remain useful campaign-quality signals, but missing, red, stale, non-ancestor, or dirty evidence does not fail validate-base. Producing the three-stage evidence is not a release-cut precondition.

Running the optional campaign:

make uat-sweep CONFIG=tests/uat/configs/release-gate-01-native-dataframe.yaml
make uat-sweep CONFIG=tests/uat/configs/release-gate-02-docker-nonoltp.yaml   # after stage 1 completes
make uat-sweep CONFIG=tests/uat/configs/release-gate-03-docker-oltp.yaml     # after stage 2 completes
make uat-gate-check STAGE1=<run-dir> STAGE2=<run-dir> STAGE3=<run-dir>
# review the campaign report; it is historical evidence, not a release input

See docs/operations/uat-framework.md “Three-stage UAT campaign” for campaign ordering and its report checklist. The emergency override below applies only to the blocking canary check.

Emergency override is admin-only: set repository variables RELEASE_READINESS_OVERRIDE_SHA to the exact release PR head SHA and RELEASE_READINESS_OVERRIDE_REASON to the incident/approval record. Remove both variables after the release. API outages, stale canaries, or ruleset drift must not be bypassed with an undocumented local change.

What release-cut does

  1. Cuts a vX.Y.Z branch off develop (develop itself is never modified).

  2. Bumps the 6 version sources via scripts/update_version.py and generates the CHANGELOG entry via scripts/generate_changelog_entry.py --since-ref origin/release. The release note boundary is the current release branch patch delta against release, not git log origin/release..HEAD ancestry and not the latest tag reachable from develop, because release-finalize intentionally does not replay release commits onto develop.

  3. Opens $EDITOR on CHANGELOG.md for hand-curation when a terminal is attached, then gates on generate_changelog_entry.py --check-curation, which inspects the drafted section’s text: it rejects verbatim commit subjects (trailing (#NNNN)), the manual-edit placeholder, and more than 60 bullets. Curation is always required — the draft is the raw origin/release..HEAD delta, hundreds of commits whenever release lags develop. Headless runs stop here: hand-curate the section, then re-run make release-cut (see “Resuming or aborting a cut” below). RELEASE_ALLOW_RAW_CHANGELOG=1 accepts the raw draft deliberately.

  4. Curates the release branch — git rm’s the dev-only and deferred release paths while retaining the curated Results Explorer publication inputs: results-data/, results-explorer/, _project/scripts/explorer_pipeline/, _project/scripts/explorer_publish.py, and _project/scripts/results_explorer_snapshot_invariants.py. The rest of _project/, _blog/, agent configs, and dev-tooling root files remain development-only; the exact list is enforced by the release-cut: Makefile target and scripts/check_release_curation.py. For v0.3.0, landing/ and docs/blog/ stay in the release tree so /prompts/ and promoted release posts ship. Explorer browser/seed/ submission/sync workflows remain development-only.

  5. Commits a single Release vX.Y.Z commit on vX.Y.Z.

  6. Merges origin/release into vX.Y.Z with -s ours --allow-unrelated-histories. release and develop have unrelated roots and diverge permanently, so without this the release PR is unmergeable, GitHub cannot compute a merge ref, and no CI runs at all. -s ours keeps the curated release tree byte-for-byte (the merge is guarded on that) and only records release as a second parent, which release-finalize’s squash-merge then collapses away.

  7. Pushes and opens a PR against release.

  8. Sweeps prior v* branches on origin (option-c lifecycle: keep until superseded, then auto-delete on the next release-cut).

Resuming or aborting a cut

Steps 1-3 are idempotent, so an interrupted cut is resumed by re-running the same command from the vX.Y.Z branch:

make release-cut VERSION=X.Y.Z      # reuses the branch, keeps the CHANGELOG section

The branch is reused rather than recreated, the version bump and uv lock re-apply to the same values, and an existing ## [X.Y.Z] section is left untouched — so a section you curated between runs survives. To throw the cut away instead:

make release-cut-abort VERSION=X.Y.Z   # reset, return to develop, delete the branch

release-cut-abort refuses once vX.Y.Z exists on origin, and refuses to run from any branch other than vX.Y.Z or develop. release-cut likewise refuses to resume a branch that already carries its Release vX.Y.Z commit.

Changelog summarization shells out to the claude CLI. It is skipped automatically inside a Claude Code session (where the nested call blocks until its 120s timeout) and whenever BENCHBOX_CHANGELOG_SUMMARIZE=0; set BENCHBOX_CHANGELOG_SUMMARIZE=1 to force it. Skipping only means the draft is raw commit subjects, which step 3 requires you to curate anyway.

What release-finalize does

  1. Finds the open release PR for vX.Y.Z.

  2. Checks the required PR status list once and refuses to continue unless both validate-base and release-required-result are present and green. Missing means the ruleset/workflow contract is broken; pending means wait in GitHub Actions and rerun the command. release-finalize does not poll.

  3. Squash-merges the PR. (Ruleset release-only also blocks the merge unless validate-base and release-required-result are green.)

  4. Fast-forwards release and tags vX.Y.Z.

  5. Pushes the tag — which fires .github/workflows/release.yml: dependency-bounds → build (with SOURCE_DATE_EPOCH from the tag commit) → publish (PyPI trusted publisher) → github-release → test-installation (cross-platform pip install verification).

  6. Leaves develop untouched. Dev-only paths persist on develop by design (per A3 in _project/decisions/single-repo-migration.md); the release squash on release does not need to be replayed onto develop.

Syncing develop’s version. release-cut/release-finalize never modify develop (step 6), so its declared version does not track releases on its own — D5’s original “rebase develop onto main” step became impossible once release/develop diverged as unrelated roots. After a release publishes, bump develop to match on a branch off develop: scripts/update_version.py --version X.Y.Z --update-pyproject, then uv lock, then PR back to develop. This realigns all six version sources (benchbox/__init__.py, pyproject.toml, the three Current release: doc markers, and the landing/index.html badge) with the latest published release, so develop no longer trails PyPI.

Push-to-release jobs are post-merge signals. They may still start when release advances, but they are not pre-publish evidence: the tag push follows the successful release PR merge and .github/workflows/release.yml begins from that public tag. If a post-merge release check fails after the tag is pushed, handle it as a patch release or incident; do not treat the already-published release as if it had been blocked.

Recovering from common failures

  • validate-base or release-required-result is missing: stop. The release-only ruleset or release workflow contract is out of sync; do not finalize until both stable required contexts exist.

  • validate-base or release-required-result is pending or failed: wait for GitHub Actions or fix on a feature branch off develop, PR back to develop, then re-run make release-cut (the option-c sweep will delete the stale vX.Y.Z branch automatically).

  • UAT campaign evidence is missing, stale, red, dirty, or non-ancestor: follow up with the optional three-stage campaign and review its report (see “UAT matrix campaign evidence”). This does not block release-finalize and does not use the emergency override.

  • Release canary is missing, stale, or red: inspect the latest release-canary.yml run. If the non-fast canary failed, fix through develop; if ruleset drift failed, update the live GitHub ruleset or this runbook so they match. Use the emergency override variables only with an explicit incident/approval record.

  • Wheel content is wrong: adjust pyproject.toml / MANIFEST.in excludes on develop, then cut a patch release. PyPI rejects re-uploads of an existing version, so always bump.

  • release.yml fails after tag push: investigate via gh run view, fix the underlying issue, bump to the next patch version, and re-cut. Do not force-push or re-tag; the tag is already public.

Recovering from a broken PyPI release

PyPI versions are immutable: once X.Y.Z is published you cannot re-upload or delete-and-replace it, even to fix a broken build. This is generalized guidance — _project/TODO/main/active/release-recovery-v0-3-1.yaml is the worked precedent (see below) for what happens when this bites in practice: v0.3.0 published 2026-05-16 with a broken clean import (ModuleNotFoundError: No module named 'pandas') and sat as PyPI-latest, uncaught, for over a month, because nightly’s package-install test built a wheel from develop rather than checking the artifact actually on PyPI. (.github/workflows/release-canary.yml’s pypi-latest-installability job now closes that specific detection gap going forward — see “Release canary and ruleset drift” above.)

Two response options once a bad version is confirmed on PyPI, and when to use each:

  • Yank the broken version. PyPI’s “yank” (pip index / PyPI project UI) hides a release from default dependency resolution without deleting it — existing installs that already pinned the bad version are unaffected, but new pip install benchbox resolutions skip it and fall back to the latest non-yanked version. Use this when there is a safe prior version to fall back to and the fix is not ready yet.

  • Ship a .postN or the next patch. This repo’s actual precedent is a full next patch release (v0.3.0 -> v0.3.1), not a .postN suffix, per release-recovery-v0-3-1. Use this when the fix is ready quickly and a monotonic forward release is cleaner than yanking (e.g. there is no good prior fallback version, or you want users who already resolved the bad version to get the fix on their next install/upgrade rather than silently falling back).

Either path is a judgment call based on the incident (is there a safe fallback version, is a fix already close) — it is not automated by the canary itself; a canary failure is a signal to make that call, not a trigger for an automatic rollback action.

The forward-fix path uses the same flow as any other release — there is no separate recovery procedure:

git checkout develop && git pull
make release-cut VERSION=X.Y.Z
# review the PR; wait for validate-base and release-required-result
make release-finalize VERSION=X.Y.Z

See release-recovery-v0-3-1 for the worked example of diagnosing a broken PyPI-latest release, confirming the fix on develop, and cutting the recovery version through this same flow.

Reference

  • Makefile targets: release-cut, release-finalize.

  • Workflow: .github/workflows/release.yml.

  • Canary workflow: .github/workflows/release-canary.yml.

  • Release-readiness gate: scripts/release_readiness_check.py.

  • Ruleset drift gate: scripts/ruleset_drift_check.py.

  • Curation drift guard: scripts/check_release_curation.py (runs in lint.yml on every PR).

  • Version updater: scripts/update_version.py.

  • Changelog generator: scripts/generate_changelog_entry.py.

  • Architecture record: _project/decisions/single-repo-migration.md (D5 / A3 / A4 / A5, plus the amendments for the 2-command flow, the develop-not-modified rule, and the v0.3.0 release-scope curation).