ADR: published-results as a Slim Corpus-Only Branch¶
Status¶
Accepted (2026-05-03). The W2 force-push of published-results to the slim
orphan commit 00260bd46 has shipped (pre-slim tip preserved as
refs/backups/published-results-pre-slim-20260503 on origin). The W3 mirror
PR #166 is open as draft against published-results. The W4 sync workflow
(.github/workflows/sync-results-data-to-published.yml) lands together
with this ADR update.
Date¶
2026-05-03
Context¶
published-results is the long-lived branch the Phase 2 contributor flow
targets (see docs/operations/results-phase-2-runbook.md).
The runbook and docs/contributing-results.md both tell external
submitters to open PRs against it. The submission validator workflow
(.github/workflows/validate-submission.yml) is wired to fire only on
PRs whose base ref is published-results.
The branch was originally created by forking develop at
520d7055d (2026-05-01, “fix(results): support per-bundle published
manifests” PR #95). Since then develop has accumulated 1,674 commits
(~3 months of work spanning every part of BenchBox) without any of those
commits being mirrored to published-results. As a consequence:
published-resultscurrently tracks 13,809 files, the overwhelming majority of which (benchbox/,tests/,_project/,_binaries/, every.github/workflows/*.ymlother than the submission validator) have nothing to do with the corpus.Opening a corpus PR against
published-resultsfrom adevelop-based branch produces a 1,678-commit diff dominated by unrelated develop drift, which is unreviewable.The actual explorer build pipelines (
docs.ymlandresults-explorer-browser.yml) trigger only onmain(push and pull-request).published-resultsdoes not feed any explorer build today; merges to it are convention-only.
The 2026-05-02 UAT corpus integration TODO
(results-explorer-uat-corpus-integrate-validated-bundles) ran into all
three problems and ultimately landed via a develop PR (#164) with the
slim-down deferred to this ADR’s parent TODO. We now need a durable
shape for published-results so the runbook’s contributor flow becomes
practically usable instead of inheriting develop drift.
Decision¶
published-results is a slim corpus-only branch. Its tracked
contents are fixed by allowlist; everything else is excluded.
Allowlist¶
Path |
Why it lives on |
|---|---|
|
The corpus itself: schema-v2 bundle JSONs and per-bundle manifest sidecars. |
|
Generated index, regenerated by the contributor on every submission. |
|
Corpus-flow guidance distinct from the develop README. |
|
Per-cohort curation notes accumulated over time. |
|
Maintainer-run seed lane spec. |
|
The cohort-depth gate enforced by CI. |
|
Local-clone hygiene for the corpus directory. |
|
Per-bundle validator entrypoint — a thin CLI wrapper that must run inside the submission CI without shipping the rest of |
|
The shared validator implementation used by both develop and this branch. Mirrored here (the single |
|
Inventory generator — same rationale as above. |
|
The submission CI gate. |
|
Repository-root ignore — kept minimal. |
|
Required for any public repo branch; near-zero churn. |
|
Slim, points contributors at |
|
A |
Total expected file count: on the order of hundreds, dominated by
the results-data/bundles/* set. Anything not on this list is removed
in the slim-down.
Explicit exclusions (deleted in the slim-down)¶
benchbox/(the package source) — except the single mirrored filebenchbox/validation/bundle.py(see the allowlist above). The rest of the package is not needed to validate or display bundles;validate_submission.pyis a thin wrapper over that one shared module.tests/— the develop-side test suite. The corpus depth and per-bundle validators run againstresults-data/directly._project/— TODO/DONE/handoffs/blind-spots. Project tracking belongs on develop._binaries/— TPC-H/DS binaries. Not required for corpus validation._sources/— vendored upstream sources. Not required for corpus validation._blog/,docs/,landing/,examples/,results-explorer/,docker/— develop-side artefacts; the explorer is built frommainand feeds the public site, not frompublished-results.All other
.github/workflows/*.ymlexceptvalidate-submission.yml— the slim branch only needs the submission validator workflow.All other top-level dotfiles,
Makefile,pyproject.toml,uv.lock, etc. — develop’s tooling is irrelevant to a corpus-only branch.
Validator invocation contract¶
generate_corpus_inventory.py is stdlib-only Python. validate_submission.py
is a thin wrapper that imports the shared implementation from
benchbox/validation/bundle.py (mirrored onto this branch), with an
importlib file-loader fallback if benchbox is not importable; that shared
module’s own dependencies are stdlib-only (hashlib, json, sys,
argparse, pathlib, decimal, collections, datetime), but it also
tries one benchbox.* import — benchbox.core.results.schema_policy — and
falls back to a standalone policy check when that import fails (as it will on
this slim branch, which has no installed BenchBox). That is an optional
BenchBox import, not an absence of one: develop intentionally uses the
central schema policy when benchbox is importable there, while
published-results always takes the standalone fallback path. CI invokes
both scripts with uv run --no-project --python 3.11, so neither needs
project metadata or an installed BenchBox to run.
The current validate-submission.yml invokes them via
uv run -- python scripts/<script>.py, which expects a pyproject.toml
in the working directory. To keep published-results slim and keep
CI working on day one, the slim-down changes the workflow on
published-results to invoke the validators with uv run --no-project --python 3.11 -- python scripts/<script>.py. The --no-project flag
tells uv not to look for or install a project; --python 3.11 pins
the Python version (the develop-side workflow today gets this from
actions/setup-python@v5 and an unpinned uv run, which is fragile
even there).
This means pyproject.toml and uv.lock are not added to the
allowlist and stay excluded. The on-branch copy of
validate-submission.yml is the one that has to change; that change
is part of W2 of the parent TODO (it lands together with the
orphan-branch reset).
Python version pinning¶
.python-version is not in the allowlist because it does not
exist in the develop tree today. After the slim-down, the Python
version validate-submission.yml runs against is pinned by the
workflow file itself (actions/setup-python@v5 with
python-version: "3.11" and the matching --python 3.11 flag passed
to uv run --no-project). If a future change adds a
.python-version file to develop, the slim-branch maintenance plan
needs to be updated to mirror it.
Trust labels¶
The trust-label resolver in scripts/generate_corpus_inventory.py
already derives the label from the presence of a per-bundle
<stem>.manifest.json sidecar. This decision does not change that
contract. After the slim-down, published-results will continue to
host the same mix of maintainer-run (seed corpus, no sidecar) and
community-submission (manifest sidecar present) entries.
Vendored scripts: source of truth and divergence detection¶
validate_submission.py and generate_corpus_inventory.py exist on
develop (the canonical home) and are vendored to published-results
for self-contained CI. Sync is automated by
.github/workflows/sync-results-data-to-published.yml:
when develop’s copy of either script changes, the workflow opens a
draft mirror PR vs published-results carrying the change. Maintainers
review and flip ready when the develop change is intended to surface on
the public corpus branch.
This means published-results will normally be at most one mirror PR
behind develop on the vendored scripts. If a develop PR modifies one
of the scripts but the mirror PR is left as a draft (e.g. waiting on a
matching slim-branch contributor PR), divergence is bounded and visible
in the open-PR list against published-results.
If the workflow itself ever misfires (e.g. detects no drift when there
is one), the manual recovery is to trigger it via workflow_dispatch
from the develop branch. In the absence of any sync at all, develop
remains authoritative — published-results copies are treated as
read-only mirrors, and any develop-side bug fix in the validators is
the canonical fix.
The workflow file itself is NOT auto-mirrored — this is permanent, not a gap to close¶
sync-results-data-to-published.yml’s automated mirror only covers
scripts/validate_submission.py, benchbox/validation/bundle.py, and
scripts/generate_corpus_inventory.py (the vendored scripts). It
deliberately does not include
.github/workflows/validate-submission.yml itself, and never will:
GITHUB_TOKEN cannot push changes under .github/workflows/ regardless
of the permissions: block granted to it — this is a hard-coded GitHub
Actions restriction (the token needs an actual workflows OAuth scope,
which GITHUB_TOKEN never carries), not a configuration choice that
could be relaxed by editing the sync workflow.
Practical effect: whenever validate-submission.yml changes on develop
(new CI logic, a new guard step, an invocation change), a maintainer
must manually diff and re-apply it onto published-results
(git diff origin/develop:.github/workflows/validate-submission.yml origin/published-results:.github/workflows/validate-submission.yml),
preserving the slim-branch’s --no-project --python 3.11 invocation —
exactly the one-time sync PR #985 did. There is no automated drift
alarm for this specific file; treat every develop-side PR that touches
validate-submission.yml as carrying an implicit “and re-sync
published-results by hand” follow-up.
Consequences¶
Positive
Future contributor PRs vs
published-resultsproduce small, reviewable diffs (typically a handful of bundle JSONs + manifests + inventory regen).The runbook’s claim that “contributor PRs target
published-results” matches reality; the convention becomes practical.published-resultshistory becomes corpus-only over time, making it trivially auditable for what has shipped to the public corpus.Removing the develop drift eliminates a class of merge surprise where unrelated develop refactors landed on a branch external contributors were expected to base off.
Negative
The slim-down is a force-push to a shared branch. Any in-flight contributor PR or local fork referencing the old
published-resultshistory will need to rebase. We will announce the change inCONTRIBUTING.mdon the slim branch, indocs/contributing-results.mdondevelop, and in the runbook before the force-push.Vendoring
validate_submission.pyandgenerate_corpus_inventory.pyintroduces a sync surface. Mitigated by W4 of the parent TODO and by treating develop as authoritative.The branch loses the May-1 history. Project history overall is preserved on
developandmain;published-results’s own log is intentionally short and corpus-focused.
Alternatives considered¶
Leave
published-resultsas a stale develop snapshot. Status quo. Rejected because the contributor flow is broken in practice (giant unreviewable diffs) and the disparity will only grow.Keep
published-resultssynced with develop continuously. Rejected because the branch then carries develop’s full surface permanently, defeating the slim-corpus design intent and forcing contributors to clone the entire BenchBox source tree just to add a bundle. Also conflates corpus PR review with code PR review.Retire
published-results; route all submissions throughdevelop. Rejected becausedevelop’s CI surface is much heavier (lint, typecheck, fast-test) and would gate corpus contributions on passing develop-side checks that have nothing to do with the bundle. The corpus needs its own lightweight CI gate.Define the slim contents but skip the force-push, slimming incrementally via
git rmPRs. Rejected because the interim state is worse than either endpoint — contributors would still see drift, and incremental cleanup PRs are bigger maintainer toil than a one-time orphan-branch reset.Drop
uv runfromvalidate-submission.ymland callpython3 scripts/<script>.pydirectly. Stronger expression of slim intent than the chosen approach: the validator’s shared module is stdlib-only, so plainpython3works withoutuv,pyproject.toml, oruv.lock. Considered but not chosen because the project-wide convention (CLAUDE.md/AGENTS.md) is “always useuvfor Python execution” and breaking that convention on a single workflow file would be surprising for maintainers. The chosen approach (uv run --no-project --python 3.11) preserves the convention while still keeping the slim branch free of project metadata. If the convention ever changes, this alternative becomes the natural follow-up.
Operational footprint¶
One-time force-push of
published-resultsto a new orphan commit containing only the allowlisted paths. Documented in W2 of the parent TODO; gated on user authorization.W3 of the parent TODO mirrors the UAT-20260502 corpus delta (188 community-submission bundles + 188 manifests + regenerated inventory
slim README) onto the post-slim
published-results.
W4 of the parent TODO ships
.github/workflows/sync-results-data-to-published.yml. The workflow watches develop for changes under the slim-branch allowlist paths (results-data/bundles/, the corpus docs, the two vendored validators) and opens a draft mirror PR vspublished-results. The mirror PR never auto-merges; maintainers review and flip ready when the change is intended to surface on the public corpus branch.
Maintenance protocol¶
Re-review this ADR when:
The corpus inventory schema changes (new top-level field that needs to live alongside
bundles/).A new validator or generator script becomes load-bearing for submission CI and would need to be vendored.
The Phase 2 contributor flow is replaced by a hosted ingest service (Phase 3) — at which point
published-resultsmay be retired rather than maintained.
References¶
Parent TODO:
_project/DONE/main/published-results-slim-down-and-corpus-mirror.yaml(completed)Originating handoff:
_project/handoffs/results-explorer-uat-corpus-integration-20260503.mdRunbook:
docs/operations/results-phase-2-runbook.mdContributor guide:
docs/contributing-results.mdSubmission validator:
scripts/validate_submission.pyInventory generator:
scripts/generate_corpus_inventory.pyCorpus depth gate:
results-data/validate_corpus.py