Hosted Results Contract

Created: 2026-04-01

Overview

This document defines the identity model, visibility states, trust labels, cohort eligibility rules, privacy policy, and consumer contracts for the BenchBox hosted results platform. It covers all three rollout phases:

Phase 1 - Static explorer MVP. All results are maintainer-curated. No community submissions, no auth, no hosted API.
Phase 2 - PR-based community contributions. Community submitters open pull requests against results-data/; GitHub Actions validates; maintainers merge.
Phase 3 - Hosted API with auth, rate limiting, and self-service submission.

Consumers of this contract: the benchbox submit CLI command, the hosted API (Phase 3), and the results explorer frontend. Each section concludes with a phase annotation indicating when the rules apply.

1. Result Identity Model

1.1 Identifiers

Three distinct identifiers exist for every result entry. They serve different purposes and have different lifetimes.

`bundle_hash`

A SHA-256 digest of the raw canonical schema-v2 bundle JSON bytes, computed before any normalization or storage transformation.

Property	Value
Computation	`sha256(raw_bundle_bytes)` - bytes as written by `benchbox run`
Stability	Immutable. The hash changes only if the bundle contents change.
Uniqueness	Content-addressable. Two identical runs produce the same hash.
Phase introduced	Phase 2 (PR CI validation); formally tracked from Phase 3
Used for	Deduplication, integrity verification, idempotent re-upload detection

The bundle_hash is included in the submission manifest that benchbox submit packages alongside the bundle. CI uses it to detect re-submitted identical bundles.

`submission_id`

A server-generated UUID (v4) assigned at first ingest of a bundle.

Property	Value
Computation	Generated by ingest service on first receipt
Stability	Stable across re-uploads of the same bundle (same `bundle_hash` → same `submission_id`)
Uniqueness	Unique per distinct bundle content (deterministically derived from `bundle_hash`)
Phase introduced	Phase 3 (Phase 2 uses the PR number as a submission handle)
Used for	API responses, status polling, audit trail

Re-uploading a bundle with a known bundle_hash returns the existing submission_id rather than creating a new one. This makes the ingest endpoint idempotent with respect to content.

`public_result_id`

A stable, human-readable permalink slug that identifies a result in the public corpus. In the explorer publication pipeline this is the same value as result_id on manifest entries and published bundle filenames.

Property	Value
Format	`{benchmark}-{platform}-sf{scale_factor}-{yyyymmdd}-{sha8}` (e.g., `tpch-duckdb-sf1.0-20260315-a1b2c3d4`)
`sha8`	First 8 hex characters of `sha256(published_bundle_bytes)`. Published bytes are the anonymized, canonical-JSON payload written to `bundles/{public_result_id}.json` — not the raw private capture.
Stability	Permanent once minted for a given set of published bytes. Same published bytes always re-derive the same id. Permanence attaches at publication (content address of public artifact), not at local run capture. See `docs/development/adr/adr-public-result-id-permanence.md`.
Uniqueness	Unique within the public corpus. Identical published digests under the same id are idempotent (publish once). Distinct published content that collides on the same id fails the build closed (`DuplicateResultIdError`); the Phase 1 pipeline does not append `-{n}`.
Minting	Phase 1: by the explorer static pipeline when projecting `results-data/` into the public corpus. Phase 2+: after validation passes, using the same format over the published payload.
Phase introduced	Phase 1
Used for	Stable URLs (`/results/r/{public_result_id}`), manifest index, compare links, `bundles/{public_result_id}.json`

The public_result_id is the only identifier that is stable across all phases and safe to use in external links. CLI tools and frontend must use this identifier for any URL or reference intended to be shared or bookmarked.

Any change that re-derives the ID of an already-public result needs an explicit alias or redirect so existing links keep resolving.

Identifier Summary

Identifier	Stability	Phase	Purpose
`bundle_hash`	Content hash, immutable	2+	Dedup, integrity
`submission_id`	UUID, stable per bundle	3	API tracking
`public_result_id`	Slug, permanent	1+	Public URLs, index

1.2 Submission and Publication State Model

Submission processing, archive acceptance, presentation policy, and observed deployment are separate state dimensions. No single published flag determines whether a result is accepted, visible, ranking-eligible, or live.

Status Definitions

Dimension and state	Meaning	Visible to public
Acceptance: `pending`	Received; awaiting validation	No
Acceptance: `validated`	Passed automated checks; awaiting acceptance	No
Acceptance: `accepted`	Validator-clean input exists at the pinned accepted-archive revision	Controlled independently
Acceptance: `rejected`	Failed validation or declined by maintainer; includes a reason string	No
Promotion: `not_requested`	No desired-state promotion currently targets the accepted input	Previous live state remains
Promotion: `promotion_pending`	Desired state names accepted input without a matching live receipt	Previous live state remains
Promotion: `live`	A fresh attested receipt proves the intended artifact and corpus are publicly observable	According to visibility policy
Promotion: `promotion_failed`	Build, deploy, or observation failed to produce a matching receipt	Previous live state remains
Presentation: `active`	Normal visibility and ranking policy applies	According to visibility policy
Presentation: `withdrawal_requested`	Authorized suppression is desired but not yet receipt-confirmed	Previous live state remains
Presentation: `withdrawn`	A receipt confirms suppression from public presentation and ranking	Tombstone only if a public ID existed; otherwise no public surface
Presentation: `readmission_requested`	Authorized restoration is desired but not yet receipt-confirmed	Existing tombstone if a public ID existed; otherwise no public surface

Status Transition Rules

Acceptance: pending → validated → accepted
pending → rejected
validated → rejected
Promotion: not_requested → promotion_pending → live
Promotion: promotion_pending → promotion_failed
Promotion: promotion_failed → promotion_pending
Promotion: live → promotion_pending
Presentation: active → withdrawal_requested → withdrawn
Presentation: withdrawn → readmission_requested → active

Promotion state is scoped to the latest target generation. A failed target may be retried or replaced by returning to promotion_pending. A receipt-confirmed live result may start a newer target generation by moving the latest attempt to promotion_pending; the prior current_live_generation and receipt remain observed live until the new target receives a matching receipt.

Acceptance remains an audit fact when promotion fails or presentation is withdrawn. Withdrawal must be carried into later desired state so ordinary promotion cannot resurrect it. Re-admission requires a new authorized policy event, not a contributor-controlled field. That event records readmission_requested; the result remains suppressed until a matching live receipt confirms the restoration generation, at which point presentation returns to active. A prior public ID keeps its tombstone while readmission is pending; a never-public private result continues to have no public surface.

published is retained only as a legacy API compatibility term. New contracts must report acceptance, promotion, and presentation separately, and must never translate a Git merge directly into live.

Additional rules:

Idempotent re-upload: Submitting a bundle whose bundle_hash already maps to an accepted result returns the existing identifiers and no new submission_id. A legacy "already_published" API response may be returned only when a matching live receipt exists; otherwise the response reports the orthogonal acceptance and promotion state.
Rejection reason: Every rejected transition must carry a human-readable reason string. Reasons are visible to the submitter but not to the public.
Withdrawn tombstone: A withdrawn result that previously had a public_result_id retains it as a stable tombstone stub with content fields redacted, and its public detail route returns 410 Gone. A never-public private result has no public ID or route, so its withdrawal is recorded only in the authenticated submission status. See Section 4.4.

Phase Assignment

State	Phase 1	Phase 2	Phase 3
`pending`	Not used (no ingest)	PR open, CI running	API received
`validated`	Not used	CI passed, awaiting merge	Async validation passed
`accepted`	Maintainer input accepted into the archive	Valid PR merged to `published-results`	Ingest acceptance committed
`promotion_pending`	Desired manifest lacks matching receipt	Same	Same
`live`	Matching attested public receipt	Same	Same
`promotion_failed`	Promotion ended without matching receipt	Same	Same
`rejected`	Not used	CI failed or PR closed without merge	Validation failed
Presentation: `withdrawal_requested` / `withdrawn` / `readmission_requested`	Authorized policy and receipt-confirmed suppression or restoration	Same	API or admin policy event

2. Visibility States and Trust Labels

2.1 Visibility States

Six visibility states control who can see a result and whether it appears in search indexes or compare views.

Visibility is orthogonal to archive acceptance, provenance trust, withdrawal, ranking eligibility, and deployment state.

State	Who can see it	Indexed	Phase available
`private`	Submitter only	No	Phase 3
`unlisted`	Anyone with the direct URL	No	Phase 3
`public-self-reported`	Everyone	Yes	Phase 2+
`public-vendor-reported`	Everyone	Yes	Phase 2+
`public-curated`	Everyone	Yes	Phase 1+
`public-verified`	Everyone	Yes	Future (reserved)

private - The result bundle is stored but the public_result_id is not minted. Useful for submitters who want to validate their results before publishing. Not available in Phase 1 or Phase 2 (no auth layer exists).

unlisted - The public_result_id is minted and the detail page is accessible, but the result does not appear in the manifest index, search results, or compare view browse. Useful for sharing a single result with a specific audience. Not available before Phase 3.

public-self-reported - Fully public and indexed. Results submitted via community PR (Phase 2) or self-service API (Phase 3) start in this state. The result carries a “Self-Reported” label (see Section 2.2).

public-vendor-reported - Fully public and indexed. Results produced by the platform vendor itself (rather than BenchBox maintainers or an arms-length community contributor). The vendor has a direct interest in the outcome, so the result carries a distinct “Vendor Supplied” label (see Section 2.2). Unlike public-self-reported, vendor-reported results are ranking-eligible (see Section 3.4): the conflict of interest is disclosed via the badge rather than by excluding the result from ranked tables. The vendor label is applied by maintainers and cannot be self-asserted: vendor-reported results are accepted only from maintainers.

public-curated - Fully public and indexed. Results generated and committed by BenchBox maintainers carry this visibility. This is the only state available in Phase 1.

public-verified - Reserved for future third-party attestation workflows. No results carry this state at launch; the attestation mechanics are not yet defined.

2.2 Trust Labels

Trust labels communicate the provenance and review depth of a result to explorer visitors. Labels appear on result detail pages, compare view headers, and manifest index cards.

Label	Display text	Visibility state	Meaning
`maintainer-run`	“Maintainer Run”	`public-curated`	Run by BenchBox maintainers in a controlled environment
`community-submission`	“Community Submission”	`public-self-reported`	Submitted by an arms-length community contributor via PR or API
`vendor-supplied`	“Vendor Supplied”	`public-vendor-reported`	Produced by the platform vendor itself; applied under maintainer control, never self-asserted
`verified`	“Verified”	`public-verified`	Attested by a third party (reserved, not yet in use)

Trust labels are informational. They do not imply that one result is more accurate than another; they communicate who ran the benchmark and under what level of review.

The trust label is derived from a result’s result_source (internal → maintainer-run, community → community-submission, vendor → vendor-supplied). This mapping, the visibility states, and the funding vocabulary (Section 2.4) are defined once in benchbox/core/results/provenance.py; the CLI, submission validator, corpus inventory, and explorer read model all import it rather than re-deriving the strings.

2.4 Funding Disclosure

Independent of result_source, every result may declare how the run was funded. Funding is orthogonal to provenance — a vendor-supplied result may be employer-funded, and a community result may be on a free trial. Funding is a public, informational disclosure; it does not affect ranking eligibility.

Funding value	Meaning
`employer`	Paid for by the runner’s employer
`personal`	Paid for out of pocket by an individual
`free-trial`	Run on a free trial or promotional credits
`vendor-sponsored`	The platform vendor paid for or sponsored the run
`grant`	Funded by a research or institutional grant
`unspecified`	Not disclosed (the default when no funding is declared)

Funding is declared by the producer at run time and carried through submission; unspecified is always a valid value.

2.3 Phase-by-Phase Display Rules

Phase 1

All results carry the public-curated visibility state and the maintainer-run trust label. No label selection logic is needed; the label is hardcoded in the static build pipeline.

Phase 2

Results fall into one of two categories based on their submission path:

Submission path	Visibility state	Trust label
Committed by maintainer CI	`public-curated`	Maintainer Run
Merged from community PR	`public-self-reported`	Community Submission

The explorer renders both labels. A legend on the browse and compare pages explains the distinction.

Phase 3

All six visibility states apply. The full label spectrum is rendered. The explorer must handle the case where a compare view contains results with different trust labels and display a per-result label badge rather than a single cohort-level label.

3. Cohort Eligibility and Ranking Rules

3.1 Comparison Cohort Definition

A valid comparison cohort is a set of results that can be meaningfully compared side-by-side in the explore compare view.

Required for a valid cohort:

Field	Requirement
`canonical_benchmark`	Must be identical across all results. `star_schema` is the legacy raw alias for canonical `ssb`; raw `benchmark` remains available for audit.
`scale_factor`	Must be identical across all results in the cohort (e.g., `1.0`)
`canonical_phase`	Must be identical for ranking identity. It is normalized raw `test_type` (lowercased, with `standard` folded into `power`), or explicit `unknown` when provenance is absent.
`stream_count`	Throughput phase only. Must be identical; throughput results run with different numbers of streams are separate cohorts. It is the number of entries in `phases.throughput_test.stream_results`, and absent for every other phase.
For completed throughput phases, a configured `run.streams` count (or
`config.streams` when absent from `run`) must equal the derived `stream_count`.
Older bundles without either configured count retain the legacy interpretation.
A mismatch excludes the result with `stream_count_mismatch`; corpus validation
rejects completed phases with the mismatch. Failed or partial phases are
excluded by phase status and may contain fewer recorded streams than configured.

A completed throughput phase must also record stream IDs that are exactly the integers 1..S, each with success: true. For TPC-H and TPC-DS, every stream must also record all 22 or 99 canonical queries as usable measurements (SUCCESS or PASS, at least 1 ms, run_type measurement), TPC-DS a/b variant pairs cover their base query, power-phase queries on stream 0 are ignored, and at least two streams are required. A completed phase that fails these checks is excluded with incomplete_throughput_streams and carries no Throughput@Size. Unsuccessful streams keep the phase-status exclusion.

These fields form the ranking cohort key. Raw submitted benchmark and phase values are never rewritten; the derived identity is generated centrally by the publish pipeline and consumed by the explorer. Results with different canonical values cannot appear in the same ranking claim.

The ranking metric is chosen per benchmark family and phase. TPC-H and TPC-DS power cohorts rank on power_score and their throughput cohorts rank on throughput_at_size (Throughput@Size); both rank higher-is-better. Every other cohort ranks on display_geomean_ms, lower-is-better.

3.2 Hard Blocks

The following mismatches are hard blocks. The compare view refuses to render and returns an error message explaining why the comparison is invalid.

Mismatch	Error message shown
Different `benchmark`	“Cannot compare results from different benchmarks. Select results from the same benchmark.”
Different `scale_factor`	“Cannot compare results at different scale factors. Select results at the same scale factor.”

Hard blocks apply in all phases. There is no override mechanism. The frontend must enforce this check client-side before attempting to render a comparison; the API (Phase 3) must enforce it server-side as well.

3.3 Soft Warnings

The following mismatches are soft warnings. The compare view renders but displays a prominent inline warning banner explaining which fields differ and how that may affect interpretation.

Mismatch	Warning message
Different `query_subset`	“Warning: these results use different query subsets. Totals and rankings reflect different query sets and may not be directly comparable.”
Different `tuning_mode`	“Warning: these results use different tuning modes (`tuned` vs `notuning`). Performance differences may reflect tuning rather than platform capability.”

Soft warnings do not prevent rendering. The compare view shows all selected results with the warning banner at the top. Users may dismiss the warning after reading it.

Additional fields that produce soft warnings if they differ: driver_version (if the submitter has pinned a non-default driver version). Phase differences are not a soft ranking mismatch: they select separate cohorts. A compare view may render mixed phases as evidence, but it must suppress winner/ranking claims and identify the phase mismatch explicitly.

3.4 Ranking Eligibility

Not all results are eligible for inclusion in ranked tables or aggregate leaderboard views within a cohort.

Visibility state	Eligible for ranked tables
`public-curated`	Yes
`public-vendor-reported`	Yes (badged)
`public-verified`	Yes (reserved)
`public-self-reported`	No
`unlisted`	No
`private`	No

public-self-reported results appear in browse views and can be selected for pairwise comparison, but they are excluded from any sorted table that is presented as a ranking. This prevents unverified results from affecting the official leaderboard within a cohort.

public-vendor-reported results are ranking-eligible, but always render with the distinct “Vendor Supplied” badge so the conflict of interest is visible in the ranked table. The disclosure is the badge, not exclusion.

Ranking totals and cohort-relative speedups count only ranking-eligible rows with non-null primary metrics. Visible rows that are not eligible keep their raw metrics for browse and comparison surfaces, but carry rank = null and do not define the cohort’s best or slowest ranked value.

The explorer must not display a ranked table that mixes public-curated and public-self-reported results without clear visual separation. If a cohort contains only public-self-reported results, the view renders as a browse list, not a ranked table.

4. Privacy, Redaction, and Retention

4.1 Public Fields

The following fields are always public for any live result with a public visibility state. They cannot be redacted by the submitter or admin.

Field	Notes
`benchmark`	Benchmark family (e.g., `tpch`)
`platform`	Platform identifier (e.g., `duckdb`)
`scale_factor`	Numeric scale factor (e.g., `1.0`)
`query_timings`	Per-query execution time in milliseconds
`validation_status`	Per-query validation result (passed/failed/skipped)
`submission_date`	Date the result was accepted into the archive
`trust_label`	The server-controlled provenance label assigned by policy
`funding`	How the run was funded (see Section 2.4); `unspecified` when not declared
`public_result_id`	The stable permalink slug
`query_subset`	Which queries were run (full set or named subset)
`tuning_mode`	Whether platform tuning was applied
`client_link`	Optional nested locality metrics under `environment.client_link` (`client_cloud`, `client_region`, `statement_overhead_ms`); safe non-identifying metadata

4.2 Redactable Fields

The following fields may be redacted by the submitter before submission or requested for redaction after publication. Admins may also redact any field. Redacted fields are replaced with the sentinel value "[redacted]" in all public-facing views.

Field	Who may redact	Notes
`system_hostname`	Submitter or admin	The machine name where the benchmark ran
`os_version`	Submitter or admin	Exact OS version string (e.g., `Ubuntu 22.04.3 LTS`)
`private_config_values`	Submitter or admin	Any platform-option key=value pairs the submitter marks private
`contributor_email`	Submitter or admin	Only collected in Phase 3; display name is shown instead
`ip_address`	Admin only (never public)	Stored for abuse detection; never exposed

Redaction does not change archive acceptance. Redaction before publication is reflected in the anonymized published bytes and therefore in the minted public_result_id (the sha8 is sha256 over published bytes). Redaction of an already-live result that changes published bytes (replacing a field with "[redacted]") is a compatibility event per docs/development/adr/adr-public-result-id-permanence.md: it mints a new content-addressed public_result_id (new sha8) and the old URL is preserved via an explicit alias or redirect to the new bundle, so existing links remain resolvable. Presentation visibility and ranking remain separate policy decisions; only the specific field value is hidden.

4.3 Retention Policy

State or asset	Retention
Accepted source bundle	Retained indefinitely by default. No automatic expiry.
`pending` / `validated`	Retained for 90 days from submission date, then purged if not promoted.
`rejected`	Retained for 30 days from rejection date, then purged.
`withdrawn` presentation	For a previously public result, its tombstone is retained indefinitely. For a never-public private result, no public tombstone exists. Non-sensitive audit evidence is retained indefinitely. Accepted source bytes remain preserved during withdrawal.

The accepted archive is retained independently of public presentation. Submitters may request withdrawal at any time (see Section 4.4); withdrawal does not silently rewrite the historical acceptance record or authorize source-byte deletion.

4.4 Data Removal

Submitters may request withdrawal of an accepted or live result at any time. There is no deadline. Withdrawal is available before and after publication.

Withdrawal process:

Phase 1: File an issue or contact maintainers. The maintainer records the withdrawal policy and republishes presentation without deleting accepted source bytes.
Phase 2: Same as Phase 1, or close the original PR if acceptance has not occurred.
Phase 3: For a result with a minted public ID, call DELETE /v1/results/{public_result_id} with a valid auth token. For a never-public private result, call DELETE /v1/submissions/{submission_id}. Either path may also be initiated through support.

Effect of withdrawal:

A result with a minted public ID is removed from active browse, search, ranking, and compare indexes within 7 days of the withdrawal request. Its stable ID remains in a separate tombstone lookup registry.
A never-public private result is addressed only by authenticated submission_id. Withdrawal must not mint or expose a public_result_id. A matching receipt confirms that no public-ID mapping, public route, or public index membership exists; only then is presentation recorded as withdrawn for the submitter. Because no public URL ever existed, no public tombstone is created.
Derived public read models suppress the result. Accepted source bytes are kept unless a separately approved erasure request requires their deletion.
For a result that had a public ID, a tombstone remains in the separate lookup registry indefinitely. The tombstone contains only: public_result_id, status: withdrawn, and withdrawn_at (date). All other fields are redacted. The detail page at the stable URL returns the tombstone with a 410 Gone HTTP status.

For results that had public IDs, the tombstone ensures that existing links and citations do not silently resolve to a different result. External parties who bookmarked the URL see a clear signal that the result was withdrawn rather than a 404 or an unrelated result.

5. Consumer Contract Reference

5.1 CLI Contract

The benchbox submit command is the primary submission interface for Phase 2 and Phase 3. This section defines what the CLI must do and what it can expect in return.

What `benchbox submit` packages

The CLI packages a submission bundle consisting of:

Component	Description
Canonical schema-v2 bundle	The `.json` file produced by `benchbox run`, unmodified
Companion files	Any files referenced in the bundle (query plans, etc.), if present
Submission manifest	A `submission.json` sidecar with contributor metadata

Result bundle schema policy

The canonical bundle is schema-v2 JSON. The current BenchBox producer writes top-level "version": "2.2". Public PR submission validation accepts numeric schema version family 2.x so a future producer minor can be submitted before the hosted contract document is updated, but it rejects missing or malformed versions.

Downstream consumers are stricter than public ingest:

Consumer	Accepted versions	Failure behavior
Runtime loader	`"2.0"`, `"2.1"`, `"2.2"`	Rejects with the runtime loader schema policy and asks for re-export.
Explorer static pipeline	`"2.0"`, `"2.1"`, `"2.2"`	Rejects before manifest/detail projection to avoid silent field drops.
Normalizer	`"2.0"`, `"2.1"`, `"2.2"` as v2; other shapes as legacy	Uses best-effort legacy extraction for v1.x and unknown shapes.

Allowed nested metadata blocks

Benchmark results may include optional nested metadata blocks within canonical schema-v2 sections:

environment.client_link: Allowed nested metadata block disclosing client-to-platform locality and statement round-trip overhead (collection_status, source, client_cloud, client_region, and statement_overhead_ms). This block contains non-identifying topological and driver overhead metrics only; raw network identifiers (IP addresses, hostnames, and port numbers) are strictly excluded from published bundles.

The submission manifest schema:

{
  "submission_tool_version": "<'benchbox/' + semver string>",
  "submitted_at": "<ISO 8601 UTC timestamp>",
  "bundle_file": "<filename of primary result JSON>",
  "bundle_hash": "<sha256 of primary result JSON>",
  "companion_hashes": {
    "<companion filename>": "<sha256 of companion file>"
  },
  "benchmark": "<benchmark name from result>",
  "platform": "<platform name from result>",
  "scale_factor": "<scale factor from result>",
  "phase": 2,
  "submission_path": "PR-based",
  "submitted_by": "<GitHub username or display name - Phase 2: PR author; Phase 3: authenticated actor>",
  "submission_notes": "<optional free-text, max 500 chars>"
}

Field notes:

submitted_by, submission_notes: Phase 2 - optional; Phase 3 - required.
benchmark, platform, scale_factor, bundle_file, phase, submission_path: All phases - required, extracted from result metadata.

`bundle_hash` computation

The CLI computes bundle_hash as the SHA-256 of the primary result JSON bytes. Companion files are covered separately through companion_hashes, a mapping of companion filename to SHA-256. The manifest itself is excluded because it is created after these hashes are computed. Hashes must be recomputed at submission time, not cached from a prior run, to detect post-run modifications.

Dry-run behavior

When invoked with --dry-run, benchbox submit:

Computes bundle_hash and validates it against the bundle contents.
Runs all client-side cohort and schema validations.
Prints a summary of what would be submitted (benchmark, platform, scale, hash, estimated submission manifest).
Does NOT write to results-data/, open a PR, or contact the API.
Exits with code 0 if validation passes, non-zero if it fails.

Dry-run is the recommended way to verify a submission before committing.

Phase 2 output mode (`--output`)

benchbox submit --output ./submission/ writes the submission bundle to a local directory for manual PR submission. No network calls are made. The output directory contains:

submission/
  bundle/
    <result_filename>.json
    <result_filename>.plans.json   (if present)
    <result_filename>.tuning.json  (if present)
  <result_filename>.manifest.json
  CONTRIBUTING.md

The contributor opens a PR against published-results by copying the contents of bundle/ into results-data/bundles/ and placing <result_filename>.manifest.json alongside the bundle files. See CONTRIBUTING.md in the output directory for step-by-step instructions.

Phase 3 API mode

When --service [URL] is passed, the CLI POSTs the submission bundle directly to the hosted API. Authentication comes from benchbox auth login keyring storage, BENCHBOX_SUBMIT_TOKEN, or the lower-precedence BENCHBOX_SERVICE_TOKEN; dry runs do not require credentials.

POST /v1/submissions
Content-Type: multipart/form-data

bundle:       <bundle.json bytes>
manifest:     <submission.json bytes>
companions[]: <companion files, if any>

Expected responses:

HTTP status	Meaning	CLI behavior
`202 Accepted`	Submission received; `submission_id` in body	Print submission_id and poll URL
`200 OK`	`bundle_hash` already accepted; body uses the complete status payload below, including presentation and current-live receipt fields	Print acceptance, latest promotion attempt, and independently observed live state; exit 0
`409 Conflict`	Bundle in `pending` or `validated` state; `submission_id` in body	Print status and poll URL
`422 Unprocessable`	Schema or hash validation failed; `errors` array in body	Print errors, exit non-zero
`429 Too Many Requests`	Rate limited; `Retry-After` in headers	Print wait time, exit non-zero

Status polling contract

After a Phase 3 submission, the CLI may poll for status:

GET /v1/submissions/{submission_id}/status

Response:

{
  "submission_id": "<uuid>",
  "bundle_hash": "<sha256>",
  "acceptance_status": "<pending|validated|accepted|rejected>",
  "promotion_status": "<not_requested|promotion_pending|live|promotion_failed>",
  "presentation_status": "<active|withdrawal_requested|withdrawn|readmission_requested>",
  "target_generation": "<integer or null>",
  "current_live_generation": "<integer or null>",
  "current_live_receipt": "<receipt identifier or null>",
  "current_live_observed_at": "<ISO 8601 or null>",
  "public_result_id": "<slug or null>",
  "rejection_reason": "<string or null>",
  "updated_at": "<ISO 8601>"
}

The idempotent POST /v1/submissions 200 OK response MUST return this same status payload, with submission_id null when no submission record exists. It must not return a reduced acceptance-and-promotion response because the CLI exits without polling on this path and would otherwise lose the prior observed live generation.

promotion_status describes the latest requested target generation. It does not replace the currently observed live generation while that target is pending or failed. current_live_generation, current_live_receipt, and current_live_observed_at identify the last receipt-confirmed generation independently; all three are null when no generation has ever been proven live. After a successful promotion they advance atomically to the target generation. This lets consumers report both a failed or pending promotion attempt and the prior generation that remains publicly live. presentation_status remains an independent policy and observation dimension: withdrawal_requested is desired suppression, while withdrawn requires a matching receipt that confirms suppression.

The CLI polls with exponential backoff (initial 5s, max 60s, up to 10 attempts) and prints the final status to stdout.

For compatibility, a legacy "status": "already_published" response may be emitted only when a matching attested live receipt exists. Accepted-but-not-live results must not be described as already published.

5.2 API Contract (Phase 3)

The hosted API is available at api.benchbox.dev.

Endpoint surface

Method	Path	Auth required	Description
`POST`	`/v1/submissions`	Yes	Submit a new bundle
`GET`	`/v1/submissions/{submission_id}/status`	Yes	Poll submission status
`GET`	`/v1/results`	No	List published results (paginated)
`GET`	`/v1/results/{public_result_id}`	No	Get result detail
`DELETE`	`/v1/results/{public_result_id}`	Yes (owner or admin)	Withdraw a result
`DELETE`	`/v1/submissions/{submission_id}`	Yes (owner or admin)	Withdraw a never-public private result
`GET`	`/v1/cohorts`	No	List available cohorts (benchmark + scale combinations)
`GET`	`/v1/compare`	No	Compare results by IDs (cohort validation enforced)

All public read endpoints (GET /v1/results, GET /v1/results/{id}, GET /v1/cohorts, GET /v1/compare) are unauthenticated. Rate limits apply to all endpoints; authenticated requests receive higher limits.

Response envelope

All API responses use a consistent envelope:

{
  "data": { ... },
  "error": null,
  "meta": {
    "request_id": "<uuid>",
    "timestamp": "<ISO 8601>"
  }
}

Error responses set data: null and populate error:

{
  "data": null,
  "error": {
    "code": "VALIDATION_FAILED",
    "message": "bundle_hash mismatch",
    "details": [ ... ]
  },
  "meta": { ... }
}

Rate limits

Endpoint group	Unauthenticated	Authenticated
Read endpoints	60 req/min	600 req/min
Submission endpoints	Not available	10 req/min
Status polling	Not available	120 req/min

Rate limit headers are returned on every response: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset.

5.3 Frontend/Explorer Contract

The explorer frontend consumes the static manifest JSON and the DuckDB database file. In Phase 3 it may also call the read API. This section defines what the frontend must know about visibility, trust, cohorts, and redaction.

Visibility state rendering

Visibility state	Render in browse index	Render in compare	Show detail page
`private`	No	No	No (404)
`unlisted`	No	No	Yes (direct URL only)
`public-self-reported`	Yes	Yes	Yes
`public-curated`	Yes	Yes	Yes
`public-verified`	Yes	Yes	Yes

The frontend must not expose private or unlisted results in any index or search view. For unlisted, the detail page is accessible if the user has the direct URL, but the result must not appear in autocomplete, search, or browse.

Trust label rendering rules

Every result card, detail page header, and compare view row must display the trust label badge.
In a compare view containing results with different trust labels, each result row displays its own badge. No aggregate or cohort-level badge is shown.
The badge for public-curated is rendered in the primary brand color.
The badge for public-self-reported is rendered in a neutral/muted color.
The verified badge (reserved) is rendered in a distinct accent color when implemented.
A legend explaining all trust labels is accessible from every page that displays them (persistent tooltip or a “What does this mean?” link).

Cohort validation rules for compare view

The frontend enforces cohort rules before rendering:

Hard block check: If any two selected results differ in canonical benchmark or scale_factor, show an error and refuse to render the comparison. If canonical phases differ, render evidence only and suppress winner/ranking claims. Link to the result detail pages so the user can verify the raw provenance. Raw aliases such as ssb and star_schema are one benchmark family for this check.
Soft warning check: If any two selected results differ in query_subset or tuning_mode, show a dismissible warning banner above the comparison table. Do not block rendering.
Ranking eligibility check: If the cohort contains any public-self-reported results, render the table as a browse-style list (no rank column, no “ranking” label). If all results are public-curated or public-verified, a rank column and total-duration sort are permitted.

Privacy and redaction rendering

Redacted fields display the sentinel string "[redacted]" in the UI, styled in a muted color (e.g., gray italic text). They are not hidden entirely.
The detail page must not display fields that are absent from the result JSON (some results omit optional fields entirely; this is distinct from redaction).
Withdrawn results that had a public ID render a dedicated “Result Withdrawn” page at the stable URL with tombstone data and a brief explanation. HTTP 410 is returned for API consumers; the frontend shows a human-readable message.
A never-public private result has no public route or frontend tombstone. Its withdrawal status is visible only through its authenticated submission record.