Skip to content

Concurrency Contract Proof Matrix

This matrix binds each concurrency claim to the lowest repository layer that can falsify it. It does not certify live adapter behavior or an external MCP deployment. Local durable-job tests do not certify TLS termination, deployment, rollback, operator response, or a production database adapter.

Repository evidence

InvariantProducer to consumerFailure injectionProving evidence
Expected-result publication is single-flight and validation policy is run-localexpected-results provider to TPC runnerconcurrent exact, loose, skip, and disabled runstests/unit/core/expected_results/test_singleflight_run_policy.py
Failed or incomplete throughput work cannot publish a valid scorestream runner to official resultfailed dictionary and object results carrying misleading metricstests/unit/core/test_tpc_reporting_execution.py, tests/unit/core/throughput/test_containment.py
A throughput adapter is admitted only from the canonical manifest capabilityplatform manifest to stream admissionundeclared adapter and unsupported manifest entrytests/unit/platforms/test_throughput_session_capability_sweep.py, tests/unit/platforms/throughput_session_capability_snapshot.json
Requested TPC-DI phases fail the aggregate outcome when any phase failsphase result to enhanced pipeline resultfailed data, SCD, and incremental phasestests/integration/test_tpcdi_phase3_benchmark.py
A failed TPC-DI file transform fails the ETL run and names the failed files, sequentially and in parallelper-file transform to run_etl_pipeline result and batch statusone unreadable source file among healthy files; every file failing in paralleltests/unit/tpcdi/test_transform_outcomes.py
An enhanced TPC-DI data phase with a missing processor or no input files is not reported as successprocessor and generated-file inventory to phase resultno connection-bound processors; empty file liststests/unit/tpcdi/test_transform_outcomes.py
A later phase cannot start while throughput workers may still runstream runner outstanding state to phase-boundary gate and combined sequencerstimed-out running stream; maintenance requested after ittests/unit/core/throughput/test_containment.py (TestOutstandingOwnershipState, TestPhaseBoundaryGate, TestOfficialBenchmarkContainment, TestCombinedSequencerContainment)
Termination is observed from worker futures, never assumedretained futures to boundary releaserunning future released mid-wait; no observable handlestests/unit/core/throughput/test_containment.py (TestAwaitQuiescence)
Contained work blocks reuse of the measurement connection for post-measurement probesadapter containment flag to plan capture and link probeworker still running when the bounded cleanup wait expirestests/unit/platforms/test_base_adapter.py (TestDeferredConnectionClose)
The measurement connection closes only after contained workers endadapter close path to retained worker futures, after result capture has consumed the throughput resultblocked worker released after the close requesttests/unit/platforms/test_base_adapter.py (TestDeferredConnectionClose)
Independent-connection adapters give each stream its own session; shared-cursor adapters share one by declarationadapter capability to stream connection factorytemp-table and session-setting leakage across streams; undeclared overridetests/integration/test_throughput_session_isolation.py
TPC-DI generation is repeatable for one seed across repeated requestsgeneration request to fact filesreuse one generator after its nested RNG has advancedtests/unit/test_tpcdi_phase1.py
BigQuery TPC-DI relative dates use valid interval syntaxSQLite query source to BigQuery executionAQ7 DATE('now', '-90 days') rewrite and parsetests/unit/platforms/test_bigquery_adapter.py
Only the current MCP owner may publish or completedurable attempt to result row and artifactstale owner after lease fencingtests/unit/mcp/test_durable_jobs.py, tests/integration/mcp/test_durable_jobs.py
Cancellation or lease loss never proves database work stoppeddurable attempt to admission capacitycancellation plus expired lease while the executor remains blockedtests/integration/mcp/test_durable_jobs.py
Unknown work retains capacity until a durable quiescence attestationrecovery and retention to replacement claimretention expiry before and after worker quiescencetests/unit/mcp/test_durable_jobs.py, tests/integration/mcp/test_durable_jobs.py
Durable global and per-principal running caps hold across concurrent processesclaim transaction to shared SQLite statetwo spawned processes claiming at a cap of one, globally and for one principaltests/integration/mcp/test_durable_jobs.py (test_multiprocess_claims_respect_global_running_limit, test_multiprocess_claims_respect_per_principal_running_limit)
Durable queue bounds hold across repository handlessubmit and retry transactions to shared SQLite stateglobal and per-principal queue limits, each through separate repository handles in one process; not proven across processestests/unit/mcp/test_durable_jobs.py
A durable job stores and exposes a shared outcome derived from its resultresult payload to job row and status toolfailed queries, failed phases, failed validation, no result, rejected request, quarantined outstanding worktests/unit/mcp/test_durable_jobs.py (test_job_stores_and_exposes_shared_outcome_derived_from_result, test_public_outcome_reports_non_terminal_and_quarantined_states, test_recovered_published_artifact_keeps_its_derived_outcome)
Outstanding throughput work is detected from the real result export shape and from the result object when both exports failresult export to job workerphases.throughput_test.outstanding_work payload; exporter and payload builder both failingtests/unit/mcp/test_run_response_outstanding_work.py
Work that quiesced before publication is published annotated; any reported stream the worker did not track, or cannot parse, keeps the job quarantinedstream runner results and exported response to job publicationstale outstanding export of finished work; untracked stream id beside a tracked one; unparseable id; untracked id while tracked work still runstests/unit/mcp/test_durable_jobs.py (test_stale_outstanding_export_of_quiesced_work_publishes_an_annotated_result, test_outstanding_streams_the_worker_never_tracked_keep_the_job_quarantined, test_untracked_stream_keeps_the_job_unobserved_even_while_tracked_work_still_runs, test_unparseable_stream_id_quarantines_instead_of_escaping_as_a_failure)
Leaked throughput work releases its capacity only when its futures finishstream runner futures to worker attestation to claim admissionblocked stream future released after the job was quarantined; no observable handlestests/unit/mcp/test_durable_jobs.py (test_leaked_throughput_work_attests_quiescence_when_its_futures_finish, test_leaked_work_without_observable_handles_stays_quarantined, test_leaked_work_quarantines_even_when_the_executor_raises_or_the_response_hides_it, test_quiescence_attestation_is_retried_and_logged_when_the_store_fails)
Recovery never deletes the staging directory of a fenced attempt that may still writerecovery fence to staging directoryexpired lease with a live writer; purge before and after quiescencetests/unit/mcp/test_durable_jobs.py (test_recovery_leaves_fenced_attempt_staging_until_quiescence_and_purge)
The load-testing executor bounds its waits, stops abandoned streams between queries, reports them as outstanding, and times durations and latencies on the monotonic clockstream futures to run resultstream blocked past the drain bound; query over its timeout; wall clock stepping backwards; non-positive timeouttests/unit/core/load_testing/test_executor.py (TestExecutorBoundedWaitsAndClocks, TestExecutorLatencyAndAbandonedStreams)
Fairness and per-principal FIFO do not depend on wall-clock orderdurable enqueue and service sequences to claim selectionwall-clock rollback between submissions and claimstests/unit/mcp/test_durable_jobs.py
Capacity reporting does not expose another tenant’s job identitydurable capacity state to MCP callertwo authenticated principals with quarantined worktests/integration/mcp/test_durable_jobs.py

The concurrency tests use events, fenced row transitions, database-owned sequences, or separate processes as synchronization oracles. Sleeps may bound a test, but a sleep alone is not evidence that work stopped or that ownership changed.

Adapter evidence

The platform manifest is the single declaration source for stream capability. Repository tests establish the contract wiring and focused behavior for the adapters with existing session implementations. They do not establish live service equivalence for every platform.

  • Shared-cursor support is declared only for the established embedded or Spark-style implementations pinned by the capability snapshot.
  • Independent-session support is declared only where an adapter already has a tested new_stream_connection implementation or inherits one from its wire family.
  • All other manifest adapters are unsupported until adapter-specific tests or UAT prove database, catalog, schema, credentials, settings, overlap, and cleanup equivalence.
  • No adapter in this surface currently claims native mid-query cancellation.

Live container and cloud evidence belongs to docs/operations/uat-framework.md. An unsupported classification is a safe product limit, not proof that the platform can never support throughput.

Known fail-closed boundary

An unknown attempt consumes durable capacity until its displaced worker attests quiescence. Three cases never attest, and BenchBox does not release the fence merely because retention time elapsed:

  • The worker process dies after losing its lease and before it can attest.
  • The worker process restarts or exits while leaked futures are still running. The attesting thread dies with the process and nothing else observes the futures.
  • The job returned a result that reports outstanding throughput work and the worker does not hold futures for every outstanding stream the result reports, for example a custom executor that does not go through StreamRunner, a stream id the worker never tracked, or an id that cannot be parsed.

A worker that did observe the leaked futures, on a normal return or after the executor raised, attests from a background thread once they all finish, then removes the attempt’s staging directory. Attestation is retried with backoff and failures are logged. Until then each such job holds one running slot for its principal and one global slot. No operator release path exists. A future one must obtain explicit termination evidence before it can release capacity; until then, manual resubmission is unsafe.

Staging directories of fenced attempts are not deleted at recovery. The displaced worker removes its own directory when it finishes, and the retention purge removes any leftover once the job is terminal.

Load-testing executor limit

An abandoned load-testing stream stops only between queries. A query that never returns keeps its worker thread alive, and the standard library joins pool threads at interpreter exit, so a hung query can still delay process exit. The run result lists such streams in outstanding_stream_ids with cleanup_state set to outstanding.

Quiesced work in a published result

Throughput work that the stream runner flagged as outstanding can finish before the job publishes. When every reported outstanding stream id was tracked by the worker and all tracked futures are done, the job publishes normally. The published response moves the stream ids from stream_ids (or outstanding_stream_ids) to quiesced_stream_ids, sets cleanup_state to quiesced, and sets quiesced_before_publication to true. The canonical result file named by mcp_metadata.result_file was written at export time and may still say outstanding. mcp_metadata.result_file_note states that the result file reflects the moment of export. A reported id the worker never tracked keeps the job quarantined.

Job outcome for rows written before the outcome field

Job rows written before the outcome column existed have no stored outcome. For a completed row with a retained artifact the status tool derives the outcome from that artifact. Rows without an artifact report a null outcome.