BenchBox Architecture

High-level overview of BenchBox’s design, components, and extension points.

Design Philosophy

BenchBox is built on three core principles:

Separation of Concerns: Benchmark logic, platform adapters, and query execution are independent layers
Extensibility: New benchmarks and platforms can be added without modifying core code
Standards Compliance: TPC benchmarks follow official specifications for reproducibility

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                       CLI / Python API                       │
│                    (User Entry Points)                       │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Orchestration Layer                        │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Run Configuration   │   Query Selection             │  │
│  │  Parameter Generation│   Result Collection           │  │
│  └──────────────────────────────────────────────────────┘  │
└──────────────────┬──────────────────────────────────────────┘
                   │
     ┌─────────────┼─────────────┐
     │             │             │
┌────▼────┐  ┌────▼────┐  ┌────▼────┐
│Benchmark│  │Platform │  │ Results │
│  Layer  │  │ Adapter │  │  Model  │
└─────────┘  └─────────┘  └─────────┘
     │             │             │
┌────▼────────────▼─────────────▼────┐
│       Database Connection          │
│     (Platform-Specific Driver)     │
└────────────────────────────────────┘

Repository map

The repository separates the installable product from its documentation, tests, examples, and project operations:

Path	Purpose
`benchbox/`	Python package, benchmark implementations, platform adapters, CLI, and result model
`docs/`	User guides, concepts, reference material, design records, and contributor documentation
`tests/`	Unit, integration, end-to-end, performance, and live test suites
`examples/`	Runnable examples, notebooks, tuning files, and workflow patterns
`results-explorer/`	Browser application for published benchmark results
`results-data/`	Public result corpus and its validation metadata

This map describes ownership boundaries rather than every directory. Use the linked component documentation below for implementation details.

Core Components

1. Benchmark Layer

Location: Public wrappers live in benchbox/*.py on benchbox.base.BaseBenchmark. Core implementations live under benchbox/core/{benchmark}/.

Responsibility: Encapsulates benchmark-specific logic

Key Classes:

BaseBenchmark: Public wrapper base in benchbox/base.py for benchmark construction and the run_with_platform API. It remains a supported public base; it is not a retired execution model.
{Benchmark}: Implementation class (e.g., TPCH, TPCDS)
{Benchmark}Generator: Data generation logic
{Benchmark}Queries: Query templates and parameterization

A benchmark class knows how to generate data (the dbgen invocation), how to retrieve queries (with parameter substitution), its schema definitions, and its validation rules.

Example:

from benchbox import TPCH

See: Custom Benchmarks Guide

2. Platform Adapter Layer

Location: benchbox/platforms/{platform}/

Responsibility: Abstracts database-specific connection and execution logic

Key Classes:

{Platform}Adapter: Main adapter class (e.g., DuckDBAdapter)
{Platform}Connection: Connection wrapper implementing DatabaseConnection
Platform-specific optimizations (bulk loading, query hints)

An adapter handles connection management, data loading strategies (COPY, INSERT, external tables), query execution and error handling, and result collection and formatting.

Example:

from benchbox.platforms.duckdb import DuckDBAdapter

Supported Adapters:

DuckDB (embedded, local files)
ClickHouse (native protocol)
Databricks (SQL Warehouse, Unity Catalog)
BigQuery (serverless, cloud storage)
Snowflake (warehouse, stages)
Redshift (serverless, provisioned)
SQLite (testing, minimal datasets)

See: Platform Selection Guide

3. Results Model

Location: benchbox/core/results/

Responsibility: Structured representation of benchmark execution results

Key Classes:

QueryResult: Schema-facing query row model used by compatibility and validation boundaries (benchbox/core/schemas.py); it is not the canonical in-memory execution object.
QueryExecution: Canonical in-memory result for one executed query (benchbox/core/results/models.py), including timing, status, validation, resource usage, and optional plan data.
BenchmarkResults: Aggregate benchmark result with timing, metadata, validation, and query-result summaries.
ExecutionPhases: Setup, power test, throughput test phases
ValidationResult: Data quality and correctness checks

Schema:

{
    "benchmark_name": "TPC-H",
    "platform": "DuckDB",
    "scale_factor": 1.0,
    "execution_id": "tpch_1234567890",
    "timestamp": "2025-10-12T10:30:00Z",
    "total_execution_time": 45.2,
    "query_results": [
        {
            "query_id": "q1",
            "execution_time": 2.1,
            "status": "SUCCESS",
            "row_count": 4
        }
    ],
    "execution_phases": { ... },
    "validation_status": "PASSED"
}

See: Result Schema Reference

4. Connection Abstraction

Location: benchbox/core/connection.py

Responsibility: Unified interface for database operations

Key Interface:

class DatabaseConnection(ABC):
    @abstractmethod
    def execute(self, query: str) -> Any:
        pass

    @abstractmethod
    def fetchall(self, cursor) -> list:
        pass

    @abstractmethod
    def close(self) -> None:
        pass

execute runs a query and returns a cursor or result, fetchall fetches all results from a cursor, and close closes the connection. All platform adapters implement this interface, enabling benchmark code to remain platform-agnostic.

5. Data Generation

Location: benchbox/core/{benchmark}/generator.py

Responsibility: Create synthetic benchmark data

Mechanisms:

TPC Benchmarks: Invoke official C binaries (dbgen, dsdgen, datagen)
Custom Benchmarks: Python-based generation using Faker, NumPy, Pandas
Output Formats: Parquet (default), CSV, JSON

Example: TPC-H uses the official dbgen binary, and generate() returns a list of .parquet files. Custom benchmarks use Python, and the coffee shop generator generates data with Faker.

generator = TPCHGenerator(scale_factor=1.0, output_dir="./data")
file_paths = generator.generate()

generator = CoffeeShopGenerator(scale_factor=0.001)
file_paths = generator.generate()

See: Data Generation Guide

6. CLI Orchestration

Location: benchbox/cli/

Responsibility: Command-line interface and workflow orchestration

Commands: the examples run a benchmark end-to-end, generate data only, do a dry run that previews queries, and check dependencies.

benchbox run --benchmark tpch --platform duckdb --scale 1

benchbox datagen --benchmark tpcds --scale 0.1

benchbox run --benchmark tpch --dry-run ./output

benchbox check-deps --matrix

Orchestrator Flow:

Parse CLI arguments and config files
Validate platform dependencies
Initialize benchmark and platform adapter
Execute benchmark phases (setup, queries, validation)
Collect and save results
Generate reports

See: CLI Quick Start

Data Flow

End-to-End Execution Flow

1. User Command
   ├── benchbox run --benchmark tpch --platform duckdb --scale 1
   └── Parsed by CLI → creates RunConfiguration

2. Benchmark Initialization
   ├── TPCH(scale_factor=1.0)
   ├── Checks if data exists at output_dir
   └── generate_data() if needed

3. Platform Adapter Setup
   ├── DuckDBAdapter()
   ├── Establishes database connection
   └── Creates schema (CREATE TABLE statements)

4. Data Loading Phase
   ├── Platform-specific bulk load (COPY FROM, external tables)
   ├── Validates row counts
   └── Creates indexes/constraints

5. Query Execution Phase
   ├── For each query in benchmark:
   │   ├── Get query text with parameters
   │   ├── Execute via platform adapter
   │   ├── Measure execution time
   │   └── Collect results
   └── Aggregate timing statistics

6. Results Collection
   ├── Create BenchmarkResults object
   ├── Include execution metadata (platform info, system profile)
   ├── Validate results (if validation rules exist)
   └── Save to JSON file

7. Output
   ├── Print summary to console
   ├── Save detailed JSON to output_dir
   └── Optional: Upload to cloud storage

Extension Points

Adding a New Benchmark

Create benchbox/core/{benchmark}/ directory
Implement classes:
- {Benchmark}BenchmarkImpl(BaseBenchmark)
- {Benchmark}Generator
- {Benchmark}Queries
- {Benchmark}Schema
Register in benchbox/__init__.py

See: Custom Benchmarks Guide

Adding a New Platform

Create benchbox/platforms/{platform}/ directory
Implement:
- {Platform}Adapter
- {Platform}Connection(DatabaseConnection)
Add platform extras to pyproject.toml
Add the platform’s canonical entry to benchbox/core/platform_manifest.py, including one explicit support_status. The registry’s _OPTIONAL_ADAPTERS and _build_platform_metadata() paths are live pass-throughs, not independent registration authorities.

External Table Support

Platforms can optionally support --table-mode external by setting supports_external_tables = True and implementing create_external_tables(). In external mode the runner skips native schema creation and load_data(), calling create_external_tables() instead to register views or external table definitions over staged Parquet files.

Shared logic for Hive-style external tables (Trino, Presto) lives in HiveExternalTableMixin (benchbox/platforms/base/external_table_mixin.py). Shared logic for managed Spark external tables (Athena Spark, EMR Serverless, Dataproc Serverless, Glue) lives in SparkExternalTableMixin (benchbox/platforms/base/cloud_spark/external_tables.py). Platforms that need pre-flight validation (e.g. Athena checking for an S3 bucket) can implement validate_external_table_requirements(), which the runner calls before proceeding.

See: Adding New Platforms

Adding Query Parameter Variants

TPC benchmarks support query variants with different parameter substitutions. The first call returns a query with random parameters (the default), the second returns a query with specific parameters, and the third generates multiple variants for a seed sweep:

query = benchmark.get_query("q1")

query = benchmark.get_query("q1", params={"date": "1998-09-02", "quantity": 24})

variants = benchmark.generate_query_variants("q1", count=5, seed_start=42)

See: TPC Patterns Usage

Design Patterns

1. Adapter Pattern

Platform adapters isolate database-specific logic, allowing benchmarks to remain platform-agnostic.

2. Benchmark Lifecycle

The lifecycle is orchestrated by the core run service and lifecycle runner: configuration and platform setup flow through execute_run(...), which calls run_benchmark_lifecycle(...). BaseBenchmark remains the public wrapper base and benchmark-facing API boundary; it is not the owner of the entire runtime execution workflow.

3. Strategy Pattern

Data loading strategies vary by platform (bulk COPY, INSERT batches, external tables) but implement a common interface.

4. Factory Pattern

get_platform_adapter(name) creates appropriate adapter instances based on configuration.

Performance Considerations

Data Loading Optimization

Parquet Format: Columnar storage for fast analytical queries
Bulk Loading: Platform-specific optimizations (DuckDB COPY, ClickHouse INSERT, BigQuery external tables)
Parallel Loading: Multi-threaded data ingestion where supported

Query Execution Optimization

Connection Pooling: Reuse connections across queries
Result Streaming: Fetch results incrementally for large result sets
Query Compilation: Use platform-specific prepared statements

Memory Management

Scale Factor Validation: Prevent OOM by validating scale vs. available memory
Streaming Results: Don’t materialize full result sets unless needed
Cleanup: Explicit connection and resource cleanup

Security Considerations

Credential Management

Environment variables for sensitive data
Support for platform-specific auth (OAuth, IAM, service accounts)
No credentials in configuration files or logs

SQL Injection Prevention

Parameterized queries where possible
Input validation on query IDs and parameters
No direct string concatenation into SQL

Data Privacy

Support for synthetic data only (no real customer data)
Anonymization support for custom benchmarks
Data residency controls for cloud platforms

Testing Architecture

Test Layers

Unit Tests (tests/unit/): Component-level testing
Integration Tests (tests/integration/): Database interaction tests
Live Tests (tests/integration/platforms/): Real cloud platform testing

See: Testing Guide

Workflow Concepts - Typical benchmarking workflows
Data Model - Result schema and data structures
Platform Selection Guide - Choosing a platform
Custom Benchmarks - Creating new benchmarks
Adding Platforms - Platform development

BenchBox Architecture

Design Philosophy

Architecture Diagram

Repository map

Core Components

1. Benchmark Layer

2. Platform Adapter Layer

3. Results Model

4. Connection Abstraction

5. Data Generation

6. CLI Orchestration

Data Flow

End-to-End Execution Flow

Extension Points

Adding a New Benchmark

Adding a New Platform

External Table Support

Adding Query Parameter Variants

Design Patterns

1. Adapter Pattern

2. Benchmark Lifecycle

3. Strategy Pattern

4. Factory Pattern

Performance Considerations

Data Loading Optimization

Query Execution Optimization

Memory Management

Security Considerations

Credential Management

SQL Injection Prevention

Data Privacy

Testing Architecture

Test Layers

Related Documentation