Skip to content

Search docs and blog

Frequently Asked Questions (FAQ)

This page answers common questions about BenchBox. For troubleshooting specific errors, see the Troubleshooting Guide.

General Questions

What is BenchBox?

BenchBox is a benchmarking toolbox that makes it simple to benchmark analytical (OLAP) databases. It provides industry-standard benchmarks like TPC-H, TPC-DS, and others in a single Python tool with minimal setup required.

Is BenchBox production-ready?

BenchBox is currently Beta software. Core functionality is stable (TPC-H, TPC-DS with DuckDB and major cloud platforms). APIs may still change before 1.0. We recommend thorough testing before using BenchBox in production. See the main README for details on Beta status.

The default wheel includes a benchbox.experimental namespace (nl2sql, aiml-functions, multiregion, gpu, load_testing). This namespace is not part of the supported Beta surface - it has no stability guarantees and is not integrated with the benchmark registry or CLI. Use it at your own risk. load_testing in particular is unsupported load-testing/workload-analysis research code, not a supported concurrency-testing feature.

Which databases does BenchBox support?

BenchBox supports:

  • Local: DuckDB, SQLite, ClickHouse (local mode)
  • Cloud: Databricks, Snowflake, BigQuery, Redshift

See the Platform Selection Guide and Comparison Matrix for detailed requirements and capabilities.

Which benchmarks are available?

BenchBox includes 22 public-discovery benchmarks, with examples across:

  • TPC Standards: TPC-H, TPC-DS, TPC-DI, TPC-DS-OBT, TPC-H Skew, TPC-Havoc
  • Academic Benchmarks: SSB, AMPLab, JoinOrder
  • Industry and real-world workloads: ClickBench, H2ODB, NYC Taxi, FlightData, TSBS DevOps, CoffeeShop
  • BenchBox Primitives and AI/ML: Read Primitives, Write Primitives, Metadata Primitives, Transaction Primitives, AI Primitives, Vector Search, TPC-H Data Vault

See the Benchmarks Catalog for feature details and selection guidance.

How do I cite BenchBox in academic work?

BenchBox is MIT-licensed open-source software. When citing BenchBox in academic papers:

Harris, Joe. (2025). BenchBox: A Benchmarking Toolbox for Analytical Databases.
GitHub repository: https://github.com/BenchBox-dev/benchbox

Please also cite the underlying benchmark specifications (TPC-H, TPC-DS, etc.) when reporting results.

Installation & Setup

How do I install BenchBox?

The recommended installation method uses uv:

Terminal window
uv add benchbox

For specific platforms, install extras:

Terminal window
uv add benchbox --extra databricks --extra snowflake

See the Installation Guide for complete instructions.

Why use uv instead of pip?

Both work fine! uv is faster and handles dependency resolution more reliably, but standard pip is fully supported:

Terminal window
python -m pip install benchbox

Do I need to compile TPC tools manually?

No. BenchBox includes pre-compiled TPC-H and TPC-DS data generation tools (dbgen, dsdgen, dsqgen) for all major platforms (macOS, Linux, Windows) in the _binaries/ directory. These are automatically used when you run benchmarks.

Can I run BenchBox without any database installed?

Yes! Install BenchBox with the [duckdb] extra and you can run benchmarks immediately - DuckDB creates databases in-process with no server setup.

Terminal window
uv add benchbox --extra duckdb
uv run -- benchbox run --platform duckdb --benchmark tpch --scale 0.01

How do I verify my installation?

Run these commands to verify:

Terminal window
benchbox --version


benchbox profile


benchbox check-deps --matrix

The first command checks the BenchBox version. The second profiles your environment. The third checks platform dependencies.

Running Benchmarks

What scale factor should I use?

It depends on your goal:

  • Quick testing: 0.01 or 0.1 (completes in seconds/minutes)
  • Development: 1 (standard size, ~1GB for TPC-H)
  • Performance testing: 10, 100, or higher (requires more time and resources)
  • TPC-DS note: Use scale ≥ 1 only (fractional scale causes binary segfaults)

Use benchbox profile to get scale factor recommendations for your system.

How long does a benchmark take?

Time varies significantly based on scale factor, platform, hardware, and benchmark complexity:

  • Scale factor: Larger scale factors = more data = longer runtimes
  • Query complexity: TPC-DS has 99 complex queries; TPC-H has 22 simpler queries
  • Platform: Cloud platforms add network latency and cluster startup time
  • Hardware: CPU cores, memory, and disk speed all affect performance

Use --dry-run to preview configurations without running queries. Run benchmarks to establish baselines for your specific environment. See Dry Run Mode.

Can I run just specific queries?

Yes, use the --queries flag:

Terminal window
benchbox run --platform duckdb --benchmark tpch --scale 1 --queries 1,6,13

Or run specific phases:

Terminal window
benchbox run --platform duckdb --benchmark tpch --scale 1 --phases power

See CLI Quick Start for all options.

How do I run benchmarks in CI/CD?

Add --non-interactive to skip prompts and use specific output directories:

Terminal window
benchbox run \
  --non-interactive \
  --platform duckdb \
  --benchmark tpch \
  --scale 0.1 \
  --output results/ci-run-${{ github.run_id }}

See CI/CD Integration for complete examples.

Where are benchmark results stored?

By default, results are stored in benchmark_runs/ with timestamped subdirectories:

benchmark_runs/
├── 20250103_143022_tpch_duckdb_sf1/
│   ├── manifest.json
│   ├── results/
│   │   └── power_test_results.json
│   └── logs/

Specify a custom location with --output:

Terminal window
benchbox run --output /path/to/results --platform duckdb --benchmark tpch --scale 1

Can I reuse generated data?

Yes! By default, BenchBox checks for existing data and skips regeneration. To force regeneration:

Terminal window
benchbox run --force --platform duckdb --benchmark tpch --scale 1

Generated data is stored in benchmark_runs/*/data/ by default.

Platform-Specific Questions

How do I configure cloud platform credentials?

Use environment variables (recommended) or the interactive setup wizard:

Environment variables:

Terminal window
export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_TOKEN="your-token"
export SNOWFLAKE_ACCOUNT="your-account"
export SNOWFLAKE_USER="your-user"
export SNOWFLAKE_PASSWORD="your-password"

Setup wizard:

Terminal window
benchbox platforms setup

See the Platform Selection Guide for platform-specific details.

Why is my cloud benchmark slow?

Common causes:

  • Cold start: First query on cloud warehouses is often slower (cluster startup)
  • Small warehouse: Increase warehouse/cluster size (e.g., XSmall to Small)
  • Network latency: Data transfer between cloud storage and compute
  • Cache warming: Run with --phases generate,load,power first

See Performance Optimization for tuning tips.

Can I use BenchBox with self-hosted databases?

Yes! BenchBox currently supports:

  • ClickHouse: Fully supported via --platform clickhouse-server (local or remote)
  • DuckDB: Fully supported for local/embedded use via --platform duckdb
  • SQLite: Fully supported for local/embedded use via --platform sqlite

For databases not yet supported, see the Development Roadmap. You can also create custom platform adapters - see Adding New Platforms for guidance.

Do cloud benchmarks cost money?

Yes. Running benchmarks on cloud platforms incurs costs for:

  • Compute (warehouse/cluster runtime)
  • Storage (data stored)
  • Data transfer (network egress)

Start with small scale factors and monitor costs. Use --dry-run to plan resource usage without executing queries.

Data & Query Questions

Are the TPC benchmarks official/compliant?

BenchBox implements TPC-H and TPC-DS queries and data generation according to specifications, but BenchBox is not TPC-certified and results cannot be published as official TPC results.

For official TPC compliance and publication:

  • Results must be audited by the TPC
  • Specific hardware and configuration requirements apply
  • See TPC.org for official benchmarking

BenchBox is designed for internal performance testing and comparison.

Can I modify the queries?

Yes, but be aware that modifying standard benchmark queries defeats the purpose of standardized benchmarking. For custom workloads, consider:

  • Creating custom benchmarks (see Custom Benchmarks)
  • Using query variants (some benchmarks support this)
  • Forking queries for experimentation

Query files are located in benchbox/core/{benchmark}/queries/.

How does query translation work?

BenchBox automatically translates queries to platform-specific SQL dialects using SQLGlot. The translation handles:

  • Function name differences (e.g., SUBSTRING vs SUBSTR)
  • Date/time functions
  • Type casting
  • Platform-specific syntax

In rare cases, manual tuning may be needed for platform-specific SQL syntax.

Can I use compressed data formats?

Yes! BenchBox supports multiple compression formats:

  • Parquet (recommended for most platforms)
  • CSV with compression (gzip, zstd)
  • Platform-native formats

Configure with tuning files or command-line options. See the Data Generation guide for details.

Troubleshooting & Errors

I get “command not found: benchbox”

The benchbox CLI is not in your PATH. Solutions:

  1. Ensure your virtual environment is activated
  2. Add Python’s scripts directory to PATH
  3. Run with uv run -- benchbox instead

See the Troubleshooting Guide Installation Issues section.

Benchmark fails with “missing dependency”

Install the required platform extras:

Terminal window
uv add benchbox --extra databricks

Run benchbox check-deps --matrix to see all platform dependencies and their status.

Queries fail with syntax errors

This may indicate a query translation issue. Try:

  1. Check the platform is officially supported
  2. Review query logs in benchmark_runs/*/logs/
  3. Report the issue with query details

See the Troubleshooting Guide Query Execution Issues section.

TPC-DS fails at scale < 1

TPC-DS data generation tools have a known issue with fractional scale factors (causes segfault). Always use scale ≥ 1 for TPC-DS:

Terminal window
benchbox run --platform duckdb --benchmark tpcds --scale 1

This is a limitation of the upstream TPC-DS tools, not BenchBox.

Advanced Features

Can I run benchmarks in parallel?

Yes! Some benchmarks support throughput tests with concurrent queries. Set the stream count with --concurrency (a hidden option on run, shown by benchbox run --help-topic all):

Terminal window
benchbox run --official --benchmark tpch --platform duckdb --scale 1 \
  --phases throughput --concurrency 4 --seed 42

The throughput driver floors the stream count at the TPC minimum of 2. See Power Run & Concurrent Queries.

How do I compare results across runs?

BenchBox stores results in JSON format with full run metadata. You can:

  • Parse JSON files programmatically
  • Use the Python API for analysis
  • Build custom comparison tools

See Result Schema and API Reference.

Can I create custom benchmarks?

Yes! BenchBox provides APIs for creating custom benchmarks:

from benchbox.base import BaseBenchmark


class MyCustomBenchmark(BaseBenchmark):
    pass

Implement the required methods in the subclass body.

See Custom Benchmarks Guide for complete instructions.

How do I automate performance regression detection?

BenchBox integrates with CI/CD for automated regression detection:

  1. Run benchmarks in CI on each commit
  2. Compare results against baseline
  3. Fail builds when performance regresses

See Performance Monitoring for setup examples.

Contributing & Support

How can I contribute?

Contributions are welcome! See the root CONTRIBUTING.md file for:

  • Code contribution guidelines
  • Development setup
  • Testing requirements
  • Pull request process

You can also contribute by:

  • Reporting bugs and issues
  • Improving documentation
  • Sharing feedback and use cases

Where do I report bugs?

Create an issue on GitHub with:

  • BenchBox version (benchbox --version)
  • Platform and benchmark being used
  • Complete error messages and logs
  • Minimal reproduction steps

How do I request new features?

Open a feature request on GitHub describing:

  • The use case and motivation
  • Proposed behavior or API
  • Alternative solutions you’ve considered
  • Any relevant examples from other tools

Where can I ask questions?

For questions not covered in the documentation:

  1. Check existing GitHub issues and discussions
  2. Open a new discussion or issue
  3. Provide context and specific details

Support is provided on a best-effort basis during Beta development.

See Also

BenchBox is open source under the MIT License.