Skip to content

Search docs and blog

Getting Started in 5 Minutes

Follow these steps to install BenchBox, verify your environment, and run a reproducible benchmark. Everything below works on macOS, Linux, and Windows with Python 3.11+.

Step 0 - Prerequisites

  1. Install uv (recommended) or make sure you have pip available.
  2. Ensure DuckDB can create temporary files in your working directory. No other services are required for the quick start.
Terminal window
mkdir benchbox-demo && cd benchbox-demo

Creating an isolated project directory is optional.

Step 1 - Install BenchBox with DuckDB

Install the duckdb extra. DuckDB is an optional dependency - a plain uv add benchbox installs the core package with SQLite only, and every command in this guide uses DuckDB.

Terminal window
uv add benchbox --extra duckdb

Prefer pip? python -m pip install "benchbox[duckdb]" (quote the brackets so your shell does not expand them).

Verify the CLI is on your PATH:

Terminal window
uv run -- benchbox --version

For other platforms, extras, and dependency checks, see the dedicated installation guide.

Step 2 - Profile Your Environment

The profile command confirms CPU, memory, and available adapters. BenchBox uses this information to suggest scale factors and concurrency levels.

Terminal window
uv run -- benchbox profile

Look for the Available Databases table. duckdb should be Ready because Step 1 installed the duckdb extra; if it is not, re-run the Step 1 install command. If you plan to use cloud platforms later, run uv run -- benchbox check-deps --platform <name>, or use the guided benchbox setup --platform <name> wizard once you have credentials handy.

Step 3 - Run Your First Benchmark

Run a minimal TPC-H benchmark to generate data, load it into DuckDB, and execute the standard power test.

Terminal window
uv run -- benchbox run \
  --platform duckdb \
  --benchmark tpch \
  --scale 0.01

Unattended execution? Add --non-interactive to skip prompts. BenchBox stores outputs under benchmark_runs/ by default.

Step 4 - Inspect the Results

Summarize the most recent run:

Terminal window
uv run -- benchbox results --limit 1

The results display shows timing, validation status, and per-query metrics from your benchmark execution.

Step 5 - Export Results (Optional)

Share your results in different formats without re-running the benchmark:

Terminal window
uv run -- benchbox export --last --format csv


uv run -- benchbox export --last --format html --output-dir ./reports/


uv run -- benchbox export --last --format json --format csv --format html

The commands export to CSV for spreadsheet analysis, generate an HTML report for sharing with your team, and export to all formats.

The export command is useful for:

  • Creating shareable HTML reports for stakeholders
  • Analyzing query performance in spreadsheets (Excel, Google Sheets)
  • Archiving results in multiple formats
  • Converting between formats without re-running expensive benchmarks

Optional - DataFrame Platforms

BenchBox also supports benchmarking DataFrame libraries using their native APIs. This enables direct comparison between SQL and DataFrame execution paradigms.

Quick Start with DataFrames

Terminal window
uv add benchbox --extra polars
uv run -- benchbox run --platform polars-df --benchmark tpch --scale 0.01


uv add benchbox --extra pandas
uv run -- benchbox run --platform pandas-df --benchmark tpch --scale 0.01

Polars and pandas each require an extra (polars and pandas); neither ships in the base install.

Compare SQL vs DataFrame

Terminal window
uv run -- benchbox run --platform duckdb --benchmark tpch --scale 0.1
uv run -- benchbox run --platform polars-df --benchmark tpch --scale 0.1

These commands run the same benchmark with different paradigms: DuckDB runs SQL, and Polars runs DataFrame operations.

For more details, see the DataFrame Platforms Guide.

Optional - Python API in 15 Lines

import duckdb
from benchbox import TPCH


conn = duckdb.connect(":memory:")
benchmark = TPCH(scale_factor=0.01, output_dir="./tpch_data")


benchmark.generate_data()
conn.execute(benchmark.get_create_tables_sql())


for table, path in benchmark.tables.items():
    conn.execute(
        f"COPY {table} FROM '{path}' (DELIMITER '|' NULL '' HEADER FALSE);"
    )


rows = conn.execute(benchmark.get_query(1)).fetchall()
print(f"Query 1 returned {len(rows)} rows")

Next Steps

Essential Guides

Understanding BenchBox

Expanding Your Testing

Troubleshooting

BenchBox is open source under the MIT License.