Everyone says you should. BenchBox makes it easy to do. Use as CLI, Python library, or MCP for AI assistants. No Docker. No compilers. Just pip install.
# CLI - Quick benchmarking
benchbox run --platform duckdb --benchmark tpch
# MCP - AI assistant integration
"Run TPC-H on DuckDB" # Claude executes via MCP
# Library - Deep Python integration
from benchbox import TPCHWhy BenchBox
Minimal requirements. A Python-only core with optional extras for cloud platforms. Install and go.
TPC standards, academic, industry, real-world, time-series, SQL primitives, AI/ML, and experimental workloads.
SQL databases (DuckDB, ClickHouse, cloud platforms) and DataFrame libraries (Polars, Pandas).
Automatic SQL translation. The same benchmark queries run on local and cloud databases.
From 10 MB to 100 TB. No need to download pre-generated files.
An MCP server for Claude Code and other AI assistants. Run benchmarks with natural language.
From measurements to answers
The Results Explorer turns BenchBox output into a browsable view of performance. Use the public corpus to understand how platforms behave on the same workload, or open your own result file in the browser before you share it.
Open the Results ExplorerTPC-H · Scale factor 10
| Version | Geomean | Power@Size |
|---|---|---|
| DuckDB v2.0.0… | 128 ms | 281,041 |
| DuckDB v1.5.5 | 152 ms | 236,191 |
| DuckDB v1.4.4 | 168 ms | 213,021 |
| DuckDB v1.3.2 | 189 ms | 198,175 |
Browse by benchmark or platform, then narrow the public corpus to results that match the workload and scale you care about.
Compare compatible runs, inspect query-level timings, and see where one platform gains or loses time instead of relying on a single headline number.
Open a BenchBox result JSON file locally in the browser. Review its charts and run receipt without uploading it or adding it to the public rankings.
Workloads
Official industry standards for comparing databases
Standard data warehouse benchmark with 22 business intelligence queries.
TPC.org Standard ↗Complex decision support benchmark modeling multi-dimensional analysis.
TPC.org Standard ↗Data integration benchmark testing ETL operations and data quality.
TPC.org Standard ↗Research benchmarks from academia
Simplified data warehouse testing basic OLAP functionality.
Original Paper ↗Big data benchmark focusing on machine learning and analytics workloads.
Original Site ↗Query optimizer testing with complex multi-table joins from IMDB data.
Original Paper ↗Real-world benchmarks from practitioners
Analytics benchmark with real-world web analytics queries.
Official Site ↗Data science benchmark testing aggregation and join performance.
Official Site ↗Small analytical benchmark modeling a coffee shop with sales and inventory.
Original Post ↗Benchmarks built on public real-world datasets
Real-world transportation analytics with geospatial and temporal queries.
NYC TLC Data ↗US domestic flight analytics with on-time performance and carrier metrics.
US BTS TranStats ↗Workloads for time-series databases and columnar monitoring engines
Time-series benchmark for infrastructure monitoring and observability.
Original Repo ↗Fundamental database operation testing
Read operation testing across 26 categories.
Comprehensive write operation testing including bulk loads and merges.
ACID transaction testing with isolation levels, commits, and rollbacks.
Catalog introspection via INFORMATION_SCHEMA, SHOW, DESCRIBE, and PRAGMA.
SQL-based AI functions (Snowflake Cortex, BigQuery ML, Databricks AI) with cost controls.
Vector similarity and other AI-shaped analytical workloads
Vector similarity search with kNN, ANN, and filtered queries on embedding data.
Experimental benchmarks for specialized testing
Query optimizer stress testing with 220 syntax variants of TPC-H queries.
Test optimizer behavior on non-uniform data distributions (Zipfian, exponential).
TPC-DS queries on a single denormalized "One Big Table" schema.
TPC-H adapted for Data Vault 2.0 modeling (Hub-Link-Satellite pattern).
Engines
In-process and local columnar databases
In-process analytical database. Built-in, no setup required.
Embedded ClickHouse via chDB for in-process columnar analytics.
Self-hosted ClickHouse for high-performance columnar analytics.
Apache Arrow-native query engine for in-memory analytics.
High-performance analytical database from TUM (formerly Umbra).
Traditional relational databases
The world's most advanced open source relational database.
Embedded transactional database for testing and CI/CD.
DuckDB-powered Postgres extension for vectorized OLAP on Postgres tables.
Columnar storage extension for Postgres with Iceberg and Delta Lake support.
Enterprise cloud data warehouses and lakehouses
Multi-cloud data warehouse with elastic compute.
Data Intelligence Platform with lakehouse architecture.
Google Cloud serverless data warehouse.
Managed ClickHouse with serverless and dedicated options.
AWS cloud data warehouse (serverless or provisioned).
AWS serverless query service for S3 data lakes.
Microsoft Azure analytics service.
Fabric SQL warehouse with OneLake-native storage.
SQL endpoint over Fabric lakehouses on OneLake.
Cloud data warehouse optimized for analytics.
Open source cloud-native data warehouse written in Rust.
Serverless cloud DuckDB with hybrid local/cloud execution.
Enterprise Trino with managed cloud offering.
Open source MPP and federated SQL engines
Distributed SQL query engine for federated data.
Distributed SQL query engine (Meta's fork).
Distributed SQL engine for large-scale data processing.
MPP analytical database with real-time ingestion and MySQL protocol.
High-performance columnar MPP engine for sub-second analytics.
Rust-based Spark-compatible engine on DataFusion via Spark Connect.
Native C++ Spark acceleration layer powered by Meta's Velox.
Distributed SQL with real-time analytics and MySQL protocol.
Cloud-managed Spark for lakehouse and data lake analytics
Serverless Spark with Hudi, Iceberg, and Delta Lake support.
Serverless data integration service with Spark ETL.
Serverless Spark with automatic scaling.
Interactive Spark notebooks on Amazon Athena.
Managed Spark and Hadoop clusters on Google Cloud.
Serverless Spark with no cluster management.
SaaS Spark with OneLake integration.
Enterprise Spark pools with ADLS Gen2.
PySpark API compatibility on Snowflake.
Optimized for time-stamped data
Time-series database built on PostgreSQL.
Time-series database for metrics and events.
High-performance time-series database with Postgres wire protocol.
Native DataFrame APIs instead of SQL
Fast Rust-based DataFrame library with lazy evaluation.
Reference Python DataFrame implementation.
Apache Spark DataFrame API for distributed computing.
Arrow-native DataFrame with lazy evaluation.
Parallel computing library with DataFrame API.
GPU-accelerated DataFrame library from NVIDIA.
Databricks PySpark DataFrame execution on serverless or classic compute.
Storage
Convert benchmark data to modern columnar formats for optimized storage, ACID transactions, and time travel capabilities.
Columnar storage for fast analytics
Industry-standard columnar format with universal platform support.
High-performance columnar format optimized for analytical workloads.
ACID transactions, time travel, and schema evolution
Open table format with ACID transactions and time travel on Parquet.
Modern table format with hidden partitioning and schema evolution.
Table format optimized for incremental processing and record-level updates.
DuckDB's native table format with ACID transactions and optimal performance.
# Convert TPC-H data to different formats
benchbox convert --input ./tpch_sf1 --format parquet
benchbox convert --input ./tpch_sf1 --format delta
benchbox convert --input ./tpch_sf1 --format iceberg
benchbox convert --input ./tpch_sf1 --format ducklakeAI agents
BenchBox includes an MCP (Model Context Protocol) server for Claude Code and other AI assistants. Run benchmarks with natural language instead of memorizing CLI flags.
Pick a surface, platform, benchmark, and scale. Copy the prompt into your coding agent. Defaults to a safe local DuckDB + TPC-H quickstart.
Open the prompt builderclaude mcp add benchbox -- uv run python -m benchbox.mcpWhat benchmarks are available?
Which platforms support TPC-DS?
Explore 22 benchmarks and 50 platforms without reading documentation.
Run TPC-H on DuckDB at scale 0.1
Compare Polars and Pandas on SSB
Run benchmarks without memorizing CLI syntax or options.
Which queries were slowest?
Compare results from my last two runs
Get AI-powered analysis of performance patterns and regressions.
Install
uv add benchbox# Quick TPC-H benchmark on DuckDB
benchbox run --platform duckdb --benchmark tpch --scale 0.1
# Preview on cloud before spending credits
benchbox run --platform databricks --benchmark tpch --dry-run ./previewfrom benchbox import TPCH
# Initialize and generate benchmark data
tpch = TPCH(scale_factor=0.1)
data_files = tpch.generate_data()
# Get schema and queries
create_sql = tpch.get_create_tables_sql()
query = tpch.get_query(1) # Q1: Pricing Summary ReportDatabase benchmarking made simple.