Welcome to BenchBox’s documentation!

BenchBox is a comprehensive database benchmarking toolkit supporting TPC-H, TPC-DS, TPC-DI, and other industry-standard benchmarks across SQL and DataFrame platforms.

Getting Started

• BenchBox Documentation
• Installation & Environment Setup
• Getting Started in 5 Minutes
• BenchBox CLI - Quick Reference
• MCP Integration Guide
• Tutorials

User Guide

• Configuration Handbook
  • Minimal Example
  • File Locations & Schema
  • Execution Profiles
  • Environment Overrides
  • Validating Configuration
  • Related Commands
• Data Generation
  • Overview
  • Basic Data Generation
  • Smart Data Generation
  • Benchmark-Specific Generation
  • Data Validation
  • Performance Optimization
  • Configuration Options
  • Advanced-level Usage
  • Error Handling
  • Best Practices
  • Supported Benchmarks
  • Future Enhancements
  • Troubleshooting
• Dry Run Mode
  • Overview
  • Quick Start
  • Output Structure
  • Console Output
  • Features
  • Integration Patterns
  • Troubleshooting
  • Best Practices
  • API Reference
• Examples Directory Guide
  • Quick Navigation
  • Directory Structure
  • Which Example Should I Use?
  • Detailed Guides
  • Example Categories Explained
  • Running the Examples
  • Additional Resources
  • See Also
• Understanding Benchmark Results
  • Result Overview
  • Viewing Results
  • Key Metrics
  • Understanding Validation
  • TPC Metrics
  • Comparing Results
  • Result File Locations
  • Interpreting Slow Queries
  • Next Steps
• Troubleshooting Guide
  • Quick Diagnosis
  • Installation Issues
  • Connection Issues
  • Authentication Failures
  • Catalog Not Found (Presto/Trino)
  • Permission Errors
  • Query Timeouts
  • Memory Issues
  • Data Generation Issues
  • Data Loading Issues
  • Platform-Specific Issues
  • General Tips
  • Getting Help
  • Related Documentation
• Frequently Asked Questions (FAQ)
  • General Questions
  • Installation & Setup
  • Running Benchmarks
  • Platform-Specific Questions
  • Data & Query Questions
  • Troubleshooting & Errors
  • Advanced Features
  • Contributing & Support
  • See Also
• Concepts
  • Core Concepts
  • Comparisons
  • Understanding BenchBox
  • Related Documentation

Benchmarks

• TPC Standards
  • About the Transaction Processing Performance Council
  • Why TPC Benchmarks Matter
  • TPC Benchmarks in BenchBox
  • Included Benchmarks
  • See Also
• Academic Benchmarks
  • The Role of Academic Research
  • Why Academic Benchmarks Matter
  • Academic Benchmarks in BenchBox
  • Included Benchmarks
  • See Also
• Industry Benchmarks
  • The Rise of Vendor Benchmarks
  • Why Industry Benchmarks Matter
  • Interpreting Vendor Benchmarks
  • Industry Benchmarks in BenchBox
  • Included Benchmarks
  • See Also
• BenchBox Primitives
  • Why Primitives?
  • Design Philosophy
  • Use Cases for Primitives
  • Primitive Categories
  • Included Benchmarks
  • See Also
• BenchBox Experimental
  • What Makes a Benchmark “Experimental”?
  • Why Include Experimental Benchmarks?
  • Using Experimental Benchmarks
  • Experimental Benchmarks in BenchBox
  • Included Benchmarks
  • See Also

Platforms

• Platform Selection Guide
  • Quick Start: Choose Your Path
  • Platform Extras and Installation Matrix
  • Presto Lineage: Trino vs PrestoDB vs Athena
  • SQL vs DataFrame: When to Choose Each
  • Detailed Selection Criteria
  • Selection Criteria Summary
  • Platform Combination Strategies
  • Migration Strategies
  • Common Anti-Patterns
  • Getting Started
  • Expert Tips
  • Summary
  • See Also
• Multi-Platform Database Support
  • Supported Platforms
  • Quick Start
  • DuckDB
  • Apache DataFusion
  • ClickHouse
  • Databricks
  • BigQuery
  • Redshift
  • Snowflake
  • SQLite
  • Troubleshooting
  • See Also
• Platform Comparison Matrix
  • Platform Overview
  • Quick Decision Guide
  • Deployment Modes
  • Local/Embedded Platforms
  • Cloud Data Warehouses
  • Distributed SQL Engines
  • Relational & Time-Series Databases
  • Managed Spark Services
  • DataFrame Platforms
  • Hybrid Platforms (SQL + DataFrame)
  • Benchmark Support Matrix
  • Scale Factor Recommendations
  • Cost Comparison (All Categories)
  • Use Case Recommendations
  • Migration Complexity
  • Summary by Category
  • Resources
• SQL Platforms
  • Local/Embedded Engines
  • Distributed SQL Engines
• DataFrame Platforms
  • DataFrame Platforms
  • Polars Platform
  • Pandas DataFrame Platform
  • Modin DataFrame Platform
  • Dask DataFrame Platform
  • cuDF DataFrame Platform
  • PySpark Platform (SQL & DataFrame)
  • DataFusion DataFrame Platform
• Cloud Platforms
  • AWS Athena Platform
  • Firebolt Platform
  • Microsoft Fabric
  • Azure Analytics Platforms

Guides

• TPC Benchmark Guides
  • Official Benchmark Guides
  • Cross-Cutting Guides
  • Additional Resources
  • Related Documentation
  • TPC Resources
• Table Format Guides
  • Overview
  • Why Format Choice Matters for Benchmarks
  • File Formats vs Table Formats
  • Format Comparison
  • When to Consider Format Impact
  • Choosing the Right Format
  • Guide Index
  • Related Documentation
• Platform Comparison Guide
  • Overview
  • Quick Start
  • Command Reference
  • Usage Examples
  • Programmatic API
  • Platform Categories
  • Best Practices
  • Interpreting Results
  • Troubleshooting
  • Query Plan Comparison
  • Migration from compare-dataframes
  • Migration from compare-plans
  • Related Documentation
• Cloud Storage Support
  • Overview
  • Installation
  • Authentication Setup
  • Basic Usage
  • Credential Validation
  • Data Flow
  • Platform Integration
  • Best Practices
  • Troubleshooting
  • Examples
  • Limitations
  • Manifest-Based Upload Validation
• Data Compression in BenchBox
  • Overview
  • Quick Start
  • Compression Types
  • Performance Results
  • Compression Levels
  • Integration with Data Generators
  • Platform Adapter Support
  • Advanced Usage
  • Best Practices
  • Troubleshooting
  • API Reference
  • Examples Repository
• Advanced Topics
  • Quick Links
  • Recommended Workflow
  • Additional References
• BenchBox Visualization Architecture
  • Goals and Constraints
  • Library Selection
  • Module Architecture
  • Chart Taxonomy
  • Style Guide (Publication-Ready Defaults)
  • CLI Design (benchbox visualize)
  • Export and Performance
  • Dependency Footprint
  • Related Documentation

Reference

• BenchBox CLI Reference
  • Global Options
  • Commands Overview
  • Quick Start
  • Documentation Sections
• Python API Reference
  • Overview
  • Quick Start
  • API Organization
  • Common Patterns
  • Type Hints
  • Configuration Classes
  • See Also
• MCP Server Reference
  • Running the Server
  • Tools
  • Prompts
  • Resources
  • Error Handling
  • Related Documentation
• Result Export Formats
  • Export Commands
  • JSON Format (Schema v1.1)
  • CSV Format
  • HTML Format
  • Loading Results in Python
  • Visualization Examples
  • Schema Versioning
  • Anonymization
  • Integration Examples
  • Related Documentation
• Benchmark Result Schema (v1.1)
  • Top-Level Structure
  • Results Payload
  • Export Block
  • Validation
  • Compatibility Notes
• API Reference

Development

• Getting Started with Development
  • Development Guide
  • BenchBox Testing Guide
  • Live Integration Tests
• Architecture & Design
  • Design & Architecture
  • BenchBox Architecture Design Document
  • BenchBox Repository Structure
  • Import Patterns and Lazy Loading
• Platform Development
  • Adding New Platform Adapters
  • Adding a New DataFrame Platform
  • Runtime Module Architecture Overview
  • DB API 2.0: Foundation of BenchBox Platform Support
• BenchBox Architecture Design Document
  • 1. High-Level Architecture
  • 2. Core Interfaces and Abstract Classes
  • 3. Module Structure and Dependencies
  • 4. Design Patterns and Extensibility
  • 5. Key Design Decisions
  • 6. Class/Interface Diagram
  • 7. Implementation Considerations
  • 8. Roadmap and Future Extensions
  • 9. Conclusion
• Testing Documentation
  • Test Documentation
  • Test Categories
  • Related Documentation

Browse by Tag

• By Audience
  • beginner (19)
  • intermediate (77)
  • advanced (28)
  • contributor (25)
• By Benchmark
  • tpc-h (13)
  • tpc-ds (10)
  • tpc-di (5)
  • tpc-havoc (1)
  • tpch-skew (1)
  • ssb (2)
  • clickbench (2)
  • h2odb (2)
  • join-order (1)
  • amplab (1)
  • nyctaxi (1)
  • coffeeshop (1)
  • datavault (1)
  • tsbs-devops (1)
  • read-primitives (1)
  • write-primitives (1)
  • transaction-primitives (1)
  • metadata-primitives (1)
  • ai-primitives (1)
  • custom-benchmark (20)
• By Platform
  • duckdb (4)
  • sqlite (2)
  • postgresql (1)
  • datafusion (1)
  • snowflake (3)
  • databricks (3)
  • bigquery (2)
  • redshift (1)
  • motherduck (1)
  • starburst (1)
  • clickhouse (3)
  • trino (1)
  • presto (1)
  • firebolt (1)
  • timescaledb (1)
  • influxdb (1)
  • athena (1)
  • aws-glue (1)
  • emr-serverless (1)
  • athena-spark (1)
  • dataproc (1)
  • dataproc-serverless (1)
  • azure (1)
  • fabric (1)
  • fabric-spark (1)
  • synapse-spark (1)
  • spark (2)
  • pyspark (1)
  • pandas (1)
  • polars (2)
  • dask (1)
  • modin (1)
  • cudf (1)
  • datafusion-df (1)
• By Platform Type
  • sql-platform (19)
  • dataframe-platform (16)
  • cloud-platform (20)
  • embedded-platform (3)
  • cloud-storage (3)
• By Content Type
  • guide (93)
  • tutorial (5)
  • reference (73)
  • concept (30)
  • quickstart (4)
• By Feature
  • architecture (1)
  • cli (16)
  • cloud (3)
  • e2e (1)
  • python-api (37)
  • configuration (3)
  • data-generation (3)
  • performance (11)
  • tuning (3)
  • validation (12)
  • testing (5)
  • visualization (6)