BenchBox
Home Docs Blog Results Instruct an agent GitHub Run benchmark
Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
BenchBox 0.3.1 documentation
BenchBox 0.3.1 documentation

Getting Started

  • BenchBox Documentation
  • Installation & Environment Setup
  • Getting Started in 5 Minutes
  • BenchBox CLI - Quick Reference
  • MCP Integration Guide
  • Tutorials
    • Your First Benchmark
    • Understanding Benchmark Results
    • Comparing Platforms
    • DataFrame Benchmarking Quickstart

User Guide

  • Configuration Handbook
  • Data Generation
  • Dry Run Mode
  • Examples Directory Guide
  • Understanding Benchmark Results
  • Troubleshooting Guide
  • Frequently Asked Questions (FAQ)
  • Concepts
    • BenchBox Architecture
    • Benchmarking Workflows
    • Data Model
    • Glossary
    • Database Benchmarking Tools Compared

Benchmarks

  • TPC Standards
    • TPC-H Benchmark
    • TPC-DS Benchmark
    • TPC-DI Benchmark
  • Academic Benchmarks
    • Star Schema Benchmark (SSB)
    • AMPLab Big Data Benchmark
    • Join Order Benchmark
  • Industry Benchmarks
    • ClickBench (ClickHouse Analytics Benchmark)
    • H2O DB Benchmark
    • CoffeeShop Benchmark
  • Real-World Data Benchmarks
    • NYC Taxi OLAP Benchmark
    • Flight Data Benchmark
  • Time-Series Benchmarks
    • TSBS DevOps Benchmark
  • BenchBox Primitives
    • Read Primitives Benchmark
    • Write Primitives Benchmark
    • Transaction Primitives Benchmark
    • Metadata Primitives Benchmark
    • AI/ML Primitives Benchmark
  • AI & ML Benchmarks
    • Vector Search Benchmark
  • BenchBox Experimental
    • TPC-Havoc Benchmark
    • TPC-H Skew Benchmark
    • TPC-DS-OBT Benchmark
    • TPC-H Data Vault Benchmark

Platforms

  • Platform Selection Guide
  • Multi-Platform Database Support
  • Platform Comparison Matrix
  • SQL Platforms
    • DuckDB Platform
    • ClickHouse Local Mode
    • ClickHouse Server Platform
    • Apache DataFusion Platform Guide
    • CedarDB Platform
    • PostgreSQL Platform
    • SQLite Platform
    • pg_duckdb Platform
    • pg_mooncake Platform
    • Trino Platform
    • PrestoDB Platform Guide
    • Apache Spark Platform
    • Apache Doris Platform
    • StarRocks Platform
    • LakeSail Sail Platform
    • Apache Gluten + Velox Platform
    • SingleStore Platform
    • TimescaleDB Platform
    • InfluxDB
    • QuestDB Platform
  • DataFrame Platforms
    • DataFrame Platforms
    • Polars Platform
    • Pandas DataFrame Platform
    • PySpark Platform (SQL & DataFrame)
    • DataFusion DataFrame Platform
    • Modin DataFrame Platform
    • Dask DataFrame Platform
    • cuDF DataFrame Platform
    • Databricks DataFrame Platform
  • Cloud Platforms
    • Snowflake Platform
    • Databricks Platform
    • BigQuery Platform
    • ClickHouse Cloud Platform
    • Redshift Platform
    • Amazon Athena Platform
    • Azure Analytics Platforms
    • Microsoft Fabric
    • Microsoft Fabric Lakehouse SQL
    • Firebolt Platform
    • Databend Platform
    • MotherDuck Platform
    • Starburst Platform
    • Onehouse Quanton Platform
    • AWS Glue Platform
    • Amazon EMR Serverless Platform
    • Amazon Athena for Apache Spark Platform
    • Google Cloud Dataproc Platform
    • Google Cloud Dataproc Serverless Platform
    • Microsoft Fabric Spark Platform
    • Azure Synapse Analytics Spark Platform
    • Snowpark Connect for Spark

Guides

  • TPC Benchmark Guides
    • TPC-H Official Benchmark Guide
    • TPC-DS Official Benchmark Guide
    • TPC-DI Deployment Guide
    • TPC-DI ETL Implementation Guide
    • TPC Maintenance Phase: Complete Guide
    • TPC Patterns Usage Guide
    • TPC Test Result Validation System
  • Table Format Guides
    • Parquet Deep Dive
    • Delta Lake Guide
    • Apache Iceberg Guide
    • Vortex Guide
  • Platform Comparison Guide
  • Cloud Storage Support
  • Data Compression in BenchBox
  • Advanced Topics
    • Power Run Iterations and Concurrent Query Execution
    • Optimizer Sniff Test Queries
    • Performance Monitoring
    • Advanced Performance Optimization Guide
    • Performance & Tuning
      • Performance Monitoring
      • Advanced Performance Optimization Guide
      • Power Run Iterations and Concurrent Query Execution
    • Open Table Formats Guide
    • Custom Benchmarks
    • CI/CD Integration Guide
    • Customization & Extensions
      • Custom Benchmarks
      • Optimizer Sniff Test Queries
  • BenchBox Visualization Architecture
    • Chart Generation Guide
    • Chart Types
    • Customization
    • Templates
    • CLI Reference: benchbox visualize

Contributing

  • Contributing Benchmark Results
  • Cutting a BenchBox release
  • Worktree pool — maintainer dev loop
  • Phase 2 Results Operations Runbook

Reference

  • BenchBox CLI Reference
    • run - Run Benchmarks
    • convert - Convert Data Formats
    • shell - Interactive SQL Shell
    • platforms - Platform Management
    • Utility Commands
    • Tuning Commands
    • Results Commands
    • submit - Submit Results
    • auth - Hosted Submission Credentials
    • visualize - Generate Charts
    • report - Historical Analysis
    • metrics - Performance Metrics
    • aggregate - Aggregate Results
    • datagen - Data Generation
    • setup - Cloud Credentials
    • show-plan / plan-history - Query Plans
    • download-answers - Answer Files
    • Configuration
    • Common Workflows
    • Troubleshooting
  • Python API Reference
    • Base Benchmark API
    • Benchmark API Reference
      • Benchmark APIs
      • TPC-H Benchmark API
      • TPC-DS Benchmark API
      • TPC-DI Benchmark API
      • SSB (Star Schema Benchmark) API
      • ClickBench Benchmark API
      • Join Order Benchmark API
      • AMPLab Big Data Benchmark API
      • H2O.ai Database Benchmark API
      • Read Primitives Benchmark API
      • Write Primitives Benchmark API
    • Results API
    • Result Analysis API
    • DuckDB Platform Adapter
    • Apache DataFusion Platform Adapter
    • SQLite Platform Adapter
    • ClickHouse Platform Adapter
    • Databricks Platform Adapter
    • BigQuery Platform Adapter
    • Snowflake Platform Adapter
    • Amazon Redshift Platform Adapter
    • Cloud Storage Integration API
    • Data Validation Utilities API
    • Utilities & Helpers
      • Utility Functions API
      • Tuning Configuration API
      • Cloud Storage Integration API
      • Result Analysis API
      • Data Validation Utilities API
      • Performance Monitoring Utilities API
      • Additional Utilities API
    • Additional Utilities API
    • Performance Monitoring Utilities API
    • Tuning Configuration API
  • MCP Server Reference
  • Result Export Formats
  • Benchmark Result Schema (v1.1)
  • API Reference

Development

  • Getting Started with Development
    • Development Guide
    • BenchBox Testing Guide
    • Live Integration Tests
  • Architecture & Design
    • Design & Architecture
      • BenchBox Architecture
      • BenchBox Repository Structure
      • Future State Proposals
        • Architecture Contract Decision Index
        • Benchmark Family Plugin Seam Future State
        • Prune publishing subsystem — COMPLETED (v0.2.1)
        • benchbox-maintainer Future State
        • benchbox-experimental Future State
        • Monitoring Optional Extra Future State
        • MCP API Formalization Future State
    • BenchBox Architecture
    • BenchBox Repository Structure
    • Import Patterns and Lazy Loading
  • Platform Development
    • Adding New Platform Adapters
    • Adding a New DataFrame Platform
    • Runtime Module Architecture Overview
    • DB API 2.0: Foundation of BenchBox Platform Support
  • BenchBox Architecture
  • Testing Documentation
    • End-to-End (E2E) Testing Guide
    • Docker Integration Tests
    • Live Integration Tests

Browse by Tag

  • By Audience
    • beginner (19)
    • intermediate (92)
    • advanced (30)
    • contributor (43)
  • By Benchmark
    • tpc-h (13)
    • tpc-ds (10)
    • tpc-di (5)
    • tpc-havoc (1)
    • tpch-skew (1)
    • ssb (2)
    • clickbench (2)
    • h2odb (2)
    • join-order (1)
    • amplab (1)
    • nyctaxi (1)
    • coffeeshop (1)
    • datavault (1)
    • tsbs-devops (1)
    • read-primitives (1)
    • write-primitives (1)
    • transaction-primitives (1)
    • metadata-primitives (1)
    • ai-primitives (1)
    • custom-benchmark (21)
  • By Platform
    • duckdb (4)
    • sqlite (2)
    • postgresql (2)
    • datafusion (1)
    • snowflake (3)
    • databricks (3)
    • bigquery (2)
    • redshift (1)
    • motherduck (1)
    • starburst (1)
    • clickhouse (3)
    • trino (1)
    • presto (1)
    • firebolt (1)
    • timescaledb (2)
    • influxdb (1)
    • athena (1)
    • aws-glue (1)
    • emr-serverless (1)
    • athena-spark (1)
    • dataproc (1)
    • dataproc-serverless (1)
    • azure (1)
    • fabric (1)
    • fabric-spark (1)
    • synapse-spark (1)
    • spark (2)
    • pyspark (1)
    • pandas (1)
    • polars (2)
    • dask (1)
    • modin (1)
    • cudf (1)
    • datafusion-df (1)
  • By Platform Type
    • sql-platform (29)
    • dataframe-platform (18)
    • cloud-platform (22)
    • embedded-platform (2)
    • cloud-storage (3)
  • By Content Type
    • guide (112)
    • tutorial (5)
    • reference (90)
    • concept (32)
    • quickstart (4)
  • By Feature
    • architecture (12)
    • cli (26)
    • cloud (6)
    • e2e (1)
    • python-api (37)
    • configuration (3)
    • data-generation (3)
    • performance (11)
    • tuning (3)
    • validation (14)
    • testing (7)
    • visualization (7)
Back to top
Edit this page
View this page

Tagged with: testing¶

Pages with this tag

  • CI/CD Integration Guide

  • Pytest xdist Safety

  • BenchBox Testing Guide

  • Docker Integration Tests

  • End-to-End (E2E) Testing Guide

  • Testing Documentation

  • Live Integration Tests

Next
Tagged with: visualization
Previous
Tagged with: validation
Copyright © 2025, Joe Harris
Made with Sphinx and @pradyunsg's Furo