Development Roadmap
This document describes planned platform and benchmark additions for BenchBox. Items are organized by priority and implementation phase.
Note: This roadmap reflects current planning. Timelines are not committed, and scope may evolve with community feedback and sponsorship opportunities.
Overview
BenchBox’s expansion roadmap focuses on three strategic areas:
- Platform Coverage: Adding high-performance and cloud-native analytical platforms
- Benchmark Diversity: Real-world datasets and specialized workloads
- DataFrame Support: Extending programmatic DataFrame benchmarking to all benchmarks
Current Statistics
| Category | Current | Planned |
|---|---|---|
| SQL Platforms | 39 | +0 |
| DataFrame Platforms | 9 | +0 |
| Benchmarks | 23 | +4 |
| DataFrame-enabled Benchmarks | 21 | +1 (Geospatial) |
Platform Additions
All previously planned platform additions through Phase 4 have shipped. The platform expansion roadmap is complete pending new sponsorship or community requests.
Completed Platforms
| Platform | Shipped | Modes |
|---|---|---|
| LakeSail Sail | v0.2.1 | SQL + DataFrame |
| Apache Doris | v0.2.1 | SQL |
| StarRocks | v0.2.1 | SQL |
| ClickHouse Cloud | v0.2.1 | SQL |
| Firebolt Cloud | v0.2.1 | SQL |
| QuestDB | v0.2.1 | SQL |
| Databend | v0.2.1 | SQL |
| Microsoft Fabric Spark | v0.2.x | SQL + DataFrame |
| Starburst | v0.2.x | SQL |
| TimescaleDB | v0.2.x | SQL |
Benchmark Additions
Completed Benchmarks
| Benchmark | Shipped | Notes |
|---|---|---|
| Flight Data (US aviation on-time) | v0.3.0 | 20 queries, SQL + DataFrame |
| NYC Taxi Expansion | v0.3.0 | Green, FHV, HVFHV vehicle types |
Planned Benchmarks
Geospatial Primitives
Status: Not Started | Effort: Medium | Priority: Medium
Dedicated benchmark for spatial SQL operations (ST_* functions) across platforms. NYC Taxi already includes some spatial queries; this would be a standalone primitives suite covering the full spatial SQL surface.
Scope:
- Point-in-polygon, distance, bounding box
- Spatial indexing and predicate pushdown
- Cross-platform dialect coverage (PostGIS, DuckDB spatial, ClickHouse, BigQuery)
GitHub Archive
Status: Not Started | Effort: Medium | Priority: Low
Developer activity analytics over the public GitHub event archive.
Stack Overflow Dataset
Status: Not Started | Effort: Medium | Priority: Low
Q&A and engagement analytics over the Stack Exchange data dump.
Wikipedia Pageviews
Status: Not Started | Effort: Medium | Priority: Low
Web traffic time-series patterns from Wikimedia pageview dumps.
DataFrame Support Initiative
Current State
DataFrame benchmarking is supported for 21 benchmarks across two API families:
Expression Family (Polars, PySpark, DataFusion, LakeSail):
- TPC-H, TPC-DS, TPC-DS OBT, TPC-DI, TPC-H Skew, TPC-Havoc, DataVault
- SSB, ClickBench, AMPLab, CoffeeShop, H2ODB, TSBS DevOps
- NYC Taxi, Flight Data, JoinOrder, JoinOrder Synthetic
- Read Primitives, Write Primitives, AI Primitives, Metadata Primitives
Pandas Family (Pandas, cuDF, Dask):
- Full coverage matching the Expression Family above
Transaction Primitives — restricted to ACID-capable table-format adapters only:
Delta Lake (delta-lake, delta), PySpark with Delta Lake (run via --platform pyspark-df using the Delta Lake table format), and Iceberg (iceberg). Polars,
DataFusion, and all Pandas-family adapters (pandas-df, cuDF, Dask) are
rejected at runtime because they do not support ACID transactions.
Remaining Expansion
The DataFrame initiative is substantially complete. The one remaining gap is:
| Benchmark | Status | Notes |
|---|---|---|
| Geospatial Primitives | Not Started | Blocked on the benchmark itself existing |
Implementation Timeline
Phase 1: Foundation (Completed)
MotherDuck adapter✓Platform deployment infrastructure✓Core infrastructure improvements✓Onehouse Quanton adapter✓ (multi-format: Hudi, Iceberg, Delta)Apache Hudi maintenance operations✓
Phase 2: High-Impact Platforms (Completed)
LakeSail Sail (SQL + DataFrame)✓Apache Doris✓StarRocks✓Cloud deployment modes (ClickHouse Cloud, Firebolt Cloud)✓
Phase 3: Benchmark Diversity (Completed)
Flight Data Benchmark✓NYC Taxi Expansion✓DataFrame Tier 1 (SSB, ClickBench, NYC Taxi)✓
Phase 4: Extended Coverage (Completed)
Microsoft Fabric Spark✓Starburst✓TimescaleDB✓QuestDB✓DataFrame Tiers 2–4✓
Phase 5: Specialized Workloads
- Geospatial Primitives benchmark
- Real-world datasets (GitHub Archive, Stack Overflow, Wikipedia)
How to Contribute
Requesting Platforms or Benchmarks
Open an issue with:
- Use case description
- Scale factors needed
- Platform/benchmark specifics
- Compliance requirements (if applicable)
Sponsoring Development
Enterprise users can sponsor specific platform or benchmark development. Contact the maintainers for details.
Contributing Implementations
See the Platform Development Guide and Adding New Platforms for implementation patterns.
Related Documentation
- Adding New Platform Adapters - Platform implementation guide
- Platform Development - Platform adapter architecture
Last updated: 2026-06-05