Cloud Sorted Ingestion

BenchBox supports an opt-in sorted-ingestion strategy for cloud data warehouse runs so you can compare:

• Deep sorted ingestion
• Platform-native tuning (for example clustering or Z-ORDER)
• Baseline runs with no sorted-ingestion rewrite

Strategy Controls

Use unified tuning platform_optimizations:

platform_optimizations:
  sorted_ingestion_mode: force
  sorted_ingestion_method: ctas

sorted_ingestion_mode:

• off: disable sorted ingestion
• auto: enable only when the platform capability matrix supports a method
• force: require sorted ingestion, fail if unsupported

sorted_ingestion_method:

• auto: choose platform default
• ctas: use CTAS rewrite flow where available
• z_order: Databricks Z-ORDER path
• liquid_clustering: Databricks liquid clustering path
• vacuum_sort: Redshift vacuum sort path

CLI Overrides

You can override strategy at runtime:

benchbox run \
  --platform snowflake \
  --benchmark tpch \
  --scale 1 \
  --tuning ./unified_tuning.yaml \
  --sorted-ingestion-mode force \
  --sorted-ingestion-method ctas

Comparison Workflow

1. Run baseline:

benchbox run --platform snowflake --benchmark tpch --scale 1 --tuning notuning

2. Run platform-native tuning:

benchbox run --platform snowflake --benchmark tpch --scale 1 --tuning ./snowflake-native.yaml

3. Run deep-sorted ingestion:

benchbox run --platform snowflake --benchmark tpch --scale 1 --tuning ./snowflake-sorted.yaml

4. Compare result files:

benchbox compare <baseline.json> <native.json> <sorted.json>

Platform Notes

• Snowflake: supports ctas sorted-ingestion path when mode is enabled.
• Databricks: supports ctas, z_order, and liquid_clustering strategy methods. TPC templates now distinguish legacy databricks_z_order from databricks_liquid_auto; do not combine Liquid Clustering with ZORDER or per-table partitioning fields.
• Redshift: supports ctas and vacuum_sort.
• Amazon Athena and Azure Synapse Analytics: support ctas method.
• BigQuery: forcing sorted ingestion currently returns an actionable error for CTAS-hook execution path.
• ClickHouse Cloud: post-load sorted ingestion is unsupported; use table ORDER BY at create time.

Result Metadata

BenchBox writes sorted-ingestion metadata into execution_metadata.sorted_ingestion:

• Configured mode/method
• Resolved mode/method
• Platform capability details
• Applied table list/count
• Total sorted-ingestion apply time