Skip to content

Configuration

BenchBox supports configuration through multiple sources with the following precedence:

  1. Command-line arguments (highest priority)
  2. Environment variables
  3. Configuration files
  4. Default values (lowest priority)

Configuration File Format

BenchBox uses YAML configuration files. Default location: ~/.benchbox/config.yaml

Example configuration:

output:
compression:
enabled: true
type: zstd
level: 3
formats:
- json
- csv
platforms:
databricks:
enabled: true
warehouse_id: "abc123"
bigquery:
enabled: true
project_id: "my-project"
tuning:
default_mode: notuning
enable_constraints: false

Platform Configuration

Each platform can have specific configuration options:

Databricks:

platforms:
databricks:
warehouse_id: "warehouse-id"
catalog: "main"
schema: "benchbox"

BigQuery:

platforms:
bigquery:
project_id: "my-project"
dataset: "benchbox"
location: "US"

Snowflake:

platforms:
snowflake:
account: "account-name"
compute_resource: "COMPUTE_WH"
database: "BENCHBOX"
schema: "PUBLIC"

Environment Variables

BenchBox recognizes these environment variables:

Core Settings

These variables override the corresponding settings in the configuration file (equivalent to editing ~/.benchbox/config.yaml):

VariableConfig PathTypeDefaultDescription
BENCHBOX_DATABASE_PREFERREDdatabase.preferredstringduckdbPreferred database platform
BENCHBOX_SCALE_FACTORbenchmarks.default_scalefloat0.01Default scale factor
BENCHBOX_VERBOSEexecution.verbosebooleantrueEnable verbose output (true/1/yes/on)
BENCHBOX_MAX_WORKERSexecution.max_workersinteger4Maximum parallel worker threads
BENCHBOX_TUNING_CONFIGtuning.default_config_filestring-Path to tuning configuration file. Used when --tuning tuned resolves with no explicit file (see Tuning Commands)
BENCHBOX_OUTPUT_DIRruntime path resolverstringworktree-anchored benchmark_runs rootRuntime output root override; it is not exposed as load_config().output.directory
BENCHBOX_MEMORY_LIMIT_GBexecution.memory_limit_gbinteger0 (auto)Memory limit in GB; 0 means no limit

Boolean parsing: true, 1, yes, on (case-insensitive) are treated as true.

General Settings

  • BENCHBOX_NON_INTERACTIVE=true: Enable non-interactive mode
  • BENCHBOX_NO_COMPRESSION=true: Disable data compression
  • BENCHBOX_CONFIG_PATH=/path/to/config.yaml: Custom config file location

Advanced Settings

These variables control lower-level behaviors, useful for CI, offline environments, or advanced workflows:

VariableDescription
BENCHBOX_DATA_DIROverride the default local data directory for generated benchmark files
BENCHBOX_CACHE_DIROverride the cache directory for DataFrame benchmark data
BENCHBOX_TUNING_PATHOverride the tuning file search root; BenchBox looks for {platform}/{benchmark}_tuned.yaml under this directory
BENCHBOX_QUERY_VALIDATION_MODETPC-DS query validation mode: exact, loose, range, or skip (default: skip)
BENCHBOX_NO_DOWNLOADSet to 1 or true to disable automatic download of expected-answer files; useful in offline/airgapped environments
BENCHBOX_ANSWERS_URLOverride the base URL for downloading expected-answer archives (default: GitHub releases)
BENCHBOX_ENABLE_EXPERIMENTALSet to 1 or true to show experimental platforms in the platform registry
BENCHBOX_DATA_ORG_MAX_IN_MEMORY_BYTESMaximum bytes held in memory when sorting generated data files (integer; default: 512 MB)

Platform Authentication

benchbox setup --platform <name> saves cloud credentials, and benchbox run reads the saved credentials. The variables below are read by setup when you accept its environment auto-detection prompt; setting them does not authenticate run by itself.

Databricks:

  • DATABRICKS_TOKEN: Authentication token
  • DATABRICKS_HOST: Workspace URL

BigQuery:

  • GOOGLE_APPLICATION_CREDENTIALS: Service account key file path (required by setup)
  • BIGQUERY_PROJECT: Google Cloud project ID (required by setup)
  • BIGQUERY_DATASET, BIGQUERY_LOCATION, BIGQUERY_STORAGE_BUCKET: Optional dataset, location and staging bucket

Snowflake:

  • SNOWFLAKE_ACCOUNT: Account identifier
  • SNOWFLAKE_USERNAME: Username
  • SNOWFLAKE_PASSWORD: Password
  • SNOWFLAKE_WAREHOUSE, SNOWFLAKE_DATABASE: Warehouse and database (required by setup)
  • SNOWFLAKE_SCHEMA, SNOWFLAKE_ROLE: Optional schema and role

Redshift:

  • REDSHIFT_HOST, REDSHIFT_USERNAME, REDSHIFT_PASSWORD, REDSHIFT_DATABASE: Connection details (required by setup)
  • REDSHIFT_PORT, REDSHIFT_SCHEMA, REDSHIFT_S3_BUCKET, REDSHIFT_IAM_ROLE: Optional port, schema and S3 staging settings
  • AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION: Optional AWS credentials and region for S3 staging

ClickHouse:

  • CLICKHOUSE_HOST: Server hostname
  • CLICKHOUSE_USER: Username
  • CLICKHOUSE_PASSWORD: Password

Platform-Specific Options

Each platform supports specific options via --platform-option KEY=VALUE:

Universal Keys (All Platforms)

These keys are available for every platform:

KeyDescription
driver_versionPin the Python driver package to a specific version (e.g. 1.2.0). Useful for reproducing results or testing a specific connector release.
driver_auto_installWhen true, automatically installs the requested driver_version via uv if the package is not already present.

Example: pin DuckDB driver and auto-install it:

Terminal window
benchbox run --platform duckdb --benchmark tpch \
--platform-option driver_version=1.2.0 \
--platform-option driver_auto_install=true

Example: pin Snowflake connector version:

Terminal window
benchbox run --platform snowflake --benchmark tpch \
--platform-option driver_version=3.12.0 \
--platform-option driver_auto_install=true \
--platform-option compute_resource=COMPUTE_WH

The Snowflake account, user, and password come from benchbox setup --platform snowflake; they are not --platform-option keys.

Saved Credentials File

benchbox setup saves platform credentials in ~/.benchbox/credentials.yaml. The file is plain YAML and is not encrypted. It holds secrets such as Snowflake, Redshift, SingleStore and Athena passwords, AWS keys and the Databricks token, and any process you run can read it. Only the hosted-submission token (benchbox auth login) is kept in the operating system keyring.

A value written as ${VAR} is read from the environment when benchbox run loads the file. A later save of the file, such as benchbox setup or benchbox setup --remove, writes the resolved values in place of the ${VAR} references, so a secret read from the environment ends up in the file.

snowflake:
account: ${SNOWFLAKE_ACCOUNT}
user: ${SNOWFLAKE_USERNAME}
password: ${SNOWFLAKE_PASSWORD}
warehouse: ${SNOWFLAKE_WAREHOUSE}

benchbox run reads Snowflake settings only from this file. The SNOWFLAKE_* variables are read only by benchbox setup; ${SNOWFLAKE_*} references in the file are how run picks them up from the environment.

Compute Settings

On platforms that run on a named compute resource, compute_resource names it and compute_size sets its size. BenchBox takes each setting from the first place that has it:

  1. --platform-option on the command line;
  2. the saved credential profile written by benchbox setup;
  3. the option’s default.

A value on the command line overrides the saved profile. Two spellings that disagree in the same place, such as warehouse=A and compute_resource=B on one command line, stop the run before any work and name both values.

On Redshift, compute_resource names the workgroup when the host is a Serverless endpoint (*.redshift-serverless.amazonaws.com) and the cluster when it is a provisioned endpoint (*.redshift.amazonaws.com). With any other host, use workgroup_name or cluster_identifier. On Databricks, compute_resource names the SQL warehouse and sets the HTTP path to /sql/1.0/warehouses/<id>.

The --platform name:mode deployment selector follows the same rule: it overrides a deployment key, such as LakeSail sail_mode, in the saved profile.

Athena Spark Runtime Version

On Athena Spark, the Spark-enabled workgroup sets the engine version. A session cannot choose it, so BenchBox never sends a version when it starts one.

KeyDescription
runtime_versionSpark engine version you expect the workgroup to run (e.g. PySpark engine version 3). Results record it as runtime_version with runtime_version_source: requested. Change the version on the workgroup itself. engine_version is a deprecated alias, removed in 0.6.0.
Terminal window
benchbox run --platform athena-spark --benchmark tpch \
--platform-option compute_resource=my-spark-workgroup \
--platform-option s3_staging_dir=s3://my-bucket/benchbox \
--platform-option "runtime_version=PySpark engine version 3"

ClickHouse Options

Choose the deployment with the platform name, not an option: --platform clickhouse-local (or clickhouse:local) runs embedded chDB, and --platform clickhouse-server (or clickhouse:server) connects to a server. The mode option is a deprecated alias for this choice.

clickhouse-local accepts:

  • data_path=/path/to/dir: Directory for persistent chDB storage
  • optimize_after_load=true: Run OPTIMIZE TABLE ... FINAL after each table loads (tuned runs only)

clickhouse-server accepts:

  • host=localhost: Server hostname
  • port=9000: Native protocol port
  • secure=true: Enable TLS encryption
  • username=default and password=...: Authentication
  • optimize_after_load=true: As above

Examples:

Terminal window
benchbox run --platform clickhouse-local --benchmark tpch \
--platform-option data_path=./chdb-data
benchbox run --platform clickhouse-server --benchmark tpch \
--platform-option secure=true \
--platform-option port=9440

View Platform Details

Use benchbox platforms status to see platform information and capabilities:

Terminal window
benchbox platforms status clickhouse
benchbox platforms status databricks

Configuration Sections Reference

Complete reference for all settings in ~/.benchbox/config.yaml (or ./benchbox.yaml). Unset values fall back to the defaults shown here.

system

System profiling and detection settings.

SettingTypeDefaultDescription
auto_profilebooleantrueAutomatically detect system capabilities on startup
save_profilebooleantruePersist the detected system profile to disk
profile_cache_hoursinteger24Hours to cache the system profile before re-detecting

database

Database connection settings.

SettingTypeDefaultDescription
preferredstringduckdbDefault platform when none is specified on the CLI
connection_timeoutinteger30Connection timeout in seconds
auto_detectbooleantrueAutomatically detect available database platforms

benchmarks

Default benchmark execution parameters.

SettingTypeDefaultDescription
default_scalefloat0.01Default scale factor
timeout_minutesinteger60Maximum benchmark execution time
max_memory_gbinteger8Maximum memory allocation hint in GB
continue_on_errorbooleanfalseContinue running remaining queries when one fails

output

Result output and export settings.

SettingTypeDefaultDescription
formatslist[json, console]Output format list
directorystringruntime-resolvedResults directory; resolved from the runtime output root rather than a legacy config key
timestamp_formatstring%Y%m%d_%H%M%STimestamp format for result filenames
submit_to_servicebooleanfalseLegacy placeholder; use benchbox submit --service for hosted upload
service_urlstringhttps://api.benchbox.dev/v1Default hosted results service URL placeholder
compression.enabledbooleantrueEnable result file compression
compression.typestringzstdCompression algorithm (zstd, gzip)
compression.levelintegernullCompression level; null uses the algorithm default

execution

Performance and execution control settings.

SettingTypeDefaultDescription
parallel_queriesbooleanfalseExecute queries in parallel
max_workersinteger4Maximum worker threads for parallel execution
memory_limit_gbinteger0Memory cap in GB; 0 means no limit (auto)
verbosebooleantrueEnable verbose progress output

execution.power_run - settings for multi-iteration power runs:

SettingTypeDefaultDescription
iterationsinteger3Number of measurement iterations per query
warm_up_iterationsinteger1Warm-up iterations before measurement (not included in stats)
timeout_per_iteration_minutesinteger60Timeout for each iteration
fail_fastbooleanfalseStop immediately when any iteration fails
collect_metricsbooleantrueCollect resource metrics during execution

execution.concurrent_queries - settings for throughput / concurrent-stream runs:

SettingTypeDefaultDescription
enabledbooleanfalseEnable concurrent query streams
max_concurrentinteger2Number of concurrent query streams
query_timeout_secondsinteger300Per-query timeout in concurrent mode
stream_timeout_secondsintegerunsetTimeout in seconds for each throughput stream in benchbox run. Unset keeps the benchmark default (3600 for TPC-H, 7200 for TPC-DS)
cancel_on_timeoutbooleanfalseSignal a timed-out throughput stream to stop before its next query
retry_failed_queriesbooleantrueRetry failed queries in concurrent streams
max_retriesinteger3Maximum retry attempts per query

Only stream_timeout_seconds and cancel_on_timeout are read by benchbox run. The other settings are read only through ExecutionConfigHelper.

tuning

Table tuning and optimization settings.

SettingTypeDefaultDescription
default_config_filestringnullDefault tuning YAML file used when --tuning tuned finds no explicit file; see Tuning Commands for the full resolution order
validate_on_loadbooleantrueValidate tuning configurations when loaded
allow_platform_incompatiblebooleanfalseAllow tuning configs that contain platform-incompatible directives

Example full configuration file:

system:
profile_cache_hours: 48
database:
preferred: duckdb
benchmarks:
default_scale: 1.0
timeout_minutes: 120
continue_on_error: true
output:
directory: ./results
compression:
enabled: true
type: zstd
execution:
max_workers: 8
memory_limit_gb: 16
power_run:
iterations: 5
warm_up_iterations: 2
tuning:
default_config_file: ./tuning/my_tuning.yaml