Configuration
BenchBox supports configuration through multiple sources with the following precedence:
- Command-line arguments (highest priority)
- Environment variables
- Configuration files
- Default values (lowest priority)
Configuration File Format
BenchBox uses YAML configuration files. Default location: ~/.benchbox/config.yaml
Example configuration:
output: compression: enabled: true type: zstd level: 3 formats: - json - csv
platforms: databricks: enabled: true warehouse_id: "abc123"
bigquery: enabled: true project_id: "my-project"
tuning: default_mode: notuning enable_constraints: falsePlatform Configuration
Each platform can have specific configuration options:
Databricks:
platforms: databricks: warehouse_id: "warehouse-id" catalog: "main" schema: "benchbox"BigQuery:
platforms: bigquery: project_id: "my-project" dataset: "benchbox" location: "US"Snowflake:
platforms: snowflake: account: "account-name" compute_resource: "COMPUTE_WH" database: "BENCHBOX" schema: "PUBLIC"Environment Variables
BenchBox recognizes these environment variables:
Core Settings
These variables override the corresponding settings in the configuration file (equivalent to editing ~/.benchbox/config.yaml):
| Variable | Config Path | Type | Default | Description |
|---|---|---|---|---|
BENCHBOX_DATABASE_PREFERRED | database.preferred | string | duckdb | Preferred database platform |
BENCHBOX_SCALE_FACTOR | benchmarks.default_scale | float | 0.01 | Default scale factor |
BENCHBOX_VERBOSE | execution.verbose | boolean | true | Enable verbose output (true/1/yes/on) |
BENCHBOX_MAX_WORKERS | execution.max_workers | integer | 4 | Maximum parallel worker threads |
BENCHBOX_TUNING_CONFIG | tuning.default_config_file | string | - | Path to tuning configuration file. Used when --tuning tuned resolves with no explicit file (see Tuning Commands) |
BENCHBOX_OUTPUT_DIR | runtime path resolver | string | worktree-anchored benchmark_runs root | Runtime output root override; it is not exposed as load_config().output.directory |
BENCHBOX_MEMORY_LIMIT_GB | execution.memory_limit_gb | integer | 0 (auto) | Memory limit in GB; 0 means no limit |
Boolean parsing: true, 1, yes, on (case-insensitive) are treated as true.
General Settings
BENCHBOX_NON_INTERACTIVE=true: Enable non-interactive modeBENCHBOX_NO_COMPRESSION=true: Disable data compressionBENCHBOX_CONFIG_PATH=/path/to/config.yaml: Custom config file location
Advanced Settings
These variables control lower-level behaviors, useful for CI, offline environments, or advanced workflows:
| Variable | Description |
|---|---|
BENCHBOX_DATA_DIR | Override the default local data directory for generated benchmark files |
BENCHBOX_CACHE_DIR | Override the cache directory for DataFrame benchmark data |
BENCHBOX_TUNING_PATH | Override the tuning file search root; BenchBox looks for {platform}/{benchmark}_tuned.yaml under this directory |
BENCHBOX_QUERY_VALIDATION_MODE | TPC-DS query validation mode: exact, loose, range, or skip (default: skip) |
BENCHBOX_NO_DOWNLOAD | Set to 1 or true to disable automatic download of expected-answer files; useful in offline/airgapped environments |
BENCHBOX_ANSWERS_URL | Override the base URL for downloading expected-answer archives (default: GitHub releases) |
BENCHBOX_ENABLE_EXPERIMENTAL | Set to 1 or true to show experimental platforms in the platform registry |
BENCHBOX_DATA_ORG_MAX_IN_MEMORY_BYTES | Maximum bytes held in memory when sorting generated data files (integer; default: 512 MB) |
Platform Authentication
benchbox setup --platform <name> saves cloud credentials, and benchbox run reads the saved credentials. The variables below are read by setup when you accept its environment auto-detection prompt; setting them does not authenticate run by itself.
Databricks:
DATABRICKS_TOKEN: Authentication tokenDATABRICKS_HOST: Workspace URL
BigQuery:
GOOGLE_APPLICATION_CREDENTIALS: Service account key file path (required by setup)BIGQUERY_PROJECT: Google Cloud project ID (required by setup)BIGQUERY_DATASET,BIGQUERY_LOCATION,BIGQUERY_STORAGE_BUCKET: Optional dataset, location and staging bucket
Snowflake:
SNOWFLAKE_ACCOUNT: Account identifierSNOWFLAKE_USERNAME: UsernameSNOWFLAKE_PASSWORD: PasswordSNOWFLAKE_WAREHOUSE,SNOWFLAKE_DATABASE: Warehouse and database (required by setup)SNOWFLAKE_SCHEMA,SNOWFLAKE_ROLE: Optional schema and role
Redshift:
REDSHIFT_HOST,REDSHIFT_USERNAME,REDSHIFT_PASSWORD,REDSHIFT_DATABASE: Connection details (required by setup)REDSHIFT_PORT,REDSHIFT_SCHEMA,REDSHIFT_S3_BUCKET,REDSHIFT_IAM_ROLE: Optional port, schema and S3 staging settingsAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_DEFAULT_REGION: Optional AWS credentials and region for S3 staging
ClickHouse:
CLICKHOUSE_HOST: Server hostnameCLICKHOUSE_USER: UsernameCLICKHOUSE_PASSWORD: Password
Platform-Specific Options
Each platform supports specific options via --platform-option KEY=VALUE:
Universal Keys (All Platforms)
These keys are available for every platform:
| Key | Description |
|---|---|
driver_version | Pin the Python driver package to a specific version (e.g. 1.2.0). Useful for reproducing results or testing a specific connector release. |
driver_auto_install | When true, automatically installs the requested driver_version via uv if the package is not already present. |
Example: pin DuckDB driver and auto-install it:
benchbox run --platform duckdb --benchmark tpch \ --platform-option driver_version=1.2.0 \ --platform-option driver_auto_install=trueExample: pin Snowflake connector version:
benchbox run --platform snowflake --benchmark tpch \ --platform-option driver_version=3.12.0 \ --platform-option driver_auto_install=true \ --platform-option compute_resource=COMPUTE_WHThe Snowflake account, user, and password come from
benchbox setup --platform snowflake; they are not --platform-option keys.
Saved Credentials File
benchbox setup saves platform credentials in ~/.benchbox/credentials.yaml. The file is plain YAML and is not encrypted. It holds secrets such as Snowflake, Redshift, SingleStore and Athena passwords, AWS keys and the Databricks token, and any process you run can read it. Only the hosted-submission token (benchbox auth login) is kept in the operating system keyring.
A value written as ${VAR} is read from the environment when benchbox run loads the file. A later save of the file, such as benchbox setup or benchbox setup --remove, writes the resolved values in place of the ${VAR} references, so a secret read from the environment ends up in the file.
snowflake: account: ${SNOWFLAKE_ACCOUNT} user: ${SNOWFLAKE_USERNAME} password: ${SNOWFLAKE_PASSWORD} warehouse: ${SNOWFLAKE_WAREHOUSE}benchbox run reads Snowflake settings only from this file. The SNOWFLAKE_* variables are read only by benchbox setup; ${SNOWFLAKE_*} references in the file are how run picks them up from the environment.
Compute Settings
On platforms that run on a named compute resource, compute_resource names
it and compute_size sets its size. BenchBox takes each setting from the first
place that has it:
--platform-optionon the command line;- the saved credential profile written by
benchbox setup; - the option’s default.
A value on the command line overrides the saved profile. Two spellings that
disagree in the same place, such as warehouse=A and compute_resource=B
on one command line, stop the run before any work and name both values.
On Redshift, compute_resource names the workgroup when the host is a
Serverless endpoint (*.redshift-serverless.amazonaws.com) and the cluster
when it is a provisioned endpoint (*.redshift.amazonaws.com). With any other
host, use workgroup_name or cluster_identifier. On Databricks,
compute_resource names the SQL warehouse and sets the HTTP path to
/sql/1.0/warehouses/<id>.
The --platform name:mode deployment selector follows the same rule: it
overrides a deployment key, such as LakeSail sail_mode, in the saved profile.
Athena Spark Runtime Version
On Athena Spark, the Spark-enabled workgroup sets the engine version. A session cannot choose it, so BenchBox never sends a version when it starts one.
| Key | Description |
|---|---|
runtime_version | Spark engine version you expect the workgroup to run (e.g. PySpark engine version 3). Results record it as runtime_version with runtime_version_source: requested. Change the version on the workgroup itself. engine_version is a deprecated alias, removed in 0.6.0. |
benchbox run --platform athena-spark --benchmark tpch \ --platform-option compute_resource=my-spark-workgroup \ --platform-option s3_staging_dir=s3://my-bucket/benchbox \ --platform-option "runtime_version=PySpark engine version 3"ClickHouse Options
Choose the deployment with the platform name, not an option:
--platform clickhouse-local (or clickhouse:local) runs embedded chDB, and
--platform clickhouse-server (or clickhouse:server) connects to a server.
The mode option is a deprecated alias for this choice.
clickhouse-local accepts:
data_path=/path/to/dir: Directory for persistent chDB storageoptimize_after_load=true: RunOPTIMIZE TABLE ... FINALafter each table loads (tuned runs only)
clickhouse-server accepts:
host=localhost: Server hostnameport=9000: Native protocol portsecure=true: Enable TLS encryptionusername=defaultandpassword=...: Authenticationoptimize_after_load=true: As above
Examples:
benchbox run --platform clickhouse-local --benchmark tpch \ --platform-option data_path=./chdb-data
benchbox run --platform clickhouse-server --benchmark tpch \ --platform-option secure=true \ --platform-option port=9440View Platform Details
Use benchbox platforms status to see platform information and capabilities:
benchbox platforms status clickhousebenchbox platforms status databricksConfiguration Sections Reference
Complete reference for all settings in ~/.benchbox/config.yaml (or ./benchbox.yaml). Unset values fall back to the defaults shown here.
system
System profiling and detection settings.
| Setting | Type | Default | Description |
|---|---|---|---|
auto_profile | boolean | true | Automatically detect system capabilities on startup |
save_profile | boolean | true | Persist the detected system profile to disk |
profile_cache_hours | integer | 24 | Hours to cache the system profile before re-detecting |
database
Database connection settings.
| Setting | Type | Default | Description |
|---|---|---|---|
preferred | string | duckdb | Default platform when none is specified on the CLI |
connection_timeout | integer | 30 | Connection timeout in seconds |
auto_detect | boolean | true | Automatically detect available database platforms |
benchmarks
Default benchmark execution parameters.
| Setting | Type | Default | Description |
|---|---|---|---|
default_scale | float | 0.01 | Default scale factor |
timeout_minutes | integer | 60 | Maximum benchmark execution time |
max_memory_gb | integer | 8 | Maximum memory allocation hint in GB |
continue_on_error | boolean | false | Continue running remaining queries when one fails |
output
Result output and export settings.
| Setting | Type | Default | Description |
|---|---|---|---|
formats | list | [json, console] | Output format list |
directory | string | runtime-resolved | Results directory; resolved from the runtime output root rather than a legacy config key |
timestamp_format | string | %Y%m%d_%H%M%S | Timestamp format for result filenames |
submit_to_service | boolean | false | Legacy placeholder; use benchbox submit --service for hosted upload |
service_url | string | https://api.benchbox.dev/v1 | Default hosted results service URL placeholder |
compression.enabled | boolean | true | Enable result file compression |
compression.type | string | zstd | Compression algorithm (zstd, gzip) |
compression.level | integer | null | Compression level; null uses the algorithm default |
execution
Performance and execution control settings.
| Setting | Type | Default | Description |
|---|---|---|---|
parallel_queries | boolean | false | Execute queries in parallel |
max_workers | integer | 4 | Maximum worker threads for parallel execution |
memory_limit_gb | integer | 0 | Memory cap in GB; 0 means no limit (auto) |
verbose | boolean | true | Enable verbose progress output |
execution.power_run - settings for multi-iteration power runs:
| Setting | Type | Default | Description |
|---|---|---|---|
iterations | integer | 3 | Number of measurement iterations per query |
warm_up_iterations | integer | 1 | Warm-up iterations before measurement (not included in stats) |
timeout_per_iteration_minutes | integer | 60 | Timeout for each iteration |
fail_fast | boolean | false | Stop immediately when any iteration fails |
collect_metrics | boolean | true | Collect resource metrics during execution |
execution.concurrent_queries - settings for throughput / concurrent-stream runs:
| Setting | Type | Default | Description |
|---|---|---|---|
enabled | boolean | false | Enable concurrent query streams |
max_concurrent | integer | 2 | Number of concurrent query streams |
query_timeout_seconds | integer | 300 | Per-query timeout in concurrent mode |
stream_timeout_seconds | integer | unset | Timeout in seconds for each throughput stream in benchbox run. Unset keeps the benchmark default (3600 for TPC-H, 7200 for TPC-DS) |
cancel_on_timeout | boolean | false | Signal a timed-out throughput stream to stop before its next query |
retry_failed_queries | boolean | true | Retry failed queries in concurrent streams |
max_retries | integer | 3 | Maximum retry attempts per query |
Only stream_timeout_seconds and cancel_on_timeout are read by benchbox run. The other settings are read only through ExecutionConfigHelper.
tuning
Table tuning and optimization settings.
| Setting | Type | Default | Description |
|---|---|---|---|
default_config_file | string | null | Default tuning YAML file used when --tuning tuned finds no explicit file; see Tuning Commands for the full resolution order |
validate_on_load | boolean | true | Validate tuning configurations when loaded |
allow_platform_incompatible | boolean | false | Allow tuning configs that contain platform-incompatible directives |
Example full configuration file:
system: profile_cache_hours: 48
database: preferred: duckdb
benchmarks: default_scale: 1.0 timeout_minutes: 120 continue_on_error: true
output: directory: ./results compression: enabled: true type: zstd
execution: max_workers: 8 memory_limit_gb: 16 power_run: iterations: 5 warm_up_iterations: 2
tuning: default_config_file: ./tuning/my_tuning.yamlRelated
- Run Command - Platform and tuning options during execution
- Platforms Command - Platform management and setup
- Platform Documentation - Detailed platform guides