Testing Documentation

Documentation for testing BenchBox functionality, including live integration tests.

Test Documentation

• Pytest xdist Safety - Why BenchBox caps local xdist workers and how to validate changes
• Test tiers and duration budgets - Measured T1/T2/T3 policy and quarantine markers
• Docker Integration Tests - Running real tests against database engines in Docker containers (free, no credentials)
• Live Integration Tests - Running integration tests against live cloud database platforms

Test Categories

Unit Tests

Fast, isolated tests for individual components without external dependencies.

make test-unit
uv run -- python -m pytest -m unit

The two commands are alternatives.

Integration Tests

Tests that verify interaction between components, may use embedded databases.

make test-integration
uv run -- python -m pytest -m "integration and not live_integration"

The two commands are alternatives.

E2E Tests

End-to-end tests that validate complete benchmark workflows through the CLI.

uv run -- python -m pytest -m e2e_quick

uv run -- python -m pytest -m e2e_local

uv run -- python -m pytest tests/e2e/

The e2e_quick marker runs quick E2E tests in dry-run mode, e2e_local runs local platform E2E tests with full execution, and the last command runs all E2E tests.

E2E tests cover:

• CLI option validation (--benchmark, --scale, --phases, --queries, etc.)
• Error handling for invalid parameters
• Result file validation and schema compliance
• Local platforms (DuckDB, SQLite, DataFusion)
• Cloud platforms (dry-run mode for Snowflake, BigQuery, etc.)
• DataFrame platforms (Polars, Pandas, Dask)

See E2E Testing Guide for detailed information.

Docker Integration Tests

Tests that execute real queries against database engines running in Docker containers. No cloud credentials needed: just Docker.

make test-docker-clickhouse

make test-docker-all

The first command tests a single platform and the second tests all Docker platforms.

See Docker Integration Tests for platform list and setup.

Live Integration Tests (Cloud)

Tests that require live database credentials and cloud platforms.

make test-live
uv run -- python -m pytest -m live_integration

The two commands are alternatives.

See Live Integration Tests for detailed setup instructions.

Related Documentation

• Development Guide - Development environment setup
• Testing Guide - Test organization and strategies
• Pytest xdist Safety - xdist root cause, cap behavior, and validation checklist
• CI/CD Integration - Automated testing workflows