Testing Documentation

Tags contributor testing

Documentation for testing BenchBox functionality, including live integration tests.

Test Documentation

Test Categories

Unit Tests

Fast, isolated tests for individual components without external dependencies.

make test-unit
# or
uv run -- python -m pytest -m unit

Integration Tests

Tests that verify interaction between components, may use embedded databases.

make test-integration
# or
uv run -- python -m pytest -m "integration and not live_integration"

E2E Tests

End-to-end tests that validate complete benchmark workflows through the CLI.

# Quick E2E tests (dry-run mode)
make test-e2e-quick
# or
uv run -- python -m pytest -m e2e_quick

# Local platform E2E tests (full execution)
uv run -- python -m pytest -m e2e_local

# All E2E tests
uv run -- python -m pytest tests/e2e/

E2E tests cover:

  • CLI option validation (--benchmark, --scale, --phases, --queries, etc.)

  • Error handling for invalid parameters

  • Result file validation and schema compliance

  • Local platforms (DuckDB, SQLite, DataFusion)

  • Cloud platforms (dry-run mode for Snowflake, BigQuery, etc.)

  • DataFrame platforms (Polars, Pandas, Dask)

See E2E Testing Guide for detailed information.

Live Integration Tests

Tests that require live database credentials and cloud platforms.

make test-live
# or
uv run -- python -m pytest -m live_integration

See Live Integration Tests for detailed setup instructions.