Skip to content

Search docs and blog

Azure Synapse Analytics Spark Platform

Azure Synapse Analytics is Microsoft’s enterprise analytics platform providing integrated Spark, SQL, and Data Explorer capabilities. BenchBox integrates with Synapse Spark pools via the Livy API for benchmark execution with ADLS Gen2 storage.

Features

  • Enterprise - Mature platform with extensive enterprise features
  • ADLS Gen2 - Azure Data Lake Storage integration
  • Spark Pools - Dedicated pools with configurable sizing
  • Entra ID - Azure Active Directory authentication
  • Integration - Native integration with Synapse SQL pools

Installation

This installs the Azure cloud-Spark dependencies, including azure-identity, azure-storage-file-datalake, and requests.

Terminal window
uv add benchbox --extra cloud-spark-azure

Prerequisites

  1. Azure Synapse Analytics workspace
  2. Spark pool created in the workspace
  3. ADLS Gen2 storage account linked to workspace
  4. Azure Entra ID authentication configured:
    • Interactive: az login
    • Service principal: Environment variables
    • Managed identity: On Azure VMs

Configuration

CLI Usage

The BenchBox CLI does not accept the Synapse workspace or storage settings, so run Synapse Spark benchmarks through the Python API described below. The CLI can still preview the generated queries without connecting to Azure:

Terminal window
benchbox run --platform synapse-spark --benchmark tpch --dry-run ./preview

Adapter Options

These are SynapseSparkAdapter keyword arguments.

OptionDefaultDescription
workspace_namerequiredSynapse workspace name
spark_pool_namerequiredSpark pool name
storage_accountrequiredADLS Gen2 storage account name
storage_containerrequiredADLS Gen2 container name
storage_pathbenchboxPath within container for staging
tenant_id-Azure tenant ID (for service principal)
livy_endpointauto-derivedCustom Livy API endpoint URL
timeout_minutes60Statement timeout in minutes

Python API

The example initializes the adapter with a workspace and storage (tenant_id is optional), verifies the connection, creates the schema, loads data to ADLS and creates tables, executes a query via Livy, and cleans up the session.

from benchbox.platforms.azure import SynapseSparkAdapter


adapter = SynapseSparkAdapter(
    workspace_name="my-synapse-workspace",
    spark_pool_name="sparkpool1",
    storage_account="mystorageaccount",
    storage_container="benchbox",
    tenant_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
)


adapter.create_connection()


adapter.create_schema("tpch_benchmark")


adapter.load_data(
    tables=["lineitem", "orders", "customer"],
    source_dir="/path/to/tpch/data",
)


result = adapter.execute_query("SELECT COUNT(*) FROM lineitem")


adapter.close()

Execution Model

Synapse Spark executes benchmarks via the Livy API:

  1. Pool Startup - Spark pool started (or uses running pool)
  2. Session Creation - Livy session created in pool
  3. Statement Execution - SQL submitted as Livy statements
  4. Result Retrieval - Results returned via Livy output
  5. Session Cleanup - Session closed after completion

Spark Pool Configuration

Node Sizes

SizevCoresMemoryUse Case
Small432 GBDevelopment, small datasets
Medium864 GBGeneral benchmarking
Large16128 GBLarge scale factors
XLarge32256 GBEnterprise workloads
XXLarge64512 GBMaximum performance

Auto-Pause and Auto-Scale

Auto-pause: Pool pauses after idle timeout (saves costs)
Auto-scale: Pool scales nodes based on workload
Min nodes: Minimum nodes to keep warm
Max nodes: Maximum nodes for scaling

Authentication

Synapse Spark uses Azure Entra ID (Azure AD) for authentication:

Interactive Login (Development)

Terminal window
az login

Service Principal (Automation)

Terminal window
export AZURE_CLIENT_ID=app-client-id
export AZURE_CLIENT_SECRET=app-client-secret
export AZURE_TENANT_ID=tenant-id

Managed Identity (Azure VMs)

No configuration needed - automatically uses VM’s managed identity.

Cost Estimation

Synapse Spark uses vCore-hour billing:

Node SizevCoresPrice/Hour
Small4~$0.22
Medium8~$0.44
Large16~$0.88
XLarge32~$1.76
Scale FactorData SizeEst. RuntimeEst. Cost*
0.01~10 MB~15 min~$0.30
1.0~1 GB~1 hour~$2.00
10.0~10 GB~3 hours~$8.00
100.0~100 GB~8 hours~$25.00

*Estimates based on 3-node Medium pool. Actual costs vary.

Workspace Setup

Creating a Spark Pool

  1. Go to Synapse Studio
  2. Navigate to Manage > Apache Spark pools
  3. Click New to create pool
  4. Configure:
    • Name: e.g., “sparkpool1”
    • Node size: Medium (8 vCores)
    • Auto-scale: Enable
    • Auto-pause: Enable (15 min idle)

Linking Storage

  1. Go to Manage > Linked services
  2. Click New > Azure Data Lake Storage Gen2
  3. Configure:
    • Name: Primary storage link
    • Authentication: Managed identity
    • Account: Your ADLS Gen2 account

Troubleshooting

Common Issues

Authentication fails:

  • Run az login for interactive authentication
  • Check service principal credentials
  • Verify tenant ID is correct

Spark pool not starting:

  • Check pool auto-start is enabled
  • Verify pool isn’t at max capacity
  • Review Azure resource quotas

Storage access denied:

  • Check Storage Blob Data Contributor role
  • Verify managed identity is configured
  • Check container exists and is accessible

Session timeout:

  • Increase timeout_minutes for large scale factors
  • Check pool hasn’t auto-paused
  • Review session idle timeout settings

Comparison with Other Azure Platforms

AspectSynapse SparkFabric SparkDatabricks
DeploymentPaaSSaaSSaaS
StorageADLS Gen2OneLakeDBFS/Unity
BillingvCore-hoursCapacity UnitsDBUs
IntegrationSynapse SQLPower BIMLflow
MaturityEstablishedNewEstablished

BenchBox is open source under the MIT License.