Read Primitives — Exchange Shuffle

Read Primitives benchmark · all Read Primitives queries

Data is redistributed based on the join keys so matching rows end up on the same node

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  ps1.ps_partkey,
  ps1.ps_suppkey AS supp1,
  ps2.ps_suppkey AS supp2
FROM partsupp AS ps1
JOIN partsupp AS ps2
  ON ps1.ps_partkey = ps2.ps_partkey
WHERE
  ps1.ps_suppkey < ps2.ps_suppkey AND ps1.ps_supplycost > 100

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def exchange_shuffle_expression_impl(ctx: DataFrameContext) -> Any:
    col, lit = ctx.col, ctx.lit
    partsupp = ctx.get_table("partsupp")
    ps1 = partsupp.select(
        col("ps_partkey").alias("ps_partkey"), col("ps_suppkey").alias("supp1"), col("ps_supplycost").alias("cost1")
    )
    ps2 = partsupp.select(col("ps_partkey").alias("ps_partkey_2"), col("ps_suppkey").alias("supp2"))
    return (
        ps1.join(ps2, left_on="ps_partkey", right_on="ps_partkey_2")
        .filter((col("supp1") < col("supp2")) & (col("cost1") > lit(100)))
        .select("ps_partkey", "supp1", "supp2")
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['exchange_shuffle'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'exchange_shuffle', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py