Read Primitives — Self Join with Hash Collision

Read Primitives benchmark · all Read Primitives queries

Self-join with hash collision handling on large table

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  l1.l_orderkey,
  COUNT(*) AS match_count
FROM lineitem AS l1
JOIN lineitem AS l2
  ON l1.l_partkey = l2.l_partkey
WHERE
  l1.l_orderkey <> l2.l_orderkey AND l1.l_shipdate = l2.l_shipdate
GROUP BY
  l1.l_orderkey
LIMIT 10000

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def shuffle_self_join_expression_impl(ctx: DataFrameContext) -> Any:
    col = ctx.col
    lineitem = ctx.get_table("lineitem")
    other = lineitem.select("l_orderkey", "l_partkey", "l_shipdate").rename({"l_orderkey": "l_orderkey_2"})
    return (
        lineitem.join(other, on=["l_partkey", "l_shipdate"])
        .filter(col("l_orderkey") != col("l_orderkey_2"))
        .group_by("l_orderkey")
        .agg(col("l_orderkey_2").count().alias("match_count"))
        .limit(10000)
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['shuffle_1mb_rows'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'shuffle_1mb_rows', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py