Read Primitives — Union All with GroupBy

Read Primitives benchmark · all Read Primitives queries

Complex join with UNION ALL and different data sources

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  source_type,
  COUNT(*) AS record_count
FROM (
  SELECT
    'ORDER' AS source_type,
    o_custkey AS cust_id
  FROM orders
  WHERE
    o_orderdate >= CAST('1995-01-01' AS DATE)
  UNION ALL
  SELECT
    'LINEITEM' AS source_type,
    o_custkey AS cust_id
  FROM orders AS o
  JOIN lineitem AS l
    ON o.o_orderkey = l.l_orderkey
  WHERE
    l.l_shipdate >= CAST('1995-01-01' AS DATE)
) AS combined
GROUP BY
  source_type

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def shuffle_union_all_groupby_expression_impl(ctx: DataFrameContext) -> Any:
    orders, lineitem = _tables(ctx, "orders", "lineitem")
    col = ctx.col
    lit = ctx.lit

    orders_source = (
        orders.filter(col("o_orderdate") >= lit(date(1995, 1, 1)))
        .select(col("o_custkey").alias("cust_id"))
        .with_columns(lit("ORDER").alias("source_type"))
    )

    lineitem_orders = (
        orders.join(lineitem, left_on="o_orderkey", right_on="l_orderkey")
        .filter(col("l_shipdate") >= lit(date(1995, 1, 1)))
        .select(col("o_custkey").alias("cust_id"))
        .with_columns(lit("LINEITEM").alias("source_type"))
    )

    return (
        orders_source.vstack(lineitem_orders).group_by("source_type").agg(col("cust_id").count().alias("record_count"))
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['shuffle_inner_join_union_all_with_groupby'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'shuffle_inner_join_union_all_with_groupby', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py