Read Primitives — Optimizer: Aggregate Pushdown

Read Primitives benchmark · all Read Primitives queries

Test aggregate pushdown before join

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  c.c_name,
  c.c_mktsegment,
  c.c_nationkey,
  COUNT(o.o_orderkey) AS order_count,
  SUM(o.o_totalprice) AS total_spent,
  AVG(o.o_totalprice) AS avg_order_value
FROM customer AS c
JOIN orders AS o
  ON c.c_custkey = o.o_custkey
WHERE
  c.c_nationkey = 15
  AND o.o_orderdate >= CAST('1995-01-01' AS DATE)
  AND o.o_orderdate < CAST('1996-01-01' AS DATE)
GROUP BY
  c.c_custkey,
  c.c_name,
  c.c_mktsegment,
  c.c_nationkey
HAVING
  SUM(o.o_totalprice) > 500000
ORDER BY
  total_spent DESC
LIMIT 50

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def optimizer_aggregate_pushdown_expression_impl(ctx: DataFrameContext) -> Any:
    customer, orders = _tables(ctx, "customer", "orders")
    col = ctx.col
    lit = ctx.lit

    return (
        customer.join(orders, col("c_custkey") == col("o_custkey"))
        .filter(col("c_nationkey") == lit(15))
        .filter(col("o_orderdate") >= lit(date(1995, 1, 1)))
        .filter(col("o_orderdate") < lit(date(1996, 1, 1)))
        .group_by("c_custkey", "c_name", "c_mktsegment", "c_nationkey")
        .agg(
            col("o_orderkey").count().alias("order_count"),
            col("o_totalprice").sum().alias("total_spent"),
            col("o_totalprice").mean().alias("avg_order_value"),
        )
        .filter(col("total_spent") > lit(500000))
        .select("c_name", "c_mktsegment", "c_nationkey", "order_count", "total_spent", "avg_order_value")
        .sort(["total_spent", "c_name"], descending=[True, False])
        .limit(50)
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['optimizer_aggregate_pushdown'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'optimizer_aggregate_pushdown', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py