Read Primitives — QUALIFY CUME_DIST

Read Primitives benchmark · all Read Primitives queries

Find lineitems in top 5% quantity per shipdate using CUME_DIST

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  l_shipdate,
  l_orderkey,
  l_linenumber,
  l_quantity,
  quantity_cumulative_dist
FROM (
  SELECT
    l_shipdate,
    l_orderkey,
    l_linenumber,
    l_quantity,
    CUME_DIST() OVER (PARTITION BY l_shipdate ORDER BY l_quantity) AS quantity_cumulative_dist,
    CUME_DIST() OVER (PARTITION BY l_shipdate ORDER BY l_quantity) AS _w
  FROM lineitem
  WHERE
    l_shipdate >= CAST('1995-01-01' AS DATE)
    AND l_shipdate < CAST('1996-01-01' AS DATE)
  ORDER BY
    l_shipdate,
    l_quantity DESC,
    l_orderkey,
    l_linenumber
) AS _t
WHERE
  _w >= 0.95

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def qualify_cume_dist_expression_impl(ctx: DataFrameContext) -> Any:
    col, lit = ctx.col, ctx.lit
    result = (
        ctx.get_table("lineitem")
        .filter((col("l_shipdate") >= lit(date(1995, 1, 1))) & (col("l_shipdate") < lit(date(1996, 1, 1))))
        .with_columns(
            ctx.window_cume_dist(order_by=[("l_quantity", True)], partition_by=["l_shipdate"]).alias(
                "quantity_cumulative_dist"
            )
        )
    )
    return (
        result.filter(col("quantity_cumulative_dist") >= lit(0.95))
        .select("l_shipdate", "l_orderkey", "l_linenumber", "l_quantity", "quantity_cumulative_dist")
        .sort(["l_shipdate", "l_quantity", "l_orderkey", "l_linenumber"], descending=[False, True, False, False])
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['qualify_cume_dist'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'qualify_cume_dist', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py