Read Primitives — Optimizer: Runtime Filter

Read Primitives benchmark · all Read Primitives queries

Test runtime filter / dynamic partition pruning

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  l.l_orderkey,
  l.l_partkey,
  l.l_suppkey,
  l.l_quantity,
  l.l_extendedprice,
  p.p_name,
  p.p_type
FROM lineitem AS l
JOIN part AS p
  ON l.l_partkey = p.p_partkey
WHERE
  p.p_type LIKE '%STEEL%'
  AND p.p_size BETWEEN 10 AND 20
  AND l.l_shipdate >= CAST('1995-01-01' AS DATE)
  AND l.l_shipdate < CAST('1996-01-01' AS DATE)
  AND l.l_quantity > 20
ORDER BY
  l.l_extendedprice DESC
LIMIT 1000

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def optimizer_runtime_filter_expression_impl(ctx: DataFrameContext) -> Any:
    lineitem, part = _tables(ctx, "lineitem", "part")
    col = ctx.col
    lit = ctx.lit

    selective_parts = part.filter(col("p_type").str.contains("STEEL")).filter(
        (col("p_size") >= lit(10)) & (col("p_size") <= lit(20))
    )
    return (
        lineitem.join(selective_parts, col("l_partkey") == col("p_partkey"))
        .filter(col("l_shipdate") >= lit(date(1995, 1, 1)))
        .filter(col("l_shipdate") < lit(date(1996, 1, 1)))
        .filter(col("l_quantity") > lit(20))
        .select("l_orderkey", "l_partkey", "l_suppkey", "l_quantity", "l_extendedprice", "p_name", "p_type")
        .sort(["l_extendedprice", "l_orderkey", "l_partkey"], descending=[True, False, False])
        .limit(1000)
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['optimizer_runtime_filter'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'optimizer_runtime_filter', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py