Read Primitives — Optimizer: Distinct Elimination

Read Primitives benchmark · all Read Primitives queries

Test DISTINCT elimination when result is already unique

Representative SQL

Rendered for DataFusion with default parameters.

SELECT DISTINCT
  o_orderkey,
  o_custkey,
  o_orderdate,
  o_totalprice
FROM orders AS o
WHERE
  o_orderdate >= CAST('1995-01-01' AS DATE)
  AND o_orderdate < CAST('1996-01-01' AS DATE)
  AND o_totalprice > 50000
ORDER BY
  o_orderkey
LIMIT 2000

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def optimizer_distinct_elimination_expression_impl(ctx: DataFrameContext) -> Any:
    orders = ctx.get_table("orders")
    col = ctx.col
    lit = ctx.lit

    return (
        orders.filter((col("o_orderdate") >= lit(date(1995, 1, 1))) & (col("o_orderdate") < lit(date(1996, 1, 1))))
        .filter(col("o_totalprice") > lit(50000))
        .select("o_orderkey", "o_custkey", "o_orderdate", "o_totalprice")
        .unique()
        .sort("o_orderkey")
        .limit(2000)
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['optimizer_distinct_elimination'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'optimizer_distinct_elimination', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py