Read Primitives — Approximate Distinct Count with Group By

Read Primitives benchmark · all Read Primitives queries

HLL distinct counts per low-cardinality group (sketch on Polars/PySpark/DataFusion; exact on Pandas/Modin/cuDF/Dask)

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  l_returnflag,
  l_linestatus,
  APPROX_DISTINCT(l_orderkey) AS unique_orders,
  APPROX_DISTINCT(l_partkey) AS unique_parts
FROM lineitem
GROUP BY
  l_returnflag,
  l_linestatus

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def approx_count_distinct_groupby_expression_impl(ctx: DataFrameContext) -> Any:
    lineitem = ctx.get_table("lineitem")
    col = ctx.col

    return lineitem.group_by("l_returnflag", "l_linestatus").agg(
        col("l_orderkey").approx_n_unique().alias("unique_orders"),
        col("l_partkey").approx_n_unique().alias("unique_parts"),
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['approx_count_distinct_groupby'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'approx_count_distinct_groupby', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py