AMPLab Q2a — Join with Filter

AMPLab benchmark · all AMPLab queries

Join uservisits to rankings with pageRank filter

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  uv.destURL,
  uv.visitDate,
  uv.adRevenue,
  r.pageRank,
  r.avgDuration
FROM uservisits AS uv
JOIN rankings AS r
  ON uv.destURL = r.pageURL
WHERE
  r.pageRank > 1000 AND uv.visitDate >= '2000-01-01'
ORDER BY
  r.pageRank DESC
LIMIT 100

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def q2a_expression_impl(ctx: DataFrameContext) -> Any:
    params = get_parameters("Q2a")
    threshold = params.get("pagerank_threshold", 1000)
    start_date = params.get("start_date", date(2000, 1, 1))
    limit_rows = params.get("limit_rows", 100)

    uservisits = ctx.get_table("uservisits")
    rankings = ctx.get_table("rankings")
    col = ctx.col
    lit = ctx.lit

    return (
        uservisits.filter(col("visitDate").cast_date() >= lit(start_date))
        .join(rankings, left_on="destURL", right_on="pageURL")
        .filter(col("pageRank") > lit(threshold))
        .select("destURL", "visitDate", "adRevenue", "pageRank", "avgDuration")
        .sort("pageRank", descending=True)
        .limit(limit_rows)
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import AMPLab

bench = AMPLab(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['2a'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('amplab', '2a', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/amplab/queries.py