Read Primitives — Correlation Analysis

Read Primitives benchmark · all Read Primitives queries

Correlation analysis between numeric columns

Representative SQL

Rendered for DataFusion with default parameters.

SELECT
  CORR(l_quantity, l_extendedprice) AS qty_price_correlation,
  COVAR_POP(l_quantity, l_discount) AS qty_discount_covariance,
  COVAR_SAMP(l_tax, l_extendedprice) AS tax_price_covariance,
  REGR_SLOPE(l_extendedprice, l_quantity) AS price_qty_slope,
  REGR_INTERCEPT(l_extendedprice, l_quantity) AS price_qty_intercept,
  REGR_R2(l_extendedprice, l_quantity) AS regression_r_squared
FROM lineitem
WHERE
  l_shipdate >= CAST('1995-01-01' AS DATE)
  AND l_shipdate < CAST('1996-01-01' AS DATE)

Representative DataFrame

Expression family (Polars, PySpark, DataFusion), default parameters.

def statistical_correlation_expression_impl(ctx: DataFrameContext) -> Any:
    import polars as pl

    lf = ctx.get_table("lineitem").native.filter(
        (pl.col("l_shipdate") >= date(1995, 1, 1)) & (pl.col("l_shipdate") < date(1996, 1, 1))
    )
    slope = pl.cov("l_quantity", "l_extendedprice", ddof=0) / pl.col("l_quantity").var(ddof=0)
    intercept = pl.col("l_extendedprice").mean() - slope * pl.col("l_quantity").mean()
    return lf.select(
        pl.corr("l_quantity", "l_extendedprice").alias("qty_price_correlation"),
        pl.cov("l_quantity", "l_discount", ddof=0).alias("qty_discount_covariance"),
        pl.cov("l_tax", "l_extendedprice", ddof=1).alias("tax_price_covariance"),
        slope.alias("price_qty_slope"),
        intercept.alias("price_qty_intercept"),
        (pl.corr("l_quantity", "l_extendedprice") ** 2).alias("regression_r_squared"),
    )

Get the exact query for your run

The blocks above are representative. To get the precise statement a specific platform runs at a specific scale, substitute your own values:

from benchbox import ReadPrimitives

bench = ReadPrimitives(scale_factor=0.01)            # your scale factor
queries = bench.get_queries(dialect="datafusion")   # your platform's dialect; drop dialect= for native SQL
print(queries['statistical_correlation'])

For the DataFrame form:

from benchbox.core.query_catalog import get_dataframe_render

print(get_dataframe_render('read_primitives', 'statistical_correlation', family="expression").source)  # or family="pandas"

Source

Query text: benchbox/core/read_primitives/queries.py