Live-DB revalidation — v0.8.0
Running featurizer against the three live triage databases
(dirtyduck, chicago311, donorschoose) after the
0.7.0 perf fixes and the 0.8.0 sharding rework — 2026-07-12.
wide (~36.8k columns), which
crashed the PostgreSQL backend in every previous snapshot. Two root causes, both found
by measurement: (1) 0.7.0 — correlated two-window drift (ADR-0012) and the
never-analyzed as_of_dates cardinality (ADR-0013) made all-agg 10–100×
slower than it should be; (2) 0.8.0 — order-based column chunking gave every
group query a CTE closure spanning most of the plan, and PostgreSQL's planning memory
is superlinear in same-statement window-function count (measured cliff: ~675
window columns plan in ~5s, ~1350 OOM-kill the backend at a plain EXPLAIN).
Groups are now packed by dependency lineage with a 500-window budget (ADR-0005
amendment), and a pre-flight guardrail (warn_plan_size) names doomed
configs at render time.1 · Test results
| tier | command | result |
|---|---|---|
| DB-free | just test-fast | 591 passed (guardrail, lineage-clustering, window-budget and re-join regression tests added) |
| Integration (seeded) | just test-integration | 435 passed, 0 failed — value equality holds under the new partitioning |
| Type check | just typecheck | 0 errors |
2 · Live full-cohort materialization (measured)
Each config materialized through featurizer's grouped path
(to_dataframe(connection=…)) at one as-of date, read-only.
narrow = the triage config's own primitives; all-agg = the full
65-aggregator default set with identity; wide = all-agg × 14
transformers (lags, rolling, EMA, cusum…). "was" = the v0.6.0
snapshot (2026-07-08).
| database | variant | features | shards | rows | seconds | was (v0.6.0) | status |
|---|---|---|---|---|---|---|---|
| dirtyduck | narrow | 147 | 1 | 22,169 | 3.0 | 3.4 | materialized |
| dirtyduck | all-agg | 272 | 1 | 22,169 | 7.5 | 356.8 | materialized |
| dirtyduck | wide | 1,252 | 1 | 22,169 | 63.2 | ln crash | materialized |
| chicago311 | narrow | 28 | 1 | 30,654 | 0.8 | 1.0 | materialized |
| chicago311 | all-agg | 191 | 1 | 30,654 | 6.0 | 10.1 | materialized |
| chicago311 | wide | 907 | 1 | 30,654 | 49.2 | ln crash | materialized |
| donorschoose | narrow | 175 | 1 | 3,000 | 0.4 | 1.5 | materialized |
| donorschoose | all-agg | 1,063 | 1 | 3,000 | 7.6 | 281.1 | materialized |
| donorschoose | wide | 36,802 | 32 | 3,000 | 470.1 | backend crash | materialized |
shards = actual column-group queries emitted (lineage + window-budget packing); duplicate output column names = 0 in every cell. Per-DB detail (config, live schema, feature samples with real values): dirtyduck · chicago311 · donorschoose.
3 · Read this honestly
all-agg is now uniformly seconds on every live DB (was 10–357s), and wide
completes everywhere — but donorschoose wide's ~8 minutes is survival, not
speed: ~36.8k columns across 32 group queries is an extreme, atypical config.
The pre-flight guardrail exists precisely so that configs in that regime announce
their cost at render time instead of dying minutes into execution. The known open
edge: to_tables() (CREATE TABLE AS) would hit PostgreSQL's 8160-byte
heap-row limit on a 1400-numeric-column group; the SELECT/fetch paths measured here
are immune.