FAQ & troubleshooting

The questions below are the ones the rest of the documentation already answers, collected in one place. Each answer links to its canonical source — an ADR, a reference page, or the code — so you can go deeper. If your question isn’t here, open an issue.

Installation & compatibility

How do I install it? Is it on PyPI?

There is no PyPI package — deliberately. The name is generic (featurizer derives from dssg/featurizer), so distribution goes through GitHub releases on ccd-ia/featurizer instead. Three ways to install:

# 1. From source (the development path)
git clone https://github.com/ccd-ia/featurizer && cd featurizer && uv sync

# 2. Straight from git into your own project
uv add "git+https://github.com/ccd-ia/featurizer.git"
#   or: pip install "git+https://github.com/ccd-ia/featurizer.git"

# 3. A pinned wheel from a tagged release
#   grab the .whl asset from github.com/ccd-ia/featurizer/releases

Optional extras live behind uv sync --extra <name>: viz (diagnostic plots), bridge (the φ-bridge precompute companion), parquet (Arrow output). The SQL spine never imports them.

Does it work with MySQL, SQLite, DuckDB, or BigQuery?

No — featurizer emits PostgreSQL-dialect SQL and is validated only against PostgreSQL. The generated queries lean on Postgres-specific features: LEFT JOIN LATERAL for as-of joins, ordered-set aggregates (percentile_cont, mode() within group), bool_and/bool_or, PostGIS ST_* for spatial features, and enum catalog introspection for categoricals. Run the SQL on a real PostgreSQL instance (the test suite spins up an ephemeral postgres:16 container via just db-up).

Which Python and PostgreSQL versions are supported?

The CI-tested matrix (v1.0): Python 3.10–3.13 on the DB-free tier, and PostgreSQL 14, 16, and 17 on the integration tier, where the generated SQL actually executes. Versions outside the matrix may work but are not verified — we only claim what we test.

What does “stable” mean since 1.0?

A written commitment, not a vibe: the config schema, the Featurizer public surface and return shapes, the output-naming contract, the imputation contract, and the φ-bridge contract are frozen — breaking any of them requires a major version, and removals warn for at least one minor first. Internals (CTE names, SQL text, module layout) stay free to change. The freeze list, item by item, is the stability and deprecation policy in CONTRIBUTING.md.

Concepts

What does “as-of” / point-in-time correctness mean, and why should I care?

A feature is a function φ(entity, t) that may only see events with timestamp τ ≤ t. If a feature computed for a training row at date t accidentally reads events from after t, that’s data leakage: the model looks brilliant in backtest and fails in production. Featurizer’s temporal joins (mode: as_of, with an optional grace lookback) enforce the τ ≤ t boundary in SQL so leakage can’t creep in. See the φ theory page and drag the as-of date in the interactive explorable.

Why aren’t peer-group, spatial, or φ-bridge features in the primitives list?

Because they aren’t registry primitives — they’re planner passes driven by their own config blocks (peer_groups, spatial_relationships, the native 1-hop graph_relationships pass added in 0.9.0) or φ-bridge families (the featurizer/bridge/ companion: sentiment, NER counts, readability, language id, multi-metric centralities, Louvain community, embeddings, Markov surprisal). Aggregations and transformers apply uniformly across the entity graph; these families need cross-entity, second-table, or heavy-Python context the registry model doesn’t express, so they’re deliberately separate. The primitives reference covers everything that is a registry primitive; the primitives explorer lets you filter and search them; the bridge cookbook shows how to wire and extend the bridge families.

Common errors & limits

target list can have at most 1664 entries — my wide config fails

PostgreSQL caps a CTE/result target list at 1664 columns. A wide config — many primitives × many intervals × many variables — blows past that in a single monolithic query. The fix is column-group sharding: featurizer splits the feature set into groups, materializes each group’s CTE closure separately, and re-joins on the full key. It kicks in automatically past a threshold you can tune with Featurizer(..., materialize_threshold=N). See performance internals and the three render paths.

My column names look truncated or contain a ~

Generated feature names can exceed PostgreSQL’s 63-byte identifier limit. Featurizer hash-truncates anything longer to a stable, collision-safe name: the first 54 characters of the full name, a ~, then the first 8 hex chars of the MD5 of the full name — so the mapping is deterministic across runs, and two names differing only in the erased tail still get distinct columns. Internal CTE names use _ as the cap separator — a bare ~ there was a real bug (fixed in v0.8.0’s companion-CTE path).

The cut can land mid-variable-name. Truncation keeps the head (the outer wrapper prefixes), so a deep composition erases the innermost — most informative — fragment:

CUM_SUM(facilities.MEAN(inspections.ABS(inspections.kw_rod~978dcf98
       └ full name: CUM_SUM(facilities.MEAN(inspections.ABS(inspections.kw_rodent_complaint_flag)))

Anything that matches on physical column names (glob patterns like *(inspections.kw_*, feature-group regexes, humans reading a table) will miss these columns. Don’t parse physical names — use the feature manifest, which records for every output column the rendered column name, the full untruncated label, a truncated flag, and lineage (parents, source_alias, interval, definition). It’s available in-process (Featurizer.feature_manifest / manifest_dataframe()) and to_tables persists it as a "<schema>"."<stem>_manifest" table beside the feature tables.

Since 1.1.0, glob the label directly instead of doing that by hand:

f = Featurizer("config.yaml")

f.columns_matching("*(inspections.kw_*")   # -> ["…kw_rod~978dcf98", …] physical names
f.manifest_matching("*(inspections.kw_*")  # -> full ManifestEntry rows (label, lineage, interval)

Matching is case-sensitive fnmatch, against label, in output order. A pattern that matches nothing raises LookupError with near-miss suggestions rather than returning an empty list — a silently empty selection is the exact failure this is meant to prevent. Pass allow_empty=True when probing for optional features.

If you’re building a triage feature_groups.definitions entry, resolve it through the helper rather than writing globs against physical names — an explicit glob targeting an inner fragment aborts the run on every truncated column with “matches no feature_groups.definitions glob”, which you cannot fix by widening the glob (you’d have to match an unpredictable hash):

definitions = {"cardiac": f.columns_matching("*frecuencia_cardiaca*")}

Querying the persisted manifest table instead? Use glob_to_like — it exists because _ is a literal in a glob but a single-character wildcard in SQL LIKE, and ~95% of real labels contain one, so a hand-written translation silently over-matches:

from featurizer.manifest import glob_to_like

like, escape = glob_to_like("*(inspections.kw_*")
cur.execute(
    'select column_name, feature_group from "sch"."feat_manifest" '
    "where label like %s escape %s",
    (like, escape),
)

Why not truncate tail-preservingly? Keeping the innermost (alias.variable fragment instead of the outer prefix sounds glob-friendlier. Measured, it is a regression — so this is settled as won’t do, not deferred to 2.x. Run python -m benchmarks.truncation_shapes (no database needed) over the sample config — 2,104 columns, 1,217 of them truncating:

scheme	variable lost	operator lost	ambiguous	worst group
head-keeping (today)	197	0	887	8
tail-keeping (proposed)	0	1171	1027	78
both-ends (26 + hash + 26)	0	0	311	6

ambiguous counts columns whose visible, hash-elided text duplicates another column’s. No scheme has a correctness bug — the hash keeps every column unique under all three; this is purely legibility.

Tail-keeping recovers every variable name by erasing the operator stack, which is what actually distinguishes sibling columns in a wide config: its worst case is 78 columns all rendering as …s.ABS(measurements.frecuencia_cardiaca)|interval=P2W)), every ABS/PCT_CHANGE_1/ROLLING_* × MAX/MEAN/MEDIAN/MIN/STDDEV/SUM collapsed into one string. It fixes the variable glob and breaks the operator glob.

A both-ends shape does dominate the status quo — and still isn’t worth it. It leaves 311 ambiguous columns, so you would keep needing the manifest anyway, while renaming 57.8% of columns and silently invalidating every downstream feature cache and column-matching config. The naming contract including 63-byte capping is frozen, so that rename also costs a major version and a deprecation cycle. Paying all that to go from “physical names are unparseable” to “physical names are slightly less unparseable” is a bad trade.

The real conclusion: 63 bytes cannot hold a compositional name, and every fixed-window shape just picks which half to blind you to. The manifest is not a workaround for a naming scheme we got wrong — it is the design. Treat physical column names as opaque handles. Full reasoning and reopen criteria live in .out-of-scope/tail-preserving-truncation.md; tests/test_truncation_shapes.py pins the ordering so the conclusion cannot silently invert.

A categorical one-hot column is missing, or has a value my data never contains

That’s by design. Featurizer builds the categorical vocabulary from the column’s PostgreSQL enum labels — it never scans the data to discover values. This makes the feature matrix split-blind: the same columns appear whether you featurize the train split, the test split, or a single row, so train/serve schemas can’t drift. A value present in your enum but absent from a given slice still gets its (all-zero) column; a value in your data but not the enum is a modeling error to fix upstream. See the categoricals notebook. Imputation of the resulting matrix is opt-in, not automatic.

column "entityid" does not exist — but the table has entityId

The column was created with quotes, so PostgreSQL stored its capitals. pandas.DataFrame.to_sql does this for every column; so does any DDL that writes "entityId". A config that declares entityId is read as PostgreSQL reads an unquoted identifier: folded to entityid, which is not the stored name.

Ask for the exact name by putting the quotes inside the YAML string:

variables:
  '"entityId"': {type: numeric}

The rule is in the configuration reference. A column created by unquoted DDL needs none of this: totalAmount in the config matches the stored totalamount.

Two events share a timestamp. Which one comes first?

Since 1.3.0 the order is fixed, and the same on every read. A window orders by the entity’s temporal index, then by the entity’s other identifier columns (its id first, then relationship keys and index-typed variables), then by every declared variable of the entity. So lag_1, a transition, a run length, an autocorrelation, a rolling window or an as-of lookup over two rows on the same date gives the same value whether the table was just loaded, clustered or vacuumed. Two rows that tie in all of that are the same row twice.

Before 1.3.0 two such rows came out in the physical order of the table, and the value of an order-dependent primitive could change without the data changing: issue #66 measured it on 36 columns of 1,437 of 3,000 entities. If you use the sequence, autocorrelation, difference or lag families on an engine older than 1.3.0, make the temporal index unique within a key, or upgrade. The row order every window and every as-of lookup uses is on the concepts page.

cosinor_amplitude_weekly is NULL for a series that has data

Every timestamp of that series falls on the same weekly phase: rows a whole number of weeks apart, which a date index on one weekday produces. The sine and cosine the amplitude regresses on are then constant, and before 1.3.0 the primitive divided by their rounding error and returned numbers around 1e15 (#67). It returns NULL when the basis has no spread. A series that sits on two or more phases gets a value.

My matrix has fewer rows than entities × dates

Your target declares a temporal_ix, and some of its rows are dated after some of your as-of dates. A row that does not exist yet at a date is not emitted under it: a store that opens in October has no row for an as-of date in July. Releases up to 1.2 emitted such rows, with empty aggregates. The planner logs a warning that names the target when this applies.

If you want every target row under every date, leave temporal_ix off the target, or left-join the matrix onto your own entities × dates table. The reasons are under the target’s read on the concepts page.

row is too big: size …, maximum size 8160 — but only with to_tables

A PostgreSQL heap row must fit one 8 KiB page (~8160 bytes). Fetching a 1,000+-column result with to_dataframe / to_arrow / to_parquet never hits this — result rows stream without page storage — but to_tables runs create table … as, and ~1,000 fixed-width 8-byte feature columns (bigint/float8, which TOAST cannot move out of line) overflow the page.

Since v1.0 to_tables pre-flights every column group with a row-width estimate (8 bytes per column + tuple header + null bitmap) and, when a group would exceed the ~8000-byte budget, automatically re-partitions into more, narrower tables — you get extra <stem>_group_<NNN> tables instead of the error, all still re-joinable on (as_of_date, id) and correctly tagged in the manifest. The estimate is a documented heuristic: text and wide numeric values are variable-width, so a pathological config could still trip PostgreSQL — if it does, lower the group width yourself by splitting your config, and please report it. See performance internals.

A generated query takes tens of seconds on a small table

That is PostgreSQL’s JIT compiler. A query whose estimated cost is over jit_above_cost (100,000 by default) has every expression of its target list compiled before the first row is read, and a generated query has hundreds of aggregate expressions. The compile time does not depend on the data: issue #53 measured 55.6 s for one query over five rows, and 0.1 s with jit = off.

to_dataframe, to_arrow, to_parquet and to_tables turn jit off around their own statements and put your value back afterwards, on your connection= as well. If you execute the SQL yourself (query, query_groups), turn it off in the same transaction:

set local jit = off;

The measurement on the three live databases is in the internals page.

Cannot yet materialize the oversized synth … as-of LATERAL join

A forward temporal relationship (temporal: {mode: as_of} pulling the most recent parent record) renders as a correlated LEFT JOIN LATERAL. When the same entity’s synth CTE also grows past the materialization threshold (issue #7 temp-table sharding), featurizer refuses rather than emit subtly-wrong SQL: flattening a correlated LATERAL into shards is feature work, deliberately out of scope for 1.0. The error is the boundary, and it names the two workarounds:

1. Narrow that entity’s primitive/interval breadth so its synth stays under the limit (fewer aggregations, transformers, or intervals on the entity carrying the as-of join), or
2. raise the as-of relationship to the target entity, where no materialization is needed.

See the temporal-joins section of the configuration reference.

Development & docs

Why do the tutorial notebooks show outputs but never execute in CI?

The docs site renders each notebook from its committed, executed outputs — the outputs that were validated against a live database — and never runs it during the build. The GitHub Pages workflow has no PostgreSQL, and re-executing would either fail or silently diverge. The committed outputs are the source of truth; a count-parity test guards the generated pages against drift.

How do I run the tests? Some are skipped

Tests are tiered via just:

just test-fast           # DB-free — runs anywhere
just db-up               # ephemeral postgres:16 container
just test-integration    # needs the database
just seed && just test-realistic   # realistic-dataset tier (integration + slow)

Integration tests skip automatically when no database is configured — that’s expected on a fresh checkout, not a failure.

Why does basedpyright ignore aggregations.py and transformations.py?

Those two modules define the primitive variants with heavy dynamic patterns (metaprogrammed classes, SQL-string templating) that the type checker can’t follow without noise. They’re listed in pyrightconfig.json’s ignore set on purpose; the rest of the codebase type-checks under standard mode.