Aligning an External Cohort

Every example in this section continues with the well-known Adult Income dataset [1] used throughout the rest of this documentation, and with the Logistic Regression, Decision Tree, and Random Forest classifiers trained on it using the model_tuner library [2]. Click here to view the corresponding codebase for this workflow.

The external cohort is the American Community Survey (ACS) extract distributed by folktables [3], whose ACSIncome task was constructed specifically as a modern successor to Adult Income. It predicts the same outcome, above or below a $50,000 income threshold, from broadly the same kinds of features, which makes it a natural external validation set for a model trained on the 1994 census extract. It is also a realistic one: nothing about its column names, codings, or units matches what the Adult Income model expects.

That mismatch is the subject of this page.

Why Not a Mapping Loop

Applying a fitted model to a cohort it was not trained on is rarely a matter of calling predict. The external frame arrives in its own naming convention, collects some variables the model never saw, and omits others the model requires. The usual response is a hand-written mapping loop, which works until it does not, and it fails silently in three distinct ways:

A source column is absent. The loop reads a column that is not in the incoming frame and raises far from the line that caused it, or worse, was wrapped in a try and skipped.
A destination is misspelled. ext["capital_gain"] = ... when the model expects capital-gain. Pandas creates the new column without complaint, the model never receives the value, and nothing raises.
An expected feature is never touched. The column exists because the frame was pre-allocated, stays entirely NaN, and the model consumes it as missing for every row.

Each of these produces predictions. None of them produces an error. The resulting AUC looks plausible, which is precisely the problem.

align_features builds the aligned frame declaratively and checks all three conditions before returning anything. Once the frame is aligned, the Per-Model Ground Truth support in the evaluation functions lets the ACS cohort be scored alongside the Adult Income test set in one call.

Discovering the Feature Contract

Before anything can be aligned, the model has to be asked what it wants. get_expected_features resolves the ordered feature names a fitted estimator expects at predict time, across the shapes a model actually arrives in.

get_expected_features(model)

Parameters:

model (estimator, Pipeline, or wrapper) – Any fitted estimator, pipeline, or wrapper.

Returns:

Ordered list of feature names the model expects at predict time.

Return type:

list[str]

Raises:

FeatureContractError –

If no name information is reachable from the object or any of its wrapper attributes.
If the wrapper chain is cyclic.

Important

For a fitted ColumnTransformer, the raw pre-transform input columns are reconstructed from transformers_. That is what you build an external frame against; the estimator’s own names are post-transform and carry num__ / cat__ prefixes.
For a Pipeline, feature_names_in_ is preferred (the raw input to the pipeline). The first step is consulted only when the pipeline never recorded it.
Wrapper objects are descended automatically. The chain covers best_estimator_, estimator_, estimator, regressor_, classifier_, base_estimator, calibrated_classifiers_, model, _model, pipeline, and test_model, which between them cover model_tuner.Model [2], GridSearchCV, calibration wrappers, TransformedTargetRegressor, and most hand-rolled wrappers.

Notes

Resolution Order:
- ColumnTransformer raw input columns, when transformers_ is present.
- Pipeline: feature_names_in_, then the first step.
- Bare estimator: feature_names_in_, feature_name_, feature_names_.
- XGBoost sklearn wrapper: get_booster().feature_names.
- Native xgboost.Booster: feature_names.
- LightGBM Booster: feature_name().
- statsmodels results: exog_names.
- Wrapper attributes, recursively.
When It Raises:
- A model that was fitted on a bare NumPy array records no names, so nothing is reachable. Pass expected=[...] to align_features explicitly in that case.

Asking the Adult Income Random Forest what it expects returns the feature contract the external frame has to satisfy:

from model_metrics import get_expected_features

get_expected_features(model_rf)

Output

['age',
'fnlwgt',
'education-num',
'capital-gain',
'capital-loss',
'hours-per-week']

Building the Aligned Frame

align_features(df, model=None, col_map=None, derived=None, expected=None, *, on_unmapped='warn', on_all_nan='warn', dtype='float', copy_passthrough=False, fill=None, report_fill=True)

Parameters:

df (pandas.DataFrame) – The incoming external frame, in its own naming convention.
model (estimator, optional) – Any fitted estimator. Used to resolve expected when that is not given directly.
col_map (dict, optional) – Mapping of {source_column: destination}, or {source_column: [dest_a, dest_b]} when one source feeds several model features.
derived (dict, optional) – Mapping of {destination: callable(df) -> array} or {destination: array-like}, for features that are computed rather than renamed.
expected (iterable of str, optional) – Explicit ordered feature list. Overrides whatever the model reports, and is the escape hatch when the model carries no names.
on_unmapped (str, optional) – What to do about expected features that no mapping ever touched and that were not filled. One of "raise", "warn", or "ignore". Defaults to "warn".
on_all_nan (str, optional) – What to do about features that were mapped but came out entirely NaN, which usually means the source column exists in the external frame but was never populated. One of "raise", "warn", or "ignore". Defaults to "warn".
dtype (str, optional) – Cast applied to the result, or None to leave dtypes alone. Defaults to "float".
copy_passthrough (bool, optional) – Also carry over any column of df whose name already matches an expected feature, without needing an entry in col_map. Defaults to False.
fill (scalar, dict, or callable, optional) – Value or values for expected features that no mapping touched, meaning variables the external cohort does not collect at all. Accepts a scalar (same value for every unmapped column), a dict {name: value} (per column; columns absent from the dict stay NaN), or a callable name -> value (returning None leaves that column NaN). Default None leaves them NaN, which delegates to the model’s own missing-value handling and asserts nothing. See External Validation Example 3: Choosing a Fill.
report_fill (bool, optional) – Emit a warning listing exactly which columns were filled with what, so the assumption is visible in logs and reproducible in a write-up. Defaults to True.

Returns:

A DataFrame containing exactly expected, in order, indexed like df.

Return type:

pandas.DataFrame

Raises:

FeatureContractError –
- If a source column named in col_map is absent from df.
- If a destination is named that the model never asked for.
- If a destination would be written more than once.
- If expected contains duplicate names.
- If fill is a dict naming a column that was mapped or is not a model feature.
- If on_unmapped="raise" and any expected feature was never mapped.
- If on_all_nan="raise" and any mapped feature is entirely NaN.
ValueError –
- If neither model nor expected is provided.
- If a derived column has a length other than len(df).
- If on_unmapped or on_all_nan is not one of the three recognized modes.

Important

Supply either model (from which the contract is read) or expected (the contract stated directly). expected wins when both are given.
The three failure modes are checked before any data is written, so a misspelled destination raises rather than producing a frame that scores.
derived is for columns the external cohort must compute rather than rename, such as a unit conversion or a code crosswalk. Anything a col_map entry can express should stay in col_map, where the collision check applies.
The returned frame is ordered to match expected. This matters for plain scikit-learn estimators, which match features by position rather than by name and will otherwise silently misread a correctly named frame.

Notes

The Three Checks:
- Source absent: every key of col_map must exist in df. Raises with the sorted list of offenders.
- Unknown destination: every destination, from col_map and derived alike, must appear in expected. The error prints the full expected list, which is usually enough to spot the typo.
- Never mapped: any expected feature that no mapping touched is reported through on_unmapped. This is the one that is a warning by default, because it is often legitimate.
Collisions:
- Two sources writing the same destination raises rather than letting the second overwrite the first. This catches a copy-paste error in a long mapping dict, where the same destination is typed twice with different sources.
Genuine Missingness vs Structural Absence:
- A column that was mapped but is NaN for some rows is real per-record missingness, and is left alone.
- A column that was never mapped is structurally absent: the external cohort does not collect that variable for anybody. Only these are eligible for fill, which is why a fill dict naming a mapped column raises.

External Validation Example 1: A Straight Rename

Several ACS columns are the same quantity the Adult Income model already knows, recorded under the census PUMS variable name. AGEP is age, WKHP is hours-per-week. Those need nothing more than a rename.

copy_passthrough=True additionally picks up any column that already matches an expected feature by name, so only the genuinely different names need an entry.

from model_metrics import align_features

ext = align_features(
    X_acs,
    model=model_rf,
    col_map={
        "AGEP": "age",
        "WKHP": "hours-per-week",
    },
    copy_passthrough=True,
)

y_prob_ext = model_rf.predict_proba(ext)[:, 1]

Output

/tmp/ipykernel_212836/2168851049.py:3: UserWarning:

4 expected feature(s) never mapped, left all-NaN: ['fnlwgt', 'education-num', 'capital-gain', 'capital-loss']

Any expected feature that neither the mapping nor the passthrough covered is reported by name in a warning, so the gap is visible before the predictions are used rather than after.

External Validation Example 2: Derived Columns

Most of the ACS columns are not straight renames. SCHL records educational attainment on a 1 to 24 scale, while the Adult Income model was fitted on education-num, which runs 1 to 16. Passing SCHL through unchanged would hand the model values six points past anything it saw in training, and every master’s-and-above record would collapse into the same terminal leaves as a doctorate.

derived takes a callable applied to the whole incoming frame, so the crosswalk lives next to the mapping it belongs to rather than in a preprocessing cell three notebooks away.

# ACS SCHL (1-24) crosswalked to Adult Income education-num (1-16)
SCHL_TO_EDNUM = {
    1: 1, 2: 1, 3: 1,           # none / preschool / kindergarten
    4: 2, 5: 2, 6: 2, 7: 2,     # grades 1-4
    8: 3, 9: 3,                 # grades 5-6
    10: 4, 11: 4,               # grades 7-8
    12: 5, 13: 6, 14: 7,        # grades 9, 10, 11
    15: 8,                      # 12th, no diploma
    16: 9, 17: 9,               # HS diploma, GED
    18: 10, 19: 10,             # some college
    20: 12,                     # Associate's
    21: 13, 22: 14, 23: 15, 24: 16,
}

col_map={
    "AGEP": "age",
    "WKHP": "hours-per-week",
}

derived={
    "education-num": lambda d: d["SCHL"].map(SCHL_TO_EDNUM),
}

ext = align_features(
    X_acs,
    col_map=col_map,
    derived=derived,
    model=model_rf,

)

Output

/tmp/ipykernel_212836/3837904464.py:15: UserWarning:

3 expected feature(s) never mapped, left all-NaN: ['fnlwgt', 'capital-gain', 'capital-loss']

Note

A derived callable receives the incoming frame, not the frame under construction, so it can reference any external column regardless of whether that column is mapped. Its return value is length-checked against df.

Important

A crosswalk is a modelling decision in its own right, and the places where it is not one-to-one deserve to be stated. ACS records a single Associate’s degree code, while Adult Income splits Assoc-voc (11) from Assoc-acdm (12); the mapping above sends all of them to 12. Choosing 11 instead is equally defensible and will move the predictions slightly. Write the choice down rather than leaving it implicit in a dict.

External Validation Example 3: Choosing a Fill

Some model features have no counterpart in the external cohort at all. The Adult Income model was fitted on capital-gain and capital-loss; ACS PUMS records no realized-capital-gains variable, so nothing in the incoming frame can supply them. What to do about that is the most consequential decision in the whole alignment, and it is a modelling decision rather than a formatting one.

The default, fill=None, leaves the column NaN. For a model that handles missing values natively (CatBoost, XGBoost, HistGradientBoosting) this delegates to the model’s own learned handling and asserts nothing about the external records. It is the honest default and usually the right one.

Filling asserts something:

ext = align_features(
    X_acs,
    model=model_rf,
    col_map=col_map,
    derived=derived,
    fill={"capital-gain": 0, "capital-loss": 0},
)

Output

/home/lshpaner/Python_Projects/model_metrics/metrics_venv/lib/python3.12/site-packages/IPython/core/interactiveshell.py:3747: UserWarning:

2 unmapped column(s) filled rather than left missing: capital-gain=0, capital-loss=0. This asserts a value the external cohort never measured; record it wherever the results are reported.

/tmp/ipykernel_212836/629044355.py:1: UserWarning:

1 expected feature(s) never mapped, left all-NaN: ['fnlwgt']

Filling those with 0 asserts “nobody in this cohort had any capital gain or loss”. Zero happens to be the modal value in Adult Income for both columns, so the fill looks harmless. It is not: the nonzero tail of capital-gain is almost purely >50K, which is why it typically ranks among the model’s strongest features. Zeroing it removes the model’s best positive signal, the external predictions come back systematically low, and they come back low in a way that correlates with race and sex, because capital income does. A fairness audit will pick that up and attribute it to the model.

With report_fill=True (the default), a warning names every column and its value, so the assumption appears in the notebook log:

UserWarning: 2 unmapped column(s) filled rather than left missing:
capital-gain=0, capital-loss=0. This asserts a value the external cohort
never measured; record it wherever the results are reported.

Important

Fix the fill in advance, from what is known about the cohort. Selecting it by comparing downstream AUC is fitting a preprocessing choice to the evaluation set, and the resulting performance estimate is optimistically biased in exactly the way external validation is supposed to guard against.

Notes

Scalar, dict, or callable:
- A scalar fills every unmapped column with the same value. Rarely what you want unless the features are homogeneous, for example a block of one-hot indicators where 0 means “level not set”.
- A dict fills per column and leaves anything absent from the dict as NaN, which is the right shape when some absent features have a defensible default and others do not.
- A callable name -> value suits a regular naming convention; returning None for a name leaves that column NaN.
Interaction with the model:
- A model that cannot consume NaN, such as a plain RandomForestClassifier, forces the question, since leaving the column missing is not an option. That constraint is a property of the estimator, not evidence that filling is correct.
What not to fill:
- Features whose absence is the whole finding. When the external cohort does not collect what the model leans on most, a fill produces a number where the honest answer is that this model cannot be validated on this cohort as specified. Refitting on the shared feature set gives a worse model whose degradation actually means something.
A note on ``fnlwgt``:
- Adult Income models are frequently fitted on fnlwgt, a CPS survey weight that is a function of the sampling design and carries no information about the individual. It has no external counterpart worth constructing, and its presence in the feature set is usually an artifact of feeding every numeric column to the estimator. It is worth removing from the model rather than filling.

External Validation Example 4: Failing Loudly

For a pipeline run rather than exploratory work, promote both reports to exceptions so a silently degraded frame cannot reach the evaluation step.

ext = align_features(
    X_acs,
    model=model_rf,
    col_map=mapping,
    derived=derived,
    on_unmapped="raise",
    on_all_nan="raise",
)

Output

---------------------------------------------------------------------------
FeatureContractError                      Traceback (most recent call last)
Cell In[25], line 1
----> 1 ext = align_features(
   X_acs,
   model=model_rf,
   col_map=col_map,

File ~/Python_Projects/model_metrics/src/model_metrics/align_features.py:291, in align_features(df, model, col_map, derived, expected, on_unmapped, on_all_nan, dtype, copy_passthrough, fill, report_fill)
still_nan = [c for c in unmapped if c not in filled]
# --- report gaps ------------------------------------------------------- #
--> 291 _report(
   on_unmapped,
   f"{len(still_nan)} expected feature(s) never mapped, left all-NaN: {still_nan}",
   still_nan,
)
all_nan = [
   c for c in expected if c not in unmapped and c not in filled and ext[c].isna().all()
]
_report(
   on_all_nan,
   f"{len(all_nan)} mapped feature(s) are entirely NaN after alignment: {all_nan}",
   all_nan,
)

File ~/Python_Projects/model_metrics/src/model_metrics/align_features.py:358, in _report(mode, msg, payload)
   return
if mode == "raise":
--> 358     raise FeatureContractError(msg)
if mode == "warn":
   warnings.warn(msg, stacklevel=3)

FeatureContractError: "3 expected feature(s) never mapped, left all-NaN: ['fnlwgt', 'capital-gain', 'capital-loss']"

on_unmapped="raise" requires every model feature to be accounted for, by a mapping, a derivation, or an explicit fill. on_all_nan="raise" catches the subtler case where a mapping is syntactically valid but the source column turns out to be empty in this cohort.

External Validation Example 5: Scoring Both Cohorts Together

Once the ACS frame is aligned, the Adult Income test set and the external cohort can be passed to any evaluation function as a pair, using the per-model list form of y described in Per-Model Ground Truth. The two cohorts have different row counts and share no observations, which is exactly what that form exists to express.

import numpy as np
from model_metrics import (
    combine_plots,
    show_roc_curve,
    show_pr_curve,
    show_calibration_curve,
)

p_int = model_rf.predict_proba(X_test)[:, 1]
p_ext = model_rf.predict_proba(ext)[:, 1]

INT_TITLE = "Adult Income 1994 (internal)"
EXT_TITLE = "ACS (external)"

SHARED = {
    "y_prob": [p_int, p_ext],
    "y": [np.asarray(y_test).ravel(), y_acs],
    "model_title": [INT_TITLE, EXT_TITLE],
    "overlay": True,
    "curve_kwgs": {
        INT_TITLE: {"color": "black", "linewidth": 1.5},
        EXT_TITLE: {"color": "#C1440E", "linewidth": 1.5},
    },
}

combine_plots(
    plot_calls=[
        (show_roc_curve, {**SHARED, "title": "Discrimination"}),
        (show_pr_curve, {**SHARED, "legend_metric": "ap",
                         "title": "Precision-Recall"}),
        (show_calibration_curve, {**SHARED, "bins": 10,
                                  "title": "Calibration"}),
    ],
    n_cols=3,
    n_rows=1,
    figsize=(19, 5.5),
    suptitle="Adult Income classifier: internal vs external validation",
)

Output

Classifiers AUROC with Youden Operating Point

Important

Read the three panels differently. AUC ROC is invariant to the positive rate and transfers cleanly. Average Precision and the Brier score both move with prevalence, so a worse external value does not by itself indicate a worse model. This matters more than usual here: the positive rate in Adult Income is roughly 24%, while ACSIncome sits closer to 40% depending on the state and survey year, because the same nominal $50,000 threshold is a very different threshold in 2018 dollars than in 1994 dollars. The calibration panel is where that shift shows most clearly.

Exceptions

exception FeatureContractError: Raised when a mapping does not satisfy a model’s expected feature set. Subclasses KeyError, so existing except KeyError handlers continue to catch it, while code that wants to distinguish a contract violation from an ordinary missing key can catch the specific type.

from model_metrics import align_features, FeatureContractError

try:
    ext = align_features(
        X_acs,
        model=model_rf,
        col_map=mapping,
        on_unmapped="raise",
    )
except FeatureContractError as exc:
    print(f"External frame does not satisfy the model contract:\n{exc}")