Changelog

Version 0.0.6a5

Updated

• funnel_plot.py: risk-adjusted funnel plots for any grouping unit and binary outcome. Each group’s observed event count is compared against the count expected from its case mix and plotted against volume, with exact Poisson control limits that widen for low-volume groups. Returns a per-group frame with observed, expected, n, oe, and an outlier flag.

• group_oe, poisson_limits, estimate_phi: public helpers for the aggregation, the control limits, and the Spiegelhalter overdispersion factor.

• Overdispersion: overdispersion=True inflates the limits by sqrt(phi). The estimated phi is appended to the title, custom or default, and stored on the returned frame as g.attrs["phi"].

• Separate drawing and flagging thresholds: limit_alphas controls which control bands are drawn, flag_alpha controls which points are flagged against their own expected count.

• curve_kwgs and point_kwgs: dict-based styling for the reference line, control bands, and inlier/outlier markers, merged over the defaults and validated against unknown keys.

• Standard plotting inputs: title, labels, axis limits, fonts, gridlines, legend, save, and ax for composition into combine_plots, all routed through the shared plot_utils helpers.

• test_funnel_plot.py: 53 tests covering the aggregation and limit math, flagging behavior, the overdispersion path, the style-dict contract, and the save and axes plumbing.

Version 0.0.6a4

Added

• funnel_plot.py: new module for risk-adjusted funnel plots. Each group’s observed event count is compared against the count expected from its case mix and plotted against group volume, with exact Poisson control limits that widen for low-volume groups. Works for any grouping unit (center, surgeon, region, school, branch) and any binary outcome, given one row per case with a group column, a 0/1 outcome column and a predicted probability column.

  Public functions:

  • funnel_plot(df, group_col, ...): draws the plot and returns the per-group frame g with observed, expected, n, oe and an outlier flag, indexed by group_col. Flagged groups are g[g["outlier"]].

  • group_oe(df, group_col, y_col, p_col, min_volume=1): the aggregation step, exposed so callers can get g without plotting.

  • poisson_limits(expected_grid, alpha, phi=1.0): exact Poisson control limits for an O/E ratio, as multiples of the target O/E = 1.

  • estimate_phi(g): overdispersion factor from the dispersion of standardized O/E residuals (Spiegelhalter 2005).

• Overdispersion: overdispersion=True inflates the control limits by sqrt(phi), widening them when there is genuine between-group variation beyond chance. The estimated phi appears in the default title, and estimate_phi is public so it can be inspected before deciding.

• Separate drawing and flagging thresholds: limit_alphas controls which control bands are drawn, flag_alpha controls which points are flagged. Flagging tests each group against its own expected count rather than against the smoothed band.

• min_volume: drops small groups before any computation, keeping a handful of cases from dominating the O/E axis.

• annotate_outliers: labels flagged points with their group id. Off by default, since the returned frame is easier to read.

• Standard plotting inputs: title, xlabel, ylabel, xlim, ylim, figsize, label_fontsize, tick_fontsize, title_fontsize, text_wrap, gridlines, grid_kwgs, show_legend, legend_loc, legend_fontsize, save_plot, image_path_png, image_path_svg, image_filename, dpi, show_plot, ax. Saving, title, axis limits and legend route through the shared save_plot_images, apply_plot_title, apply_axis_limits and apply_legend helpers in plot_utils.

• curve_kwgs and point_kwgs: dict-based styling. curve_kwgs takes "target" (the O/E = 1 reference line) and "limits" (the control bands, either one dict applied to every band or a list paired positionally with limit_alphas). point_kwgs takes "inlier" and "outlier". Both merge over the defaults, so a partial dict changes one property and leaves the rest alone, and values pass straight through to ax.plot and ax.scatter.

• ax support: passing an existing axis skips figure sizing, tight_layout, saving and plt.show(), leaving those to the caller. Composes into subplot grids and into combine_plots.

• Example notebook covering the default call, reading the returned frame, overdispersion, control limits vs flagging, styling, both save routes, side-by-side outcomes via ax, and the O/E distribution.

Full Changelog: https://github.com/lshpaner/model_metrics/compare/0.0.5a12…0.0.6a4

Version 0.0.6a3

MLflow-backed model loading with no hard-coded experiment ids, run ids, or paths. Reads a local MLflow FileStore tree directly from disk; mlflow itself is not a dependency.

Two problems it solves that the plain MLflow client does not:

• Several tracking roots may live under one project (mlruns/preprocessing, mlruns/models, …). All are indexed, not just one.

• The recorded artifact_location often points into a directory that no longer exists, because mlruns/ trees get copied between machines and projects. Every MLflow artifact API then resolves to a missing path. Artifacts are instead rebased onto the store root actually found on disk.

Configuration. All optional, settable by environment variable or by calling configure() before first use:

• MODEL_REGISTRY_ROOT: directory to walk. Defaults to the working directory, walking up to the repository boundary to find an mlruns/ tree, so the registry works when called from a notebooks/ subdirectory. The walk stops at .git, so an unrelated mlruns/ higher up the filesystem is never picked up silently.

• MODEL_REGISTRY_TARGET: outcome token stripped from artifact folder names. Artifact folders are conventionally named <algo>_<TARGET>, so setting this turns rf_income into rf and you group by algorithm rather than by outcome. The token is stripped wherever it appears, so rf_income_no_sex becomes rf_no_sex and ablation variants stay distinguishable.

• MODEL_REGISTRY_STORES: comma-separated store prefixes eligible to win a metric comparison.

• MODEL_REGISTRY_FILE: artifact filename, default model.pkl.

• MODEL_REGISTRY_TOLERANT: set to 0 to make version mismatches raise instead of warn.

Naming. Several runs may share one artifact folder, so the folder gives the algo (lr, dt, rf) and the MLflow run name gives the variant (rf_orig, rf_smote, rf_orig_no_sex). variant is the addressable key; algo is what you group by.

Stores. The index is always global. available(), rank(), load(), load_all() and variants() see every model on disk, so nothing is hidden. STORES constrains only best_per_algo() and load_best_per_algo(), so a superseded store cannot win a comparison it was never meant to be in. Prefixes are project-relative and matched on whole path segments, so mlruns/models matches mlruns/models/12345 but not mlruns/models_old/12345. A substring test would wrongly accept the latter, which is the exact confusion this setting exists to prevent. store_summary() shows every store on disk with its model count and whether it is currently eligible, so exclusions are visible before a result is trusted.

Selection policy. The default is "newest" (latest start_time wins a tie). Selecting by best test metric is available but off by default, because picking the max-metric run is model selection on the evaluation set, and the winning score is optimistically biased as a result. load_best_per_algo() carries that warning in its docstring: fine for exploration, but pin the variant by name for anything you report.

select_on_validation() and load_selected() provide the defensible path, which is to choose champions on validation data and then report test metrics for those winners only. score_candidates() accepts a single (X, y) pair, a dict keyed by algo or variant for models that need different feature spaces, or a callable taking the entry, so ablation variants can be scored against their own matrices in the same sweep. Failures are captured per row rather than aborting the sweep, so one unloadable artifact doesn’t cost you the other twenty scores.

Cross-version loading. Artifacts are often written under a different scikit-learn than the one installed, and pickles reference private sklearn classes by module path. Those classes get renamed or removed between versions, so a plain load dies with something like AttributeError: Can't get attribute '_RemainderColsList' on <module 'sklearn.compose._column_transformer'>. Two layers handle the drift:

Tolerant unpickling. Missing sklearn internals are synthesized rather than raising. Every load goes through _load_model(), which tries the configured loader, retries with the known shims installed, then falls back to a _TolerantUnpickler that synthesizes any remaining sklearn class on demand. Non-sklearn missing classes still raise, since synthesizing those would be guessing.

Post-load fitted-state repair. Shimming fixes unpickling but not fitted state: newer transform() code reads instance attributes that an older fit() never wrote. repair_estimator() walks the whole pipeline graph, covering steps, transformers, nested estimators, calibrated classifiers, and plain instance attributes, and restores those attributes to the value that reproduces the original training-version behavior rather than the current version’s. Two repairs ship:

• SimpleImputer._fill_dtype, set to statistics_.dtype, which makes the newer astype() call a no-op and matches versions that did not cast at all. This one is gated on a probe of the installed sklearn: the attribute was introduced partway through the 1.x line, so on versions that never had it, its absence in a fitted imputer is normal and repairing anyway would fill repaired() with noise and warn about perfectly healthy models.

• ColumnTransformer remainder columns. When the pickle references _RemainderColsList and the installed sklearn no longer defines it, the synthesized stand-in unpickles as an empty list. Under remainder="passthrough" that silently drops every passthrough column, and the failure surfaces far downstream as ValueError: Feature shape mismatch, expected: 25, got 15. The remainder set is fully determined by fitted state, so it is rebuilt exactly as sklearn computes it at fit time, sorted(set(range(n_features_in_)) - union(explicit transformer cols)), and written to all three places that must agree: _transformer_to_input_indices, _remainder, and transformers_. Repairing _remainder alone is not enough, because transform() reads transformers_ when the transformer is fitted.

Both layers warn on use, and shimmed() / repaired() report what a given process had to touch. None of this makes cross-version loading supported; it makes it survivable long enough to verify.

Verification. verify_entry() recomputes ROC AUC, average precision, and Brier score for a loaded model and compares them against what was logged. This is the check that makes a cross-version load defensible: if the recomputed score matches the logged one, the estimator state survived whatever shimming and repair was needed. If it doesn’t, don’t use the model.

The split argument must correspond to the data you pass. Comparing a logged validation score against metrics recomputed on the test set will always disagree and means nothing. Default tol is 1e-3 rather than something tighter, because MLflow often stores metrics rounded to three decimals. If no logged key matches the requested split, it warns rather than silently returning an empty frame. verify_all() runs the check across every best_per_algo() champion in one call, reporting one pass/fail per model plus the worst absolute delta observed, so a bad load cannot hide behind an aggregate.

Optional dependencies. model_tuner’s loadObjects is used when present, so anything it does at save time is mirrored at load time; otherwise it falls back to joblib (which reads plain pickles too), then to pickle itself. backend() reports which is active. PyYAML is used when present, with a built-in flat-scalar parser covering every field the module actually reads (experiment_id, name, run_id, run_uuid, run_name, lifecycle_stage, start_time). MLflow’s meta.yaml files are flat key/value documents, so the fallback is not a compromise.

Diagnostics. diagnose() prints the resolved configuration, active backend, YAML parser, shims and repairs applied so far, the store summary, and the full inventory; it also runs as __main__. is_available() gates registry-dependent code paths without raising. Failure messages distinguish “search root does not exist” from “no tracking store found anywhere under it” from “stores found but no artifact under that filename”, each naming the specific setting to change, and including a note that remote tracking servers and SQL backends are out of scope and need the mlflow client instead.

Public surface. available(), rank(), resolve(), load() / load_model(), load_all(), variants(), algos(), experiments(), metric_names(), resolve_metric(), best_per_algo(), load_best_per_algo(), score_candidates(), select_on_validation(), load_selected(), verify_entry(), verify_all(), store_summary(), configure(), set_stores(), stores(), refresh(), backend(), shimmed(), repaired(), is_available(), diagnose().

resolve_metric() deserves a note: it maps a loose metric name onto the key actually logged, normalizing spaces and hyphens to underscores so average_precision finds test_average_precision, valid ap, or test Average Precision. Without that normalization a raw substring test never matches and the default metric argument to best_per_algo() raises on a perfectly ordinary store. It raises on an ambiguous match rather than silently ranking on the wrong split.

New helper in metrics_utils.py. Resolves the ground-truth array for a given model index, so a list of y arrays can be passed alongside a list of y_prob arrays.

def _y_for_model(y, idx, n_models, n_expected, name):
    ...

A list or tuple whose length matches the model count, with every element sized, is treated as per-model ground truth. Anything else is treated as a single shared y used by every model, preserving existing behavior. The n_models > 1 guard prevents a one-model call with a plain Python list of labels from being sliced by index.

The helper also raises on a length mismatch between y and the prediction array, naming the model:

ValueError: Length mismatch for ACS CA 2018 (external): y has 195663 rows, predictions have 9769.

Motivation: external validation. Comparing a model on its internal test set against an independent cohort means two arrays of different lengths with no shared observations. Previously y was consumed whole by every curve while y_prob was indexed per model, so overlaying two cohorts either raised an opaque numpy error (setting an array element with a sequence ... detected shape was (2,) + inhomogeneous part) or silently produced a plausible but meaningless figure.

Wired into the X is None branch of summarize_model_performance (classification path), show_confusion_matrix, show_roc_curve, show_pr_curve, show_lift_chart, show_gain_chart, show_calibration_curve, plot_threshold_metrics, and show_residual_diagnostics.

New parameters:

• model_colors: colors used per model in a multi-model overlay, cycled if shorter than the model count.

• metric_colors: metric-to-color mapping used when color encodes the metric. Unspecified keys keep their defaults.

• metric_linestyles: metric-to-linestyle mapping used when color encodes the model.

• legend_loc (default "bottom"): previously hardcoded.

• legend_ncol (default 3): previously hardcoded.

• legend_bbox_to_anchor: explicit anchor override. When omitted, the anchor now scales with the number of legend rows so multi-model overlays clear the x-axis label automatically.

Curve colors were hardcoded (red, green, blue, purple) with no way to override them. In overlay mode every model drew the same four colors, making the cohorts indistinguishable.

Color now encodes whichever dimension is ambiguous. With a single curve set, meaning a single plot, subplots, or a one-model overlay, color encodes the metric and matches the previous red/green/blue/purple scheme. In a multi-model overlay, color encodes the model and each metric gets a distinct linestyle, so two cohorts on one axes stay separable. Both mappings are overridable.

curve_kwgs previously accepted only a flat dict applied to every curve, which could not express per-model color. It now also accepts a dict keyed by model title, or a list aligned with the models, matching the convention in the rest of the show_* family.

The flat form is detected by its values not being dicts, so the legacy {"linestyle": "-", "linewidth": 1} form keeps working. Note that routing this through normalize_curve_styles would have silently resolved the flat form to empty per-model styles, since that helper assumes a dict is keyed by model title.

When no estimator is supplied, the number of models is now taken from the number of prediction arrays passed:

if model is None:
    if y_prob is not None:
        models = [None] * len(y_prob)
    elif y_pred is not None:
        models = [None] * len(y_pred)
    else:
        models = [None]

Previously [model] collapsed to a single entry, so passing a list of probability arrays silently evaluated only the first and returned a one-column table.

y_prob and y_pred normalization was also tightened: a bare list of scalars is now wrapped as one model’s predictions before any length is read off it, rather than being mistaken for a list of models.

In overlay mode, apply_legend() fired at the end of _plot_single while the model-name prefixing loop ran after that call returned. Each legend rebuild therefore captured the previous model already renamed and the current one still bare, and the final model never got prefixed at all.

Labels are now set at draw time and the prefixing loop is removed. The legend is applied once after the model loop completes. _plot_single returns its legend-entry count so the caller can size the legend after all models are drawn.

axvline(x=0.5) and the model-threshold line lived inside _plot_single, so N models produced N identical verticals and N duplicate legend entries.

Reference lines are properties of the axes rather than of any one model, so in overlay mode they are now emitted on the first pass only. A model_threshold shared by every model is drawn once; distinct per-model thresholds are each drawn and labeled with their model name.

Now takes idx so it can resolve per-model color and decide whether to draw reference lines. All three call sites (overlay, subplots, single plot) pass it.

• New page: External Validation, documenting align_features, get_expected_features, and FeatureContractError. Worked examples use the Adult Income models against an ACS cohort, covering straight renames, the SCHL to education-num crosswalk, and the fill decision for capital-gain / capital-loss, which ACS does not record.

• New page: Model Registry, covering all 28 public functions grouped by task: configuration, naming, discovery, stores, selection, metric-based selection, validation-set selection, cross-version loading, verification, and the ModelEntry record.

• New section in Performance Assessment: Per-Model Ground Truth, referenced from all nine functions that accept the per-model y list.

• resid_diagnostics_to_dataframe now has a function directive; it was used in Residual Diagnostics Example 4 but never documented.

• combine_plots: height_ratios added to the documented signature. It was already used in Combined Plot Grids Example 2 but absent from the parameter list. The hspace / wspace descriptions were corrected: both switch the figure to constrained_layout, which supersedes tight_layout.

• overlap_crosstab: decimal_places added to the documented signature. It was already used in Example 3 but absent from the parameter list.

• index.rst: fixed a stray backtick in the tqdm prerequisite that rendered as tqdm`` on the built page.

The legend_loc="bottom" branch of apply_legend calls fig.set_figheight(current_height + 2) on first use, guarded by a _resized_for_bottom_legend flag. Inside combine_plots this grabs the shared figure and grows the entire grid, not just the panel that asked for a bottom legend. Panels using a bottom legend end up sitting well below their axes.

Workaround: pass legend_bbox_to_anchor explicitly to bypass the row-count anchor math, or set fig._resized_for_bottom_legend = True before the call.

Proposed fix, not yet applied:

if len(fig.axes) <= 1 and not hasattr(fig, "_resized_for_bottom_legend"):
    ...

This keeps standalone behavior and makes the resize a no-op inside combine_plots, which owns its own layout.

The if overlay: block sets labels and applies the legend but never draws a baseline. show_roc_curve and show_calibration_curve both draw their reference lines in the equivalent block.

This matters most for cross-cohort comparison, where the positive rate differs between curves. A PR curve’s chance level is the positive rate, so without a per-curve baseline the cohort with the lower prevalence looks worse than it is. A per-curve axhline at y_true.mean() would fix it.

Version 0.0.6a2

Added

Feature grouping (all three feature-selection functions)

• feature_groups and column_transformer parameters on show_cumulative_feature_importance, show_feature_pareto, and show_cumulative_feature_performance. Rolls expanded columns (one-hot levels) up into their parent variable so a categorical’s importance is summed rather than fragmented across its indicators. Accepts a dict {column: parent}, a callable, or "auto" to infer from a fitted ColumnTransformer.

• In the performance sweep, grouping redefines n_features to count variables rather than transformed columns: blanking a group NaNs all its member columns together, and the returned feature list holds parent names. This makes k correspond to inputs a user would actually have to collect.

• Four new helpers in feature_selection_utils.py: _infer_groups_from_column_transformer, _resolve_feature_groups, _group_importance_df, _expand_groups. Group inference reads ct.transformers_ and matches output names against each branch’s input columns by longest prefix, so names containing spaces or underscores (e.g. cat__CKD Stage_CKD Stage 4) resolve correctly without string splitting.

Scores table

• show_cumulative_feature_performance now returns a features column in the per-k scores table (model mode), giving the ordered top-k feature names for each row. The exact set behind any score can be read off directly, e.g. df.loc[df["ROC AUC"] >= 0.84, "features"].iloc[0].

• New include_retained_pct parameter adds a "<metric> (% of full)" column per metric, expressing each score as a percentage of its full-model value. Lower-is-better metrics (Brier, MAE, RMSE, MSE) are inverted so 100% always means “matches the full-feature model”.

Retain controls

• New retain parameter accepts a dict coupling metric and threshold in one place, e.g. retain={"average_precision": 0.98}. Supersedes retain_metric / retain_threshold, which remain fully supported.

Styling controls

• curve_kwgs and marker_kwgs on show_cumulative_feature_performance accept either a flat dict (applied to every curve) or a per-metric mapping keyed by input metric name, e.g. {"roc_auc": {"color": "navy"}, "average_precision": {"marker": "s"}}. Per-metric form is detected when all values are dicts.

• bar_kwgs on show_cumulative_feature_importance and show_feature_pareto; curve_kwgs and threshold_kwgs on show_feature_pareto; retain_kwgs on show_cumulative_feature_performance. All merge over the hardcoded defaults.

Changed

• Helpers consolidated into feature_selection_utils.py. All nine private helpers now live in a single module, and show_cumulative_feature_importance / show_cumulative_feature_performance import them rather than carrying duplicate inline definitions. Removes the drift risk of maintaining two copies of the same function.

• Marker styling honoured in both smoothing paths. _plot_perf_series previously hardcoded "o" at size 5 for the markers in the smoothed branch, so marker options silently did nothing when smooth=True. Markers now inherit each line’s resolved colour and respect marker_kwgs in both branches.

• ``features`` is always the final column of the scores table, regardless of which optional columns are enabled.

• Retain-metric validation now raises a plain ValueError (a short-lived custom exception class was removed as unnecessary), and the message lists the input metric keys the caller passed (roc_auc, brier) rather than internal display labels (ROC AUC, Brier Score (lower is better)).

• ``show_feature_pareto`` threshold summary moved off the plot: the “N features to X%” label is printed to the console instead of drawn at the crossover, matching show_cumulative_feature_importance and keeping the crossover region uncluttered. The reference and crossover guide lines remain.

Fixed

• Partial ``order=`` silently inflated scores. In show_cumulative_feature_performance, features of X that did not appear in an explicit order list were never blanked, so at k = len(order) nothing was blanked at all and the reported score was the full model. Passing a 14-feature subset of a 90-feature model therefore returned the full-model AUC at every step, mislabelled as a top-k sweep. The retained set is now exactly ordered_features[:k] and everything else in X is blanked at every k; a one-time note reports how many features are held out. Verified against hand-computed ground truth in both grouped and ungrouped modes.

• ``retain`` silently overrode explicit arguments. Passing retain together with retain_threshold or retain_metric discarded the latter without warning. This now raises.

Validation

show_cumulative_feature_performance was verified end-to-end against manual recomputation: per-k scores for ROC AUC, average precision and Brier; the features column against the ranked prefix; the retention table cell for cell; the retain_threshold marker’s k* and returned feature list; grouped mode with one-hot columns blanked as a unit; and the regression path (R² and MAE). On a real fitted XGBoost pipeline the function reported ROC AUC 0.80974773 and average precision 0.31730261 against hand-computed 0.8097477289163151 and 0.3173026098536764 for the same 20-variable set.

Version 0.0.6a1

All notable changes from this development cycle. Format follows Keep a Changelog. Suggested version bump: 0.0.5a12 to 0.0.6a1.

Added

• new contributor/co-author, Arthur Funnell

Feature-selection plotting suite (model_evaluator.py)

• show_cumulative_feature_importance: production version of the previously orphaned cumaltive_feat_importance_plot. Ranks features by importance and plots horizontal bars with running cumulative-percentage annotations. Supports model_tuner wrappers, plain sklearn estimators, and pipelines. Params include threshold, show_annotation, feature_names, clean_names, return_df, return_features, and full ax / combine_plots compatibility.

• show_cumulative_feature_performance: traces how a fitted model’s performance holds up as it is restricted to its top-k most important features, without retraining. Excluded features are set to NaN and the fixed model is re-scored, so it requires a NaN-native model (CatBoost, XGBoost, HistGradientBoosting). Two input modes: model + X (runs the ablation) or y_probs (a dict of precomputed per-k predictions). Classification and regression, a selectable metric set, display in {absolute, cumulative_gains}, x_as_percent, smooth (PCHIP), retain_threshold markers, and return_df / return_retention / return_features.

• show_feature_pareto: classic Pareto chart (descending importance bars with a smooth cumulative curve and threshold crossover on a twin axis) plus a display="curve" concentration form (cumulative % of features vs cumulative % of importance, with a uniform-importance diagonal). Shares ranking and label helpers with the importance function.

Feature-selection helpers (new module feat_importance_utils.py)

• _resolve_feature_importance_values: resolves importances from a final estimator across CatBoost get_feature_importance(), sklearn feature_importances_, and linear coef_ magnitude (binary and multiclass).

• _rank_features_by_importance: single ranking path used by all three feature functions. Handles wrappers, plain estimators, and pipelines; resolves feature names from explicit feature_names, get_feature_names(), a pipeline preprocessor, feature_names_in_, or a generic fallback.

• _clean_feature_labels: display-only relabeling from a dict or callable.

• _retained_fractions and _plot_perf_series: retention math and optional-smoothing plot helper for the performance curve.

Styling and axis controls

• Per-line styling kwargs that merge over the hardcoded defaults, so thickness, color, and style are overridable (e.g. curve_kwgs={"linewidth": 3}): bar_kwgs on show_cumulative_feature_importance; bar_kwgs, curve_kwgs, threshold_kwgs on show_feature_pareto; curve_kwgs, retain_kwgs on show_cumulative_feature_performance.

• show_cumulative_feature_performance: in display="absolute", R² is automatically placed on a secondary right-hand axis when plotted alongside target-unit error metrics (MAE, RMSE), so its [0, 1] range stays legible; colors and legend are merged across the two axes. Classification metrics (all bounded to [0, 1]) are unaffected.

Metrics registry (metrics_utils.py)

• Metric registry (_CLASSIFICATION_METRICS, _REGRESSION_METRICS, _ALL_METRICS, _DEFAULT_METRICS) covering roc_auc, average_precision, brier, precision, recall, f1, accuracy (classification) and r2, mae, rmse, mse (regression), with _resolve_task and _validate_metrics_for_task.

• Added accuracy_score to the sklearn imports (the only metric not already imported).

DeLong test (new module delong_test.py)

• delong_roc_test: exact DeLong test for two correlated ROC AUCs using the Sun & Xu (2014) fast algorithm (_compute_midrank, _fast_delong). Nonparametric, accounts for the full between-AUC covariance, validated to match a paired bootstrap. Returns (auc1, auc2, p_value).

Arbitrary-format figure saving (plot_utils.py)

• save_plot_images now saves to any matplotlib-supported format when image_filename carries an extension (.pdf, .jpg, .tiff, .eps, and so on), matching eda_toolkit’s _save_figure.

Tests

• test_cumulative_feature_importance.py, test_cumulative_feature_performance.py, test_feature_pareto.py, test_resolve_feature_importance_values.py, test_save_plot_images.py, and test_delong_test.py.

• conftest.py: forces the Agg backend before pyplot import and neutralizes plt.show suite-wide, preventing headless/WSL hangs.

Changed

• save_plot_images moved from metrics_utils.py to plot_utils.py (it is a figure utility). Public from model_metrics import save_plot_images is unaffected via the package star imports; explicit submodule imports were repointed.

• hanley_mcneil_auc_test moved into the new delong_test.py module, alongside delong_roc_test.

• show_roc_curve AUC comparison: the delong= parameter now runs the actual DeLong test (delong_roc_test) instead of the approximation. A new hanley_mcneil= parameter runs the Hanley & McNeil approximation. Both are handled by a single routing loop, and the group_category guard covers both.

• Figure save trigger: providing image_filename triggers a save even when save_plot=False; legacy save_plot + image_path_png / image_path_svg behavior is unchanged.

• Feature-selection helpers extracted into the new feat_importance_utils.py (_resolve_feature_importance_values, _rank_features_by_importance, _clean_feature_labels, _plot_perf_series, _retained_fractions). They previously lived in model_evaluator.py, which now imports them. Private, so package-level access is unchanged.

• Pareto threshold summary moved off the plot: show_feature_pareto no longer draws the “N features to X%” label on the axes; it prints the count to the console instead (matching show_cumulative_feature_importance), keeping the crossover uncluttered. The reference and crossover guide lines remain.

Fixed

• Multi-model default plots dropped every model but the first: in the default (non-overlay, non-subplots) path, show_roc_curve, show_pr_curve, show_lift_chart, show_gain_chart, show_calibration_curve, and plot_threshold_metrics reused the ax parameter across the per-model loop, so only the first model was shown and saved. Each now captures the caller’s ax once and creates a fresh figure per model.

• Importance function had no test coverage: test_cumulative_feature_importance.py had been overwritten with the performance tests, leaving show_cumulative_feature_importance untested. Restored a suite covering the importance function and the ranking helpers.

• Removed the hardcoded model_2022 global from the importance plot (its original TODO), replacing it with a required model parameter.

• Importance summary printed a pandas Styler repr under print(); now renders a plain percent-formatted table that displays correctly outside notebooks.

• Importance and performance functions failed on plain sklearn estimators (RandomForestRegressor and friends) because they assumed a model_tuner wrapper; the shared ranking helper now handles bare estimators and pipelines.

• Removed dead code from the importance plot (unused bar handle, an unused crossover variable that could IndexError, and a no-op axvline).

• Fixed a duplicated average_precision_score import in the sklearn import block.

• Corrected an RMSE definition to reuse np.sqrt(mean_squared_error(...)) inline rather than a redundant helper, matching compute_regression_metrics.

• Family validation now rejects cross-task metrics (for example mae on a classifier) with a specific, multi-line message naming the offending metrics and the valid options.

• Test suite: replaced HistGradientBoostingClassifier fixtures with NaN-tolerant stubs to eliminate an OpenMP deadlock that hung pytest under WSL/conda, and made model fixtures module-scoped where safe.

Deprecated

• hanley_mcneil_auc_test remains available but is documented as a parametric approximation. Prefer delong_roc_test for reported model comparisons.

Migration notes

• Update imports of save_plot_images to from model_metrics.plot_utils import save_plot_images (or the top-level from model_metrics import save_plot_images).

• Update imports of hanley_mcneil_auc_test to from model_metrics.delong_test import hanley_mcneil_auc_test.

• Behavior change: show_roc_curve(delong=[a, b]) now performs the exact DeLong test. For the previous approximation behavior, use show_roc_curve(hanley_mcneil=[a, b]).

• Ensure __init__.py exposes the new module, for example from .delong_test import *, so from model_metrics import delong_roc_test resolves.

• Update imports of the feature-selection helpers (_resolve_feature_importance_values, _rank_features_by_importance, _clean_feature_labels, _plot_perf_series, _retained_fractions) to from model_metrics.feat_importance_utils import .... model_evaluator.py re-imports them, so submodule-qualified imports via model_evaluator still resolve, but pointing at the real module is preferred.

Version 0.0.5a12

Three small but useful changes to the crosstab pair this round.

decimal_places parameter on overlap_crosstab

New keyword-only argument controlling how many decimal places the returned cells round to. Defaults to None (no rounding) for backward compatibility. Most useful with normalize=True to control proportion display:

ct = overlap_crosstab(
    y_true=y_test,
    y_prob_a=y_prob_lr,
    y_prob_b=y_prob_rf,
    normalize=True,
    decimal_places=2,
)

Rounding runs after normalize and mask_impossible, so NaN cells from mask_impossible=True are left untouched. Integer counts (default mode without normalize) round to themselves and stay visually unchanged.

Independent font-size control in the summary panel

The right-hand swap-summary panel of plot_overlap_crosstab now exposes two separate font-size knobs that were previously coupled:

• summary_body_fontsize (new): controls the italic body lines beneath each colored headline. Defaults to None, which falls back to summary_fontsize - 1 (the original derived size, so the visual default is unchanged).

• label_fontsize (existing, expanded): already controlled the matrix row and column headers; now also drives the bold colored summary headlines via label_fontsize + 5. Default label_fontsize=12 gives 17pt headlines, matching the previous summary_fontsize + 6 = 17 exactly.

The two knobs are independent. Bumping one does not move the other. Driven by the need for larger summary headlines on dense slides without inflating the body sentences.

plot_overlap_crosstab(
    y_true=y_test,
    model_a=lr, model_b=rf, X_a=X_test,
    label_fontsize=18,
    summary_body_fontsize=14,
)

Whitespace removal above the matrix

_draw_crosstab_matrix had two layout issues that conspired to leave 30% or more empty space above the matrix in larger figure sizes:

1. ylim extended to n + 1.4 when the topmost content (the label_b axis label) only needed about n + 0.75. That left 0.65 units of empty data space inside the axes.

2. set_aspect("equal") used the default anchor "C", which centered the squared axes box in its subplot cell, sending leftover space upward when matplotlib’s default top margin already biased the layout.

Fixes:

• Tightened ax.set_ylim from (-0.3, n + 1.4) to (-0.3, n + 0.85).

• Changed ax.set_aspect("equal") to ax.set_aspect("equal", anchor="N").

Both apply unconditionally to every render mode: table_only, full view, no-summary, no-legend, and combine_plots-nested.

Tests added

17 new tests across test_metrics_utils.py and test_model_evaluator_addtl.py pinning the changes above. Coverage breakdown:

• overlap_crosstab decimal_places: 5 tests (rounds with normalize, None preserves precision, NaN cells survive, decimal_places=0 rounds to whole numbers, no-op on integer counts).

• _draw_crosstab_summary font sizes: 6 tests (default 17pt headlines, explicit label_fontsize scales headlines, body defaults to fontsize-1, explicit body_fontsize wins, two knobs independent, backward-compat positional args).

• plot_overlap_crosstab integration: 3 tests (summary_body_fontsize accepted and applied, label_fontsize flows into summary headlines).

• _draw_crosstab_matrix layout regression: 3 tests (ylim tight, anchor is N, end-to-end public-function anchor check).

Version 0.0.5a11

A summary of the overlap_crosstab and plot_overlap_crosstab functions added to the overlap family on this iteration, plus the supporting helpers, the font-handling infrastructure, and small touch-ups to existing overlap functions in metrics_utils.py and model_evaluator.py.

What the module adds

Two sibling functions extending the overlap family to a fourth view of the same comparison. Where overlap_summary gives per-category marginal counts and plot_overlap_venns shows per-category region overlap, the crosstab pair shows the full joint distribution of (model_a category, model_b category) in one 4x4 frame.

• overlap_crosstab returns a 4x4 pandas DataFrame indexed by {TP, FP, FN, TN} on both axes, counting every observation that lands in each (row category, column category) pair. The diagonal is the agreement count; the off-diagonal valid cells are the TP-swap and FP-swap pairs.

• plot_overlap_crosstab renders the same crosstab as a styled matplotlib figure with three cell colorings (green for agreement, red for swaps, blue-gray for the eight structurally impossible cells), an optional right-hand summary panel surfacing the derived swap story, and an optional legend strip beneath the matrix.

Both share the same input-parameter block as the rest of the overlap family, so the same prediction inputs that drive a Venn or a summary table also drive the crosstab without modification.

Public functions

overlap_crosstab

overlap_crosstab(
    y_true,
    y_pred_a=None,
    y_pred_b=None,
    *,
    y_prob_a=None,
    y_prob_b=None,
    model_a=None,
    model_b=None,
    X_a=None,
    X_b=None,
    threshold_a=None,
    threshold_b=None,
    score_a=None,
    score_b=None,
    label_a="Model A",
    label_b="Model B",
    normalize=False,
    mask_impossible=False,
    verbose=False,
)

Returns a 4x4 DataFrame indexed by {TP, FP, FN, TN} with the same four columns. verbose=True prints the cell-meaning legend plus the derived swap summary before returning. normalize=True divides every cell by the total observation count. mask_impossible=True sets the eight structurally impossible cells to NaN instead of leaving them at 0.

plot_overlap_crosstab

plot_overlap_crosstab(
    y_true,
    y_pred_a=None,
    y_pred_b=None,
    *,
    y_prob_a=None,
    y_prob_b=None,
    model_a=None,
    model_b=None,
    X_a=None,
    X_b=None,
    threshold_a=None,
    threshold_b=None,
    score_a=None,
    score_b=None,
    label_a="Model A",
    label_b="Model B",
    title=None,
    table_only=False,
    show_summary=True,
    show_legend=True,
    colors=None,
    cell_fontsize=14,
    label_fontsize=12,
    title_fontsize=14,
    summary_fontsize=11,
    font=None,
    figsize=None,
    save_plot=False,
    image_path_png=None,
    image_path_svg=None,
    image_filename=None,
    ax=None,
)

Renders the crosstab as a matplotlib figure. Calls overlap_crosstab internally, then dispatches to three private helpers (_draw_crosstab_matrix, _draw_crosstab_summary, _draw_crosstab_legend) for the three regions of the figure.

Structural-impossibility insight

Of the 16 cells in the 4x4 grid, only 8 are reachable. Because y_true is fixed per observation:

• actual positives can only be TP or FN for each model

• actual negatives can only be TN or FP

So any cell mixing a positive-subpop category (TP, FN) on one axis with a negative-subpop category (FP, TN) on the other is structurally impossible. A single observation cannot be TP for one model and FP for the other, since y_true is either 1 or 0, not both.

The valid cells partition cleanly:

• Diagonal (4 cells): agreement, both models put the observation in the same confusion-matrix category. (TP, TP), (FP, FP), (FN, FN), (TN, TN).

• Off-diagonal, positive-subpop (2 cells): TP swap pair. (TP, FN) and (FN, TP). One model catches an actual positive the other misses.

• Off-diagonal, negative-subpop (2 cells): FP swap pair. (FP, TN) and (TN, FP). One model false-alarms on an actual negative the other clears.

The structural-impossibility check is encoded in _draw_crosstab_matrix (cells where row category and column category are in different subpops get the impossibility color) and in overlap_crosstab’s mask_impossible flag (NaN those cells in the returned DataFrame instead of leaving them at 0).

Prediction-input modes

Same three-way exclusivity as the rest of the family. For each side (a, b) supply exactly one of:

1. y_pred_* (binary predictions, used as is)

2. y_prob_* (positive-class probabilities, thresholded internally via threshold_*, default 0.5)

3. model_* + X_* (predictions via get_predictions; the model’s stored .threshold attribute is looked up automatically, with threshold_* as an explicit override and score_* selecting a non-default score key from the model’s .threshold dict)

Supplying more than one on the same side raises ValueError. The y_prob_* path defaults to 0.5 when no threshold_* is set, since there is no model object available on that path to source a stored threshold from.

Display controls (plot_overlap_crosstab)

• title controls the figure title and its agreement-rate subtitle. None defaults to "{label_a} vs {label_b}: Prediction Overlap". Pass "" to suppress.

• table_only is the one-flag shortcut for a bare matrix: suppresses the title, the side summary panel, and the bottom legend strip. Equivalent to passing title="", show_summary=False, show_legend=False.

• show_summary toggles the right-hand swap-summary text block (shared TP / FN / TN / FP counts, TP-swap and FP-swap breakdowns).

• show_legend toggles the color-legend strip beneath the matrix.

• colors accepts a dict with keys "agree", "disagree", "impossible" to override the default cell colors. Unspecified keys keep their defaults (soft green, soft pink, soft blue-gray).

• Font sizes are split across four parameters: cell_fontsize for the cell counts, label_fontsize for the row and column headers, title_fontsize for the title block, summary_fontsize for the side panel.

Default figsize behavior

figsize=None triggers a three-way default based on which display regions are active:

• (5, 5) when table_only=True (bare square matrix)

• (11, 6.5) when show_summary=True (matrix plus side panel)

• (7, 6.5) otherwise (matrix plus legend, no side panel)

The matrix uses set_aspect("equal") so cells always render square. Passing an asymmetric figsize (e.g. (10, 4)) produces a figure of that exact size with whitespace on the wider dimension, since cells will not stretch to fill.

When the title is suppressed (table_only=True or title=""), tight_layout is invoked with rect=[0, 0, 1, 1] so the matrix uses the full figure height. When the title is active, rect=[0, 0, 1, 0.88] reserves the top 12% for the title and subtitle.

Font handling

A new module-level _FONT_ALIASES dict in metrics_utils.py maps common cross-platform font names (Arial, Helvetica, Times New Roman, Consolas, Cascadia Code, Source Code Pro, Fira Code, JetBrains Mono, Menlo, Monaco, Georgia, Verdana, Calibri, Cambria, Tahoma, Garamond) to fallback chains ending in a font that is guaranteed installable (DejaVu Sans or DejaVu Sans Mono, both shipped with matplotlib).

_resolve_font_family(font) expands the input to a chain, filters it to the fonts actually findable on the system (via font_manager.findfont(name, fallback_to_default=False)), and returns the surviving list. Behavior summary:

• font=None returns None (the caller leaves rcParams untouched).

• font="Arial" (or any aliased name) always succeeds, because the alias chain ends in DejaVu Sans.

• font="UnknownFontName" (not in the alias map, not installed) raises ValueError with a useful message listing the supported aliases and pointing at how to check what is installed.

• font=42 raises TypeError.

In plot_overlap_crosstab, the resolved font is applied via matplotlib.rc_context({"font.family": resolved_font}), so the override scopes to the call only and does not modify the global rcParams.

The asymmetry between aliased and unaliased fonts is intentional. Aliased fonts have a curated fallback path that gracefully degrades to something installable on every platform, so library users do not need to install Microsoft fonts on Linux to use font="Arial". Unaliased fonts are taken at the user’s word and validated literally, so typos and missing fonts surface as errors rather than getting silently swapped to DejaVu Sans.

Supporting helpers

• _print_overlap_crosstab_legend(ct, label_a, label_b) (in metrics_utils.py) prints both the static cell-meaning legend (what each cell type represents, which cells are structurally impossible) and the dynamic swap summary derived from the raw integer counts (agreement rate, shared TP / FN / TN / FP, TP and FP swap breakdowns with net deltas).

• _draw_crosstab_matrix(ax, ct, label_a, label_b, colors, cell_fontsize, label_fontsize) (in metrics_utils.py) renders the 4x4 grid: per-cell color via Rectangle patches, per-cell text via ax.text with the count formatted by :,, row and column headers, and the axis-level labels. Uses a plain hyphen rather than an em dash for the NaN cell text under mask_impossible=True.

• _draw_crosstab_summary(ax, label_b, stats, fontsize) (in metrics_utils.py) renders the right-hand text block with four colored headlines (shared FN, shared TN, TP swap, FP swap) and their italic body lines, all relative to label_b since the swap framing reads naturally with the newer or candidate model in the _b slot.

• _draw_crosstab_legend(ax, colors, fontsize) (in metrics_utils.py) renders the horizontal color-legend strip using matplotlib’s Patch and ax.legend.

• _resolve_font_family(font) (in metrics_utils.py) is the font alias expander documented above.

• _FONT_ALIASES (module-level dict in metrics_utils.py) is the alias table.

Touch-ups to existing overlap functions

• overlap_summary: restored the n_{label_a} and n_{label_b} columns (the per-model totals for the category, equal to both + {label}_only) after a brief detour to a 5-column partition-only form. The 7-column form was restored because the per-model totals are exactly what the Venn diagram shows beneath each circle, making the table-to-figure cross-reference direct.

• _print_overlap_summary_legend: updated to document both the partition identity (both + {a}_only + {b}_only + outside == subpop) and the per-model decomposition (n_{label} == both + {label}_only) explicitly, since the difference between n_{label} (full venn circle) and {label}_only (exclusive crescent) is the most common reader-confusion point.

• overlap_table: tightened the index parameter to raise TypeError with a helpful message when a scalar (e.g. a column name string) is passed instead of an array-like, since pd.Index(scalar) produces the cryptic Index(...) must be called with a collection error.

• _draw_crosstab_matrix axis-label spacing: dropped the axis-level label_b y-position from n + 0.9 to n + 0.55 so the column-axis label sits at the same visible distance from the column headers as the row-axis label sits from the row headers.

• _draw_crosstab_summary: applied the :, thousands separator to every number in the swap-summary block (the swap headlines and the body lines were originally missing the comma format, leaving numbers like 1249 in a sea of 1,249 and +1,079).

Design decisions baked in

• Same input-parameter block as the rest of the overlap family. Swapping between plot_overlap_venns, overlap_summary, overlap_table, overlap_crosstab, and plot_overlap_crosstab for the same comparison is a one-character edit.

• Three exclusive input modes per side, no implicit threshold lookup on the y_prob path. The y_prob path defaults to 0.5 because there is no model object to source a stored threshold from. Users who want tuned-threshold behavior with probabilities in hand pass threshold_* explicitly, or use the model path.

• Strict ValueError on unfound unaliased fonts; graceful degradation for aliased fonts. Library users do not need to install Microsoft fonts on Linux to call font="Arial", but typos and unrecognized fonts loudly fail rather than silently rendering in DejaVu Sans.

• Font override scoped via mpl.rc_context. The font parameter applies only to the current call and never leaks into the global rcParams, so other figures in the same notebook are unaffected.

• table_only as the one-flag shortcut for bare matrices. Sets title="", show_summary=False, show_legend=False together, because those are the three knobs anyone toggles together when preparing a slide-ready figure.

• aspect=”equal” on the matrix. Cells stay square at every figure size, which is the right tradeoff for the symmetry-reading use case (the TP-swap and FP-swap pairs read off the diagonal best when cells are square).

• n_{label} columns retained in overlap_summary despite redundancy with the partition columns. The per-model totals match exactly what the venn diagram shows beneath each circle, making the table-to-figure cross-reference direct.

• Plot function calls data function internally rather than duplicating the counting logic. plot_overlap_crosstab invokes overlap_crosstab to build the DataFrame, then renders. Single source of truth for the counts; test the data function and the figure inherits correctness for free.

• Swap summary framed relative to label_b. “Tab+Text catches 5 extra, misses 3” reads naturally when the newer or candidate model is in the _b slot. The narrative arrow points from the baseline (_a) to the candidate (_b).

Version 0.0.5a10

Three sibling functions for comparing two binary classifiers head to head:

• plot_overlap_venns renders equal-area Venn diagrams across any subset of {“TP”, “FP”, “FN”, “TN”} categories. Each panel shows the overlap between the two models’ predictions within the relevant subpopulation (actual positives for TP/FN, actual negatives for FP/TN).

• overlap_table returns a per-observation DataFrame classifying each row as TP/FP/FN/TN for both models, with an agreement flag. Useful for drilling into specific observations or merging with other patient-level data.

• overlap_summary returns a four-row DataFrame indexed by {TP, FP, FN, TN} with per-category counts of the venn regions. Useful as numeric companion to the figure.

All three share the same input-parameter block, so swapping between table, summary, and figure for the same comparison is a one-character edit.

Public functions

plot_overlap_venns

plot_overlap_venns(
    y_true,
    y_pred_a=None,
    y_pred_b=None,
    *,
    y_prob_a=None,
    y_prob_b=None,
    model_a=None,
    model_b=None,
    X_a=None,
    X_b=None,
    threshold_a=None,
    threshold_b=None,
    score_a=None,
    score_b=None,
    categories=("FN", "TN"),
    label_a="Model A",
    label_b="Model B",
    titles=None,
    title_pad=None,
    label_kwgs=None,
    inner_fontsize=12,
    outer_fontsize=12,
    title_fontsize=11,
    figsize=None,
    ncols=None,
    pad=1.08,
    h_pad=None,
    w_pad=None,
    colors=None,
    alpha=0.4,
    save_plot=False,
    image_path_png=None,
    image_path_svg=None,
    image_filename=None,
    ax=None,
)

overlap_table

overlap_table(
    y_true,
    y_pred_a=None,
    y_pred_b=None,
    *,
    y_prob_a=None,
    y_prob_b=None,
    model_a=None,
    model_b=None,
    X_a=None,
    X_b=None,
    threshold_a=None,
    threshold_b=None,
    score_a=None,
    score_b=None,
    label_a="Model A",
    label_b="Model B",
    index=None,
)

Returns a DataFrame with columns y_true, {label_a}_pred, {label_b}_pred, {label_a}_category, {label_b}_category, agree.

overlap_summary

overlap_summary(
    y_true,
    y_pred_a=None,
    y_pred_b=None,
    *,
    y_prob_a=None,
    y_prob_b=None,
    model_a=None,
    model_b=None,
    X_a=None,
    X_b=None,
    threshold_a=None,
    threshold_b=None,
    score_a=None,
    score_b=None,
    label_a="Model A",
    label_b="Model B",
    verbose=False,
)

Returns a four-row DataFrame indexed by {TP, FP, FN, TN} with seven columns: n_{label_a}, n_{label_b}, both, {label_a}_only, {label_b}_only, outside, subpop. Two identities hold every row:

partition:    both + {a}_only + {b}_only + outside == subpop
per-model:    n_{label} == both + {label}_only

verbose=True prints a human-readable column legend via _print_overlap_summary_legend.

Prediction-input modes

Each side (a, b) accepts exactly one of three input modes:

1. y_pred_* (binary predictions, used as is)

2. y_prob_* (positive-class probabilities, thresholded internally via threshold_*, default 0.5)

3. model_* + X_* (predictions generated via get_predictions; the model’s stored .threshold attribute is looked up automatically, with threshold_* as an explicit override and score_* selecting a non-default key from the model’s .threshold dict)

Supplying more than one of these on the same side raises ValueError.

When mixing model objects without tuned thresholds, the system falls back to a 0.5 default silently, so plain sklearn classifiers work without extra setup. When mixing this view with the standalone confusion matrix from show_confusion_matrix, pass the same threshold_* value to both to keep the partition counts aligned.

label_kwgs (visibility toggles for plot_overlap_venns)

A single dict parameter controls all text decorations on each Venn panel. Six boolean keys, all default True:

show_title         heading line above the diagram
show_subtitle      auto stats line beneath the heading
show_set_labels    model names beneath each circle
show_set_totals    "FN total: X" line beneath model names
show_inner_count   the number inside each region
show_inner_role    the "both miss" / model-name role text inside each region

Pass only the keys you want to override. Common combinations:

# slide deck: title only, no chrome
label_kwgs={
    "show_subtitle":   False,
    "show_set_totals": False,
    "show_inner_role": False,
}

# bare venn: just circles and numbers
label_kwgs={
    "show_title":      False,
    "show_set_labels": False,
    "show_set_totals": False,
    "show_inner_role": False,
}

Default figsize behavior

When figsize=None (the default) and the function creates its own figure, _venn_default_figsize computes the panel size from title text length:

panel_w = max(5.5, max_chars * (title_fontsize * 0.009) + 0.6)
panel_h = 5.0

Width grows with title length but floors at 5.5 inches to avoid inter-column gaps. Height stays fixed at 5 inches to keep room for the title, the circles, and the bottom set labels without overflowing into the next row. Width and height are decoupled, so a long title widens panels without compressing them vertically.

combine_plots integration

plot_overlap_venns accepts an ax= parameter. When supplied, the function does not create its own figure. It releases the host axes and carves a sub-gridspec from the host axes’ slot, then adds new sub-axes inside that sub-gridspec for each category panel.

Recommended pattern for mixing into a full evaluation suite: add one plot_overlap_venns entry to combine_plots’s plot_calls per category, each with a single-element categories tuple. Nesting all four inside a single panel works but produces visually cramped output when the figure also contains other plots competing for width.

For per-row spacing requirements, combine_plots switches to constrained_layout when hspace is passed. height_ratios=[1, 0.7, 1, 1, 1.4, 1.4] plus hspace=0.15 is a reasonable starting point for the evaluation-suite plus four-venn-rows case.

Supporting helpers in metrics_utils.py

_VENN_CATEGORY_SPEC
    Module-private dict mapping each category key to display metadata:
    title, subpop_val, in_set_val, both_role, outside_label, subpop_name.

_venn_blend(c1, c2)
    RGB midpoint of two matplotlib color specs (via to_rgb).

_venn_resolve_side(side, y_pred, model, X, *, y_true=None, y_prob=None,
                   threshold=None, score=None)
    Three-way exclusivity check on (y_pred, y_prob, model). Returns the
    integer 1-D prediction array for one side. Model path routes through
    get_predictions for threshold-aware prediction.

_venn_category_counts(y_true, y_pred_a, y_pred_b, cat)
    Returns (a_only, b_only, both, outside, n_sub) integer counts for one
    category. Shared backend for plot_overlap_venns and overlap_summary.

_venn_default_figsize(counts_per_cat, categories, titles, label_kwgs,
                      title_fontsize, ncols, nrows)
    Computes per-panel figsize from title text length. Reads show_title
    and show_subtitle from label_kwgs to skip text estimation when titles
    are hidden.

_draw_one_venn(ax, cat, counts, label_a, label_b,
               inner_fontsize, outer_fontsize, title_fontsize,
               colors, alpha, *,
               title_override, title_pad, label_kwgs)
    Renders one category's overlap Venn into the provided axes. Handles
    set-label assembly, inner region text composition, color blending,
    and title formatting per the label_kwgs toggles.

_overlap_table_categorize(yt, yp)
    Per-observation TP/FN/FP/TN classification used by overlap_table.

_print_overlap_summary_legend(label_a, label_b)
    Human-readable column legend for overlap_summary. Documents both the
    partition identity (sum-to-subpop) and the per-model decomposition
    (n_{label} = both + {label}_only) since the difference between
    n_{label} (full venn circle) and {label}_only (exclusive crescent) is
    the most common reader-confusion point.

Design decisions baked in

• Threshold default uses the model’s stored value, not 0.5. A venn comparison should show each model at its real operating point. Models without a .threshold attribute fall back to 0.5 silently.

• Single ``threshold_*`` parameter per side, not the library’s ``model_threshold`` plus ``custom_threshold`` pair. The pair only made sense for functions that needed to distinguish “use stored” from “override scalar”. The venn API has one knob.

• Three exclusive input modes per side (y_pred, y_prob, model). Forces an explicit choice rather than silent ambiguity if multiple are passed.

• Pure-data companions return DataFrames, not styled objects. Preserves chaining and downstream .loc access. Legend printing is opt-in via verbose=True or a separate helper call.

• n_{label} columns retained in overlap_summary despite redundancy with the partition columns. The per-model totals match exactly what the venn diagram shows beneath each circle, making the table-to-figure cross-reference direct.

• Bare-string category accepted (e.g. ``categories=”FN”``). Avoids the footgun where iterating the string yields per-character category lookups.

• Single function for the venn (not a singular/plural split). combine_plots integration uses plot_overlap_venns with a single-element categories tuple rather than introducing a separate plot_overlap_venn function.

Version 0.0.5a9

Important

Corrected package name in pyproject.toml from model_metrics_dev to model_metrics. The 0.0.5a8 release was inadvertently published under the wrong package name and could not be recalled in time; 0.0.5a9 supersedes it.

New Features

• Added ax=None parameter to seven single-plot functions: show_roc_curve, show_pr_curve, show_confusion_matrix, show_calibration_curve, show_lift_chart, show_gain_chart, and plot_threshold_metrics. When a pre-created matplotlib.axes.Axes object is supplied, each function draws onto that axes and suppresses its internal plt.show() and save_plot_images calls. All existing call signatures remain fully backward-compatible — the default ax=None preserves prior behavior exactly.

• Added combine_plots function that assembles multiple single-plot function calls into a shared subplot figure. Accepts a list of (func, kwargs) tuples, pre-allocates one axes per panel, and passes each axes to the corresponding function via the ax parameter. Supports configurable grid layout (n_cols, n_rows), custom figsize, suptitle, tight_layout, and full save_plot_images integration. Unused trailing panels are hidden automatically. Panels that raise exceptions render an inline error message rather than aborting the figure.

• Added tick_fontsize parameter to combine_plots. Both label_fontsize and tick_fontsize are now automatically injected into each panel function via inspect.signature, giving uniform typography across the grid without per-panel repetition. Per-panel overrides in plot_calls always take precedence.

• Added hspace and wspace parameters to combine_plots for explicit row and column spacing control. When either is supplied, constrained_layout is activated and spacing is applied via fig.get_layout_engine().set() after tight_layout, correctly handling panels with legend_loc="bottom" that place legends outside the axes bbox.

• Added height_ratios parameter to combine_plots, passed through gridspec_kw, allowing individual rows to be resized independently. Useful when mixing plot types of different natural heights (e.g., confusion matrices alongside full-height curve panels).

• Fixed the overlay path in show_roc_curve, show_pr_curve, show_lift_chart, show_gain_chart, show_calibration_curve, and plot_threshold_metrics. Previously, overlay=True called plt.figure() unconditionally, creating a standalone figure instead of drawing onto the supplied ax. All six functions now check ax is not None before creating a figure, correctly routing overlay draws into combine_plots panel grids.

Bug Fixes

• Fixed combine_plots axes flattening logic. The previous if num_plots == 1 guard incorrectly wrapped a numpy ndarray of axes in a list when plt.subplots(1, N) was called with N > 1 and only one plot call was provided, causing an AttributeError: 'numpy.ndarray' object has no attribute 'plot'. The fix inspects the axes object directly using hasattr rather than relying on num_plots.

• Fixed show_confusion_matrix axes reference conflict. The loop variable ax used in the subplots branch shadowed the user-supplied ax parameter. The user-supplied value is now stashed as _user_ax before the loop runs to prevent clobbering.

• Fixed show_confusion_matrix ignoring model_threshold when passed as a list and X is provided. Previously the full list was forwarded to get_predictions which expects a scalar or dict, causing incorrect threshold application. The list is now indexed per model (model_threshold[idx]) in the X-provided branch.

• Fixed show_confusion_matrix raising TypeError: 'float' object is not subscriptable when model_threshold was passed as a scalar float in the X is None branch. The branch now checks isinstance before indexing, correctly handling scalar, list, and dict threshold inputs.

Version 0.0.5a8

New Features

• Added ax=None parameter to seven single-plot functions: show_roc_curve, show_pr_curve, show_confusion_matrix, show_calibration_curve, show_lift_chart, show_gain_chart, and plot_threshold_metrics. When a pre-created matplotlib.axes.Axes object is supplied, each function draws onto that axes and suppresses its internal plt.show() and save_plot_images calls. All existing call signatures remain fully backward-compatible — the default ax=None preserves prior behavior exactly.

• Added combine_plots function that assembles multiple single-plot function calls into a shared subplot figure. Accepts a list of (func, kwargs) tuples, pre-allocates one axes per panel, and passes each axes to the corresponding function via the ax parameter. Supports configurable grid layout (n_cols, n_rows), custom figsize, suptitle, tight_layout, and full save_plot_images integration. Unused trailing panels are hidden automatically. Panels that raise exceptions render an inline error message rather than aborting the figure.

• Added tick_fontsize parameter to combine_plots. Both label_fontsize and tick_fontsize are now automatically injected into each panel function via inspect.signature, giving uniform typography across the grid without per-panel repetition. Per-panel overrides in plot_calls always take precedence.

• Added hspace and wspace parameters to combine_plots for explicit row and column spacing control. When either is supplied, constrained_layout is activated and spacing is applied via fig.get_layout_engine().set() after tight_layout, correctly handling panels with legend_loc="bottom" that place legends outside the axes bbox.

• Added height_ratios parameter to combine_plots, passed through gridspec_kw, allowing individual rows to be resized independently. Useful when mixing plot types of different natural heights (e.g., confusion matrices alongside full-height curve panels).

• Fixed the overlay path in show_roc_curve, show_pr_curve, show_lift_chart, show_gain_chart, show_calibration_curve, and plot_threshold_metrics. Previously, overlay=True called plt.figure() unconditionally, creating a standalone figure instead of drawing onto the supplied ax. All six functions now check ax is not None before creating a figure, correctly routing overlay draws into combine_plots panel grids.

Bug Fixes

• Fixed combine_plots axes flattening logic. The previous if num_plots == 1 guard incorrectly wrapped a numpy ndarray of axes in a list when plt.subplots(1, N) was called with N > 1 and only one plot call was provided, causing an AttributeError: 'numpy.ndarray' object has no attribute 'plot'. The fix inspects the axes object directly using hasattr rather than relying on num_plots.

• Fixed show_confusion_matrix axes reference conflict. The loop variable ax used in the subplots branch shadowed the user-supplied ax parameter. The user-supplied value is now stashed as _user_ax before the loop runs to prevent clobbering.

• Fixed show_confusion_matrix ignoring model_threshold when passed as a list and X is provided. Previously the full list was forwarded to get_predictions which expects a scalar or dict, causing incorrect threshold application. The list is now indexed per model (model_threshold[idx]) in the X-provided branch.

• Fixed show_confusion_matrix raising TypeError: 'float' object is not subscriptable when model_threshold was passed as a scalar float in the X is None branch. The branch now checks isinstance before indexing, correctly handling scalar, list, and dict threshold inputs.

Version 0.0.5a7

Summary

This version drops support for Python 3.7.4 and sets the minimum required Python version to 3.8. Python 3.7 reached end-of-life in June 2023 and is no longer supported by the library. Users on Python 3.7.x must upgrade before installing this version.

This version also delivers four major workstreams across the library: a ground-up hardening of ModelCalculator, full multi-model support for plot_threshold_metrics, a new image_filename parameter across all eight plotting functions, and Python 3.8 compatibility restoration.

---

1. ModelCalculator

_extract_final_model

The original branching logic was ambiguous and failed silently for several real-world model wrapper patterns. The method now resolves wrappers in a strict, documented priority order:

1. Plain dict with a "model" key (e.g. model_tuner pkl format {"model": <Model>}) unwrapped first before any other check.

2. sklearn.pipeline.Pipeline via hasattr(model, "steps"), extracting the last step.

3. Objects with an estimator attribute (e.g. model_tuner Model objects wrapping a CalibratedClassifierCV).

4. Objects with a model attribute (e.g. custom wrapper classes).

5. Standalone sklearn-compatible objects with predict, predict_proba, or decision_function.

The dict unwrap path was entirely absent before this change, causing AttributeError: 'dict' object has no attribute 'predict' when loading pkl files saved in model_tuner format.

generate_predictions

The prediction block previously called model.predict() and model.predict_proba() directly on the raw object retrieved from model_dict, bypassing _extract_final_model. This caused the same AttributeError on dict-wrapped models. The block now routes through the already-resolved estimator variable for all prediction calls, while retaining the model.threshold check on the original object since that attribute lives on the model_tuner wrapper, not the inner estimator.

_add_metrics

Replaced type(y_test_m) == pd.DataFrame with isinstance(y_test_m, pd.DataFrame) per Python best practices. Also corrected squeeze(axis=0) to squeeze(axis=1), which is the correct axis for collapsing a single-column DataFrame into a Series.

_get_shap_explainer (new helper)

Replaced the generic shap.Explainer auto-detection call, which fired internal probe warnings on every invocation, with an explicit helper that selects the correct explainer class based on model attributes:

• shap.TreeExplainer for tree models (tree_ or estimators_).

• shap.LinearExplainer for linear models (coef_).

• shap.KernelExplainer via _make_predict_proba_wrapper for everything else.

A guard at the top raises ValueError immediately for models without predict_proba so unsupported models fail with a clear message rather than crashing inside SHAP internals.

_make_predict_proba_wrapper (new helper)

KernelExplainer internally converts DataFrames to numpy before calling the model function, which caused StandardScaler (fitted with named columns) to emit UserWarning: X does not have valid feature names on every SHAP call. The new wrapper re-attaches the original column names before passing the array to predict_proba, eliminating the warning at the source rather than suppressing it.

_calculate_shap_values

Global SHAP previously iterated row-by-row via itertuples, calling the explainer once per sample. This has been replaced with a single batched explainer(X_transformed) call, which is orders of magnitude faster on datasets of any meaningful size.

Fixed the multi-class SHAP averaging from .mean(axis=0).mean(axis=1) to .mean(axis=2).mean(axis=0), which is the correct reduction order for a (n_samples, n_features, n_classes) tensor.

Added pipeline unwrapping at the top of the method so the method handles dict/Pipeline-wrapped models passed directly rather than only pre-unwrapped estimators.

The include_contributions row-wise path now consistently returns top-N {feature: shap_value} dicts sorted by absolute value, matching the coefficient path. Previously it returned a flat dict over all features regardless of top_n.

_calculate_coefficients

Added a secondary Pipeline unwrap after _extract_final_model for cases where the extracted model is itself a Pipeline (e.g. model_tuner objects where .estimator is a CalibratedClassifierCV wrapping a Pipeline). Without this, coef_ lookup failed on the Pipeline object rather than its final step.

The include_contributions=False path now returns top-N feature-name lists rather than dicts, making it symmetric with the SHAP path and ensuring subset_results column content is consistent regardless of which explainability method is used.

Note

This is a behavior change for any code consuming the default output of _calculate_coefficients as dicts. Pass include_contributions=True to restore the previous dict output.

---

2. plot_threshold_metrics

Multi-model support

The function previously accepted only a single model or y_prob array. It now accepts lists of models, y_prob arrays, and thresholds and supports three display modes:

• Single: one plot per model, unchanged from previous behavior.

• Overlay: all models on a single shared axes, with curve labels prefixed by model name for disambiguation.

• Subplots: one subplot per model in an auto-sized grid.

New parameters: model_title, overlay, subplots, n_cols, n_rows, suptitle, suptitle_y, and model_threshold (now accepts a list).

Model title defaulting

When model_title=None and model objects are provided, titles now default to the model class name via extract_model_name() rather than generic “Model 1”, “Model 2” index labels. When only y_prob arrays are provided the index fallback is retained since there is nothing to extract a name from.

n_rows / n_cols auto-derivation

When n_rows is explicitly provided but n_cols is left at its default of 2, n_cols is now automatically derived as ceil(num_models / n_rows). Previously specifying n_rows=1 with 3 models still produced a 2-column grid because n_cols was never recalculated.

y_test flattening

precision_recall_curve and roc_curve inside _plot_single now receive np.asarray(y_test).ravel() to prevent ValueError: Found input variables with inconsistent numbers of samples when y_test is a single-column DataFrame loaded from parquet.

suptitle / title separation

suptitle controls the overall figure heading above all subplots. title controls per-subplot headings. The two can be set independently: passing title="" suppresses per-subplot titles while still showing the suptitle, and vice versa. Previously there was no way to have both levels of titling independently.

---

3. image_filename Save Integration

Updated save_plot_images

Added three new parameters: image_filename, fig, and dpi.

Saving is now triggered when either save_plot=True or image_filename is provided, so callers no longer need to set save_plot=True just to use a custom filename. When image_filename is provided it takes precedence over the auto-generated filename. The function now calls fig.savefig() targeting the correct figure object rather than plt.savefig(), which targeted whatever the current active figure happened to be at call time.

All eight plotting functions updated

image_filename=None added to the signature immediately after image_path_svg and threaded through every save_plot_images call site across show_confusion_matrix, show_roc_curve, show_pr_curve, show_lift_chart, show_gain_chart, show_calibration_curve, plot_threshold_metrics, and show_residual_diagnostics (22 call sites in total).

The if save_plot: guards that previously wrapped save_plot_images calls in show_calibration_curve (group path) and show_residual_diagnostics (both save sites) have been removed since save_plot_images now handles the trigger logic internally.

All eight function docstrings updated to document image_filename immediately after image_path_svg.

---

4. Python 3.8 Compatibility and Version Floor Change

The minimum supported Python version has been raised from 3.7.4 to 3.8. This aligns with the broader Python ecosystem where 3.7 has been end-of-life since June 2023 and several upstream dependencies no longer ship 3.7-compatible wheels.

fastparquet replaced with pyarrow

fastparquet fails to build on Python 3.8 due to a Cython ndarray type identifier incompatibility that affects all released versions. The dependency has been replaced with pyarrow>=11.0.0,<=14.0.2, which is the last release with official Python 3.8 wheels. pd.read_parquet() uses pyarrow automatically so no code changes were required.

pip upgrade required for Python 3.8 environment

The venv_3_8 environment shipped with pip 19.2.3 (2019), which cannot parse modern pyproject.toml files used by packages like ninja and scipy. This caused cascading build failures for scikit-learn and pyarrow. Running pip install --upgrade pip before pip install -r requirements.txt unblocks the full install.

typing imports added to plot_utils.py

Optional, List, Dict, Union, and Tuple were used in type annotations but not imported. Python 3.10+ allows X | None union syntax without importing from typing but Python 3.8 requires the explicit import. Added from typing import Optional, List, Dict, Union, Tuple to resolve NameError: name 'Optional' is not defined on import.

---

5. Test Suite

pytest collection conflict resolved

Both py_scripts/test_model_calculator.py and unittests/test_model_calculator.py share the same basename, causing pytest to fail at collection with an import file mismatch error. Fixed by adding __init__.py to both py_scripts/ and unittests/ and clearing stale __pycache__ artifacts.

test_model_calculator fixes

test_calculate_coefficients: updated assertion from isinstance(row, dict) to isinstance(row, list) since the default path now returns feature-name lists, not dicts.

test_extract_final_model_wrapped_model: resolved by adding the hasattr(model, "model") branch to _extract_final_model. No test change needed.

test_calculate_shap_unsupported_model: resolved by adding the predict_proba guard at the top of _get_shap_explainer so unsupported models raise ValueError before reaching KernelExplainer.

test_calculate_shap_unexpected_shape and test_rowwise_shap_output_unexpected_type: both monkeypatched shap.Explainer which is no longer called. Updated to monkeypatch ModelCalculator._get_shap_explainer instead.

test_model_evaluator fixes

test_plot_threshold_metrics_with_lookup and test_plot_threshold_metrics_all_lookup_metrics: both asserted "Best threshold" in captured.out. The print format now prefixes the model name (e.g. "LogisticRegression -- best threshold for..."). Updated assertions to "best threshold" in captured.out which matches both the old and new format.

Final result: 204 collected, 204 passed. Coverage: metrics_utils 79%, model_calculator 87%, model_evaluator 83%, partial_dependence 89%, plot_utils 70%.

---

6. Test and Usage Scripts

test_model_calculator.py (py_scripts)

Standalone .py equivalent of the test notebook with section headers, coloured PASS/FAIL labels, and .to_string() DataFrame output for clean terminal readability. Paths are anchored to __file__ via SCRIPT_DIR and PROJECT_ROOT so the script runs correctly regardless of which directory python is invoked from.

test_model_calculator.ipynb

Updated to use load_breast_cancer() (30 real named features) instead of make_classification() (generic feature_0..9 labels) so SHAP and coefficient outputs show meaningful feature names. Added max_iter=10000 to LogisticRegression to suppress the convergence warning that cluttered notebook output.

Version 0.0.5a6

Bug Fixes

• Fixed interactive Plotly plot not rendering in Jupyter notebooks unless save_plots was set. Display and saving are now fully decoupled; the plot always renders regardless of save_plots.

• Removed duplicate HTML save block that existed inside the static plot section.

New Features

• Added x_label_map and y_label_map parameters for mapping raw axis values to human-readable tick labels; useful for encoded or numeric categorical features.

• Added modebar_image_format parameter ("png", "svg", "jpeg", "webp") to control the download format of the Plotly modebar camera button. Defaults to "png".

Improvements

• Docstring updated to document x_label_map, y_label_map, and modebar_image_format.

• Raises section expanded to cover all ValueError conditions, including invalid save_plots, missing image paths, missing HTML paths, invalid plot_type, and invalid modebar_image_format.

• Update to plot_3d_pdp docstring

• Adds full categorical feature support to plot_3d_pdp while preserving backward compatibility with numeric grids. The function now renders categorical axes correctly in both Matplotlib and Plotly by mapping categories to numeric surface positions and overlaying labels. Custom label mapping is supported for cleaner presentation. Interactive hover and axis ticks now display true category names. HTML export logic was refactored to prevent duplicate writes and ensure reliable saving across plot modes.

Version 0.0.5a5

• Update to plot_3d_pdp docstring

• Adds full categorical feature support to plot_3d_pdp while preserving backward compatibility with numeric grids. The function now renders categorical axes correctly in both Matplotlib and Plotly by mapping categories to numeric surface positions and overlaying labels. Custom label mapping is supported for cleaner presentation. Interactive hover and axis ticks now display true category names. HTML export logic was refactored to prevent duplicate writes and ensure reliable saving across plot modes.

Version 0.0.5a4

• Adds full categorical feature support to plot_3d_pdp while preserving backward compatibility with numeric grids. The function now renders categorical axes correctly in both Matplotlib and Plotly by mapping categories to numeric surface positions and overlaying labels. Custom label mapping is supported for cleaner presentation. Interactive hover and axis ticks now display true category names. HTML export logic was refactored to prevent duplicate writes and ensure reliable saving across plot modes.

Version 0.0.5a3

Major Features Added

• Axis Limits: Added xlim and ylim parameters for standardizing axis ranges across multiple model comparisons

• Bottom Legend Support: Automatic figure height adjustment when legend_loc="bottom" to prevent legend overlap with x-axis labels

• Multi-Model Layout: Improved default layout for multiple models with plot_type="all" - now arranges as one row per model (6 columns × N rows) instead of mixed layout

Bug Fixes

• Heteroskedasticity Tests: Fixed categorical variable handling in all tests (Breusch-Pagan, White, Goldfeld-Quandt) by encoding categorical columns before running tests

• Legend Formatting:

  • Fixed legend duplication bug in Scale-Location plot (removed test name prepending since interpretations already contain test names)

  • Fixed histogram legend to use apply_legend() for consistent formatting

  • Fixed legend kwargs not being properly passed to Scale-Location plot

• Group Category Handling: Fixed KeyError when group_category column not in X DataFrame by properly checking column existence before filtering predictor columns

• Index Alignment: Fixed AssertionError when using external group_category array by ensuring Series index matches X.index

• Python 3.8 Compatibility: Fixed LaTeX rendering error in Scale-Location y-axis label by replacing \text{} with \mathrm{} for matplotlib <3.3 compatibility

Enhancements

• Scale-Location Y-Axis: Changed to LaTeX notation r"$\sqrt{|\text{Std. Residuals}|}$" for better readability

• Histogram Overlay: Removed normal distribution overlay from histogram_type="frequency" for cleaner, simpler visualization (overlay still present for histogram_type="density")

• Text Wrapping: Added text_wrap parameter support to all subplot titles (previously only worked for suptitles)

• Helper Functions:

  • Created apply_axis_limits() helper in plot_utils.py

  • Enhanced apply_legend() to handle bottom legend resizing with flag-based prevention of multiple resizes

• Refactoring: Refactored plot_threshold_metrics() to use apply_plot_title() and apply_legend() helpers for consistency

Version 0.0.5a2

Testing Improvements

• Added 150+ comprehensive unit tests covering:

  • Edge cases and error handling

  • Parameter validation

  • Different input pathways (model vs y_prob vs y_pred)

  • Group category functionality

  • Styling and customization options

  • Integration scenarios

Code Quality

• Refactored monolithic 3866-line model_metrics.py into modular components:

  • model_evaluator.py - Main plotting and evaluation functions

  • metrics_utils.py - Utility functions and calculations

  • plot_utils.py - Plotting helper functions

• Improved code maintainability and organization

• Enhanced error messages and validation

• Operating point visualization with two methods:

  • operating_point_method='youden' - Youden’s J statistic

  • operating_point_method='closest_topleft' - Closest to top-left corner

• DeLong test support for AUC comparison between models via delong parameter

• Legend ordering - Proper organization: AUC curves → Random Guess → Operating Points

• Custom operating point styling via operating_point_kwgs

Residual Diagnostics Expansion

• New plot types:

  • 'influence' - Influence plot with Cook’s distance bubbles

  • 'predictors' - Individual residual plots for each predictor

• Heteroskedasticity testing with multiple methods:

  • 'breusch_pagan' - Breusch-Pagan test

  • 'white' - White’s test

  • 'goldfeld_quandt' - Goldfeld-Quandt test

  • 'spearman' - Spearman rank correlation

  • 'all' - Run all tests

• LOWESS smoothing via show_lowess parameter

• Centroid visualization with two modes:

  • User-defined groups via group_category

  • Automatic K-means clustering via n_clusters

• Histogram types:

  • histogram_type='frequency' - Raw counts (default)

  • histogram_type='density' - Probability density with normal overlay

• Diagnostics table - Comprehensive model diagnostics via show_diagnostics_table

• Return diagnostics - Programmatic access via return_diagnostics=True

Group Category Support

• All classification plots now support group_category parameter:

  • ROC curves with per-group AUC and counts

  • PR curves with per-group metrics

  • Calibration curves with per-group calibration

• Residual diagnostics support group visualization with centroids

• Summary performance supports grouped classification metrics

Gain Chart Enhancement

• Gini coefficient calculation and display via show_gini parameter

• Custom decimal places for Gini via decimal_places parameter

Legend Customization

• Legend location now supports:

  • Standard matplotlib locations (‘best’, ‘upper right’, etc.)

  • 'bottom' - Places legend below plot (perfect for group categories)

• Automatic legend ordering for better readability

summarize_model_performance

• Added include_adjusted_r2 for regression models

• Added group_category for grouped classification metrics

• Added overall_only for regression to show only aggregate metrics

• Improved coefficient ordering (intercept first)

• Better handling of feature importances for tree-based models

show_confusion_matrix

• Added show_colorbar parameter (default: False)

• Added labels parameter to toggle TN/FP/FN/TP labels

• Improved font size controls (inner_fontsize, label_fontsize, tick_fontsize)

show_roc_curve

• Added show_operating_point and operating_point_method

• Added operating_point_kwgs for custom styling

• Added delong parameter for AUC comparison

• Added group_category for stratified analysis

• Added legend_loc parameter

show_pr_curve

• Added legend_metric parameter (‘ap’ or ‘aucpr’)

• Added group_category for stratified analysis

• Added legend_loc parameter

show_calibration_curve

• Added show_brier_score parameter (default: True)

• Added brier_decimals for formatting

• Added group_category for stratified analysis

• Added legend_loc parameter

show_gain_chart

• Added show_gini parameter (default: False)

• Added decimal_places for Gini formatting

plot_threshold_metrics

• Added lookup_metric and lookup_value for threshold optimization

• Added model_threshold to highlight specific thresholds

• Added baseline_thresh to toggle baseline line

• Added custom styling: curve_kwgs, baseline_kwgs, threshold_kwgs, lookup_kwgs

show_residual_diagnostics

• Added plot_type options: ‘all’, ‘fitted’, ‘qq’, ‘scale_location’, ‘leverage’, ‘influence’, ‘histogram’, ‘predictors’

• Added heteroskedasticity_test with multiple test options

• Added show_lowess for trend lines

• Added lowess_kwgs for LOWESS styling

• Added group_category for stratified analysis

• Added group_kwgs for custom group styling

• Added show_centroids and centroid_kwgs

• Added centroid_type (‘clusters’ or ‘groups’)

• Added n_clusters for automatic clustering

• Added histogram_type (‘frequency’ or ‘density’)

• Added show_diagnostics_table and return_diagnostics

• Added show_plots to disable plotting

• Added show_outliers and n_outliers for labeling

• Added legend_loc parameter

• Added legend_kwgs to control legend display for groups, centroids, clusters, and het_tests

• Added kmeans_rstate for reproducible clustering

• Added n_cols and n_rows for custom subplot layouts

• Added point_kwgs for scatter point styling (supports edgecolor, linewidth, etc.)

Bug Fixes

• Fixed confusion matrix colorbar removal when show_colorbar=False

• Fixed duplicate text handling in confusion matrix displays

• Fixed legend placement for grouped visualizations

• Fixed text wrapping for long titles

• Fixed LOWESS exception handling (now fails gracefully)

• Fixed feature importance display for tree-based models

• Fixed coefficient ordering in regression output

• Fixed empty metric columns in regression feature importance rows

Documentation Improvements

• Comprehensive docstrings for all major functions

• Parameter descriptions with examples

• Error message improvements for better debugging

• Type hints and validation error messages

• Usage examples in docstrings

Testing

• Test suite expanded from ~50 tests to 152 tests

• Coverage increased from 50% to 86% on core modules

• All edge cases and error conditions tested

• Integration tests for real-world workflows

• Parametrized tests for systematic coverage

Performance

• No performance regressions

• Modular code structure improves maintainability

• Efficient calculation caching where applicable

Migration Guide

From 0.0.5a1 to 0.0.5a2:

No changes required - all existing code will work as before. New features are opt-in:

Version 0.0.5a1

Added

• Operating Point Visualization for ROC Curves: Added show_operating_point parameter to display optimal classification thresholds on ROC curves with two methods:

  • youden: Youden’s J statistic (maximizes TPR - FPR)

  • closest_topleft: Point closest to top-left corner (minimizes distance to perfect classifier)

  • Configurable via operating_point_method and operating_point_kwgs parameters

  • Operating points display threshold values in legends and appear as markers on curves

• Gini Coefficient for Gain Charts: Added automatic calculation and display of Gini coefficient in show_gain_chart()

  • Prints Gini coefficient for each model (default: 3 decimal places)

  • Displays in legend labels across all plot modes (overlay, subplots, single)

  • Configurable via show_gini and decimal_places parameters

• Legend Location Control: Added legend_loc parameter to all plotting functions for flexible legend positioning

  • Supports standard matplotlib locations ('lower right', 'upper left', 'best', etc.)

  • Special ‘bottom’ option places legend below plot with proper spacing

  • Available in: show_roc_curve(), show_pr_curve(), show_calibration_curve(), show_lift_chart(), show_gain_chart()

Improved

• Legend Ordering for ROC Curves: Standardized legend entry order across all plot modes

  • Order: Model curves with AUC → Random Guess baseline → Operating points

  • Ensures consistent, intuitive legend presentation

• Overlay Mode for ROC Curves: Enhanced operating point display in overlay plots

  • Combined AUC and operating point threshold in single legend entry

  • Format: “Model Name (AUC = 0.XX, Op = 0.XX)”

  • Operating point markers appear on curves without duplicate legend entries

Technical Details

• Operating points calculated post-ROC curve generation using optimal threshold selection

• Gini coefficient derived from area under gain curve: Gini = 2 × AUGC - 1

• Legend positioning uses bbox_to_anchor for 'bottom' placement with dynamic spacing

• All changes maintain backward compatibility with existing code

Version 0.0.4a10

Refactored and stabilized the summarize_model_performance function to improve consistency across classification and regression workflows while preserving the exact formatting logic for printed outputs and regression coefficient display.

Changes

• Consolidated redundant metric computation into dedicated helper functions for classification and regression metrics.

• Ensured regression coefficients, intercepts, and feature importances are retained and ordered correctly in the final DataFrame output.

• Fixed grouped classification output so Model Threshold always appears last, and group headers correctly reflect category names.

• Added conditional handling for grouped classification to prevent KeyError when the "Model" column is absent.

• Preserved the original manual formatting block to maintain Leon’s custom printing logic for both classification and regression:

  • Right-aligned all table columns for readability.

  • Retained separator-based visual formatting and model-wise breaks.

  • Preserved coefficient and intercept reporting behavior exactly as before, ensuring regression results remain interpretable and consistent.

Impact

• Classification and regression now produce stable, well-ordered, and readable summaries.

• Grouped and non-grouped runs behave consistently without disrupting regression coefficient output.

• Backward compatibility with previous console and DataFrame output formats maintained.

Version 0.0.4a9

This release introduces a new parameter, brier_decimals, to the show_calibration_curve() function, allowing users to control the number of decimal places displayed for the Brier score.

Changes Made

• Added brier_decimals parameter (default: 3) next to show_brier_score.

• Updated Brier score display logic to format using round(brier_score, brier_decimals).

• Improved readability and precision consistency across calibration plots.

Impact

• No breaking changes.

• Users now have finer control over Brier score precision in calibration curve visualizations.

Quick Example

from model_metrics import show_calibration_curve
show_calibration_curve(model, X, y, show_brier_score=True, brier_decimals=4)

Version 0.0.4a8

Summary:

Updated hanley_mcneil_auc_test() function to perform a large-sample z-test for comparing correlated AUCs, based on Hanley & McNeil (1982), an analytical approximation of DeLong’s test.

Key Changes:

• Implemented hanley_mcneil_auc_test() with parameters:

  • y_true, y_scores_1, y_scores_2 for AUC comparison.

  • Optional model_names, verbose, and return_values arguments for flexible use.

• Added formatted, human-readable print output (when verbose=True).

• Enabled optional programmatic access with return_values=True.

• Adopted NumPy-style docstring for clarity and consistency.

• Integrated helper into show_roc_curve() to enable AUC significance testing when the delong argument is provided.

Notes: This helper can also be used as a standalone function for independent AUC comparison between two models, outside of visualization workflows.

Version 0.0.4a7

• DeLong’s test (Hanley & McNeil approximation)

  • Implemented a new helper function hanley_mcneil_auc_test() for approximate DeLong’s AUC comparison.

  • Integrated the helper inside show_roc_curve() to optionally print AUC differences and p-values between two models.

  • Added corresponding pytest coverage under test_show_roc_curve_with_delong().

• Group category support

  • Added the group_category input to summarize_model_performance() to generate subgroup-level performance summaries.

  • Enables stratified metric reporting for fairness or demographic analysis.

Version 0.0.4a6

Reworded the print message inside plot_threshold_metrics() for clarity.

Old:

print(
      f"Best threshold for {lookup_metric} = "
      f"{round(lookup_value, decimal_places)} is: "
      f"{round(best_threshold, decimal_places)}"
)

New:

print(
    f"Best threshold for target {lookup_metric} of "
    f"{round(lookup_value, decimal_places)} is "
    f"{round(best_threshold, decimal_places)}"
)

This removes the equals sign and colon, and adds “target” for a smoother, more descriptive sentence.

Version 0.0.4a8

• Added a minimal type check to ensure y_prob is always a list at the start of each affected function:

• summarize_model_performance

• show_calibration_curve

• show_confusion_matrix

• show_lift_chart

• show_gain_chart

• show_roc_curve

• show_pr_curve

# Ensure y_prob is always a list of NumPy arrays
if isinstance(y_prob, np.ndarray):
   y_prob = [y_prob]

This allows y_prob[0] indexing to work whether the caller provides a single NumPy array or a list of arrays.

• Updated unittests

Version 0.0.4a4

• Corrected README to reflect the current version.

• Previous release did not update the README properly because the file was not saved before publishing.

• No functional changes to the library.

Version 0.0.4a3

• Added missing scipy (>=1.8,<=1.14.0) requirement to the README.

Version 0.0.4a2

This version updates pyproject.toml and requirements.txt to restrict SciPy to >=1.8,<=1.14.0.

• Prevents installation of scipy==1.14.1+, which removes _lazywhere and breaks statsmodels.

• Keeps compatibility with model_tuner and Colab environments.

• Bumps package version for release.

• Updated scipy dependency to >=1.8,<=1.14.0

• Synced requirements.txt with updated constraints

Version 0.0.4a1

• Replaced the old grid parameter with subplots across plotting functions for consistency.

• Standardized gridline handling by replacing unconditional plt.grid() calls with plt.grid(visible=gridlines)

Why

• Aligns function signatures to use subplots consistently instead of grid.

• Makes gridline visibility configurable through a single gridlines flag.

• Cleaner charts when gridlines=False, no visual change when gridlines=True.

Version 0.0.4a

Summary

Added the ability to pass predicted probabilities (y_prob) directly into the functions in model_evaluator.py as an alternative to supplying a fitted model and feature matrix. This flexibility lets end users evaluate results in two ways:

• Using a model object with X (current behavior)

• Or passing y_prob directly (new option)

Details

• Updated all relevant evaluator functions (summarize_model_performance, plot_threshold_metrics, etc.) to accept y_prob as input.

• Added input validation: functions now check that either (model and X) or y_prob are provided, not both missing.

• Preserved existing model-based workflows for backward compatibility.

• Extended unit tests in unittests/ to cover the new probability-based path, including edge cases and validation errors.

Why

End users sometimes already have predicted probabilities from external pipelines or pre-computed experiments. This change avoids forcing them to re-supply the model, streamlining the evaluation process.

Version 0.0.3a

• Added "plotly>=5.18.0, <=5.24.1" in pyproject.toml, setup.py, README_min.md –> for partial_dependence.py functions

Version 0.0.2a

• Add show_ks_curve function and enhance summarize_model_performance by @lshpaner in https://github.com/lshpaner/model_metrics/pull/1

• Add plot_threshold_metrics Function by @lshpaner in https://github.com/lshpaner/model_metrics/pull/2

• Add pr_feature_plot and Update roc_feature_plot for Enhanced Visualization by @lshpaner in https://github.com/lshpaner/model_metrics/pull/3

• Reg table enhance by @lshpaner in https://github.com/lshpaner/model_metrics/pull/4

• Rmvd (%) from MAPE header by @lshpaner in https://github.com/lshpaner/model_metrics/pull/5

• Moved roc legend to lower right default by @lshpaner in https://github.com/lshpaner/model_metrics/pull/6

• Allow Flexible Inputs and Save Behavior for show_roc_curve() by @lshpaner in https://github.com/lshpaner/model_metrics/pull/7

• Prcurve calc tests by @lshpaner in https://github.com/lshpaner/model_metrics/pull/8

• Removed unused imports and functions by @lshpaner in https://github.com/lshpaner/model_metrics/pull/9

• changed saving nomenclature in show_confusion_matrix by @lshpaner in https://github.com/lshpaner/model_metrics/pull/10

• Fix Calibration Curve Grid Plot Behavior and Update Model Nomenclature by @lshpaner in https://github.com/lshpaner/model_metrics/pull/11

• Improved support for multiple models and group categories in calibration curve by @lshpaner in https://github.com/lshpaner/model_metrics/pull/13

• Upd. plot_threshold_metrics w/ new lookup_kwgs and legend logic by @lshpaner in https://github.com/lshpaner/model_metrics/pull/14

• Rmv. unused arguments by @lshpaner in https://github.com/lshpaner/model_metrics/pull/15

• Move PDF-related Functions from eda_toolkit to model_metrics by @lshpaner in https://github.com/lshpaner/model_metrics/pull/16

Full Changelog: https://github.com/lshpaner/model_metrics/compare/0.0.1a…0.0.2a

Version 0.0.1a

• Updated unit tests and README

• Added statsmodels to library imports

• Added coefficients and p-values to regression summary

• Added regression capabilities to summarize_model_performance

• Added lift and gains charts

• Updated versions for earlier Python compatibility