Welcome to the EDA Toolkit Python Library Documentation!

Note

This documentation is for eda_toolkit version 0.0.22.

The eda_toolkit is a comprehensive library designed to streamline and enhance the process of Exploratory Data Analysis (EDA) for data scientists, analysts, and researchers. This toolkit provides a suite of functions and utilities that facilitate the initial investigation of datasets, enabling users to quickly gain insights, identify patterns, and uncover underlying structures in their data.

Project Links

1. PyPI Page
2. GitHub Repository
3. Comprehensive Google Colab Notebook (End-to-End Examples)

Tip

A comprehensive Google Colab notebook is linked below that walks through nearly everything discussed in this documentation, including data loading, cleaning, visualization, and reporting workflows using the eda_toolkit. This notebook is designed as a practical, end-to-end companion to the documentation and can be run entirely in the browser without any local setup.

Open the Google Colab notebook

What is EDA?

Exploratory Data Analysis (EDA) is a crucial step in the data science workflow. It involves various techniques to summarize the main characteristics of the data, often with visual methods. EDA helps in understanding the data better, identifying anomalies, discovering patterns, and forming hypotheses. This process is essential before applying any machine learning models, as it ensures the quality and relevance of the data.

Purpose of EDA Toolkit

The eda_toolkit library is a comprehensive suite of tools designed to streamline and automate many of the tasks associated with Exploratory Data Analysis (EDA). It offers a broad range of functionalities, including:

• Data Management: Tools for managing directories, generating unique IDs, standardizing dates, and handling common DataFrame manipulations.

• Data Cleaning: Functions to address missing values, remove outliers, and correct formatting issues, ensuring data is ready for analysis.

• Data Visualization: A variety of plotting functions, including KDE distribution plots, stacked bar plots, scatter plots with optional best fit lines, and box/violin plots, to visually explore data distributions, relationships, and trends.

• Descriptive and Summary Statistics: Methods to generate comprehensive reports on data types, summary statistics (mean, median, standard deviation, etc.), and to summarize all possible combinations of specified variables.

• Reporting and Export: Features to save DataFrames to Excel with customizable formatting, create contingency tables, and export generated plots in multiple formats.

Key Features

• Ease of Use: The toolkit is designed with simplicity in mind, offering intuitive and easy-to-use functions.

• Customizable: Users can customize various aspects of the toolkit to fit their specific needs.

• Integration: Seamlessly integrates with popular data science libraries such as Pandas, NumPy, Matplotlib, and Seaborn.

• Documentation and Examples: Comprehensive documentation and examples to help users get started quickly and effectively.

Prerequisites

Before you install eda_toolkit, ensure your system meets the following requirements:

• Python: version 3.7.4 or higher is required to run eda_toolkit.

Additionally, eda_toolkit depends on the following packages, which will be automatically installed when you install eda_toolkit:

• jinja2: version 3.1.4 (exact version required)

• matplotlib: version 3.5.3 or higher, capped at 3.9.2

• nbformat: version 4.2.0 or higher, capped at 5.10.4

• numpy: version 1.21.6 or higher, capped at 2.1.0

• pandas: version 1.3.5 or higher, capped at 2.2.3

• plotly: version 5.18.0 or higher, capped at 5.24.0

• scikit-learn: version 1.0.2 or higher, capped at 1.5.2

• scipy: version 1.5.4 or higher, capped at 1.7.3

• seaborn: version 0.12.2 or higher, capped below 0.13.2

• tqdm: version 4.66.4 or higher, capped below 4.67.1

• xlsxwriter: version 3.2.0 (exact version required)

Installation

You can install eda_toolkit directly from PyPI:

pip install eda_toolkit

Description

This guide provides detailed instructions and examples for using the functions provided in the eda_toolkit library and how to use them effectively in your projects.

For most of the ensuing examples, we will leverage the Census Income Data (1994) from the UCI Machine Learning Repository [1]. This dataset provides a rich source of information for demonstrating the functionalities of the eda_toolkit.

[1]

Kohavi, R. (1996). Census Income. UCI Machine Learning Repository. https://doi.org/10.24432/C5GP7S.

---

Table of Contents

Getting Started

• Welcome to the EDA Toolkit Python Library Documentation!
  • Project Links
  • What is EDA?
  • Purpose of EDA Toolkit
  • Key Features
  • Prerequisites
  • Installation
• Description

Data Management

• Data Management Overview
• Data Management Techniques
  • Path directories
    • ensure_directory()
  • Adding Unique Identifiers
    • add_ids()
  • Trailing Period Removal
    • strip_trailing_period()
  • Standardized Dates
    • parse_date_with_rule()
  • DataFrame Analysis
    • dataframe_profiler()
    • Census Income Example
    • DataFrame Column Names
    • Calculation Details
  • Generating Summary Tables for Variable Combinations
    • summarize_all_combinations()
  • Saving DataFrames to Excel with Customized Formatting
    • save_dataframes_to_excel()
  • Creating Contingency Tables
    • contingency_table()
  • Generating Summaries (Table 1)
    • generate_table1()
    • Example 1: Mixed Summary Table (value_counts=False)
    • Example 2: Mixed Summary Table (value_counts=True)
    • Example 3: Group Comparisons (P-Values)
  • Highlighting Specific Columns in a DataFrame
    • highlight_columns()
  • Binning Numerical Columns
  • Group-by Imputer
    • groupby_imputer()
    • Group-wise Imputation Examples
  • Delete Inactive DataFrames
    • del_inactive_dataframes()
    • Example 1: List Active DataFrames (No Deletion)
    • Example 2: Delete Everything Except a Single DataFrame
    • Example 3: Dry Run (Preview Deletions)
    • Example 4: Include IPython Output Cache Variables
    • Example 5: Track DataFrame Memory Usage
    • Example 6: Track DataFrame Memory and Process RSS
    • Example 7: Programmatic Usage (No Console Output)

Plotting Functions

• Creating Effective Visualizations
  • Heuristics for Visualizations
  • Methodologies
• Histogram Distribution Plots
  • plot_distributions()
  • Deprecation Notice
  • KDE and Histograms Example
  • Histogram Example (Density)
  • Histogram Example (Count)
  • Histogram Example - (Mean and Median)
  • Histogram Example - (Mean, Median, and Std. Deviation)
  • KDE and Parametric Density Fit Example
• Grouped Distributions
  • grouped_distributions()
  • Grouped Density Plot Example
  • Grouped Histogram Plot Example
• Distribution Goodness-of-Fit Plots
  • distribution_gof_plots()
  • Theoretical QQ Plot Example
  • Empirical QQ Plot Example (Sample vs Reference Data)
  • CDF Plot Example (Lower and Upper Tails)
• Feature Scaling and Outliers
  • data_doctor()
  • Available Scale Conversions
  • Box-Cox Transformation Example 1
  • Box-Cox Transformation Example 2
  • Data Fraction Usage
    • Retaining a Sample for Analysis
  • Logit Transformation Example
  • Plain Outliers Example
    • Observed Outliers Sans Cutoffs
    • Treated Outliers With Cutoffs
  • RobustScaler Outliers Example
  • ECDF Distribution Example
• Stacked Crosstab Plots
  • stacked_crosstab_plot()
  • Stacked Bar Plots With Crosstabs Example
  • Pivoted Stacked Bar Plots Example
  • Non-Normalized Stacked Bar Plots Example
  • Regular Non-Stacked Bar Plots Example
• Outcome Crosstab Plots
  • outcome_crosstab_plot()
  • Outcome Crosstab Example
• Box and Violin Plots
  • box_violin_plot()
  • Box Plots: Subplots Example
  • Violin Plots: Subplots Example
  • Pivoted Violin Plots: Subplots Example
• Scatter Plots and Best Fit Lines
  • Scatter Fit Plot
    • scatter_fit_plot()
    • Regression-Centric Scatter Plots Example
    • Scatter Plots Grouped by Category Example
    • Scatter Plots (All Combinations Example)
    • Scatter Plots: Excluding Specific Combinations
      • Example 1
      • Example 2
• Correlation Matrices
  • flex_corr_matrix()
  • Triangular Correlation Matrix Example
  • Full Correlation Matrix Example

Theoretical Overview

• Gaussian Assumption for Normality
  • Histograms and Kernel Density Estimation (KDE)
• Pearson Correlation Coefficient
• Box-Cox Transformation
  • Mathematical Definition
  • Properties and Benefits
  • Practical Considerations
  • Applications in Modeling
    • Confidence Intervals for Lambda
• The Yeo-Johnson Transformation
• Median and IQR Scaling
  • Centering Data Using the Median
  • Explanation of Each Component
  • Example Calculation
• Logit Transformation
  • Purpose and Assumptions
• Parameter Estimation for Density Models and Distribution GOF Plots
  • Method of Moments (MoM)
  • Maximum Likelihood Estimation (MLE)
  • Relationship to Density Overlays
• Empirical Cumulative Distribution Function: ECDF
  • Definition
  • Interpretation
  • Statistical Properties
    • Consistency
    • Unbiasedness
    • Variance
  • Relationship to Order Statistics
  • Comparison with Histogram and KDE
  • Tail Behavior and Exceedance Functions
  • Practical Considerations

About EDA Toolkit

• ASCII Art
  • Overview
  • Features
  • ASCII Art Collection
    • print_art()
  • Examples
• Acknowledgements
• Contributors/Maintainers
• Citing EDA Toolkit
• Changelog
  • Version 0.0.22
  • Version 0.0.21
  • Version 0.0.20
    • What’s Changed
  • Version 0.0.19
  • Version 0.0.18
  • Version 0.0.17
  • Version 0.0.16
  • Version 0.0.15
    • Scatter Plot Function Updates
      • Avoid In-Place Modification of exclude_combinations
      • Improve Progress Tracking and Resolve Last Plot Saving Issue
      • Updated tqdm Saving Logic in scatter_fit_plot
  • Version 0.0.14
    • Ensure Crosstabs Dictionary is Populated with return_dict=True
    • Enhancements and Fixes for scatter_fit_plot Function
    • Fix Default Title and Filename Handling in flex_corr_matrix
  • Version 0.0.13
    • Add ValueError for Insufficient Pool Size in add_ids and Enhance ID Deduplication
    • Enhance strip_trailing_period to Support Strings and Mixed Data Types
    • Changes in stacked_crosstab_plot
    • Add Environment Detection to dataframe_columns Function
    • Add tqdm Progress Bar to dataframe_columns Function
    • Other Enhancements and Fixes
  • Version 0.0.12
    • New Features
    • Improvements
  • Version 0.0.11
  • Version 0.0.10
  • Version 0.0.9
  • Version 0.0.8
  • Version 0.0.8c
  • Version 0.0.8b
  • Version 0.0.8a
  • Version 0.0.7
  • Version 0.0.6
  • Version 0.0.5
  • Version 0.0.4
  • Version 0.0.3
  • Version 0.0.2
  • Version 0.0.1rc0
  • Version 0.0.1b0
  • Version 0.0.1b0
  • Version 0.0.1b0
  • Version 0.0.1b0
• References