{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "In the **CRISP-DM** (Cross-Industry Standard Process for Data Mining) framework, the second step following **business understanding** is **data understanding** and exploratory data analysis (EDA). This important phase involves comprehending the complexities of the dataset at hand and preparing it for subsequent modeling steps. **Descriptive statistics** play a crucial role in EDA, providing a comprehensive **snapshot** of the data's **central tendency**, **dispersion**, and **distributional** characteristics. Central tendency metrics like mean, median, and mode offer insights into the dataset's typical values, while measures of dispersion such as standard deviation and range quantify the extent of variability. Distributional statistics including skewness and kurtosis, along with box plots, illuminate the data's spread and shape. EDA is instrumental in describing the **overview** and **characteristics** of the data, ensuring its **quality checks**, and unraveling **relationships** between different features by **hypothesis testing**. This phase sets the foundation for informed decision-making in subsequent stages of the data mining process." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Problem Definition \n", "In the area of financial decision-making, a dataset named \"bank-loan\" takes center stage, focusing on the critical domain of **credit scoring**. With a pool of **700 records** derived from **bank customers** who successfully obtained loans and conscientiously repaid their installments, the dataset captures the repayment outcomes, categorized as 1 and 0 for **default statuses**. The overarching objective is to develop a robust credit scoring system, a discerning arbiter for loan approvals. This system will draw on various factors, including age, education, employment duration, tenure at the current residence, income levels, debit-to-income ratio, credit-to-debit ratio, and other debts reported at the time of loan application. By delving into the intricate details of these parameters, the aim is to construct a **predictive model** that empowers the financial institution to make informed decisions when considering loan applications, thereby optimizing risk management and ensuring the soundness of lending practices." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "df = pd.read_csv('/kaggle/input/bank-loan/Bankloan.txt')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Certainly, let's break down the meaning of each field in the \"bank-loan\" dataset:\n", "\n", "**Age:** Age in years.\n", "\n", "**Ed:** 1-Did not complete high school 2-High school degree 3-Some college 4-College degree 5-Post-undergraduate degree\n", "\n", "**Employ:** Years with current employer\n", "\n", "**Address:** Years at current address\n", "\n", "**Income:** Household income in thousands\n", "\n", "**Debtinc:** Debt to income ratio (x100)\n", "\n", "**Creddebt:** Credit card debt in thousands\n", "\n", "**Othdebt:** Other debt in thousands\n", "\n", "**Default:** The \"Default\" field is the target variable, indicating previously defaulted. It takes binary values, with 1 typically denoting a \"bad\" default status and 0 representing a \"good\" repayment history." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### EDA answers some critical questions about data \n", "\n", "These questions will come down to diagnosing potentially critical data quality issues (or general data characteristics) that may compromise the operations of the machine learning models or their performance throughout the whole data science lifecycle.\n", "\n", "* Do I have missing data? How does my missing data behave?\n", "* Are all instances labelled? Are they trustful?\n", "* Are these features consistent and informative?\n", "* Do I have enough data to train my model?\n", "* Is my data noisy? \n", "\n", "As standard machine learning algorithms expect their input to follow a few assumptions on the data (e.g., **representativeness**, **completeness**, **consistency**), these questions need to be addressed even before considering building a model. Let's consider examples related to representativeness, completeness, and consistency in a dataset for a data science project:\n", "\n", "**Representativeness:** \n", "\n", "In a dataset for customer churn prediction, representativeness would mean that the dataset includes a diverse **sample** of customers, including various demographics, usage patterns, and regions. If the dataset predominantly represents only a specific group of customers, the model might not **generalize** well to the entire customer base. \n", "\n", "**Completeness:**\n", " \n", "In a healthcare dataset tracking patient outcomes, completeness is crucial. If a significant percentage of patient records **lacks key informatio**n such as diagnosis, treatment history, or follow-up data, it can hinder the ability to draw comprehensive insights or build accurate predictive models. \n", "\n", "**Consistency:**\n", " \n", "Consider a dataset that records sales transactions, and there's a column for currency. Consistency would mean that the currency values are uniform throughout the dataset. If some entries use symbols like \"$,\" while others use \"USD,\" it introduces inconsistency. Ensuring a standardized format enhances the reliability of analyses.\n", "\n", "This is where profiling your data through and through is a required step in **every data science project**, and one that will save you a lot of **time**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### What is Data Profiling, and why is it so important?\n", "**Data Profiling** is precisely this step of diagnosing your data.\n", "\n", "Think of it as performing a regular checkup to your data to see if everything is “good to go” for model building, or if there is something that needs to be handled properly first.\n", "\n", "A standard way of profiling your data is through **Exploratory Data Analysis (EDA)**. This involves a deep exploration of the data, trying to understand its intricacies and properties as fully as possible, namely through:\n", "\n", "**Dataset Overview and Characteristics:** Determining the number of features and observations, types of features, duplicate records; \n", "\n", "**Univariate Visualization and Feature Assessment:** Analyzing descriptive statistics such as distribution, range, variation, scale, common values, missing values, possible outliers, and evaluating the need for re-indexing, reformatting, or remove values as well as other operations such as data imputation or augmentation; \n", "\n", "**Multivariate Visualization and Correlation Assessment:** Investigating **patterns** and **relationships** between features, behavior or missing values, and assessing the need for dimensionality reduction and feature selection. \n", "\n", "* This is a thorough and crucial step in the data science lifecycle, but a **very-time consuming** one as well. It’s an intimidating process for anyone who is not a trained data scientists, and even for data scientists, it might be difficult to know where to look precisely. \n", "\n", "* Without systematic processes and tools, the success of EDA gets highly dependent on the **expertise** and experience of the person conducting the analysis.\n", "\n", "* This is a giant responsibility, especially given that this is an **iterative process**, **error-prone**, with ever-changing requirements, thresholds, and constraints as the model is deployed to production. \n", "\n", "* **Data Profiling does not end** after the model is built and deployed because real-world domains are not immutable, and neither is data quality. \n", "\n", "As the new Data-Centric **AI paradigm** is extensively advocating, data profiling must become a **standard**, **systematic**, **iterative**, **continuous**, and **automated process** performed at each step of data analysis pipelines." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pandas Profiling: Automated EDA \n", "Having reached an outstanding milestone of 10K stars on GitHub, the data science community has praised Pandas Profiling as the top open-source tool for data profiling. \n", "\n", " \n", "Reference:https://medium.com/ydata-ai/auditing-data-quality-with-pandas-profiling-b1bf1919f856" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install ydata_profiling" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from ydata_profiling import ProfileReport\n", "\n", "# Generate a profile report\n", "profile = ProfileReport(df, title=\"Bankloan data EDA\", type_schema = {\"Ed\": \"categorical\", \"Default\": \"categorical\"})\n", "\n", "# Save the report to an HTML file\n", "profile.to_file(\"your_dataset_profile_report.html\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Needed Preparation Actions\n", "The obtained results from EDA suggest that the following data cleaning and preparation steps must be undertaken:\n", "* The 'age' column contains values that are out of range and should be converted to null. \n", "\n", "* The distribution of the 'age' suggests that values have been rounded, so it might be better to categorize it.\n", "* In the 'ed' column, the values '4' and '5' should be merged, as the frequency of '5' is low. \n", "* The 'default' column includes inconsistent codes and should be corrected\n", "* The missing values in 'age', 'ed' and 'income' should be imputed.\n", "* The outliers should be detected and treated.\n", "* The 'employ','address','income', 'debtinc', 'creddebt' and 'otherdebt' have skewed distribution and should be transformed or categorized.\n", "* The target field is imbalanced; it should be taken into consideration during modeling." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### EDA with Consideration of Target Field \n", "You can separate the dataset based on the target field ('default') and compare the descriptive statistics and graphs for each class (0 or 1) to unveil the relationship between input fields and the target" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from ydata_profiling import ProfileReport\n", "\n", "df_default_0 = df[df.default == \"0\"]\n", "df_default_1 = df[df.default == \"1\"]\n", "\n", "# Generate a profile report\n", "profile_0 = ProfileReport(df_default_0, title=\"Bankloan EDA 0\",minimal=True,type_schema = {\"Ed\": \"categorical\", \"Default\": \"categorical\"})\n", "profile_1 = ProfileReport(df_default_1, title=\"Bankloan EDA 1\",minimal=True, type_schema = {\"Ed\": \"categorical\", \"Default\": \"categorical\"})\n", "\n", "comparison_report = profile_0.compare(profile_1)\n", "comparison_report.to_file(\"comparison.html\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more details see https://docs.profiling.ydata.ai/latest/getting-started/concepts/" ] } ], "metadata": { "kaggle": { "accelerator": "none", "dataSources": [ { "datasetId": 4216596, "sourceId": 7273365, "sourceType": "datasetVersion" } ], "dockerImageVersionId": 30626, "isGpuEnabled": false, "isInternetEnabled": true, "language": "python", "sourceType": "notebook" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.19" } }, "nbformat": 4, "nbformat_minor": 4 }