Avaliação_de_Risco_de_Crédito_Nubank.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "[PROJETO] Avaliação de Risco de Crédito.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/carlosfab/data_science/blob/master/Avalia%C3%A7%C3%A3o_de_Risco_de_Cr%C3%A9dito_Nubank.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZNvmL_lMsAqU",
        "colab_type": "text"
      },
      "source": [
        "# Avaliação de Risco de Crédito\n",
        "\n",
        "\n",
        "Uma das área que causa mais problemas para bancos, financiadoras e empresários é a taxa de inadimplência por parte da sua carteira de clientes.\n",
        "\n",
        "Avaliar se uma pessoa irá deixar de cumprir com suas obrigações financeiras é de extrema importância, e pode causar um grande impacto no balanço da Instituição. Cada vez mais, soluções vêm sendo desenvolvidas e aprimoradas visando minimizar o risco de *default*. \n",
        "\n",
        "*Default* é o termo utilizado para indicado o não cumprimento das obrigações e/ou condições de um empréstimo (como financiamentos ou dívidas de cartão de crédito). Normalmente, o principal motivo para o descumprimento das condições de pagamento é incapacidade financeira do cliente.\n",
        "\n",
        "<p align=center>\n",
        "  <img src=\"http://sigmoidal.ai/wp-content/uploads/2019/10/2940191.jpg\" height=\"300px\">\n",
        "</p>\n",
        " \n",
        "Grandes bancos e Fintechs têm apostado cada vez mais em modelos de *Machine Learning* para previnir a inadimplência de alguns clientes.\n",
        "\n",
        "Dentre as principais instituições financeira, o [Nubank](https://nubank.com.br/sobre-nos) é uma das que mais tem se destacado no uso de Inteligência Artificial e times de *Data Science*.\n",
        "\n",
        "O conjunto de dados a ser utilizado neste Projeto de *Data Science* parte de uma competição realizada pela Startup [Nubank](https://nubank.com.br/sobre-nos) a fim de revelar talentos e potenciais contratações pela Fintech.\n",
        "\n",
        "## Contextualização do Problema\n",
        "\n",
        "Neste problema, o objetivo é prever qual a probabilidade de um cliente da Startup Nubank não coumprir com suas obrigações financeiras e deixar de pagar a sua fatura do Cartão de Crédito.\n",
        "\n",
        "<p align=center>\n",
        "<img src=\"http://sigmoidal.ai/wp-content/uploads/2019/10/Nubank_logo.png\" width=\"90px\"></p>\n",
        "  \n",
        "Vale ressaltar que essa avaliação deve ser realizada no momento em que o cliente solicita o cartão (normalmente no primeiro contato com a instituição).\n",
        "\n",
        "\n",
        "\n",
        "> OBJETIVO: Criar um modelo que forneça a probabilidade de um cliente virar inadimplente.\n",
        "\n",
        "\n",
        "Espera-se que um modelo seja capaz de minimizar as perdas financeiras do Nubank, porém minimizando também os falsos positivos.\n",
        "\n",
        "### Hipóteses Iniciais"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5PCK4RxLpqd8",
        "colab_type": "text"
      },
      "source": [
        "## Importar os Dados\n",
        "\n",
        "Os dados que serão utilizados nesta análise estão disponíveis para download por meio [deste link](http://dl.dropboxusercontent.com/s/xn2a4kzf0zer0xu/acquisition_train.csv?dl=0). Consiste basicamente em um arquivo `csv` contendo 45.000 entradas e 43 colunas.\n",
        "\n",
        "Este arquivo será importando para uma estrutura *DataFrame* utilizando a biblioteca `pandas` a fim de possibilitar sua manipulação e análise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n2X30-l1pprz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# importar os pacotes necessários\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "# definir o estilo do seaborn para os plots\n",
        "sns.set_style()\n",
        "\n",
        "# filtrar mensagens de warning\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LtQqAgvdp44q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = pd.read_csv(\"http://dl.dropboxusercontent.com/s/xn2a4kzf0zer0xu/acquisition_train.csv?dl=0\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2yTEh3oPE2KK",
        "colab_type": "text"
      },
      "source": [
        "## Análise Exploratória de Dados"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fHnIRz6DVqgj",
        "colab_type": "text"
      },
      "source": [
        "A primeira coisa que faremos após importar o *dataset* será examinar as dimensões do *DataFrame* e as primeiras entradas. Isso possibilitará criar uma consciência situacional inicial a respeito do formato de entrada e da estrutura geral dos dados.\n",
        "\n",
        "Em relação à dimensionalidade dos dados, nosso *dataset* é composto por 45 mil entradas e 43 colunas."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t1zbiuzFcdAT",
        "colab_type": "code",
        "outputId": "1fe2d8e3-bfa9-45cb-a854-21295e16bac1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# dimensões do df\n",
        "print(\"DIMENSÕES DO DATAFRAME:\")\n",
        "print(\"Linhas:\\t\\t{}\".format(df.shape[0]))\n",
        "print(\"Colunas:\\t{}\".format(df.shape[1]))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "DIMENSÕES DO DATAFRAME:\n",
            "Linhas:\t\t45000\n",
            "Colunas:\t43\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2M6b_OBwcduW",
        "colab_type": "text"
      },
      "source": [
        "Já as primeiras 5 entradas desse conjunto de dados permitem adquirirmos uma noção inicial sobre o tipo de formato, exemplos de entrada e formulação das hipóteses iniciais do processo investigativo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aA0ru6yL0njE",
        "colab_type": "code",
        "outputId": "d1ebaa16-7997-4549-b14c-2cdd8d47b94d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 635
        }
      },
      "source": [
        "# ver as 5 primeiras entradas\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ids</th>\n",
              "      <th>target_default</th>\n",
              "      <th>score_1</th>\n",
              "      <th>score_2</th>\n",
              "      <th>score_3</th>\n",
              "      <th>score_4</th>\n",
              "      <th>score_5</th>\n",
              "      <th>score_6</th>\n",
              "      <th>risk_rate</th>\n",
              "      <th>last_amount_borrowed</th>\n",
              "      <th>last_borrowed_in_months</th>\n",
              "      <th>credit_limit</th>\n",
              "      <th>reason</th>\n",
              "      <th>income</th>\n",
              "      <th>facebook_profile</th>\n",
              "      <th>state</th>\n",
              "      <th>zip</th>\n",
              "      <th>channel</th>\n",
              "      <th>job_name</th>\n",
              "      <th>real_state</th>\n",
              "      <th>ok_since</th>\n",
              "      <th>n_bankruptcies</th>\n",
              "      <th>n_defaulted_loans</th>\n",
              "      <th>n_accounts</th>\n",
              "      <th>n_issues</th>\n",
              "      <th>application_time_applied</th>\n",
              "      <th>application_time_in_funnel</th>\n",
              "      <th>email</th>\n",
              "      <th>external_data_provider_credit_checks_last_2_year</th>\n",
              "      <th>external_data_provider_credit_checks_last_month</th>\n",
              "      <th>external_data_provider_credit_checks_last_year</th>\n",
              "      <th>external_data_provider_email_seen_before</th>\n",
              "      <th>external_data_provider_first_name</th>\n",
              "      <th>external_data_provider_fraud_score</th>\n",
              "      <th>lat_lon</th>\n",
              "      <th>marketing_channel</th>\n",
              "      <th>profile_phone_number</th>\n",
              "      <th>reported_income</th>\n",
              "      <th>shipping_state</th>\n",
              "      <th>shipping_zip_code</th>\n",
              "      <th>profile_tags</th>\n",
              "      <th>user_agent</th>\n",
              "      <th>target_fraud</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>343b7e7b-2cf8-e508-b8fd-0a0285af30aa</td>\n",
              "      <td>False</td>\n",
              "      <td>1Rk8w4Ucd5yR3KcqZzLdow==</td>\n",
              "      <td>IOVu8au3ISbo6+zmfnYwMg==</td>\n",
              "      <td>350.0</td>\n",
              "      <td>101.800832</td>\n",
              "      <td>0.259555</td>\n",
              "      <td>108.427273</td>\n",
              "      <td>0.40</td>\n",
              "      <td>25033.92</td>\n",
              "      <td>36.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>65014.12</td>\n",
              "      <td>True</td>\n",
              "      <td>sjJbkqJS7cXalHLBFA+EOQ==</td>\n",
              "      <td>Ernn+uVXCMq/6ARrBCcd+A==</td>\n",
              "      <td>NCqL3QBx0pscDnx3ixKwXg==</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>N5/CE7lSkAfB04hVFFwllw==</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>07:52:34</td>\n",
              "      <td>444</td>\n",
              "      <td>outlook.com</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>51.0</td>\n",
              "      <td>leidelaura</td>\n",
              "      <td>645</td>\n",
              "      <td>(-29.151545708122246, -51.1386461804385)</td>\n",
              "      <td>Invite-email</td>\n",
              "      <td>514-9840782</td>\n",
              "      <td>57849.0</td>\n",
              "      <td>BR-MT</td>\n",
              "      <td>17528</td>\n",
              "      <td>{'tags': ['n19', 'n8']}</td>\n",
              "      <td>Mozilla/5.0 (Linux; Android 6.0.1; SGP771 Buil...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>bc2c7502-bbad-0f8c-39c3-94e881967124</td>\n",
              "      <td>False</td>\n",
              "      <td>DGCQep2AE5QRkNCshIAlFQ==</td>\n",
              "      <td>SaamrHMo23l/3TwXOWgVzw==</td>\n",
              "      <td>370.0</td>\n",
              "      <td>97.062615</td>\n",
              "      <td>0.942655</td>\n",
              "      <td>92.002546</td>\n",
              "      <td>0.24</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>39726.0</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>100018.91</td>\n",
              "      <td>False</td>\n",
              "      <td>xsd3ZdsI3356I3xMxZeiqQ==</td>\n",
              "      <td>rlWIXTBO+VOa34+SpGyhlQ==</td>\n",
              "      <td>NCqL3QBx0pscDnx3ixKwXg==</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>N5/CE7lSkAfB04hVFFwllw==</td>\n",
              "      <td>50.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>02:34:29</td>\n",
              "      <td>346</td>\n",
              "      <td>gmail.com</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>17.0</td>\n",
              "      <td>diocezio</td>\n",
              "      <td>243</td>\n",
              "      <td>(-19.687710705798963, -47.94151536525154)</td>\n",
              "      <td>Radio-commercial</td>\n",
              "      <td>251-3659293</td>\n",
              "      <td>4902.0</td>\n",
              "      <td>BR-RS</td>\n",
              "      <td>40933</td>\n",
              "      <td>{'tags': ['n6', 'n7', 'nim']}</td>\n",
              "      <td>Mozilla/5.0 (Linux; Android 5.0.2; SAMSUNG SM-...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>669630dd-2e6a-0396-84bf-455e5009c922</td>\n",
              "      <td>True</td>\n",
              "      <td>DGCQep2AE5QRkNCshIAlFQ==</td>\n",
              "      <td>Fv28Bz0YRTVAT5kl1bAV6g==</td>\n",
              "      <td>360.0</td>\n",
              "      <td>100.027073</td>\n",
              "      <td>0.351918</td>\n",
              "      <td>112.892453</td>\n",
              "      <td>0.29</td>\n",
              "      <td>7207.92</td>\n",
              "      <td>36.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>65023.65</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Ygq6MsM98oC8yceExr69Ig==</td>\n",
              "      <td>PjTIDfJsK0DKL9fO7vuW2g==</td>\n",
              "      <td>NCqL3QBx0pscDnx3ixKwXg==</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>n+xK9CfX0bCn77lClTWviw==</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>00:60:02</td>\n",
              "      <td>6</td>\n",
              "      <td>gmail.com</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9.0</td>\n",
              "      <td>veralucia</td>\n",
              "      <td>65</td>\n",
              "      <td>(-28.748023890412284, -51.867279334353995)</td>\n",
              "      <td>Waiting-list</td>\n",
              "      <td>230-6097993</td>\n",
              "      <td>163679.0</td>\n",
              "      <td>BR-RR</td>\n",
              "      <td>50985</td>\n",
              "      <td>{'tags': ['n0', 'n17', 'nim', 'da']}</td>\n",
              "      <td>Mozilla/5.0 (Linux; Android 6.0.1; SGP771 Buil...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>d235609e-b6cb-0ccc-a329-d4f12e7ebdc1</td>\n",
              "      <td>False</td>\n",
              "      <td>1Rk8w4Ucd5yR3KcqZzLdow==</td>\n",
              "      <td>dCm9hFKfdRm7ej3jW+gyxw==</td>\n",
              "      <td>510.0</td>\n",
              "      <td>101.599485</td>\n",
              "      <td>0.987673</td>\n",
              "      <td>94.902491</td>\n",
              "      <td>0.32</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>54591.0</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>68830.01</td>\n",
              "      <td>False</td>\n",
              "      <td>KET/Pmr6rHp1RJ/P9ymztw==</td>\n",
              "      <td>Cc/kWDLQH3dpHv5HU+pLVA==</td>\n",
              "      <td>NCqL3QBx0pscDnx3ixKwXg==</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiE56f...</td>\n",
              "      <td>n+xK9CfX0bCn77lClTWviw==</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>19.0</td>\n",
              "      <td>19.0</td>\n",
              "      <td>11:20:49</td>\n",
              "      <td>406</td>\n",
              "      <td>spgov.com</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>38.0</td>\n",
              "      <td>venice</td>\n",
              "      <td>815</td>\n",
              "      <td>(-17.520650158450454, -39.75801139933186)</td>\n",
              "      <td>Waiting-list</td>\n",
              "      <td>261-3543751</td>\n",
              "      <td>1086.0</td>\n",
              "      <td>BR-RN</td>\n",
              "      <td>37825</td>\n",
              "      <td>{'tags': ['n4']}</td>\n",
              "      <td>Mozilla/5.0 (Linux; Android 6.0; HTC One X10 B...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>9e0eb880-e8f4-3faa-67d8-f5cdd2b3932b</td>\n",
              "      <td>False</td>\n",
              "      <td>8k8UDR4Yx0qasAjkGrUZLw==</td>\n",
              "      <td>+CxEO4w7jv3QPI/BQbyqAA==</td>\n",
              "      <td>500.0</td>\n",
              "      <td>98.474289</td>\n",
              "      <td>0.532539</td>\n",
              "      <td>118.126207</td>\n",
              "      <td>0.18</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>60011.29</td>\n",
              "      <td>True</td>\n",
              "      <td>xsd3ZdsI3356I3xMxZeiqQ==</td>\n",
              "      <td>i036nmJ7rfxo+3EvCD7Jnw==</td>\n",
              "      <td>NCqL3QBx0pscDnx3ixKwXg==</td>\n",
              "      <td>mLVIVxoGY7TUDJ1FyFoSIZi1SFcaBmO01AydRchaEiGYtU...</td>\n",
              "      <td>n+xK9CfX0bCn77lClTWviw==</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13:39:03</td>\n",
              "      <td>240</td>\n",
              "      <td>gmail.com</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2</td>\n",
              "      <td>1.0</td>\n",
              "      <td>46.0</td>\n",
              "      <td>darzisa</td>\n",
              "      <td>320</td>\n",
              "      <td>(-16.574259446978008, -39.90990074785962)</td>\n",
              "      <td>Invite-email</td>\n",
              "      <td>102-3660162</td>\n",
              "      <td>198618.0</td>\n",
              "      <td>BR-MT</td>\n",
              "      <td>52827</td>\n",
              "      <td>{'tags': ['pro+aty', 'n19', 'da', 'b19']}</td>\n",
              "      <td>Mozilla/5.0 (Linux; Android 7.0; Pixel C Build...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                    ids  ... target_fraud\n",
              "0  343b7e7b-2cf8-e508-b8fd-0a0285af30aa  ...          NaN\n",
              "1  bc2c7502-bbad-0f8c-39c3-94e881967124  ...          NaN\n",
              "2  669630dd-2e6a-0396-84bf-455e5009c922  ...          NaN\n",
              "3  d235609e-b6cb-0ccc-a329-d4f12e7ebdc1  ...          NaN\n",
              "4  9e0eb880-e8f4-3faa-67d8-f5cdd2b3932b  ...          NaN\n",
              "\n",
              "[5 rows x 43 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1btJbujyWSKT",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "Algumas conclusões iniciais que podem ser destacadas a respeito desse análise inicial:\n",
        "\n",
        "* Existe uma coluna `id`, anônima, que representa um identificador único do cliente. Iremos eliminar esta coluna mais para frente, pois ela não contém nenhuma informação relevante.\n",
        "* A variável alvo para analisar o risco de *default* (inadimplência) está na coluna `target_default`.\n",
        "* As colunas `['score_3', 'score_4', 'score_5']` são numéricas e as colunas `['score_1', 'score_2']` estão codificadas de alguma maneira. Teremos que verificar a frente se existe uma quantidade de classes que possa ser convertida em informação útil.\n",
        "* existem outras variáveis que apresentam algum tipo de codificação, como `['reason', 'state', 'zip', 'channel', 'job_name', 'real_state']` que estão codificadas e também precisarão de alguma análise mais aprofundada para saber se é possível extrair alguma informação das mesmas.\n",
        "* A coluna `profile_tags` contém um dicionário com *tags* atribuídas a cada cliente, aparentemente.\n",
        "* A coluna `target_fraud` seria a variável alvo de outro modelo para este mesmo *dataset*, onde o objetivo seria a detecção de fraude.\n",
        "* A coluna `lat_lon` está em formato *string* contendo uma tupla com as coordenadas."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "le94H3O62I52",
        "colab_type": "code",
        "outputId": "2373968b-75a9-4731-e214-1d58cafc770c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 337
        }
      },
      "source": [
        "# ver resumo estatístico do dados numéricos\n",
        "df.describe()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>score_3</th>\n",
              "      <th>score_4</th>\n",
              "      <th>score_5</th>\n",
              "      <th>score_6</th>\n",
              "      <th>risk_rate</th>\n",
              "      <th>last_amount_borrowed</th>\n",
              "      <th>last_borrowed_in_months</th>\n",
              "      <th>credit_limit</th>\n",
              "      <th>income</th>\n",
              "      <th>ok_since</th>\n",
              "      <th>n_bankruptcies</th>\n",
              "      <th>n_defaulted_loans</th>\n",
              "      <th>n_accounts</th>\n",
              "      <th>n_issues</th>\n",
              "      <th>application_time_in_funnel</th>\n",
              "      <th>external_data_provider_credit_checks_last_2_year</th>\n",
              "      <th>external_data_provider_credit_checks_last_month</th>\n",
              "      <th>external_data_provider_credit_checks_last_year</th>\n",
              "      <th>external_data_provider_email_seen_before</th>\n",
              "      <th>external_data_provider_fraud_score</th>\n",
              "      <th>reported_income</th>\n",
              "      <th>shipping_zip_code</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>44438.000000</td>\n",
              "      <td>45000.000000</td>\n",
              "      <td>45000.000000</td>\n",
              "      <td>45000.000000</td>\n",
              "      <td>44438.000000</td>\n",
              "      <td>15044.000000</td>\n",
              "      <td>15044.000000</td>\n",
              "      <td>31200.000000</td>\n",
              "      <td>4.443800e+04</td>\n",
              "      <td>18455.000000</td>\n",
              "      <td>44303.000000</td>\n",
              "      <td>44426.000000</td>\n",
              "      <td>44438.000000</td>\n",
              "      <td>33456.000000</td>\n",
              "      <td>45000.000000</td>\n",
              "      <td>22372.0</td>\n",
              "      <td>45000.000000</td>\n",
              "      <td>29876.000000</td>\n",
              "      <td>42767.000000</td>\n",
              "      <td>45000.000000</td>\n",
              "      <td>45000.00</td>\n",
              "      <td>45000.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>347.983258</td>\n",
              "      <td>100.006797</td>\n",
              "      <td>0.500037</td>\n",
              "      <td>99.925228</td>\n",
              "      <td>0.296717</td>\n",
              "      <td>13782.341991</td>\n",
              "      <td>41.763893</td>\n",
              "      <td>34381.951154</td>\n",
              "      <td>7.160739e+04</td>\n",
              "      <td>35.069737</td>\n",
              "      <td>0.076270</td>\n",
              "      <td>0.004569</td>\n",
              "      <td>10.696701</td>\n",
              "      <td>11.068119</td>\n",
              "      <td>247.537000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.501067</td>\n",
              "      <td>0.503381</td>\n",
              "      <td>12.534781</td>\n",
              "      <td>500.399800</td>\n",
              "      <td>inf</td>\n",
              "      <td>33923.514844</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>110.429165</td>\n",
              "      <td>3.178080</td>\n",
              "      <td>0.288326</td>\n",
              "      <td>10.016549</td>\n",
              "      <td>0.101933</td>\n",
              "      <td>8077.571933</td>\n",
              "      <td>10.252705</td>\n",
              "      <td>36586.982218</td>\n",
              "      <td>5.185897e+04</td>\n",
              "      <td>21.663194</td>\n",
              "      <td>0.274052</td>\n",
              "      <td>0.079400</td>\n",
              "      <td>4.592463</td>\n",
              "      <td>4.595501</td>\n",
              "      <td>146.118398</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.114941</td>\n",
              "      <td>0.499997</td>\n",
              "      <td>126.530487</td>\n",
              "      <td>287.968873</td>\n",
              "      <td>NaN</td>\n",
              "      <td>12692.406382</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>86.191572</td>\n",
              "      <td>0.000035</td>\n",
              "      <td>60.663039</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1005.180000</td>\n",
              "      <td>36.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>4.821180e+03</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>-999.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>403.00</td>\n",
              "      <td>12001.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>270.000000</td>\n",
              "      <td>97.867036</td>\n",
              "      <td>0.252150</td>\n",
              "      <td>93.198353</td>\n",
              "      <td>0.220000</td>\n",
              "      <td>7519.502500</td>\n",
              "      <td>36.000000</td>\n",
              "      <td>10000.000000</td>\n",
              "      <td>4.501078e+04</td>\n",
              "      <td>17.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>7.000000</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>120.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>11.000000</td>\n",
              "      <td>252.000000</td>\n",
              "      <td>50948.25</td>\n",
              "      <td>22937.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>340.000000</td>\n",
              "      <td>100.018900</td>\n",
              "      <td>0.500753</td>\n",
              "      <td>99.974796</td>\n",
              "      <td>0.290000</td>\n",
              "      <td>12023.465000</td>\n",
              "      <td>36.000000</td>\n",
              "      <td>25647.000000</td>\n",
              "      <td>6.126676e+04</td>\n",
              "      <td>32.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>248.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>27.000000</td>\n",
              "      <td>503.000000</td>\n",
              "      <td>101779.00</td>\n",
              "      <td>33896.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>420.000000</td>\n",
              "      <td>102.141797</td>\n",
              "      <td>0.748816</td>\n",
              "      <td>106.640275</td>\n",
              "      <td>0.360000</td>\n",
              "      <td>19220.270000</td>\n",
              "      <td>36.000000</td>\n",
              "      <td>47086.750000</td>\n",
              "      <td>8.601923e+04</td>\n",
              "      <td>50.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>13.000000</td>\n",
              "      <td>14.000000</td>\n",
              "      <td>375.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>43.000000</td>\n",
              "      <td>747.000000</td>\n",
              "      <td>151337.25</td>\n",
              "      <td>44910.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>990.000000</td>\n",
              "      <td>113.978234</td>\n",
              "      <td>0.999973</td>\n",
              "      <td>142.192400</td>\n",
              "      <td>0.900000</td>\n",
              "      <td>35059.600000</td>\n",
              "      <td>60.000000</td>\n",
              "      <td>448269.000000</td>\n",
              "      <td>5.000028e+06</td>\n",
              "      <td>141.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>49.000000</td>\n",
              "      <td>49.000000</td>\n",
              "      <td>500.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>59.000000</td>\n",
              "      <td>1000.000000</td>\n",
              "      <td>inf</td>\n",
              "      <td>56000.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            score_3       score_4  ...  reported_income  shipping_zip_code\n",
              "count  44438.000000  45000.000000  ...         45000.00       45000.000000\n",
              "mean     347.983258    100.006797  ...              inf       33923.514844\n",
              "std      110.429165      3.178080  ...              NaN       12692.406382\n",
              "min        0.000000     86.191572  ...           403.00       12001.000000\n",
              "25%      270.000000     97.867036  ...         50948.25       22937.000000\n",
              "50%      340.000000    100.018900  ...        101779.00       33896.000000\n",
              "75%      420.000000    102.141797  ...        151337.25       44910.000000\n",
              "max      990.000000    113.978234  ...              inf       56000.000000\n",
              "\n",
              "[8 rows x 22 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_AhtnxOk_VL",
        "colab_type": "text"
      },
      "source": [
        "Olhando o resumo das principais informações estatísticas somos capazes de destacar algumas observações:\n",
        "\n",
        "* A coluna `external_data_provider_credit_checks_last_2_year` possui valores mínimos, máximos e desvio-padrão iguais a zero.\n",
        "* A coluna `reported_income` apresenta valores *inf*, que irão interferir na análise e modelo. Substituiremos valores do tipo `np.inf` por `np.nan` para trabalhar com os dados.\n",
        "* A coluna `external_data_provider_email_seen_before` apresenta o valor mínimo de -999, o que é estranho ao se considerar as outras informações. Após verificar de maneira mais aprofundada, chegou-se a conclusão que esse dados são *outliers* ou foram tratados inadequadamente. Substituiremos os valores iguais a -999 por `np.nan`.\n",
        "\n",
        "Vamos realizar o levantamento dos tipos de variáveis que compõem o *DataFrame* a fim de estabelecer a divisão entre as variáveis numéricas e as variáveis categóricas."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5EMnuET-o1k8",
        "colab_type": "code",
        "outputId": "5b4186d0-d6e8-4d65-8299-0a8df6aab000",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 799
        }
      },
      "source": [
        "# tipo das variáveis\n",
        "print(\"\\nTIPOS DAS VARIÁVEIS:\")\n",
        "print(df.dtypes)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "TIPOS DAS VARIÁVEIS:\n",
            "ids                                                  object\n",
            "target_default                                       object\n",
            "score_1                                              object\n",
            "score_2                                              object\n",
            "score_3                                             float64\n",
            "score_4                                             float64\n",
            "score_5                                             float64\n",
            "score_6                                             float64\n",
            "risk_rate                                           float64\n",
            "last_amount_borrowed                                float64\n",
            "last_borrowed_in_months                             float64\n",
            "credit_limit                                        float64\n",
            "reason                                               object\n",
            "income                                              float64\n",
            "facebook_profile                                     object\n",
            "state                                                object\n",
            "zip                                                  object\n",
            "channel                                              object\n",
            "job_name                                             object\n",
            "real_state                                           object\n",
            "ok_since                                            float64\n",
            "n_bankruptcies                                      float64\n",
            "n_defaulted_loans                                   float64\n",
            "n_accounts                                          float64\n",
            "n_issues                                            float64\n",
            "application_time_applied                             object\n",
            "application_time_in_funnel                            int64\n",
            "email                                                object\n",
            "external_data_provider_credit_checks_last_2_year    float64\n",
            "external_data_provider_credit_checks_last_month       int64\n",
            "external_data_provider_credit_checks_last_year      float64\n",
            "external_data_provider_email_seen_before            float64\n",
            "external_data_provider_first_name                    object\n",
            "external_data_provider_fraud_score                    int64\n",
            "lat_lon                                              object\n",
            "marketing_channel                                    object\n",
            "profile_phone_number                                 object\n",
            "reported_income                                     float64\n",
            "shipping_state                                       object\n",
            "shipping_zip_code                                     int64\n",
            "profile_tags                                         object\n",
            "user_agent                                           object\n",
            "target_fraud                                         object\n",
            "dtype: object\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rCVXIPfspOvb",
        "colab_type": "text"
      },
      "source": [
        "Outra análise importante para se fazer é identificar a quantidade de dados ausentes por *feature*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SPxBxNq0pQgP",
        "colab_type": "code",
        "outputId": "b1501868-32dd-481b-bd6a-7abe4c0d0a4c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 799
        }
      },
      "source": [
        "# valores ausentes\n",
        "print(\"\\nVALORES AUSENTES:\")\n",
        "print((df.isnull().sum() / df.shape[0]).sort_values(ascending=False))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "VALORES AUSENTES:\n",
            "target_fraud                                        0.966178\n",
            "last_amount_borrowed                                0.665689\n",
            "last_borrowed_in_months                             0.665689\n",
            "ok_since                                            0.589889\n",
            "external_data_provider_credit_checks_last_2_year    0.502844\n",
            "external_data_provider_credit_checks_last_year      0.336089\n",
            "credit_limit                                        0.306667\n",
            "n_issues                                            0.256533\n",
            "facebook_profile                                    0.099067\n",
            "marketing_channel                                   0.079511\n",
            "job_name                                            0.074133\n",
            "target_default                                      0.072422\n",
            "external_data_provider_email_seen_before            0.049622\n",
            "lat_lon                                             0.030289\n",
            "user_agent                                          0.016044\n",
            "n_bankruptcies                                      0.015489\n",
            "n_defaulted_loans                                   0.012756\n",
            "reason                                              0.012578\n",
            "income                                              0.012489\n",
            "real_state                                          0.012489\n",
            "state                                               0.012489\n",
            "zip                                                 0.012489\n",
            "channel                                             0.012489\n",
            "score_3                                             0.012489\n",
            "score_2                                             0.012489\n",
            "score_1                                             0.012489\n",
            "n_accounts                                          0.012489\n",
            "risk_rate                                           0.012489\n",
            "shipping_zip_code                                   0.000000\n",
            "score_4                                             0.000000\n",
            "score_5                                             0.000000\n",
            "profile_tags                                        0.000000\n",
            "score_6                                             0.000000\n",
            "application_time_in_funnel                          0.000000\n",
            "shipping_state                                      0.000000\n",
            "reported_income                                     0.000000\n",
            "application_time_applied                            0.000000\n",
            "profile_phone_number                                0.000000\n",
            "external_data_provider_fraud_score                  0.000000\n",
            "external_data_provider_first_name                   0.000000\n",
            "external_data_provider_credit_checks_last_month     0.000000\n",
            "email                                               0.000000\n",
            "ids                                                 0.000000\n",
            "dtype: float64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Phks82LosmbY",
        "colab_type": "text"
      },
      "source": [
        "Em relação à porcentagem de valores ausentes identificados neste *dataset*:\n",
        "\n",
        "* Diversas variáveis como `['target_fraud', 'last_amount_borrowed', 'last_borrowed_in_months', 'ok_since', 'external_data_provider_credit_checks_last_2_year']` possuem mais da metade dos valores ausentes.\n",
        "\n",
        "* As variáves `['external_data_provider_credit_checks_last_year', 'credit_limit', 'n_issues']` possuem entre 25-34% do seus valores ausentes.\n",
        "\n",
        "* A variável alvo `target_default` contém valores nulos que serão eliminados do *dataset*.\n",
        "\n",
        "Neste projeto, o caso mais extremo (`target_fraud`) não representa um problema, pois é uma variável alvo que não interessa para a análise de risco de inadimplência. Já as demais *features* deverão ser usadas com o devido cuidado.\n",
        "\n",
        "Uma outra análise interessante de se fazer diz respeito à contagem de valore únicos por features. Muitas vezes, variáveis numéricas podem esconder classes/categorias que melhor representariam uma *feature*, ou revelar uma quantidade elevada de classes para \"variáveis categóricas\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5T1bex6rxP9y",
        "colab_type": "code",
        "outputId": "09c711cf-fa1e-4884-a4bd-a48a2e32c396",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 799
        }
      },
      "source": [
        "# valores únicos\n",
        "print(\"\\nVALORES ÚNICOS:\")\n",
        "print(df.nunique().sort_values())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "VALORES ÚNICOS:\n",
            "external_data_provider_credit_checks_last_2_year        1\n",
            "channel                                                 1\n",
            "target_fraud                                            2\n",
            "target_default                                          2\n",
            "external_data_provider_credit_checks_last_year          2\n",
            "facebook_profile                                        2\n",
            "last_borrowed_in_months                                 2\n",
            "external_data_provider_credit_checks_last_month         4\n",
            "n_defaulted_loans                                       5\n",
            "real_state                                              5\n",
            "email                                                   6\n",
            "n_bankruptcies                                          6\n",
            "score_1                                                 7\n",
            "marketing_channel                                       9\n",
            "shipping_state                                         25\n",
            "score_2                                                35\n",
            "n_issues                                               44\n",
            "n_accounts                                             44\n",
            "state                                                  50\n",
            "external_data_provider_email_seen_before               62\n",
            "risk_rate                                              82\n",
            "score_3                                                88\n",
            "ok_since                                              100\n",
            "user_agent                                            297\n",
            "application_time_in_funnel                            501\n",
            "zip                                                   823\n",
            "external_data_provider_fraud_score                   1001\n",
            "last_amount_borrowed                                14325\n",
            "reason                                              14874\n",
            "credit_limit                                        20928\n",
            "lat_lon                                             22412\n",
            "profile_tags                                        26131\n",
            "shipping_zip_code                                   28263\n",
            "job_name                                            32265\n",
            "external_data_provider_first_name                   32886\n",
            "application_time_applied                            35543\n",
            "reported_income                                     40025\n",
            "income                                              41211\n",
            "score_4                                             45000\n",
            "score_5                                             45000\n",
            "score_6                                             45000\n",
            "profile_phone_number                                45000\n",
            "ids                                                 45000\n",
            "dtype: int64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9tdgooY3x9f4",
        "colab_type": "text"
      },
      "source": [
        "A contagem de valores únicos mostra que as colunas `external_data_provider_credit_checks_last_2_year` e `channel` apresentam um único valor possível. Como não ha maiores informações sobre cada *feature* iremos descartar as duas colunas para o modelo de *Machine Learning*.\n",
        "\n",
        "Também se percebe que `profile_phone_number` possui 45 mil valores únicos, não trazendo provavelmente nenhuma informação que possa contribuir para a análise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G_lMu2Qw7t0e",
        "colab_type": "code",
        "outputId": "292164db-3eda-4930-cf5b-fdf13c94998b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "# Proporção entre Inadimplentes no Dataset\n",
        "fig, ax = plt.subplots()\n",
        "sns.countplot('target_default', data=df, ax=ax)\n",
        "ax.set_title(\"Inadimplentes\")\n",
        "ax.set_xlabel(\"Valor\")\n",
        "ax.set_ylabel(\"Contagem\")\n",
        "plt.show()\n",
        "\n",
        "print(\"Proporção entre Inadimplentes no Dataset:\")\n",
        "print(df['target_default'].value_counts() / df.shape[0])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEWCAYAAACufwpNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAG2ZJREFUeJzt3X+0XWV95/H3hwBC6yAgKSJBQ0ta\nG2mJEIFRO7VSIdB2gl0Wsa2JlBIdsZWZjhV1KlSlq7+pWKXFkhIcK1KqJbVRTJHRMi0/goZAggwR\ntSQNEEn4JYiC3/njPFePtze5B9jnHi73/VrrrLvPdz/P3s+Gu+4n+3eqCkmSurDLqAcgSXr6MFQk\nSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUpCcoSSU5pE3/RZLfGdJ6vprkZ4exbKlrhoqe9qbij3JV\nvbGq3jPMdTxZSc5O8r9HPQ49vRkqkqTOGCqaMZK8PsnVSf44yfYkX0lyfN/8U5LckuSBJLcnecO4\n/m9NsiXJvyf5tXHzLkry3jb98iSbkvx2krtbnxOTnJDk/yXZluQdfX3PTnJZko+1dX8hyWE72IZd\nkpyZ5MtJ7klyaZJ927y57ZDc0iT/luTrSd7Z5i0C3gG8JsmDSW5s9WclubCNcXOS9yaZ1eYdkuRz\nSe5ry/pYF/8f9PRmqGimOQq4FdgP+EPgwiRp8+4Gfh7YCzgFODfJ4fDdP8r/E3glMA+Y7HDac4A9\ngAOBdwEfAn4VOAL4KeB3khzc134x8LfAvsDfAH+fZLcJlvsbwInATwPPBbYDHxjX5mXAjwHHAO9K\n8uNV9Wng94CPVdUzq2ostC4CHgUOAV4EHAv8epv3HuAzwD7AHOD9k2yzZKhoxvlaVX2oqh4DVgAH\nAPsDVNU/VtWXq+dz9P6g/lTrdxLw11V1c1V9Azh7kvV8Gzinqr4NXEIvxN5XVQ9U1XpgA9C/N3JD\nVV3W2v8pvUA6eoLlvhF4Z1VtqqpH2jhenWTXvja/W1UPV9WNwI3j1vNdSfYHTgDOqKpvVNXdwLnA\nyX3b8HzguVX1zaq6epJtlgwVzTh3jk1U1UNt8pkASY5Pck07PHUvvT+4+7U2zwXu6FvO1yZZzz0t\nuAAebj/v6pv/8Nh6m+8uu6q+A2xq6xzv+cAnktzbxngL8BgtGMdvI/DQuPWMX9ZuwJa+5f0l8ENt\n/m8DAa5Lsn78IT9pIrtO3kR6+kvyDODvgCXA5VX17SR/T++PKsAW4KC+Ls/reAjfXXaSXegdbvr3\nCdrdAfxaVf3f8TOSzJ1kHeMfSX4H8AiwX1U9+h8aV90JnNaW/TLgn5J8vqo2TrIezWDuqUg9uwPP\nALYCj7YT+Mf2zb8UeH2S+Ul+ADir4/UfkeQX22GsM+j9sb9mgnZ/AZyT5PkASWYnWTzgOu4C5rbQ\noqq20DvE9ydJ9moXAfxIkp9uy/6lJHNa3+30Quk7T3QDNTMYKhJQVQ8Av0kvPLYDvwys7Jv/KeDP\ngM8CG9vPLl0OvKat+3XAL7bzK+O9r43rM0keoBc8Rw24jr9tP+9J8oU2vYReoG5o676M3nkmgBcD\n1yZ5sK3zLVV1++PaKs048SVd0mglORs4pKp+ddRjkZ4s91QkSZ0xVCRJnfHwlySpM+6pSJI6M+Pu\nU9lvv/1q7ty5ox6GJE0rN9xww9eravZk7WZcqMydO5c1a9aMehiSNK0kmewpEoCHvyRJHTJUJEmd\nMVQkSZ0xVCRJnTFUJEmdMVQkSZ0ZWqgk2SPJdUlubC/4+d1Wv6i9G3xt+yxo9SQ5L8nGJOvGXuPa\n5i1Nclv7LO2rH5HkptbnvL7XwkqSRmCY96k8Aryiqh5s79q+Osmn2ry3VtVl49ofT+/d3/PoPcr7\nfOCoJPvSe3fFQnrvc7ghycqq2t7anAZcC6wCFgGfQpI0EkPbU2nv+X6wfd2tfXb2oLHFwMWt3zXA\n3kkOAI4DVlfVthYkq4FFbd5eVXVN9R5gdjFw4rC2R5I0uaHeUZ9kFnADcAjwgaq6Nsl/o/fmuncB\nVwJnVtUjwIF8/zvAN7XazuqbJqhPNI5lwDKA5z3vyb0F9oi3Xvyk+uvp6YY/WjLqIUhPCUM9UV9V\nj1XVAnrv2z4yyaHA24EX0Hur3L7A24Y5hjaOC6pqYVUtnD170kfXSJKeoCm5+quq7gWuAhZV1ZZ2\niOsR4K+BI1uzzcBBfd3mtNrO6nMmqEuSRmSYV3/NTrJ3m94TeCXwpXYuhHal1onAza3LSmBJuwrs\naOC+qtoCXAEcm2SfJPsAxwJXtHn3Jzm6LWsJvfd8S5JGZJjnVA4AVrTzKrsAl1bVJ5N8NslsIMBa\n4I2t/SrgBGAj8BBwCkBVbUvyHuD61u7dVbWtTb8JuAjYk95VX175JUkjNLRQqap1wIsmqL9iB+0L\nOH0H85YDyyeorwEOfXIjlSR1xTvqJUmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQk\nSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmd\nMVQkSZ0xVCRJnRlaqCTZI8l1SW5Msj7J77b6wUmuTbIxyceS7N7qz2jfN7b5c/uW9fZWvzXJcX31\nRa22McmZw9oWSdJghrmn8gjwiqo6DFgALEpyNPAHwLlVdQiwHTi1tT8V2N7q57Z2JJkPnAy8EFgE\nfDDJrCSzgA8AxwPzgde2tpKkERlaqFTPg+3rbu1TwCuAy1p9BXBim17cvtPmH5MkrX5JVT1SVV8B\nNgJHts/Gqrq9qr4FXNLaSpJGZKjnVNoexVrgbmA18GXg3qp6tDXZBBzYpg8E7gBo8+8Dnt1fH9dn\nR/WJxrEsyZoka7Zu3drFpkmSJjDUUKmqx6pqATCH3p7FC4a5vp2M44KqWlhVC2fPnj2KIUjSjDAl\nV39V1b3AVcB/BvZOsmubNQfY3KY3AwcBtPnPAu7pr4/rs6O6JGlEhnn11+wke7fpPYFXArfQC5dX\nt2ZLgcvb9Mr2nTb/s1VVrX5yuzrsYGAecB1wPTCvXU22O72T+SuHtT2SpMntOnmTJ+wAYEW7SmsX\n4NKq+mSSDcAlSd4LfBG4sLW/EPhwko3ANnohQVWtT3IpsAF4FDi9qh4DSPJm4ApgFrC8qtYPcXsk\nSZMYWqhU1TrgRRPUb6d3fmV8/ZvAL+1gWecA50xQXwWsetKDlSR1wjvqJUmdMVQkSZ0xVCRJnTFU\nJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJ\nnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0ZWqgkOSjJVUk2JFmf5C2tfnaSzUnWts8JfX3enmRj\nkluTHNdXX9RqG5Oc2Vc/OMm1rf6xJLsPa3skSZMb5p7Ko8BvVdV84Gjg9CTz27xzq2pB+6wCaPNO\nBl4ILAI+mGRWklnAB4DjgfnAa/uW8wdtWYcA24FTh7g9kqRJDC1UqmpLVX2hTT8A3AIcuJMui4FL\nquqRqvoKsBE4sn02VtXtVfUt4BJgcZIArwAua/1XACcOZ2skSYOYknMqSeYCLwKubaU3J1mXZHmS\nfVrtQOCOvm6bWm1H9WcD91bVo+PqE61/WZI1SdZs3bq1gy2SJE1k6KGS5JnA3wFnVNX9wPnAjwAL\ngC3Anwx7DFV1QVUtrKqFs2fPHvbqJGnG2nWYC0+yG71A+UhVfRygqu7qm/8h4JPt62bgoL7uc1qN\nHdTvAfZOsmvbW+lvL0kagWFe/RXgQuCWqvrTvvoBfc1eBdzcplcCJyd5RpKDgXnAdcD1wLx2pdfu\n9E7mr6yqAq4CXt36LwUuH9b2SJImN8w9lZcCrwNuSrK21d5B7+qtBUABXwXeAFBV65NcCmygd+XY\n6VX1GECSNwNXALOA5VW1vi3vbcAlSd4LfJFeiEmSRmRooVJVVwOZYNaqnfQ5BzhngvqqifpV1e30\nrg6TJD0FeEe9JKkzhookqTOGiiSpM4aKJKkzhookqTOGiiSpM4aKJKkzhookqTOGiiSpM4aKJKkz\nhookqTMDPfsryd7AEmBuf5+q+s3hDEuSNB0N+kDJVcA1wE3Ad4Y3HEnSdDZoqOxRVf9jqCORJE17\ng55T+XCS05IckGTfsc9QRyZJmnYG3VP5FvBHwDvpvVyL9vOHhzEoSdL0NGio/BZwSFV9fZiDkSRN\nb4Me/toIPDTMgUiSpr9B91S+AaxNchXwyFjRS4olSf0GDZW/bx9JknZooFCpqhVJ9gSeV1W3DnlM\nkqRpaqBzKkl+AVgLfLp9X5Bk5SR9DkpyVZINSdYneUur75tkdZLb2s99Wj1JzkuyMcm6JIf3LWtp\na39bkqV99SOS3NT6nJckj/8/gSSpK4OeqD8bOBK4F6Cq1jL55cSPAr9VVfOBo4HTk8wHzgSurKp5\nwJXtO8DxwLz2WQacD70QAs4CjmpjOGssiFqb0/r6LRpweyRJQzBoqHy7qu4bV9vp41qqaktVfaFN\nPwDcAhwILAZWtGYrgBPb9GLg4uq5Btg7yQHAccDqqtpWVduB1cCiNm+vqrqmqgq4uG9ZkqQRGDRU\n1if5ZWBWknlJ3g/8y6ArSTIXeBFwLbB/VW1ps+4E9m/TBwJ39HXb1Go7q2+aoD7R+pclWZNkzdat\nWwcdtiTpcRo0VH4DeCG9y4k/CtwPnDFIxyTPBP4OOKOq7u+f1/YwasKOHaqqC6pqYVUtnD179rBX\nJ0kz1qBXfz1E7xEt73w8C0+yG71A+UhVfbyV70pyQFVtaYew7m71zcBBfd3ntNpm4OXj6v+n1edM\n0F6SNCKDXv31D0lWjvt8OMlbkuyxgz4BLgRuqao/7Zu1Ehi7gmspcHlffUm7Cuxo4L52mOwK4Ngk\n+7QT9McCV7R59yc5uq1rSd+yJEkjMOjNj7cDs+kd+gJ4DfAA8KPAh4DXTdDnpa1+U5K1rfYO4PeB\nS5OcCnwNOKnNWwWcwPceCXMKQFVtS/Ie4PrW7t1Vta1Nvwm4CNgT+FT7SJJGZNBQeUlVvbjv+z8k\nub6qXpxk/UQdqupqYEf3jRwzQfsCTt/BspYDyyeorwEOnWzwkqSpMeiJ+mcmed7Ylzb9zPb1W52P\nSpI0LT2eR99fneTL9PY+DgbelOQH+d49J5KkGW7Qq79WJZkHvKCVbq2qb7bpPxvKyCRJ086geyrQ\newzKjwF7AIcloaouHs6wJEnT0UChkuQseveKzKd3ldbxwNX0Ho0iSRIw+In6V9O7YuvOqjoFOAx4\n1tBGJUmalgYNlYer6jvAo0n2oncX/EGT9JEkzTCDnlNZk2Rvejc63gA8CPzr0EYlSZqWBr36601t\n8i+SfJreI+fXDW9YkqTpaNBnf105Nl1VX62qdf01SZJgkj2V9rDIHwD2aw9zHHvsyl7s4N0lkqSZ\na7LDX2+g996U59I7lzIWKvcDfz7EcUmSpqGdhkpVvQ94X5LfqKr3T9GYJEnT1KAn6t+f5CXA3P4+\n3lEvSeo36B31HwZ+BFgLPNbKhXfUS5L6DHqfykJgfnvniSRJExr0jvqbgecMcyCSpOlv0D2V/YAN\nSa4DHhkrVtV/HcqoJEnT0qChcvYwByFJenoY9OqvzyXZHxh7T/11VXX38IYlSZqOBn1My0nAdcAv\nAScB1yZ59SR9lie5O8nNfbWzk2xOsrZ9Tuib9/YkG5PcmuS4vvqiVtuY5My++sFJrm31jyXZffDN\nliQNw6An6t8JvLiqllbVEuBI4Hcm6XMRsGiC+rlVtaB9VgEkmQ+cDLyw9flgkllJZgEfoPdSsPnA\na1tbgD9oyzoE2A6cOuC2SJKGZNBQ2WXc4a57JutbVZ8Htg24/MXAJVX1SFV9BdhIL7iOBDZW1e1V\n9S3gEmBxkgCvAC5r/VcAJw64LknSkAwaKp9OckWS1yd5PfCP9F4r/ES8Ocm6dnhsn1Y7ELijr82m\nVttR/dnAvVX16Lj6hJIsS7ImyZqtW7c+wWFLkiaz01BJckiSl1bVW4G/BH6yff4VuOAJrO98enfm\nLwC2AH/yBJbxuFXVBVW1sKoWzp49eypWKUkz0mRXf/0Z8HaAqvo48HGAJD/R5v3C41lZVd01Np3k\nQ8An29fNfP/riee0Gjuo3wPsnWTXtrfS316SNCKTHf7av6puGl9stbmPd2VJDuj7+ip6d+oDrARO\nTvKMJAcD8+hdbXY9MK9d6bU7vZP5K9vjYq4Cxq5AWwpc/njHI0nq1mR7KnvvZN6eO+uY5KPAy+m9\n4GsTcBbw8iQL6D2M8qv03tdCVa1PcimwAXgUOL2qHmvLeTNwBTALWF5V69sq3gZckuS9wBeBCyfZ\nFknSkE0WKmuSnFZVH+ovJvl1ei/t2qGqeu0E5R3+4a+qc4BzJqivYoKLAqrqdnpXh0mSniImC5Uz\ngE8k+RW+FyILgd3pHb6SJOm7Jnvz413AS5L8DHBoK/9jVX126COTJE07gz776yp6J8YlSdqhQW9+\nlCRpUoaKJKkzhookqTOGiiSpM4aKJKkzhookqTOGiiSpM4aKJKkzhookqTOGiiSpM4aKJKkzhook\nqTOGiiSpM4aKJKkzhookqTOGiiSpM4aKJKkzhookqTNDC5Uky5PcneTmvtq+SVYnua393KfVk+S8\nJBuTrEtyeF+fpa39bUmW9tWPSHJT63NekgxrWyRJgxnmnspFwKJxtTOBK6tqHnBl+w5wPDCvfZYB\n50MvhICzgKOAI4GzxoKotTmtr9/4dUmSptjQQqWqPg9sG1deDKxo0yuAE/vqF1fPNcDeSQ4AjgNW\nV9W2qtoOrAYWtXl7VdU1VVXAxX3LkiSNyFSfU9m/qra06TuB/dv0gcAdfe02tdrO6psmqE8oybIk\na5Ks2bp165PbAknSDo3sRH3bw6gpWtcFVbWwqhbOnj17KlYpSTPSVIfKXe3QFe3n3a2+GTior92c\nVttZfc4EdUnSCE11qKwExq7gWgpc3ldf0q4COxq4rx0muwI4Nsk+7QT9scAVbd79SY5uV30t6VuW\nJGlEdh3WgpN8FHg5sF+STfSu4vp94NIkpwJfA05qzVcBJwAbgYeAUwCqaluS9wDXt3bvrqqxk/9v\noneF2Z7Ap9pHkjRCQwuVqnrtDmYdM0HbAk7fwXKWA8snqK8BDn0yY5Qkdcs76iVJnTFUJEmdMVQk\nSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnRnaU4ol\nTb1/e/dPjHoIegp63rtumrJ1uaciSeqMoSJJ6oyhIknqjKEiSeqMoSJJ6oyhIknqzEhCJclXk9yU\nZG2SNa22b5LVSW5rP/dp9SQ5L8nGJOuSHN63nKWt/W1Jlo5iWyRJ3zPKPZWfqaoFVbWwfT8TuLKq\n5gFXtu8AxwPz2mcZcD70Qgg4CzgKOBI4ayyIJEmj8VQ6/LUYWNGmVwAn9tUvrp5rgL2THAAcB6yu\nqm1VtR1YDSya6kFLkr5nVKFSwGeS3JBkWavtX1Vb2vSdwP5t+kDgjr6+m1ptR/X/IMmyJGuSrNm6\ndWtX2yBJGmdUj2l5WVVtTvJDwOokX+qfWVWVpLpaWVVdAFwAsHDhws6WK0n6fiPZU6mqze3n3cAn\n6J0Tuasd1qL9vLs13wwc1Nd9TqvtqC5JGpEpD5UkP5jkP41NA8cCNwMrgbEruJYCl7fplcCSdhXY\n0cB97TDZFcCxSfZpJ+iPbTVJ0oiM4vDX/sAnkoyt/2+q6tNJrgcuTXIq8DXgpNZ+FXACsBF4CDgF\noKq2JXkPcH1r9+6q2jZ1myFJGm/KQ6WqbgcOm6B+D3DMBPUCTt/BspYDy7seoyTpiXkqXVIsSZrm\nDBVJUmcMFUlSZwwVSVJnDBVJUmcMFUlSZwwVSVJnDBVJUmcMFUlSZwwVSVJnDBVJUmcMFUlSZwwV\nSVJnDBVJUmcMFUlSZwwVSVJnDBVJUmcMFUlSZwwVSVJnDBVJUmcMFUlSZ6Z9qCRZlOTWJBuTnDnq\n8UjSTDatQyXJLOADwPHAfOC1SeaPdlSSNHNN61ABjgQ2VtXtVfUt4BJg8YjHJEkz1q6jHsCTdCBw\nR9/3TcBR4xslWQYsa18fTHLrFIxtJtgP+PqoB/FUkD9eOuoh6D/y93PMWeliKc8fpNF0D5WBVNUF\nwAWjHsfTTZI1VbVw1OOQJuLv52hM98Nfm4GD+r7PaTVJ0ghM91C5HpiX5OAkuwMnAytHPCZJmrGm\n9eGvqno0yZuBK4BZwPKqWj/iYc0kHlLUU5m/nyOQqhr1GCRJTxPT/fCXJOkpxFCRJHVmWp9TUbeS\nPAbc1Fc6saq+uoO2c4FPVtWhwx+ZBEmeDVzZvj4HeAzY2r4f2W6A1ogZKur3cFUtGPUgpIlU1T3A\nAoAkZwMPVtUf97dJEnrnir8z9SMUePhLk0gyN8k/J/lC+7xkgjYvTHJdkrVJ1iWZ1+q/2lf/y/as\nNqlTSQ5JsiHJR4D1wEFJ7u2bf3KSv2rT+yf5eJI17Xfz6FGN++nKUFG/PVsArE3yiVa7G3hlVR0O\nvAY4b4J+bwTe1/ZyFgKbkvx4a//SVn8M+JXhb4JmqBcA51bVfHZ+A/R5wB+2O+1PAv5qKgY3k3j4\nS/0mOvy1G/DnScaC4Ucn6PevwDuTzAE+XlW3JTkGOAK4vndEgj3pBZQ0DF+uqjUDtPtZ4Mfa7yTA\nPkn2rKqHhze0mcVQ0WT+O3AXcBi9Pdtvjm9QVX+T5Frg54BVSd4ABFhRVW+fysFqxvpG3/R36P3+\njdmjbzp4Un+oPPylyTwL2NJOfL6O3pMLvk+SHwZur6rzgMuBn6R3lc6rk/xQa7NvkoGecio9Ge13\ndXuSeUl2AV7VN/ufgNPHvrQ9cHXIUNFkPggsTXIjvePW35igzUnAzUnWAocCF1fVBuB/AZ9Jsg5Y\nDRwwRWOW3kbv8U3/Qu+VGGNOB17aLijZAJw2isE9nfmYFklSZ9xTkSR1xlCRJHXGUJEkdcZQkSR1\nxlCRJHXGUJGGIMlVSY4bVzsjyfk76fPg8EcmDZehIg3HR4GTx9VObvUnLYlPw9BTkqEiDcdlwM8l\n2R2++/6Z5wJfTHJle+LzTUkWj++Ynj9KcnNr85pWf3l7YvRKYMPUbYo0OP+1Iw1BVW1Lch1wPL1H\n15wMXAo8DLyqqu5Psh9wTZKV9f13If8ivfeGHAbsR++hnJ9v8w4HDq2qr0zVtkiPh3sq0vD0HwIb\nO/QV4Pfao2v+CTgQ2H9cv5cBH62qx6rqLuBzwIvbvOsMFD2VGSrS8FwOHJPkcOAHquoGeu+UmQ0c\n0V4zcBff/xTdyUz07DXpKcNQkYakqh4ErgKW870T9M8C7q6qbyf5GWCiJzf/M/CaJLOSzAb+C3Dd\nVIxZerIMFWm4Pkrv3MhYqHwEWJjkJmAJ8KUJ+nwCWAfcCHwW+O2qunMKxio9aT6lWJLUGfdUJEmd\nMVQkSZ0xVCRJnTFUJEmdMVQkSZ0xVCRJnTFUJEmd+f8UByOfwhxJ+AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Proporção entre Inadimplentes no Dataset:\n",
            "False    0.779556\n",
            "True     0.148022\n",
            "Name: target_default, dtype: float64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YYahAiZnyzcY",
        "colab_type": "text"
      },
      "source": [
        "## Preparação dos Dados\n",
        "\n",
        "Com as informações que obtivemos da etapa anterior, vamos prosseguir com a limpeza e preparação dos dados."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BTF7zub_JbeV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_clean = df.copy()\n",
        "\n",
        "# substituir inf por nan em reported_income \n",
        "df_clean['reported_income'] = df_clean['reported_income'].replace(np.inf,\n",
        "                                                                  np.nan)\n",
        "\n",
        "# descartar as colunas ids, target_fraud,\n",
        "# external_data_provider_credit_checks_last_2_year e channel\n",
        "df_clean.drop(labels=['ids', 'target_fraud',\n",
        "                      'external_data_provider_credit_checks_last_2_year',\n",
        "                      'channel'], axis=1, inplace=True)\n",
        "\n",
        "#  substituir -999 em external_data_provider_email_seen_before por NaN\n",
        "df_clean.loc[df_clean['external_data_provider_email_seen_before'] == -999.0,\n",
        "             'external_data_provider_email_seen_before'] = np.nan\n",
        "\n",
        "# elminar colunas sem informação aparente ou que demandam mais pesquisa\n",
        "drop_var = ['reason', 'zip', 'job_name', 'external_data_provider_first_name',\n",
        "            'lat_lon', 'shipping_zip_code', 'user_agent', 'profile_tags',\n",
        "            'application_time_applied', 'email', 'marketing_channel',\n",
        "            'profile_phone_number', 'shipping_state']\n",
        "df_clean.drop(labels=drop_var, axis=1, inplace=True)\n",
        "\n",
        "# elminar as entradas onde target_default é NaN\n",
        "df_clean.dropna(subset=['target_default'], inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k3cRAf2YtEws",
        "colab_type": "text"
      },
      "source": [
        "Apesar de não haver maiores informações sobre as *features*, podemos assumir como premissa que muitos valores são nulos devido a dois fatores:\n",
        "\n",
        "1. Há alguns itens que não são obrigatórios durante a etapa de preenchimento dos formulários, como a informação sobre o perfil do cliente no Facebook; e\n",
        "2. Não existe observação alguma referente à coluna, como é caso das colunas `['last_amount_borrowed', 'last_borrowed_in_months', 'n_issues']`.\n",
        "\n",
        "De maneira geral, para as variáveis que se enquadram no segundo caso, os valores `NaN` serão substituidos por zero. Para as variáveis numéricas, será adotado o valor da mediana, e para as categóricas o valor mais frequente."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q5qLgUuPucUR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.impute import SimpleImputer\n",
        "\n",
        "num_df = df_clean.select_dtypes(exclude='object').columns\n",
        "cat_df = df_clean.select_dtypes(include='object').columns\n",
        "\n",
        "# substituição de valores ausentes\n",
        "df_clean.last_amount_borrowed.fillna(value=0, inplace=True)\n",
        "df_clean.last_borrowed_in_months.fillna(value=0, inplace=True)\n",
        "df_clean.n_issues.fillna(value=0, inplace=True)\n",
        "\n",
        "# variáveis numéricas\n",
        "imputer = SimpleImputer(missing_values=np.nan, strategy='median')\n",
        "imputer = imputer.fit(df_clean.loc[:,num_df])\n",
        "df_clean.loc[:,num_df] = imputer.transform(df_clean.loc[:,num_df])\n",
        "\n",
        "# variáveis categóricas\n",
        "imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')\n",
        "imputer = imputer.fit(df_clean.loc[:,cat_df])\n",
        "df_clean.loc[:,cat_df] = imputer.transform(df_clean.loc[:,cat_df])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Un5NRo81vJow",
        "colab_type": "text"
      },
      "source": [
        "Podemos agora verificar que as variáveis escolhidas não possuem mais os valores nulos iniciais."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BbaPh74i6rAc",
        "colab_type": "code",
        "outputId": "7fd050c1-0f1c-447c-df9c-99699f7d7b36",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 476
        }
      },
      "source": [
        "# verificar a quantidade de dados nulos\n",
        "df_clean.isnull().sum()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "target_default                                     0\n",
              "score_1                                            0\n",
              "score_2                                            0\n",
              "score_3                                            0\n",
              "score_4                                            0\n",
              "score_5                                            0\n",
              "score_6                                            0\n",
              "risk_rate                                          0\n",
              "last_amount_borrowed                               0\n",
              "last_borrowed_in_months                            0\n",
              "credit_limit                                       0\n",
              "income                                             0\n",
              "facebook_profile                                   0\n",
              "state                                              0\n",
              "real_state                                         0\n",
              "ok_since                                           0\n",
              "n_bankruptcies                                     0\n",
              "n_defaulted_loans                                  0\n",
              "n_accounts                                         0\n",
              "n_issues                                           0\n",
              "application_time_in_funnel                         0\n",
              "external_data_provider_credit_checks_last_month    0\n",
              "external_data_provider_credit_checks_last_year     0\n",
              "external_data_provider_email_seen_before           0\n",
              "external_data_provider_fraud_score                 0\n",
              "reported_income                                    0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-CUCnaxEcbIh",
        "colab_type": "text"
      },
      "source": [
        "Para melhorar o desempenho do modelo, irei normalizar as variáveis numéricas usando `StandardScaler`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "syGkH8EwcjCg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "scaled_df = df_clean.copy()\n",
        "# normalizar as colunas numéricas\n",
        "num_cols = scaled_df.drop('target_default',\n",
        "                         axis=1).select_dtypes(exclude='object').columns\n",
        "\n",
        "scaled_df[num_cols] = StandardScaler().fit_transform(scaled_df[num_cols].values)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-JG7GN_TeCT",
        "colab_type": "text"
      },
      "source": [
        "Continuando com a preparação dos dados, irei realizar o Label Encoding para possibilitar que as variáveis categóricas possam ser codificadas em forma de *labels* e alimentar o modelo de *Machine Learning*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HZ9OBmQ3U9Xd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.preprocessing import LabelEncoder\n",
        "\n",
        "encoded_df = scaled_df.copy()\n",
        "# extrair as colunas categóricas\n",
        "cat_cols = encoded_df.select_dtypes('object').columns\n",
        "\n",
        "# codificar cada coluna categórica\n",
        "for col in cat_cols:\n",
        "  encoded_df[col+'_encoded'] = LabelEncoder().fit_transform(encoded_df[col])\n",
        "  encoded_df.drop(col, axis=1, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cg8Ho3ElWNXF",
        "colab_type": "text"
      },
      "source": [
        "Por fim, irei separar a variável alvo das demais e dividir o *dataset* entre dados de treino e teste com a função `train_test_split`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rgxiU7jyWcJH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# separar as variáveis independentes da variável alvo\n",
        "X = encoded_df.drop('target_default', axis=1).select_dtypes(exclude='object')\n",
        "y = encoded_df['target_default']\n",
        "\n",
        "# dividir o dataset entre treino e teste\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "orDbdHaBSgPk",
        "colab_type": "text"
      },
      "source": [
        "## Construção do Modelo\n",
        "\n",
        "Arbitrariamente, escolhi o XGBoost para a construção do modelo, uma vez que este vem se destacando e apresentando bons resultados em problemas similares a este.\n",
        "\n",
        "Não foi realizada uma análise mais aprofundada sobre as variáveis e nem um *tunning* de parâmetros, mas construído um modelo simples para servir de *baseline*. Ressalta-se que os valores dos parâmetros do XGBoost também foram generalistas e arbitrários."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nsN7t-H6BHtV",
        "colab_type": "code",
        "outputId": "13a17972-7b8c-4fae-de24-a41d6b7aad94",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "from xgboost import XGBClassifier\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "# instanciando o modelo e treinando o mesmo\n",
        "ml_model = XGBClassifier(learning_rate=0.01, n_estimators=1000,\n",
        "                         max_depth=3, subsample = 0.9,\n",
        "                         colsample_bytree = 0.1, gamma=1,\n",
        "                         random_state=42)\n",
        "ml_model.fit(X_train, y_train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
              "              colsample_bynode=1, colsample_bytree=0.1, gamma=1,\n",
              "              learning_rate=0.01, max_delta_step=0, max_depth=3,\n",
              "              min_child_weight=1, missing=None, n_estimators=1000, n_jobs=1,\n",
              "              nthread=None, objective='binary:logistic', random_state=42,\n",
              "              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
              "              silent=None, subsample=0.9, verbosity=1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O6R6RKaqmpsi",
        "colab_type": "text"
      },
      "source": [
        "Lembra-se que o *dataset* utilizado neste projeto é desbalanceado, o que pode gerar métricas que não representam a realidade da situação problemática do mundo real.\n",
        "\n",
        "Sem ser exaustivo, será apresentada apenas a acurácia do modelo como métrica simples de avaliação."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9W6NdGsyPShc",
        "colab_type": "code",
        "outputId": "2a5d48d1-7339-4d55-bb9c-c649f07fbbaa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# previsões e probabilidades em cima do dataset de teste\n",
        "y_pred = ml_model.predict(X_test)\n",
        "\n",
        "# ver performance do algoritmo\n",
        "print(\"\\nAccuracy Score:\")\n",
        "print (accuracy_score(y_test, y_pred))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Accuracy Score:\n",
            "0.8406132715802923\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jjzi9UH1gP5m",
        "colab_type": "text"
      },
      "source": [
        "Pode-se assumir a premissa de que este é um problema multiobjetivo de otimização, onde espera-se minimizar tanto a quantidade de *false positives* quanto maximizar a quantidade de *true positives*.\n",
        "\n",
        "Para esta etapa inicial, a utilizacão da métrica acurácia pode ser estabelecida a fim de comparação entre diferentes modelos de *Machine Learning* (ou estabelecer um *benchmark*).\n",
        "\n",
        "Como ação futura, recomenda-se a utilização de *Problem Structuring Methods* para definir pesos para cada objetivo individual a fim de estabelecer uma função objetivo para a modelagem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PqMaG41qo1Y4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}