Vocal handling_up_to_date.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploration and pre-process the data\n",
    "\n",
    "\n",
    "\n",
    "In all Machine Learning problem one of the main task is to gain suffitant knowledge about the data one is given. This will ensure to better build the features representation of a tweet and to maximize our \"computational power / prediction accuracy\" ratio.\n",
    "\n",
    "In the first part we will therefore investigate how the data is generated, how many simple words do we have, are there some duplicates, is there any odds with the tweets? How are they classified (we are given two datasets, one for the positive tweets one for negatives) i.e. what is a typical word that occurs when a :( is present in a tweet?\n",
    "\n",
    "\n",
    "In the second part, we normalize the tweets. Some tweets may contain words that are not usefull, or can be categorized. This is done be the following pipeline.\n",
    "\n",
    "1. Import all the tweets from train_pos_full.txt and train_neg_full.txt\n",
    "2. Cast the repetition of more than three following similar letter to one letter\n",
    "3. Find the representative of all words\n",
    "4. \n",
    "\n",
    "\n",
    "\n",
    "### Helper functions and files\n",
    "\n",
    "IOTweets contains the functions :\n",
    "- build_df\n",
    "- import_\n",
    "- export\n",
    "\n",
    "ProcessTweets contains the functions :\n",
    "- merging\n",
    "- filter_single_rep\n",
    "- powerful_words\n",
    "- sem_by_repr\n",
    "- sem_by_repr2\n",
    "- no_dot\n",
    "- find_repetition\n",
    "- no_s\n",
    "- stem_tweet\n",
    "- set_min_diff\n",
    "- contains\n",
    "- process_tweets_1\n",
    "- process_tweets_2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd  # Pandas will be our framework for the first part\n",
    "from nltk.stem import PorterStemmer\n",
    "import nltk\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "from IOTweets import *\n",
    "from ProcessTweets import *\n",
    "import csv\n",
    "#%pylab inline # depreceated, use individual imports instead\n",
    "\n",
    "ps = PorterStemmer()\n",
    "tknzr = TweetTokenizer(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vocab_pos = \"vocab_cut_pos.txt\"\n",
    "vocab_neg = \"vocab_cut_neg.txt\"\n",
    "\n",
    "vocab_pos_full = \"vocab_cut_pos_full.txt\"\n",
    "vocab_neg_full = \"vocab_cut_neg_full.txt\"\n",
    "vocab_test = \"vocab_test_data.txt\"\n",
    "\n",
    "path_tweets_pos = \"train_pos.txt\"\n",
    "path_tweets_neg = \"train_neg.txt\"\n",
    "\n",
    "path_tweets_pos_full = \"train_pos_full.txt\"\n",
    "path_tweets_neg_full = \"train_neg_full.txt\"\n",
    "path_tweets_test = \"test_data.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. An insight into the dataset \n",
    "We begin by analysing the non-processed dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def write_vocab(tweets, cut_threshold, file_name , bitri):\n",
    "    counter = build_vocab_counter(tweets, cut_threshold, bitri)\n",
    "    write_vocab_to_file(counter, (file_name + \"_cut=\" +str(cut_threshold) +\"_bitri=\"+str(bitri)))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:00:12.543123\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:00:14.648871\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:00:01.305080\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:02:34.135139\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:03:04.960437\n"
     ]
    }
   ],
   "source": [
    "#import tweets\n",
    "pos_tweets = import_(path_tweets_pos)\n",
    "neg_tweets = import_(path_tweets_neg)\n",
    "pos_full_tweets = import_(path_tweets_pos_full)\n",
    "neg_full_tweets = import_(path_tweets_neg_full)\n",
    "test_tweets = import_without_id(path_tweets_test)\n",
    "\n",
    "#Create vocabs\n",
    "cut_threshold = 5\n",
    "bitri = True\n",
    "\n",
    "write_vocab(pos_tweets, cut_threshold, \"raw_vocab_pos\" , bitri)\n",
    "write_vocab(neg_tweets, cut_threshold, \"raw_vocab_neg\" , bitri)\n",
    "write_vocab(test_tweets, cut_threshold, \"raw_vocab_test\" , bitri)\n",
    "write_vocab(pos_full_tweets, cut_threshold, \"raw_vocab_pos_full\" , bitri)\n",
    "write_vocab(neg_full_tweets, cut_threshold, \"raw_vocab_neg_full\" , bitri)\n",
    "\n",
    "\n",
    "# build the vocab Dataframe (DF) for full positive and negative tweets\n",
    "pos_full_df = build_df(\"raw_vocab_pos_full_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "neg_full_df = build_df(\"raw_vocab_neg_full_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "\n",
    "# build the vocab Dataframe (DF) for partial positive and negative tweets\n",
    "pos_df = build_df(\"raw_vocab_pos_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "neg_df = build_df(\"raw_vocab_neg_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "\n",
    "#Build the vocab DF for tweets to be decided (i.e. \"test\" tweets)\n",
    "test_df = build_df(\"raw_vocab_test_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cut_threshold = 5\n",
    "bitri = True\n",
    "\n",
    "pos_df = build_df(\"raw_vocab_pos_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "neg_df = build_df(\"raw_vocab_neg_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "pos_full_df = build_df(\"raw_vocab_pos_full_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "neg_full_df = build_df(\"raw_vocab_neg_full_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)\n",
    "\n",
    "#Build the vocab DF for tweets to be decided (i.e. \"test\" tweets)\n",
    "test_df = build_df(\"raw_vocab_test_cut=\"+str(cut_threshold)+\"_bitri=\"+str(bitri), bitri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Study of word pertinence \n",
    "The idea is that the greater a word ratio i, the more pertinent it will be to cosider it when trying to label a tweet that contains it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>occurence_neg</th>\n",
       "      <th>occurence_pos</th>\n",
       "      <th>difference</th>\n",
       "      <th>somme</th>\n",
       "      <th>ratio</th>\n",
       "      <th>len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1201</th>\n",
       "      <td>(photo,)</td>\n",
       "      <td>3000</td>\n",
       "      <td>3005</td>\n",
       "      <td>5</td>\n",
       "      <td>6005</td>\n",
       "      <td>0.000833</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>581</th>\n",
       "      <td>(had, a)</td>\n",
       "      <td>5724</td>\n",
       "      <td>5738</td>\n",
       "      <td>14</td>\n",
       "      <td>11462</td>\n",
       "      <td>0.001221</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1076</th>\n",
       "      <td>(all, my)</td>\n",
       "      <td>3308</td>\n",
       "      <td>3299</td>\n",
       "      <td>9</td>\n",
       "      <td>6607</td>\n",
       "      <td>0.001362</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>917</th>\n",
       "      <td>(yesterday,)</td>\n",
       "      <td>3799</td>\n",
       "      <td>3786</td>\n",
       "      <td>13</td>\n",
       "      <td>7585</td>\n",
       "      <td>0.001714</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2770</th>\n",
       "      <td>(\", .)</td>\n",
       "      <td>1394</td>\n",
       "      <td>1399</td>\n",
       "      <td>5</td>\n",
       "      <td>2793</td>\n",
       "      <td>0.001790</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>579</th>\n",
       "      <td>(class,)</td>\n",
       "      <td>5729</td>\n",
       "      <td>5754</td>\n",
       "      <td>25</td>\n",
       "      <td>11483</td>\n",
       "      <td>0.002177</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1492</th>\n",
       "      <td>(turn,)</td>\n",
       "      <td>2432</td>\n",
       "      <td>2443</td>\n",
       "      <td>11</td>\n",
       "      <td>4875</td>\n",
       "      <td>0.002256</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>(or,)</td>\n",
       "      <td>24351</td>\n",
       "      <td>24227</td>\n",
       "      <td>124</td>\n",
       "      <td>48578</td>\n",
       "      <td>0.002553</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1416</th>\n",
       "      <td>(..., but)</td>\n",
       "      <td>2566</td>\n",
       "      <td>2580</td>\n",
       "      <td>14</td>\n",
       "      <td>5146</td>\n",
       "      <td>0.002721</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2430</th>\n",
       "      <td>(next, year)</td>\n",
       "      <td>1576</td>\n",
       "      <td>1567</td>\n",
       "      <td>9</td>\n",
       "      <td>3143</td>\n",
       "      <td>0.002864</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3395</th>\n",
       "      <td>(when, he)</td>\n",
       "      <td>1163</td>\n",
       "      <td>1170</td>\n",
       "      <td>7</td>\n",
       "      <td>2333</td>\n",
       "      <td>0.003000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1385</th>\n",
       "      <td>(idea,)</td>\n",
       "      <td>2624</td>\n",
       "      <td>2640</td>\n",
       "      <td>16</td>\n",
       "      <td>5264</td>\n",
       "      <td>0.003040</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4383</th>\n",
       "      <td>(that, &lt;user&gt;)</td>\n",
       "      <td>921</td>\n",
       "      <td>927</td>\n",
       "      <td>6</td>\n",
       "      <td>1848</td>\n",
       "      <td>0.003247</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2504</th>\n",
       "      <td>(., please)</td>\n",
       "      <td>1532</td>\n",
       "      <td>1542</td>\n",
       "      <td>10</td>\n",
       "      <td>3074</td>\n",
       "      <td>0.003253</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>291</th>\n",
       "      <td>(down,)</td>\n",
       "      <td>11021</td>\n",
       "      <td>10947</td>\n",
       "      <td>74</td>\n",
       "      <td>21968</td>\n",
       "      <td>0.003369</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>678</th>\n",
       "      <td>(!, !, i)</td>\n",
       "      <td>4970</td>\n",
       "      <td>4935</td>\n",
       "      <td>35</td>\n",
       "      <td>9905</td>\n",
       "      <td>0.003534</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3585</th>\n",
       "      <td>(life, .)</td>\n",
       "      <td>1113</td>\n",
       "      <td>1121</td>\n",
       "      <td>8</td>\n",
       "      <td>2234</td>\n",
       "      <td>0.003581</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>465</th>\n",
       "      <td>(&lt;user&gt;, no)</td>\n",
       "      <td>6977</td>\n",
       "      <td>7028</td>\n",
       "      <td>51</td>\n",
       "      <td>14005</td>\n",
       "      <td>0.003642</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5067</th>\n",
       "      <td>(should, i)</td>\n",
       "      <td>804</td>\n",
       "      <td>798</td>\n",
       "      <td>6</td>\n",
       "      <td>1602</td>\n",
       "      <td>0.003745</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2192</th>\n",
       "      <td>(it, when)</td>\n",
       "      <td>1708</td>\n",
       "      <td>1695</td>\n",
       "      <td>13</td>\n",
       "      <td>3403</td>\n",
       "      <td>0.003820</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5352</th>\n",
       "      <td>(im, in)</td>\n",
       "      <td>761</td>\n",
       "      <td>755</td>\n",
       "      <td>6</td>\n",
       "      <td>1516</td>\n",
       "      <td>0.003958</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6696</th>\n",
       "      <td>(is, about)</td>\n",
       "      <td>615</td>\n",
       "      <td>610</td>\n",
       "      <td>5</td>\n",
       "      <td>1225</td>\n",
       "      <td>0.004082</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1287</th>\n",
       "      <td>(sometimes,)</td>\n",
       "      <td>2827</td>\n",
       "      <td>2851</td>\n",
       "      <td>24</td>\n",
       "      <td>5678</td>\n",
       "      <td>0.004227</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2783</th>\n",
       "      <td>(lie,)</td>\n",
       "      <td>1387</td>\n",
       "      <td>1375</td>\n",
       "      <td>12</td>\n",
       "      <td>2762</td>\n",
       "      <td>0.004345</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5994</th>\n",
       "      <td>(are, all)</td>\n",
       "      <td>681</td>\n",
       "      <td>687</td>\n",
       "      <td>6</td>\n",
       "      <td>1368</td>\n",
       "      <td>0.004386</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5135</th>\n",
       "      <td>(!, i, just)</td>\n",
       "      <td>793</td>\n",
       "      <td>786</td>\n",
       "      <td>7</td>\n",
       "      <td>1579</td>\n",
       "      <td>0.004433</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3202</th>\n",
       "      <td>(and, she)</td>\n",
       "      <td>1220</td>\n",
       "      <td>1209</td>\n",
       "      <td>11</td>\n",
       "      <td>2429</td>\n",
       "      <td>0.004529</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7547</th>\n",
       "      <td>(come, in)</td>\n",
       "      <td>544</td>\n",
       "      <td>549</td>\n",
       "      <td>5</td>\n",
       "      <td>1093</td>\n",
       "      <td>0.004575</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6712</th>\n",
       "      <td>(why, not)</td>\n",
       "      <td>613</td>\n",
       "      <td>607</td>\n",
       "      <td>6</td>\n",
       "      <td>1220</td>\n",
       "      <td>0.004918</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5873</th>\n",
       "      <td>(about, this)</td>\n",
       "      <td>696</td>\n",
       "      <td>703</td>\n",
       "      <td>7</td>\n",
       "      <td>1399</td>\n",
       "      <td>0.005004</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>529</th>\n",
       "      <td>(frame, 2, \")</td>\n",
       "      <td>6194</td>\n",
       "      <td>0</td>\n",
       "      <td>6194</td>\n",
       "      <td>6194</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>526</th>\n",
       "      <td>(2, \", wide)</td>\n",
       "      <td>6198</td>\n",
       "      <td>0</td>\n",
       "      <td>6198</td>\n",
       "      <td>6198</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>455</th>\n",
       "      <td>((, kindle, edition)</td>\n",
       "      <td>7139</td>\n",
       "      <td>0</td>\n",
       "      <td>7139</td>\n",
       "      <td>7139</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>453</th>\n",
       "      <td>(kindle, edition)</td>\n",
       "      <td>7172</td>\n",
       "      <td>0</td>\n",
       "      <td>7172</td>\n",
       "      <td>7172</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>446</th>\n",
       "      <td>((, kindle)</td>\n",
       "      <td>7261</td>\n",
       "      <td>0</td>\n",
       "      <td>7261</td>\n",
       "      <td>7261</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>434</th>\n",
       "      <td>(kit, ()</td>\n",
       "      <td>7394</td>\n",
       "      <td>0</td>\n",
       "      <td>7394</td>\n",
       "      <td>7394</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>407</th>\n",
       "      <td>(), (, paperback)</td>\n",
       "      <td>7944</td>\n",
       "      <td>0</td>\n",
       "      <td>7944</td>\n",
       "      <td>7944</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>354</th>\n",
       "      <td>(wood, frame, ()</td>\n",
       "      <td>9182</td>\n",
       "      <td>0</td>\n",
       "      <td>9182</td>\n",
       "      <td>9182</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>331</th>\n",
       "      <td>((, audio, cd)</td>\n",
       "      <td>9706</td>\n",
       "      <td>0</td>\n",
       "      <td>9706</td>\n",
       "      <td>9706</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>327</th>\n",
       "      <td>(audio, cd)</td>\n",
       "      <td>9823</td>\n",
       "      <td>0</td>\n",
       "      <td>9823</td>\n",
       "      <td>9823</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>321</th>\n",
       "      <td>((, audio)</td>\n",
       "      <td>9995</td>\n",
       "      <td>0</td>\n",
       "      <td>9995</td>\n",
       "      <td>9995</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>309</th>\n",
       "      <td>((, dvd)</td>\n",
       "      <td>10407</td>\n",
       "      <td>0</td>\n",
       "      <td>10407</td>\n",
       "      <td>10407</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>276</th>\n",
       "      <td>(wood, frame)</td>\n",
       "      <td>11775</td>\n",
       "      <td>0</td>\n",
       "      <td>11775</td>\n",
       "      <td>11775</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>((, hardcover)</td>\n",
       "      <td>16678</td>\n",
       "      <td>0</td>\n",
       "      <td>16678</td>\n",
       "      <td>16678</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>(frame, ()</td>\n",
       "      <td>21266</td>\n",
       "      <td>0</td>\n",
       "      <td>21266</td>\n",
       "      <td>21266</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>(this, frame, is)</td>\n",
       "      <td>23692</td>\n",
       "      <td>0</td>\n",
       "      <td>23692</td>\n",
       "      <td>23692</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>(frame, is)</td>\n",
       "      <td>23727</td>\n",
       "      <td>0</td>\n",
       "      <td>23727</td>\n",
       "      <td>23727</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>(custom, picture)</td>\n",
       "      <td>24002</td>\n",
       "      <td>0</td>\n",
       "      <td>24002</td>\n",
       "      <td>24002</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>(custom, picture, frame)</td>\n",
       "      <td>24002</td>\n",
       "      <td>0</td>\n",
       "      <td>24002</td>\n",
       "      <td>24002</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>(this, frame)</td>\n",
       "      <td>24685</td>\n",
       "      <td>0</td>\n",
       "      <td>24685</td>\n",
       "      <td>24685</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>(\", wide, complete)</td>\n",
       "      <td>25034</td>\n",
       "      <td>0</td>\n",
       "      <td>25034</td>\n",
       "      <td>25034</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>(wide, complete)</td>\n",
       "      <td>25035</td>\n",
       "      <td>0</td>\n",
       "      <td>25035</td>\n",
       "      <td>25035</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>(picture, frame, /)</td>\n",
       "      <td>25402</td>\n",
       "      <td>0</td>\n",
       "      <td>25402</td>\n",
       "      <td>25402</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>(frame, /, poster)</td>\n",
       "      <td>25405</td>\n",
       "      <td>0</td>\n",
       "      <td>25405</td>\n",
       "      <td>25405</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>(/, poster, frame)</td>\n",
       "      <td>25405</td>\n",
       "      <td>0</td>\n",
       "      <td>25405</td>\n",
       "      <td>25405</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>(poster, frame)</td>\n",
       "      <td>25408</td>\n",
       "      <td>0</td>\n",
       "      <td>25408</td>\n",
       "      <td>25408</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>(/, poster)</td>\n",
       "      <td>25415</td>\n",
       "      <td>0</td>\n",
       "      <td>25415</td>\n",
       "      <td>25415</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>(frame, /)</td>\n",
       "      <td>25425</td>\n",
       "      <td>0</td>\n",
       "      <td>25425</td>\n",
       "      <td>25425</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>(\", wide)</td>\n",
       "      <td>25488</td>\n",
       "      <td>0</td>\n",
       "      <td>25488</td>\n",
       "      <td>25488</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>((, paperback)</td>\n",
       "      <td>40169</td>\n",
       "      <td>0</td>\n",
       "      <td>40169</td>\n",
       "      <td>40169</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1079592 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          word  occurence_neg  occurence_pos  difference  \\\n",
       "1201                  (photo,)           3000           3005           5   \n",
       "581                   (had, a)           5724           5738          14   \n",
       "1076                 (all, my)           3308           3299           9   \n",
       "917               (yesterday,)           3799           3786          13   \n",
       "2770                    (\", .)           1394           1399           5   \n",
       "579                   (class,)           5729           5754          25   \n",
       "1492                   (turn,)           2432           2443          11   \n",
       "134                      (or,)          24351          24227         124   \n",
       "1416                (..., but)           2566           2580          14   \n",
       "2430              (next, year)           1576           1567           9   \n",
       "3395                (when, he)           1163           1170           7   \n",
       "1385                   (idea,)           2624           2640          16   \n",
       "4383            (that, <user>)            921            927           6   \n",
       "2504               (., please)           1532           1542          10   \n",
       "291                    (down,)          11021          10947          74   \n",
       "678                  (!, !, i)           4970           4935          35   \n",
       "3585                 (life, .)           1113           1121           8   \n",
       "465               (<user>, no)           6977           7028          51   \n",
       "5067               (should, i)            804            798           6   \n",
       "2192                (it, when)           1708           1695          13   \n",
       "5352                  (im, in)            761            755           6   \n",
       "6696               (is, about)            615            610           5   \n",
       "1287              (sometimes,)           2827           2851          24   \n",
       "2783                    (lie,)           1387           1375          12   \n",
       "5994                (are, all)            681            687           6   \n",
       "5135              (!, i, just)            793            786           7   \n",
       "3202                (and, she)           1220           1209          11   \n",
       "7547                (come, in)            544            549           5   \n",
       "6712                (why, not)            613            607           6   \n",
       "5873             (about, this)            696            703           7   \n",
       "...                        ...            ...            ...         ...   \n",
       "529              (frame, 2, \")           6194              0        6194   \n",
       "526               (2, \", wide)           6198              0        6198   \n",
       "455       ((, kindle, edition)           7139              0        7139   \n",
       "453          (kindle, edition)           7172              0        7172   \n",
       "446                ((, kindle)           7261              0        7261   \n",
       "434                   (kit, ()           7394              0        7394   \n",
       "407          (), (, paperback)           7944              0        7944   \n",
       "354           (wood, frame, ()           9182              0        9182   \n",
       "331             ((, audio, cd)           9706              0        9706   \n",
       "327                (audio, cd)           9823              0        9823   \n",
       "321                 ((, audio)           9995              0        9995   \n",
       "309                   ((, dvd)          10407              0       10407   \n",
       "276              (wood, frame)          11775              0       11775   \n",
       "201             ((, hardcover)          16678              0       16678   \n",
       "161                 (frame, ()          21266              0       21266   \n",
       "142          (this, frame, is)          23692              0       23692   \n",
       "141                (frame, is)          23727              0       23727   \n",
       "138          (custom, picture)          24002              0       24002   \n",
       "139   (custom, picture, frame)          24002              0       24002   \n",
       "132              (this, frame)          24685              0       24685   \n",
       "129        (\", wide, complete)          25034              0       25034   \n",
       "128           (wide, complete)          25035              0       25035   \n",
       "127        (picture, frame, /)          25402              0       25402   \n",
       "125         (frame, /, poster)          25405              0       25405   \n",
       "126         (/, poster, frame)          25405              0       25405   \n",
       "123            (poster, frame)          25408              0       25408   \n",
       "122                (/, poster)          25415              0       25415   \n",
       "121                 (frame, /)          25425              0       25425   \n",
       "120                  (\", wide)          25488              0       25488   \n",
       "67              ((, paperback)          40169              0       40169   \n",
       "\n",
       "      somme     ratio  len  \n",
       "1201   6005  0.000833    1  \n",
       "581   11462  0.001221    2  \n",
       "1076   6607  0.001362    2  \n",
       "917    7585  0.001714    1  \n",
       "2770   2793  0.001790    2  \n",
       "579   11483  0.002177    1  \n",
       "1492   4875  0.002256    1  \n",
       "134   48578  0.002553    1  \n",
       "1416   5146  0.002721    2  \n",
       "2430   3143  0.002864    2  \n",
       "3395   2333  0.003000    2  \n",
       "1385   5264  0.003040    1  \n",
       "4383   1848  0.003247    2  \n",
       "2504   3074  0.003253    2  \n",
       "291   21968  0.003369    1  \n",
       "678    9905  0.003534    3  \n",
       "3585   2234  0.003581    2  \n",
       "465   14005  0.003642    2  \n",
       "5067   1602  0.003745    2  \n",
       "2192   3403  0.003820    2  \n",
       "5352   1516  0.003958    2  \n",
       "6696   1225  0.004082    2  \n",
       "1287   5678  0.004227    1  \n",
       "2783   2762  0.004345    1  \n",
       "5994   1368  0.004386    2  \n",
       "5135   1579  0.004433    3  \n",
       "3202   2429  0.004529    2  \n",
       "7547   1093  0.004575    2  \n",
       "6712   1220  0.004918    2  \n",
       "5873   1399  0.005004    2  \n",
       "...     ...       ...  ...  \n",
       "529    6194  1.000000    3  \n",
       "526    6198  1.000000    3  \n",
       "455    7139  1.000000    3  \n",
       "453    7172  1.000000    2  \n",
       "446    7261  1.000000    2  \n",
       "434    7394  1.000000    2  \n",
       "407    7944  1.000000    3  \n",
       "354    9182  1.000000    3  \n",
       "331    9706  1.000000    3  \n",
       "327    9823  1.000000    2  \n",
       "321    9995  1.000000    2  \n",
       "309   10407  1.000000    2  \n",
       "276   11775  1.000000    2  \n",
       "201   16678  1.000000    2  \n",
       "161   21266  1.000000    2  \n",
       "142   23692  1.000000    3  \n",
       "141   23727  1.000000    2  \n",
       "138   24002  1.000000    2  \n",
       "139   24002  1.000000    3  \n",
       "132   24685  1.000000    2  \n",
       "129   25034  1.000000    3  \n",
       "128   25035  1.000000    2  \n",
       "127   25402  1.000000    3  \n",
       "125   25405  1.000000    3  \n",
       "126   25405  1.000000    3  \n",
       "123   25408  1.000000    2  \n",
       "122   25415  1.000000    2  \n",
       "121   25425  1.000000    2  \n",
       "120   25488  1.000000    2  \n",
       "67    40169  1.000000    2  \n",
       "\n",
       "[1079592 rows x 7 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "neg_full_df\n",
    "\n",
    "#merge the pos and neg vocabs and study penrtinence of words via ratio\n",
    "merged_full = merging(neg_full_df, pos_full_df)\n",
    "merged_full.sort_values(by = [\"ratio\",\"somme\"], ascending=[True, True])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### From here, we realise that some words have strangely strong occurence in the negative tweets.\n",
    "By seeing the words in context, we realised that some tweets occured more than once.\n",
    "We checked if those words were also in the test_data that we have to classify. The check was positive.\n",
    "We will therefore capture those words, (i.e. \"1gb\" or \"cd-rom\") because they are luckily to be in the test_data set and classify directly the tweets countaining those words. We will also drop all the duplicated tweets for training in order not to let on the side some other words and this will save us power computationnal efficiency.\n",
    "\n",
    "Two example of such tweets are:\n",
    "\n",
    "    1) 1.26 - 7x14 custom picture frame / poster frame 1.265 \" wide complete cherry wood frame ( 440ch this frame is manufactu ... <url>\n",
    "    \n",
    "    2) misc - 50pc diamond burr set - ceramics tile glass lapidary for rotary tools ( misc . assorted shapes and sizes for your ... <url>\n",
    "    \n",
    "Other important notice, some Tweets are almost the same but just fiew things change: Example of a tweet from amazon [here](https://www.amazon.com/Custom-Picture-Frame-Poster-Complete/dp/B004FNYSBA?SubscriptionId=AKIAJ6364XFIEG2FHXPA&tag=megaebookmall-20&linkCode=sp1&camp=2025&creative=165953&creativeASIN=B004FNYSBA) \n",
    "\n",
    "1. `3x14 custom picture frame / poster frame 1.265 \" wide complete black wood frame ( 440bk this frame is manufactur ... <url>`\n",
    "\n",
    "2. `24x35 custom picture frame / poster frame 1.265 \" wide complete green wood frame ( 440gr this frame is manufactu ... <url>`\n",
    "\n",
    "3. `22x31 custom picture frame / poster frame 2 \" wide complete black executive leather frame ( 74093 this frame is ... <url>`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Study of names with same beginning \n",
    "Let's try to have an insight of words that have the same beginning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# are words (filtered by length) whose beginning match another word close to this word?\n",
    "same_begin = list(merged_full[merged_full[\"len\"]==8][\"word\"])\n",
    "same_begin = [list(merged_full.loc[(merged_full.word.str.startswith(w))][\"word\"]) for w in same_begin]\n",
    "\n",
    "#We visualize which word would be mapped on which one \n",
    "same_begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Expected syntaxic divergences\n",
    "As expected, words like \"haha\" can be written in many ways (extended length, typo..). We know that they mean the same,it is therefore interesting to spot them in order to later stem them to the same representative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#words beginning with \"haha\" or \"ahah\" are sure to be \"instances\" of haha\n",
    "word_haha = test_df.loc[test_df.word.str.startswith(\"haha\") | test_df.word.str.startswith(\"ahah\")]\n",
    "\n",
    "\n",
    "#print(\"Occurence of the words that can be remplaced by haha = \"+ str(int(word_haha.occurence.sum()) - int(word_haha[word_haha.word == \"haha\"].occurence)))\n",
    "\n",
    "# We make the list of all those words that have this same semantic\n",
    "word_haha_list = list(word_haha.word)\n",
    "list(word_haha.word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Study duplicates\n",
    "We noticed that several tweets were duplicated in test, pos and neg tweets. We study ho many they are, and which tweets (if any) are shared between the test set and the pos/neg set.\n",
    "We noticed after running that there are no tweets shared between all tweet sets, thus we can label duplicate of the test dataset by directly looking for corresponding duplicates in pos/neg "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#load raw tweets\n",
    "pos = import_(path_tweets_pos_full)\n",
    "neg = import_(path_tweets_neg_full)\n",
    "data = import_without_id(path_tweets_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done% % % % % %%\n"
     ]
    }
   ],
   "source": [
    "common_pos_test = []\n",
    "common_neg_test = []\n",
    "\n",
    "pos = set(pos_tweets)\n",
    "neg = set(neg_tweets)\n",
    "test_tweets = set(test_tweets)\n",
    "len_test = len(test_tweets)\n",
    "for i, test_tweet in enumerate(test_tweets) :\n",
    "    if test_tweet in pos:\n",
    "        common_pos_test.append(test_tweet)\n",
    "        print(\"duplicate pos-test\")\n",
    "        print(\"\\ttweet :\" + str(pos_tweet))  \n",
    "        break\n",
    "    elif test_tweet in neg:\n",
    "        common_neg_test.append(test_tweet)\n",
    "        print(\"duplicate neg-data\")\n",
    "        print(\"\\ttweet :\" + str(pos_tweet))\n",
    "        break\n",
    "    print(\"{:.1f}\".format(i/len_test), \"%\", end='\\r')\n",
    "            \n",
    "print (\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def occurences(tweet, tweets):\n",
    "    count = 0\n",
    "    for t in tweets:\n",
    "        if t == tweet:\n",
    "            count += 1\n",
    "    return count\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 data pre-processing\n",
    "In this part we start from raw tweets and preprocess them to achieve uniformization of the words\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load raw tweets\n",
    "pos = import_(path_tweets_pos_full)\n",
    "neg = import_(path_tweets_neg_full)\n",
    "data = import_without_id(path_tweets_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1  Data Standardization\n",
    "We start by removing repetitions of more than 3 letters, and replace dots by spaces\n",
    "ex : \n",
    "\n",
    "1) `no_dot` : \"Funny.I\" becomes \"Funny I\" \n",
    "\n",
    "2) `find_repetitions` : \"I looooove\" becomes \"I love\"\n",
    "            \n",
    "Then, we define which word is equivalent to each word\n",
    "More precisely, all words of length 8 or more are truncated, and words begining with \"hah\" or \"aha\" are stemmed to \"haha\" \n",
    "\n",
    "This is done by the function `standardize_tweets`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Process data pos\n",
      "Process data neg\n",
      "Process data test\n",
      "export processed tweets to files\n"
     ]
    }
   ],
   "source": [
    "#standardize tweets\n",
    "print(\"Process data pos\")\n",
    "processed_tweets_pos_full = standardize_tweets(pos)\n",
    "print(\"Process data neg\")\n",
    "processed_tweets_neg_full = standardize_tweets(neg) \n",
    "print(\"Process data test\")\n",
    "processed_tweets_test_full = standardize_tweets(data) \n",
    "\n",
    "print(\"export processed tweets to files\")\n",
    "export(processed_tweets_pos_full,  \"cleaned_pos_full\")\n",
    "export(processed_tweets_neg_full,  \"cleaned_neg_full\")\n",
    "export(processed_tweets_test_full, \"cleaned_test_full\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Build the vocabs\n",
    "We build vocabs for positive, negative and test tweets, removing tokens present less than 5 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:02:32.115393\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:03:02.518599\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:00:01.345433\n"
     ]
    }
   ],
   "source": [
    "#build word counters\n",
    "cut_threshold = 5\n",
    "vocab_pos =  build_vocab_counter(processed_tweets_pos_full, cut_threshold, True)\n",
    "vocab_neg =  build_vocab_counter(processed_tweets_neg_full, cut_threshold, True)\n",
    "vocab_test = build_vocab_counter(processed_tweets_test_full, cut_threshold, True)\n",
    "\n",
    "#write vocabs to files\n",
    "write_vocab_to_file(vocab_pos, \"cleaned_vocab_pos_full_bitri=True\")\n",
    "write_vocab_to_file(vocab_neg, \"cleaned_vocab_neg_full_bitri=True\")\n",
    "write_vocab_to_file(vocab_test, \"cleaned_vocab_test_full_bitri=True\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 create relevant vocab \n",
    "We keep pertinent words in a special vocab called relevant vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Load dataframes before we \n",
    "neg_df = build_df(\"cleaned_vocab_pos_full_bitri=True\", True)\n",
    "pos_df = build_df(\"cleaned_vocab_neg_full_bitri=True\", True)\n",
    "\n",
    "#Merge dataframes\n",
    "merged = merging(neg_df, pos_df, False)\n",
    "\n",
    "#create relevant vocab\n",
    "create_relevant_vocab(pertinence_thres=0.3, min_count=300, dataframe=merged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Find characteristic words\n",
    "We set a list of words that appear only in neg/pos at least min_diff times such that if we see such a word in a tweet we assume it is a neg/pos tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Import Data\n",
      "Process data pos\n",
      "Process data neg\n",
      "Process data test\n",
      "build vocab data pos\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:00:10.767453\n",
      "build vocab data neg\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:00:13.057752\n",
      "build vocab test data\n",
      "removing items present less than 5 times\n",
      "Time elpased (hh:mm:ss.ms) 0:00:01.261803\n",
      "export data\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'write_relevance_to_file' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-8b9457b0a1a5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     32\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 34\u001b[0;31m \u001b[0msetup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"train_pos.txt\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"train_neg.txt\"\u001b[0m \u001b[0;34m,\u001b[0m \u001b[0;34m\"test_data.txt\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_full\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpertinence_thres_relevant\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmin_count_relevant\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m250\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-3-8b9457b0a1a5>\u001b[0m in \u001b[0;36msetup\u001b[0;34m(path_pos, path_neg, path_test, is_full, pertinence_thres_relevant, min_count_relevant, cut_threshold, bitri)\u001b[0m\n\u001b[1;32m     21\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m     \u001b[0;31m#create relevant vocab\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 23\u001b[0;31m     \u001b[0mcreate_relevant_vocab\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpertinence_thres_relevant\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmin_count_relevant\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmerged\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m     \u001b[0mdata_tweets\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mimport_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cleaned_test_bitri=\"\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbitri\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Code/EPFL (2013-2015)/Python/Machine Learning/ML_project2/Data_exploration/ProcessTweets.py\u001b[0m in \u001b[0;36mcreate_relevant_vocab\u001b[0;34m(pertinence_thres, min_count, dataframe)\u001b[0m\n\u001b[1;32m     83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m     \u001b[0mwrite_relevance_to_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrelevant\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"relevant_vocab_pert=\"\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpertinence_thres\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m\"_count=\"\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmin_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m     \u001b[0;31m#relevant.to_csv(sep=\"\\t\", path_or_buf=(\"relevant_vocab_pert=\"+str(pertinence_thres)+\"_count=\"+str(min_count)), header=True, index=False)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'write_relevance_to_file' is not defined"
",
      "4290not sucessful\n",
      "4273not sucessful\n",
      "4216not sucessful\n",
      "4107not sucessful\n",
      "4103not sucessful\n",
      "4090not sucessful\n",
      "4074not sucessful\n",
      "4039not sucessful\n",
      "3968not sucessful\n",
      "3920not sucessful\n",
      "3895not sucessful\n",
      "3868not sucessful\n",
      "3856not sucessful\n",
      "3687not sucessful\n",
      "3636not sucessful\n",
      "3627not sucessful\n",
      "3617not sucessful\n",
      "3615not sucessful\n",
      "3614not sucessful\n",
      "3541not sucessful\n",
      "3484not sucessful\n",
      "3456not sucessful\n",
      "3431not sucessful\n",
      "3400not sucessful\n",
      "3399not sucessful\n",
      "3391not sucessful\n",
      "3322not sucessful\n",
      "3279not sucessful\n",
      "3153not sucessful\n",
      "3077not sucessful\n",
      "3048not sucessful\n",
      "3023not sucessful\n",
      "3014not sucessful\n",
      "3002not sucessful\n",
      "2956not sucessful\n",
      "2950not sucessful\n",
      "2936not sucessful\n",
      "2891not sucessful\n",
      "2849not sucessful\n",
      "2789not sucessful\n",
      "2773not sucessful\n",
      "2757not sucessful\n",
      "2690not sucessful\n",
      "2689not sucessful\n",
      "2682not sucessful\n",
      "2679not sucessful\n",
      "2665not sucessful\n",
      "2645not sucessful\n",
      "2626not sucessful\n",
      "2595not sucessful\n",
      "2575not sucessful\n",
      "2553not sucessful\n",
      "2550not sucessful\n",
      "2545not sucessful\n",
      "2543not sucessful\n",
      "2533not sucessful\n",
      "2508not sucessful\n",
      "2493not sucessful\n",
      "2471not sucessful\n",
      "2421not sucessful\n",
      "2354not sucessful\n",
      "2346not sucessful\n",
      "2312not sucessful\n",
      "2311not sucessful\n",
      "2303not sucessful\n",
      "2300not sucessful\n",
      "2296not sucessful\n",
      "2274not sucessful\n",
      "2269not sucessful\n",
      "2265not sucessful\n",
      "2255not sucessful\n",
      "2254not sucessful\n",
      "2237not sucessful\n",
      "2236not sucessful\n",
      "2234not sucessful\n",
      "2229not sucessful\n",
      "2220not sucessful\n",
      "2217not sucessful\n",
      "2212not sucessful\n",
      "2205not sucessful\n",
      "2167not sucessful\n",
      "2162not sucessful\n",
      "2144not sucessful\n",
      "2137not sucessful\n",
      "2135not sucessful\n",
      "2132not sucessful\n",
      "2130not sucessful\n",
      "2113not sucessful\n",
      "2106not sucessful\n",
      "2089not sucessful\n",
      "2085not sucessful\n",
      "2078not sucessful\n",
      "2073not sucessful\n",
      "2059not sucessful\n",
      "2057not sucessful\n",
      "2014not sucessful\n",
      "2010not sucessful\n",
      "2002not sucessful\n",
      "1976not sucessful\n",
      "1967not sucessful\n",
      "1955not sucessful\n",
      "1950not sucessful\n",
      "1948not sucessful\n",
      "1946not sucessful\n",
      "1943not sucessful\n",
      "1942not sucessful\n",
      "1938not sucessful\n",
      "1933not sucessful\n",
      "1926not sucessful\n",
      "1923not sucessful\n",
      "1865not sucessful\n",
      "1836not sucessful\n",
      "1835not sucessful\n",
      "1825not sucessful\n",
      "1824not sucessful\n",
      "1823not sucessful\n",
      "1812not sucessful\n",
      "1792not sucessful\n",
      "1765not sucessful\n",
      "1764not sucessful\n",
      "1763not sucessful\n",
      "1760not sucessful\n",
      "1751not sucessful\n",
      "1738not sucessful\n",
      "1727not sucessful\n",
      "1726not sucessful\n",
      "1725not sucessful\n",
      "1721not sucessful\n",
      "1719not sucessful\n",
      "1717not sucessful\n",
      "1715not sucessful\n",
      "1708not sucessful\n",
      "1705not sucessful\n",
      "1703not sucessful\n",
      "1701not sucessful\n",
      "1700not sucessful\n",
      "1699not sucessful\n",
      "1698not sucessful\n",
      "1697not sucessful\n",
      "1696not sucessful\n",
      "1695not sucessful\n",
      "1694not sucessful\n",
      "1693not sucessful\n",
      "1691not sucessful\n",
      "1683not sucessful\n",
      "1681not sucessful\n",
      "1680not sucessful\n",
      "1677not sucessful\n",
      "1672not sucessful\n",
      "1671not sucessful\n",
      "1666not sucessful\n",
      "1662not sucessful\n",
      "1645not sucessful\n",
      "1641not sucessful\n",
      "1640not sucessful\n",
      "1637not sucessful\n",
      "1636not sucessful\n",
      "1635not sucessful\n",
      "1633not sucessful\n",
      "1632not sucessful\n",
      "1631not sucessful\n",
      "1629not sucessful\n",
      "1627not sucessful\n",
      "1626not sucessful\n",
      "1625not sucessful\n",
      "1624not sucessful\n",
      "1621not sucessful\n",
      "1620not sucessful\n",
      "1615not sucessful\n",
      "1608not sucessful\n",
      "1605not sucessful\n",
      "1599not sucessful\n",
      "1598not sucessful\n",
      "1594not sucessful\n",
      "1592not sucessful\n",
      "1587not sucessful\n",
      "1585not sucessful\n",
      "1581not sucessful\n",
      "1577not sucessful\n",
      "1573not sucessful\n",
      "1572not sucessful\n",
      "1569not sucessful\n",
      "1562not sucessful\n",
      "1561not sucessful\n",
      "1556not sucessful\n",
      "1541not sucessful\n",
      "1540not sucessful\n",
      "1539not sucessful\n",
      "1514not sucessful\n",
      "1509not sucessful\n",
      "1495not sucessful\n",
      "1493not sucessful\n",
      "1481not sucessful\n",
      "1479not sucessful\n",
      "1478not sucessful\n",
      "1471not sucessful\n",
      "1470not sucessful\n",
      "1467not sucessful\n",
      "1459not sucessful\n",
      "1458not sucessful\n",
      "1455not sucessful\n",
      "1449not sucessful\n",
      "1448not sucessful\n",
      "1446not sucessful\n",
      "1432not sucessful\n",
      "1427not sucessful\n",
      "1420not sucessful\n",
      "1410not sucessful\n",
      "1409not sucessful\n",
      "1407not sucessful\n",
      "1406not sucessful\n",
      "1404not sucessful\n",
      "1402not sucessful\n",
      "1394not sucessful\n",
      "1393not sucessful\n",
      "1383not sucessful\n",
      "1371not sucessful\n",
      "1355not sucessful\n",
      "1344not sucessful\n",
      "1341not sucessful\n",
      "1338not sucessful\n",
      "1335not sucessful\n",
      "1334not sucessful\n",
      "1332not sucessful\n",
      "1330not sucessful\n",
      "1325not sucessful\n",
      "1323not sucessful\n",
      "1322not sucessful\n",
      "1317not sucessful\n",
      "1316not sucessful\n",
      "1309not sucessful\n",
      "1306not sucessful\n",
      "1303not sucessful\n",
      "1300not sucessful\n",
      "1299not sucessful\n",
      "1298not sucessful\n",
      "1296not sucessful\n",
      "1294not sucessful\n",
      "1289not sucessful\n",
      "1287not sucessful\n",
      "1285not sucessful\n",
      "1284not sucessful\n",
      "1283not sucessful\n",
      "1275not sucessful\n",
      "1273not sucessful\n",
      "1250not sucessful\n",
      "1241not sucessful\n",
      "1238not sucessful\n",
      "1232not sucessful\n",
      "1227not sucessful\n",
      "1224not sucessful\n",
      "1222not sucessful\n",
      "1221not sucessful\n",
      "1215not sucessful\n",
      "1214not sucessful\n",
      "1212not sucessful\n",
      "1210not sucessful\n",
      "1209not sucessful\n",
      "1207not sucessful\n",
      "1205not sucessful\n",
      "1201not sucessful\n",
      "1199not sucessful\n",
      "1197not sucessful\n",
      "1195not sucessful\n",
      "1191not sucessful\n",
      "1189not sucessful\n",
      "1183not sucessful\n",
      "1177not sucessful\n",
      "1169not sucessful\n",
      "1168not sucessful\n",
      "1163not sucessful\n",
      "1157not sucessful\n",
      "1156not sucessful\n",
      "1154not sucessful\n",
      "1153not sucessful\n",
      "1152not sucessful\n",
      "1148not sucessful\n",
      "1146not sucessful\n",
      "1145not sucessful\n",
      "1144not sucessful\n",
      "1140not sucessful\n",
      "1133not sucessful\n",
      "1132not sucessful\n",
      "1131not sucessful\n",
      "1130not sucessful\n",
      "1129not sucessful\n",
      "1128not sucessful\n",
      "1127not sucessful\n",
      "1126not sucessful\n",
      "1121not sucessful\n",
      "1120not sucessful\n",
      "1118not sucessful\n",
      "1116not sucessful\n",
      "1115not sucessful\n",
      "1114not sucessful\n",
      "1101not sucessful\n",
      "1099not sucessful\n",
      "1096not sucessful\n",
      "1095not sucessful\n",
      "1094not sucessful\n",
      "1092not sucessful\n",
      "1091not sucessful\n",
      "1090not sucessful\n",
      "1089not sucessful\n",
      "1087not sucessful\n",
      "1085not sucessful\n",
      "1084not sucessful\n",
      "1078not sucessful\n",
      "1076not sucessful\n",
      "1074not sucessful\n",
      "1070not sucessful\n",
      "1065not sucessful\n",
      "1064not sucessful\n",
      "1061not sucessful\n",
      "1056not sucessful\n",
      "1054not sucessful\n",
      "1047not sucessful\n",
      "1045not sucessful\n",
      "1044not sucessful\n",
      "1042not sucessful\n",
      "1030not sucessful\n",
      "1026not sucessful\n",
      "1025not sucessful\n",
      "1012not sucessful\n",
      "1010not sucessful\n",
      "1007not sucessful\n",
      "1005not sucessful\n",
      "1003not sucessful\n",
      "1002not sucessful\n",
      "996not sucessful\n",
      "995not sucessful\n",
      "994not sucessful\n",
      "993not sucessful\n",
      "991not sucessful\n",
      "990not sucessful\n",
      "989not sucessful\n",
      "987not sucessful\n",
      "985not sucessful\n",
      "983not sucessful\n",
      "981not sucessful\n",
      "980not sucessful\n",
      "979not sucessful\n",
      "977not sucessful\n",
      "974not sucessful\n",
      "973not sucessful\n",
      "970not sucessful\n",
      "968not sucessful\n",
      "967not sucessful\n",
      "966not sucessful\n",
      "965not sucessful\n",
      "959not sucessful\n",
      "958not sucessful\n",
      "957not sucessful\n",
      "955not sucessful\n",
      "951not sucessful\n",
      "950not sucessful\n",
      "948not sucessful\n",
      "947not sucessful\n",
      "946not sucessful\n",
      "944not sucessful\n",
      "939not sucessful\n",
      "933not sucessful\n",
      "931not sucessful\n",
      "928not sucessful\n",
      "921not sucessful\n",
      "919not sucessful\n",
      "918not sucessful\n",
      "914not sucessful\n",
      "913not sucessful\n",
      "910not sucessful\n",
      "907not sucessful\n",
      "904not sucessful\n",
      "903not sucessful\n",
      "902not sucessful\n",
      "901not sucessful\n",
      "900not sucessful\n",
      "896not sucessful\n",
      "894not sucessful\n",
      "892not sucessful\n",
      "891not sucessful\n",
      "889not sucessful\n",
      "887not sucessful\n",
      "886not sucessful\n",
      "885not sucessful\n",
      "883not sucessful\n",
      "880not sucessful\n",
      "878not sucessful\n",
      "876not sucessful\n",
      "875not sucessful\n",
      "874not sucessful\n",
      "873not sucessful\n",
      "872not sucessful\n",
      "871not sucessful\n",
      "870not sucessful\n",
      "869not sucessful\n",
      "864not sucessful\n",
      "863not sucessful\n",
      "862not sucessful\n",
      "861not sucessful\n",
      "858not sucessful\n",
      "857not sucessful\n",
      "856not sucessful\n",
      "855not sucessful\n",
      "852not sucessful\n",
      "851not sucessful\n",
      "850not sucessful\n",
      "849not sucessful\n",
      "845not sucessful\n",
      "844not sucessful\n",
      "842not sucessful\n",
      "841not sucessful\n",
      "840not sucessful\n",
      "839not sucessful\n",
      "837not sucessful\n",
      "835not sucessful\n",
      "834not sucessful\n",
      "833not sucessful\n",
      "831not sucessful\n",
      "830not sucessful\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "828not sucessful\n",
      "827not sucessful\n",
      "826not sucessful\n",
      "824not sucessful\n",
      "823not sucessful\n",
      "822not sucessful\n",
      "821not sucessful\n",
      "819not sucessful\n",
      "818not sucessful\n",
      "816not sucessful\n",
      "815not sucessful\n",
      "814not sucessful\n",
      "806not sucessful\n",
      "805not sucessful\n",
      "804not sucessful\n",
      "801not sucessful\n",
      "800not sucessful\n",
      "797not sucessful\n",
      "794not sucessful\n",
      "792not sucessful\n",
      "791not sucessful\n",
      "790not sucessful\n",
      "788not sucessful\n",
      "787not sucessful\n",
      "786not sucessful\n",
      "785not sucessful\n",
      "784not sucessful\n",
      "779not sucessful\n",
      "778not sucessful\n",
      "777not sucessful\n",
      "773not sucessful\n",
      "772not sucessful\n",
      "770not sucessful\n",
      "769not sucessful\n",
      "768not sucessful\n",
      "765not sucessful\n",
      "764not sucessful\n",
      "763not sucessful\n",
      "762not sucessful\n",
      "761not sucessful\n",
      "760not sucessful\n",
      "759not sucessful\n",
      "758not sucessful\n",
      "755not sucessful\n",
      "754not sucessful\n",
      "753not sucessful\n",
      "752not sucessful\n",
      "751not sucessful\n",
      "750not sucessful\n",
      "748not sucessful\n",
      "747not sucessful\n",
      "746not sucessful\n",
      "745not sucessful\n",
      "744not sucessful\n",
      "742not sucessful\n",
      "741not sucessful\n",
      "740not sucessful\n",
      "739not sucessful\n",
      "738not sucessful\n",
      "736not sucessful\n",
      "734not sucessful\n",
      "733not sucessful\n",
      "732not sucessful\n",
      "731not sucessful\n",
      "730not sucessful\n",
      "728not sucessful\n",
      "727not sucessful\n",
      "725not sucessful\n",
      "724not sucessful\n",
      "723not sucessful\n",
      "719not sucessful\n",
      "716not sucessful\n",
      "715not sucessful\n",
      "714not sucessful\n",
      "713not sucessful\n",
      "712not sucessful\n",
      "708not sucessful\n",
      "706not sucessful\n",
      "705not sucessful\n",
      "703not sucessful\n",
      "701not sucessful\n",
      "700not sucessful\n",
      "699not sucessful\n",
      "696not sucessful\n",
      "695not sucessful\n",
      "693not sucessful\n",
      "691not sucessful\n",
      "689not sucessful\n",
      "688not sucessful\n",
      "687not sucessful\n",
      "685not sucessful\n",
      "682not sucessful\n",
      "681not sucessful\n",
      "680not sucessful\n",
      "679not sucessful\n",
      "677not sucessful\n",
      "676not sucessful\n",
      "674not sucessful\n",
      "672not sucessful\n",
      "671not sucessful\n",
      "670not sucessful\n",
      "667not sucessful\n",
      "666not sucessful\n",
      "664not sucessful\n",
      "661not sucessful\n",
      "659not sucessful\n",
      "658not sucessful\n",
      "657not sucessful\n",
      "656not sucessful\n",
      "655not sucessful\n",
      "654not sucessful\n",
      "653not sucessful\n",
      "652not sucessful\n",
      "651not sucessful\n",
      "650not sucessful\n",
      "649not sucessful\n",
      "648not sucessful\n",
      "647not sucessful\n",
      "646not sucessful\n",
      "645not sucessful\n",
      "643not sucessful\n",
      "642not sucessful\n",
      "641not sucessful\n",
      "640not sucessful\n",
      "638not sucessful\n",
      "637not sucessful\n",
      "635not sucessful\n",
      "634not sucessful\n",
      "633not sucessful\n",
      "632not sucessful\n",
      "631not sucessful\n",
      "630not sucessful\n",
      "629not sucessful\n",
      "628not sucessful\n",
      "627not sucessful\n",
      "626not sucessful\n",
      "625not sucessful\n",
      "623not sucessful\n",
      "622not sucessful\n",
      "619not sucessful\n",
      "618not sucessful\n",
      "617not sucessful\n",
      "616not sucessful\n",
      "615not sucessful\n",
      "614not sucessful\n",
      "613not sucessful\n",
      "611not sucessful\n",
      "610not sucessful\n",
      "608not sucessful\n",
      "607not sucessful\n",
      "605not sucessful\n",
      "602not sucessful\n",
      "601not sucessful\n",
      "600not sucessful\n",
      "599not sucessful\n",
      "598not sucessful\n",
      "596not sucessful\n",
      "595not sucessful\n",
      "594not sucessful\n",
      "593not sucessful\n",
      "592not sucessful\n",
      "591not sucessful\n",
      "590not sucessful\n",
      "589not sucessful\n",
      "588not sucessful\n",
      "587not sucessful\n",
      "585not sucessful\n",
      "584not sucessful\n",
      "583not sucessful\n",
      "582not sucessful\n",
      "581not sucessful\n",
      "580not sucessful\n",
      "578not sucessful\n",
      "577not sucessful\n",
      "576not sucessful\n",
      "575not sucessful\n",
      "574not sucessful\n",
      "572not sucessful\n",
      "571not sucessful\n",
      "570not sucessful\n",
      "568not sucessful\n",
      "566not sucessful\n",
      "565not sucessful\n",
      "564not sucessful\n",
      "563not sucessful\n",
      "562not sucessful\n",
      "560not sucessful\n",
      "558not sucessful\n",
      "557not sucessful\n",
      "556not sucessful\n",
      "555not sucessful\n",
      "554not sucessful\n",
      "553not sucessful\n",
      "552not sucessful\n",
      "551not sucessful\n",
      "550not sucessful\n",
      "549not sucessful\n",
      "548not sucessful\n",
      "547not sucessful\n",
      "546not sucessful\n",
      "545not sucessful\n",
      "544not sucessful\n",
      "543not sucessful\n",
      "542not sucessful\n",
      "541not sucessful\n",
      "540not sucessful\n",
      "539not sucessful\n",
      "538not sucessful\n",
      "537not sucessful\n",
      "536not sucessful\n",
      "535not sucessful\n",
      "534not sucessful\n",
      "533not sucessful\n",
      "532not sucessful\n",
      "531not sucessful\n",
      "530not sucessful\n",
      "529not sucessful\n",
      "527not sucessful\n",
      "526not sucessful\n",
      "525not sucessful\n",
      "524not sucessful\n",
      "523not sucessful\n",
      "521not sucessful\n",
      "520not sucessful\n",
      "519not sucessful\n",
      "517not sucessful\n",
      "515not sucessful\n",
      "514not sucessful\n",
      "513not sucessful\n",
      "512not sucessful\n",
      "510not sucessful\n",
      "509not sucessful\n",
      "508not sucessful\n",
      "507not sucessful\n",
      "506not sucessful\n",
      "505not sucessful\n",
      "504not sucessful\n",
      "503not sucessful\n",
      "502not sucessful\n",
      "501not sucessful\n",
      "500not sucessful\n",
      "499not sucessful\n",
      "498not sucessful\n",
      "497not sucessful\n",
      "496not sucessful\n",
      "495not sucessful\n",
      "494not sucessful\n",
      "493not sucessful\n",
      "491not sucessful\n",
      "490not sucessful\n",
      "489not sucessful\n",
      "488not sucessful\n",
      "487not sucessful\n",
      "486not sucessful\n",
      "485not sucessful\n",
      "484not sucessful\n",
      "483not sucessful\n",
      "482not sucessful\n",
      "480not sucessful\n",
      "479not sucessful\n",
      "478not sucessful\n",
      "477not sucessful\n",
      "476not sucessful\n",
      "475not sucessful\n",
      "473not sucessful\n",
      "472not sucessful\n",
      "471not sucessful\n",
      "470not sucessful\n",
      "469not sucessful\n",
      "468not sucessful\n",
      "467not sucessful\n",
      "466not sucessful\n",
      "465not sucessful\n",
      "464not sucessful\n",
      "463not sucessful\n",
      "462not sucessful\n",
      "461not sucessful\n",
      "460not sucessful\n",
      "459not sucessful\n",
      "458not sucessful\n",
      "457not sucessful\n",
      "456not sucessful\n",
      "455not sucessful\n",
      "454not sucessful\n",
      "453not sucessful\n",
      "452not sucessful\n",
      "450not sucessful\n",
      "449not sucessful\n",
      "448not sucessful\n",
      "447not sucessful\n",
      "446not sucessful\n",
      "445not sucessful\n",
      "444not sucessful\n",
      "443not sucessful\n",
      "442not sucessful\n",
      "441not sucessful\n",
      "440not sucessful\n",
      "439not sucessful\n",
      "438not sucessful\n",
      "437not sucessful\n",
      "436not sucessful\n",
      "435not sucessful\n",
      "434not sucessful\n",
      "433not sucessful\n",
      "432not sucessful\n",
      "431not sucessful\n",
      "430not sucessful\n",
      "428not sucessful\n",
      "427not sucessful\n",
      "425not sucessful\n",
      "424not sucessful\n",
      "423not sucessful\n",
      "422not sucessful\n",
      "421not sucessful\n",
      "420not sucessful\n",
      "419not sucessful\n",
      "418not sucessful\n",
      "417not sucessful\n",
      "416not sucessful\n",
      "415not sucessful\n",
      "414not sucessful\n",
      "413not sucessful\n",
      "412not sucessful\n",
      "411not sucessful\n",
      "410not sucessful\n",
      "409not sucessful\n",
      "408not sucessful\n",
      "407not sucessful\n",
      "406not sucessful\n",
      "405not sucessful\n",
      "404not sucessful\n",
      "403not sucessful\n",
      "402not sucessful\n",
      "401not sucessful\n",
      "400not sucessful\n",
      "399not sucessful\n",
      "398not sucessful\n",
      "397not sucessful\n",
      "396not sucessful\n",
      "395not sucessful\n",
      "394not sucessful\n",
      "393not sucessful\n",
      "392not sucessful\n",
      "391not sucessful\n",
      "390not sucessful\n",
      "389not sucessful\n",
      "388not sucessful\n",
      "387not sucessful\n",
      "385not sucessful\n",
      "384not sucessful\n",
      "383not sucessful\n",
      "382not sucessful\n",
      "381not sucessful\n",
      "380not sucessful\n",
      "379not sucessful\n",
      "378not sucessful\n",
      "377not sucessful\n",
      "376not sucessful\n",
      "375not sucessful\n",
      "374not sucessful\n",
      "373not sucessful\n",
      "372not sucessful\n",
      "371not sucessful\n",
      "370not sucessful\n",
      "369not sucessful\n",
      "368not sucessful\n",
      "367not sucessful\n",
      "366not sucessful\n",
      "365not sucessful\n",
      "364not sucessful\n",
      "363not sucessful\n",
      "362not sucessful\n",
      "361not sucessful\n",
      "360not sucessful\n",
      "359not sucessful\n",
      "358not sucessful\n",
      "357not sucessful\n",
      "356not sucessful\n",
      "355not sucessful\n",
      "354not sucessful\n",
      "353not sucessful\n",
      "352not sucessful\n",
      "351not sucessful\n",
      "350not sucessful\n",
      "349not sucessful\n",
      "348not sucessful\n",
      "347not sucessful\n",
      "346not sucessful\n",
      "345not sucessful\n",
      "344not sucessful\n",
      "343not sucessful\n",
      "342not sucessful\n",
      "341not sucessful\n",
      "340not sucessful\n",
      "339not sucessful\n",
      "338not sucessful\n",
      "337not sucessful\n",
      "336not sucessful\n",
      "335not sucessful\n",
      "334not sucessful\n",
      "333not sucessful\n",
      "332not sucessful\n",
      "331not sucessful\n",
      "330not sucessful\n",
      "329not sucessful\n",
      "328not sucessful\n",
      "327not sucessful\n",
      "326not sucessful\n",
      "325not sucessful\n",
      "324not sucessful\n",
      "323not sucessful\n",
      "322not sucessful\n",
      "321not sucessful\n",
      "320not sucessful\n",
      "319not sucessful\n",
      "318not sucessful\n",
      "317not sucessful\n",
      "316not sucessful\n",
      "315not sucessful\n",
      "314not sucessful\n",
      "313not sucessful\n",
      "312not sucessful\n",
      "311not sucessful\n",
      "310not sucessful\n",
      "309not sucessful\n",
      "308not sucessful\n",
      "307not sucessful\n",
      "306not sucessful\n",
      "305not sucessful\n",
      "304not sucessful\n",
      "303not sucessful\n",
      "302not sucessful\n",
      "301not sucessful\n",
      "300not sucessful\n",
      "299not sucessful\n",
      "298not sucessful\n",
      "297not sucessful\n",
      "296not sucessful\n",
      "295not sucessful\n",
      "294not sucessful\n",
      "293not sucessful\n",
      "292not sucessful\n",
      "291not sucessful\n",
      "290not sucessful\n",
      "289not sucessful\n",
      "288not sucessful\n",
      "287not sucessful\n",
      "286not sucessful\n",
      "285not sucessful\n",
      "284not sucessful\n",
      "283not sucessful\n",
      "282not sucessful\n",
      "281not sucessful\n",
      "280not sucessful\n",
      "279not sucessful\n",
      "278not sucessful\n",
      "277not sucessful\n",
      "276not sucessful\n",
      "275not sucessful\n",
      "274not sucessful\n",
      "273not sucessful\n",
      "272not sucessful\n",
      "271not sucessful\n",
      "270not sucessful\n",
      "269not sucessful\n",
      "268not sucessful\n",
      "267not sucessful\n",
      "266not sucessful\n",
      "265not sucessful\n",
      "264not sucessful\n",
      "263not sucessful\n",
      "262not sucessful\n",
      "261not sucessful\n",
      "260not sucessful\n",
      "259not sucessful\n",
      "258not sucessful\n",
      "257not sucessful\n",
      "256not sucessful\n",
      "255not sucessful\n",
      "254not sucessful\n",
      "253not sucessful\n",
      "252not sucessful\n",
      "251not sucessful\n",
      "250not sucessful\n",
      "249not sucessful\n",
      "248not sucessful\n",
      "247not sucessful\n",
      "246not sucessful\n",
      "245not sucessful\n",
      "244not sucessful\n",
      "243not sucessful\n",
      "242not sucessful\n",
      "241not sucessful\n",
      "240not sucessful\n",
      "239not sucessful\n",
      "238not sucessful\n",
      "237not sucessful\n",
      "236not sucessful\n",
      "235not sucessful\n",
      "234not sucessful\n",
      "233not sucessful\n",
      "232not sucessful\n",
      "231not sucessful\n",
      "230not sucessful\n",
      "229not sucessful\n",
      "228not sucessful\n",
      "227not sucessful\n",
      "226not sucessful\n",
      "225not sucessful\n",
      "224not sucessful\n",
      "223not sucessful\n",
      "222not sucessful\n",
      "221not sucessful\n",
      "220not sucessful\n",
      "219not sucessful\n",
      "218not sucessful\n",
      "217not sucessful\n",
      "216not sucessful\n",
      "215not sucessful\n",
      "214not sucessful\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "213not sucessful\n",
      "212not sucessful\n",
      "211not sucessful\n",
      "210not sucessful\n",
      "209not sucessful\n",
      "208not sucessful\n",
      "207not sucessful\n",
      "206not sucessful\n",
      "205not sucessful\n",
      "204not sucessful\n",
      "203not sucessful\n",
      "202not sucessful\n",
      "201not sucessful\n",
      "200not sucessful\n",
      "199not sucessful\n",
      "198not sucessful\n",
      "197not sucessful\n",
      "196not sucessful\n",
      "195not sucessful\n",
      "194not sucessful\n",
      "193not sucessful\n",
      "192not sucessful\n",
      "191not sucessful\n",
      "190not sucessful\n",
      "189not sucessful\n",
      "188not sucessful\n",
      "187not sucessful\n",
      "186not sucessful\n",
      "185not sucessful\n",
      "184not sucessful\n",
      "183not sucessful\n",
      "182not sucessful\n",
      "181not sucessful\n",
      "180not sucessful\n",
      "179not sucessful\n",
      "178not sucessful\n",
      "177not sucessful\n",
      "176not sucessful\n",
      "175not sucessful\n",
      "174not sucessful\n",
      "173not sucessful\n",
      "172not sucessful\n",
      "171not sucessful\n",
      "170not sucessful\n",
      "169not sucessful\n",
      "168not sucessful\n",
      "167not sucessful\n",
      "166not sucessful\n",
      "165not sucessful\n",
      "164not sucessful\n",
      "163not sucessful\n",
      "162not sucessful\n",
      "161not sucessful\n",
      "160not sucessful\n",
      "159not sucessful\n",
      "158not sucessful\n",
      "157not sucessful\n",
      "156not sucessful\n",
      "155not sucessful\n",
      "154not sucessful\n",
      "153not sucessful\n",
      "152not sucessful\n",
      "151not sucessful\n",
      "150not sucessful\n",
      "149not sucessful\n",
      "148not sucessful\n",
      "147not sucessful\n",
      "146not sucessful\n",
      "145not sucessful\n",
      "144not sucessful\n",
      "143not sucessful\n",
      "142not sucessful\n",
      "141not sucessful\n",
      "140not sucessful\n",
      "139not sucessful\n",
      "138not sucessful\n",
      "137not sucessful\n",
      "136not sucessful\n",
      "135not sucessful\n",
      "134not sucessful\n",
      "133not sucessful\n",
      "132not sucessful\n",
      "131not sucessful\n",
      "130not sucessful\n",
      "129not sucessful\n",
      "128not sucessful\n",
      "127not sucessful\n",
      "126not sucessful\n",
      "125not sucessful\n",
      "124not sucessful\n",
      "123not sucessful\n",
      "122not sucessful\n",
      "121not sucessful\n",
      "120not sucessful\n",
      "119not sucessful\n",
      "118not sucessful\n",
      "117not sucessful\n",
      "116not sucessful\n",
      "115not sucessful\n",
      "114not sucessful\n",
      "113not sucessful\n",
      "112not sucessful\n",
      "111not sucessful\n",
      "110not sucessful\n",
      "109not sucessful\n",
      "108not sucessful\n",
      "107not sucessful\n",
      "106not sucessful\n",
      "105not sucessful\n",
      "104not sucessful\n",
      "103not sucessful\n",
      "102not sucessful\n",
      "101not sucessful\n",
      "100not sucessful\n",
      "99not sucessful\n",
      "98not sucessful\n",
      "97not sucessful\n",
      "96not sucessful\n",
      "95not sucessful\n",
      "94not sucessful\n",
      "93not sucessful\n",
      "92not sucessful\n",
      "91not sucessful\n",
      "90not sucessful\n",
      "89not sucessful\n",
      "88not sucessful\n",
      "87not sucessful\n",
      "86not sucessful\n",
      "85not sucessful\n",
      "84not sucessful\n",
      "83not sucessful\n",
      "82not sucessful\n",
      "81not sucessful\n",
      "80not sucessful\n",
      "79not sucessful\n",
      "78not sucessful\n",
      "77not sucessful\n",
      "76not sucessful\n",
      "75not sucessful\n",
      "74not sucessful\n",
      "73not sucessful\n",
      "72not sucessful\n",
      "71not sucessful\n",
      "70not sucessful\n",
      "69not sucessful\n",
      "68not sucessful\n",
      "67not sucessful\n",
      "66not sucessful\n",
      "65not sucessful\n",
      "64not sucessful\n",
      "63not sucessful\n",
      "62not sucessful\n",
      "61not sucessful\n",
      "60not sucessful\n",
      "59not sucessful\n",
      "58not sucessful\n",
      "57not sucessful\n",
      "56not sucessful\n",
      "55not sucessful\n",
      "54not sucessful\n",
      "53not sucessful\n",
      "52not sucessful\n",
      "51not sucessful\n",
      "50not sucessful\n",
      "49not sucessful\n",
      "48not sucessful\n",
      "47not sucessful\n",
      "46not sucessful\n",
      "45not sucessful\n",
      "44not sucessful\n",
      "43not sucessful\n",
      "42not sucessful\n",
      "41not sucessful\n",
      "40not sucessful\n",
      "39not sucessful\n",
      "38not sucessful\n",
      "37not sucessful\n",
      "36not sucessful\n",
      "35not sucessful\n",
      "34not sucessful\n",
      "33not sucessful\n",
      "32not sucessful\n",
      "31not sucessful\n",
      "30not sucessful\n",
      "29not sucessful\n",
      "28not sucessful\n",
      "27not sucessful\n",
      "26not sucessful\n",
      "25not sucessful\n",
      "24not sucessful\n",
      "23not sucessful\n",
      "22not sucessful\n",
      "21not sucessful\n",
      "20not sucessful\n",
      "19not sucessful\n",
      "18not sucessful\n",
      "17not sucessful\n",
      "16not sucessful\n",
      "15not sucessful\n",
      "14not sucessful\n",
      "13not sucessful\n",
      "12not sucessful\n",
      "11not sucessful\n",
      "10not sucessful\n",
      "9not sucessful\n",
      "8not sucessful\n",
      "7not sucessful\n",
      "6not sucessful\n",
      "5not sucessful\n"
>>>>>>> df50270467d802eaa9e2fe1c86cce6e33551bf1e
     ]
    }
   ],
   "source": [
    "data_tweets = import_(\"relevant_vocab_pert=0.3_count=300\")\n",
    "\n",
    "characteristic_words(data_tweets=data_tweets, merged=merged, is_full=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Recapitulative Function\n",
    "This function does the work presented above, we created it for clarity purposes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean raw tweets\n",
    "def setup(path_pos, path_neg, path_test, is_full,  pertinence_thres_relevant, min_count_relevant, cut_threshold=5, bitri = True):\n",
    "    #path pos, path_neg, path_data : paths of the raw tweets\n",
    "    #is_full : True iff we clean \n",
    "    \n",
    "    #export cleaned tweets and their vocabulary\n",
    "    cleaned_tweets = clean_tweets(path_pos, path_neg, path_test, is_full, cut_threshold)\n",
    "    \n",
    "    if is_full:\n",
    "        full_string = \"_full\"\n",
    "    else :\n",
    "        full_string = \"\"\n",
    "    \n",
    "    \n",
    "    #load dataframes    \n",
    "    pos_df = build_df(\"cleaned_vocab_pos\"+str(full_string)+\"_bitri=\"+str(bitri), bitri)\n",
    "    neg_df = build_df(\"cleaned_vocab_neg\"+str(full_string)+\"_bitri=\"+str(bitri), bitri)\n",
    "   \n",
    "    #Merge dataframes\n",
    "    merged = merging(neg_df, pos_df, False)\n",
    "    \n",
    "    #create relevant vocab\n",
    "    create_relevant_vocab(pertinence_thres_relevant, min_count_relevant, merged)\n",
    "    \n",
    "    data_tweets = import_(\"cleaned_test_bitri=\"+str(bitri))\n",
    "    \n",
    "    #create characteristic_words\n",
    "    characteristic_words(data_tweets, merged, is_full)\n",
    "    \n",
    "    #build the global vocabulary\n",
    "    build_global_vocab(is_full, bitri, cut_threshold)\n",
    "    \n",
    "    \n",
    "setup(\"train_pos.txt\", \"train_neg.txt\" , \"test_data.txt\", is_full=False, pertinence_thres_relevant=0.3, min_count_relevant=250)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}