generate_tree_KERMIT-sentence.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using KERMIT - Building dataset -\n",
    "\n",
    "This first notebook explains how to construct the syntactic input for KERMIT_system via KERMIT encoder.\n",
    "\n",
    "There are also links from where to download used datasets or you can use a dataset of your choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Packages\n",
    "Before starting, it is essential to have the following requirements:\n",
    "- stanford-corenlp-full-2018-10-05 : which will be used to build the trees in parenthetical form.\n",
    "- KERMIT : that it is obvious to have it but we specify it anyway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Install stanfordcorenlp \n",
    "#!pip install stanfordcorenlp\n",
    "#!wget http://nlp.stanford.edu/software/stanford-corenlp-full-2018-10-05.zip\n",
    "#!unzip stanford-corenlp-full-2018-10-05.zip && y\n",
    "\n",
    "#Install KERMIT\n",
    "#!git clone https://github.com/ART-Group-it/kerMIT\n",
    "#!pip install ./kerMIT/kerMIT\n",
    "#!apt install openjdk-8-jdk --assume-yes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Statments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, pickle, ast, os\n",
    "from tqdm import tqdm\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#script for reading/writing trees\n",
    "from scripts.script import readP, writeTree\n",
    "#script for build DTK\n",
    "from scripts.script import createDTK\n",
    "#script for parse sentences\n",
    "from scripts.script import parse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-01-24 13:13:22--  http://160.80.97.56:8006/dtk_trees_multiNLI_dev_sentence_2_tot.pkl\n",
      "Connecting to 160.80.97.56:8006... failed: Connection refused.\n"
     ]
    }
   ],
   "source": [
    "!wget http://160.80.97.56:8006/dtk_trees_multiNLI_dev_sentence_2_tot.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download  Dataset\n",
    "\n",
    "The datasets used in our work have been randomly sampled as follows:\n",
    "\n",
    "* *Note: in this tutorial we use ag_news (train and test set) but remember that the user can choose others as he prefers.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "MULTI_NLI = \"../../samir/MultiNLI_with_REL_final.csv\"\n",
    "HANS = \"../../samir/HANS_with_REL_final.csv\"\n",
    "\n",
    "data_train = pd.read_csv(MULTI_NLI) #nrows=75000 skiprows=75001\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30000"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>gold_label</th>\n",
       "      <th>sentence1_binary_parse</th>\n",
       "      <th>sentence2_binary_parse</th>\n",
       "      <th>sentence1_parse</th>\n",
       "      <th>sentence2_parse</th>\n",
       "      <th>sentence1</th>\n",
       "      <th>sentence2</th>\n",
       "      <th>pairID</th>\n",
       "      <th>heuristic</th>\n",
       "      <th>subcase</th>\n",
       "      <th>template</th>\n",
       "      <th>s1_rel</th>\n",
       "      <th>s2_rel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>non-entailment</td>\n",
       "      <td>( ( The president ) ( ( advised ( the doctor )...</td>\n",
       "      <td>( ( The doctor ) ( ( advised ( the president )...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN president)) (VP (VBD...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN doctor)) (VP (VBD ad...</td>\n",
       "      <td>The president advised the doctor .</td>\n",
       "      <td>The doctor advised the president .</td>\n",
       "      <td>ex0</td>\n",
       "      <td>lexical_overlap</td>\n",
       "      <td>ln_subject/object_swap</td>\n",
       "      <td>temp1</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN-- REL1 -- president-...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN-- REL3 -- doctor-- R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>non-entailment</td>\n",
       "      <td>( ( The student ) ( ( saw ( the managers ) ) ....</td>\n",
       "      <td>( ( The managers ) ( ( saw ( the student ) ) ....</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN student)) (VP (VBD s...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS managers)) (VP (VBD...</td>\n",
       "      <td>The student saw the managers .</td>\n",
       "      <td>The managers saw the student .</td>\n",
       "      <td>ex1</td>\n",
       "      <td>lexical_overlap</td>\n",
       "      <td>ln_subject/object_swap</td>\n",
       "      <td>temp1</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN-- REL1 -- student-- ...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS-- REL3 -- managers-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>non-entailment</td>\n",
       "      <td>( ( The presidents ) ( ( encouraged ( the bank...</td>\n",
       "      <td>( ( The banker ) ( ( encouraged ( the presiden...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS presidents)) (VP (V...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN banker)) (VP (VBD en...</td>\n",
       "      <td>The presidents encouraged the banker .</td>\n",
       "      <td>The banker encouraged the presidents .</td>\n",
       "      <td>ex2</td>\n",
       "      <td>lexical_overlap</td>\n",
       "      <td>ln_subject/object_swap</td>\n",
       "      <td>temp1</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS-- REL1 -- president...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN-- REL3 -- banker-- R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>non-entailment</td>\n",
       "      <td>( ( The senators ) ( ( supported ( the actor )...</td>\n",
       "      <td>( ( The actor ) ( ( supported ( the senators )...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS senators)) (VP (VBD...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN actor)) (VP (VBD sup...</td>\n",
       "      <td>The senators supported the actor .</td>\n",
       "      <td>The actor supported the senators .</td>\n",
       "      <td>ex3</td>\n",
       "      <td>lexical_overlap</td>\n",
       "      <td>ln_subject/object_swap</td>\n",
       "      <td>temp1</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS-- REL1 -- senators-...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NN-- REL3 -- actor-- RE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>non-entailment</td>\n",
       "      <td>( ( The actors ) ( ( avoided ( the bankers ) )...</td>\n",
       "      <td>( ( The bankers ) ( ( avoided ( the actors ) )...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS actors)) (VP (VBD a...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS bankers)) (VP (VBD ...</td>\n",
       "      <td>The actors avoided the bankers .</td>\n",
       "      <td>The bankers avoided the actors .</td>\n",
       "      <td>ex4</td>\n",
       "      <td>lexical_overlap</td>\n",
       "      <td>ln_subject/object_swap</td>\n",
       "      <td>temp1</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS-- REL1 -- actors-- ...</td>\n",
       "      <td>(ROOT (S (NP (DT The) (NNS-- REL3 -- bankers--...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0      gold_label  \\\n",
       "0           0  non-entailment   \n",
       "1           1  non-entailment   \n",
       "2           2  non-entailment   \n",
       "3           3  non-entailment   \n",
       "4           4  non-entailment   \n",
       "\n",
       "                              sentence1_binary_parse  \\\n",
       "0  ( ( The president ) ( ( advised ( the doctor )...   \n",
       "1  ( ( The student ) ( ( saw ( the managers ) ) ....   \n",
       "2  ( ( The presidents ) ( ( encouraged ( the bank...   \n",
       "3  ( ( The senators ) ( ( supported ( the actor )...   \n",
       "4  ( ( The actors ) ( ( avoided ( the bankers ) )...   \n",
       "\n",
       "                              sentence2_binary_parse  \\\n",
       "0  ( ( The doctor ) ( ( advised ( the president )...   \n",
       "1  ( ( The managers ) ( ( saw ( the student ) ) ....   \n",
       "2  ( ( The banker ) ( ( encouraged ( the presiden...   \n",
       "3  ( ( The actor ) ( ( supported ( the senators )...   \n",
       "4  ( ( The bankers ) ( ( avoided ( the actors ) )...   \n",
       "\n",
       "                                     sentence1_parse  \\\n",
       "0  (ROOT (S (NP (DT The) (NN president)) (VP (VBD...   \n",
       "1  (ROOT (S (NP (DT The) (NN student)) (VP (VBD s...   \n",
       "2  (ROOT (S (NP (DT The) (NNS presidents)) (VP (V...   \n",
       "3  (ROOT (S (NP (DT The) (NNS senators)) (VP (VBD...   \n",
       "4  (ROOT (S (NP (DT The) (NNS actors)) (VP (VBD a...   \n",
       "\n",
       "                                     sentence2_parse  \\\n",
       "0  (ROOT (S (NP (DT The) (NN doctor)) (VP (VBD ad...   \n",
       "1  (ROOT (S (NP (DT The) (NNS managers)) (VP (VBD...   \n",
       "2  (ROOT (S (NP (DT The) (NN banker)) (VP (VBD en...   \n",
       "3  (ROOT (S (NP (DT The) (NN actor)) (VP (VBD sup...   \n",
       "4  (ROOT (S (NP (DT The) (NNS bankers)) (VP (VBD ...   \n",
       "\n",
       "                                sentence1  \\\n",
       "0      The president advised the doctor .   \n",
       "1          The student saw the managers .   \n",
       "2  The presidents encouraged the banker .   \n",
       "3      The senators supported the actor .   \n",
       "4        The actors avoided the bankers .   \n",
       "\n",
       "                                sentence2 pairID        heuristic  \\\n",
       "0      The doctor advised the president .    ex0  lexical_overlap   \n",
       "1          The managers saw the student .    ex1  lexical_overlap   \n",
       "2  The banker encouraged the presidents .    ex2  lexical_overlap   \n",
       "3      The actor supported the senators .    ex3  lexical_overlap   \n",
       "4        The bankers avoided the actors .    ex4  lexical_overlap   \n",
       "\n",
       "                  subcase template  \\\n",
       "0  ln_subject/object_swap    temp1   \n",
       "1  ln_subject/object_swap    temp1   \n",
       "2  ln_subject/object_swap    temp1   \n",
       "3  ln_subject/object_swap    temp1   \n",
       "4  ln_subject/object_swap    temp1   \n",
       "\n",
       "                                              s1_rel  \\\n",
       "0  (ROOT (S (NP (DT The) (NN-- REL1 -- president-...   \n",
       "1  (ROOT (S (NP (DT The) (NN-- REL1 -- student-- ...   \n",
       "2  (ROOT (S (NP (DT The) (NNS-- REL1 -- president...   \n",
       "3  (ROOT (S (NP (DT The) (NNS-- REL1 -- senators-...   \n",
       "4  (ROOT (S (NP (DT The) (NNS-- REL1 -- actors-- ...   \n",
       "\n",
       "                                              s2_rel  \n",
       "0  (ROOT (S (NP (DT The) (NN-- REL3 -- doctor-- R...  \n",
       "1  (ROOT (S (NP (DT The) (NNS-- REL3 -- managers-...  \n",
       "2  (ROOT (S (NP (DT The) (NN-- REL3 -- banker-- R...  \n",
       "3  (ROOT (S (NP (DT The) (NN-- REL3 -- actor-- RE...  \n",
       "4  (ROOT (S (NP (DT The) (NNS-- REL3 -- bankers--...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building parenthetical trees and encode in Universal Syntactic Embeddings\n",
    "Here the loaded dataset is processed, transformed into tree form and encoded via kerMIT encoder.\n",
    "\n",
    "\n",
    "In realtime the trees are saved on file, a log file is made showing the number of processed rows of the dataset and the encoded trees are saved in pickle format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building parenthetical trees for Training Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 30000/30000 [14:00<00:00, 35.67it/s]\n"
     ]
    }
   ],
   "source": [
    "#insert here your dataset name\n",
    "name = \"MultiNLI_final\"\n",
    "dataset_name = f'{name}_REL_sencence_1'\n",
    "\n",
    "\n",
    "name = 'dtk_trees_'+dataset_name+'.pkl'\n",
    "name2 = 'log_'+dataset_name+'.txt'\n",
    "name3 = 'dt_'+dataset_name+'.txt'\n",
    "\n",
    "i = 0\n",
    "cont = 0\n",
    "listTree = []\n",
    "newList = []\n",
    "oldList = []\n",
    "\n",
    "tree = \"(S)\"\n",
    "treeException = createDTK(tree)\n",
    "\n",
    "\n",
    "for line in tqdm(data_train['s1_rel']):\n",
    "    cont += 1\n",
    "    try: \n",
    "        #tree = (parse(line))\n",
    "        tree = line\n",
    "        treeDTK = createDTK(tree)\n",
    "    except Exception:\n",
    "        tree, treeDTK = \"(S)\", treeException\n",
    "    \n",
    "    listTree.append(treeDTK)   \n",
    "    #write parenthetical tree\n",
    "    writeTree(name3,tree)\n",
    "    #every 5000 shafts saves the corresponding DTKs\n",
    "    if i>5000:\n",
    "        time.sleep(1)\n",
    "        if os.path.isfile(name):\n",
    "            #append new encoded tree in pickle file            \n",
    "            oldList = readP(name) \n",
    "            newList = oldList+listTree\n",
    "        else:\n",
    "            newList = listTree\n",
    "        \n",
    "        f=open(name, 'wb')\n",
    "        \n",
    "        for x in newList:\n",
    "            pickle.dump(x, f)\n",
    "        f.close()\n",
    "\n",
    "        f=open(name2, \"a+\")\n",
    "        f.write(str(cont)+'..')\n",
    "        f.close()\n",
    "                  \n",
    "        i = 0\n",
    "        listTree = []\n",
    "        newList = []\n",
    "        oldList = []         \n",
    "    else:\n",
    "        i +=1\n",
    "    \n",
    "    if cont == 74998:\n",
    "        f=open(name2, \"a+\")\n",
    "        f.write(str(cont)+'..!!!')\n",
    "        f.close()\n",
    "    if cont == 150000:\n",
    "        f=open(name2, \"a+\")\n",
    "        f.write(str(cont)+'..!!!')\n",
    "        f.close()\n",
    "    if cont == 300000:\n",
    "        f=open(name2, \"a+\")\n",
    "        f.write(str(cont)+'..!!!')\n",
    "        f.close()\n",
    "\n",
    "#checking consistency\n",
    "if os.path.isfile(name):\n",
    "    oldList = readP(name) \n",
    "    newList = oldList+listTree\n",
    "else:\n",
    "    newList = listTree      \n",
    "f=open(name, 'wb')\n",
    "for x in newList:\n",
    "    pickle.dump(x, f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#68352"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building parenthetical trees for Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#insert here your dataset name\n",
    "dataset_name = 'multiNLI_test_matched'\n",
    "\n",
    "\n",
    "name = 'dtk_trees_'+dataset_name+'.pkl'\n",
    "name2 = 'log_'+dataset_name+'.txt'\n",
    "name3 = 'dt_'+dataset_name+'.txt'\n",
    "\n",
    "i = 0\n",
    "cont = 0\n",
    "listTree = []\n",
    "newList = []\n",
    "oldList = []\n",
    "\n",
    "tree = \"(S)\"\n",
    "treeException = createDTK(tree)\n",
    "\n",
    "\n",
    "for line in tqdm(data_test['sentence1']):\n",
    "    cont += 1\n",
    "    try: \n",
    "        tree = (parse(line))\n",
    "        treeDTK = createDTK(tree)\n",
    "    except Exception:\n",
    "        tree, treeDTK = \"(S)\", treeException\n",
    "    \n",
    "    listTree.append(treeDTK)   \n",
    "    #write parenthetical tree\n",
    "    writeTree(name3,tree)\n",
    "    #every 5000 shafts saves the corresponding DTKs\n",
    "    if i>5000:\n",
    "        time.sleep(1)\n",
    "        if os.path.isfile(name):\n",
    "            #append new encoded tree in pickle file\n",
    "            oldList = readP(name) \n",
    "            newList = oldList+listTree\n",
    "        else:\n",
    "            newList = listTree\n",
    "        \n",
    "        f=open(name, 'wb')\n",
    "        \n",
    "        for x in newList:\n",
    "            pickle.dump(x, f)\n",
    "        f.close()\n",
    "\n",
    "        f=open(name2, \"a+\")\n",
    "        f.write(str(cont)+'..')\n",
    "        f.close()\n",
    "                  \n",
    "        i = 0\n",
    "        listTree = []\n",
    "        newList = []\n",
    "        oldList = []         \n",
    "    else:\n",
    "        i +=1\n",
    "\n",
    "#checking consistency\n",
    "if os.path.isfile(name):\n",
    "    oldList = readP(name) \n",
    "    newList = oldList+listTree\n",
    "else:\n",
    "    newList = listTree      \n",
    "f=open(name, 'wb')\n",
    "for x in newList:\n",
    "    pickle.dump(x, f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}