Deep Dive - Information Extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d9a3430d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Information Extraction\n",
    "\n",
    "\n",
    "How can we extract structured information from text?\n",
    "\n",
    "This allows us to answer for example the following questions:\n",
    "\n",
    "- Who receives questions from MEPs?\n",
    "- When were the answers submitted? How long is the response time?\n",
    "- Which party is asking most questions?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5da2de",
   "metadata": {},
   "source": [
    "# Reading in data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22774886",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "016d0b71",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = './data/parliamentary-questions_2023_sample.csv'\n",
    "data = pd.read_csv(path, index_col=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7abe8ad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_question = data.question_text.values[100]\n",
    "print(sample_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f1c7168",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_answer = data.answer_text.values[110]\n",
    "print(sample_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95104977",
   "metadata": {},
   "source": [
    "# Extracting information from text\n",
    "\n",
    "## Methods\n",
    "\n",
    "- Based on document structure. This requires all documents to be structured exactly the same to avoid noise or wrong extractions\n",
    "- Named Entity Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfcfdc25",
   "metadata": {},
   "source": [
    "## Extract the recipient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22d7c559",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the question in meaningful parts\n",
    "sample_question.split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a419784",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the second part and replace unnnecessary text\n",
    "def get_recipient_from_question(question):\n",
    "    recipient_text = question.split('\\n')[1]\n",
    "    return recipient_text.replace('to the ', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5b4c31e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# apply to all data\n",
    "data['recipient'] = data['question_text'].apply(get_recipient_from_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc777fce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get an overview of the different recipients\n",
    "data.recipient.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04fe0896",
   "metadata": {},
   "source": [
    "## Digression: Regular Expressions\n",
    "\n",
    "Regular expressions (regex) are concise patterns used for searching and manipulating text. \n",
    "\n",
    "Examples:\n",
    "- Websites: `/^www/`\n",
    "- Extracting dates in the format dd/mm/yyyy: `\\d{2}/\\d{2}/\\d{4}`\n",
    "- Matching email addresses \n",
    "\n",
    "\n",
    "Tools:\n",
    "\n",
    "- RegEx Generator: https://www.autoregex.xyz/\n",
    "- RegEx Online Tester: https://regexr.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "341a4d5a",
   "metadata": {},
   "source": [
    "## Extract the date of the answer submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8df8acfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "# Create a regex to detect a date in the format (DD.MM.YYYY)\n",
    "date_regex = '(\\d{1,2}\\.\\d{1,2}\\.\\d{4})'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8ab33ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "re.findall(date_regex, sample_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae1a25d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_answer_date(answer):\n",
    "    # ensures that the answer is a string\n",
    "    if not isinstance(answer, str):\n",
    "        return None\n",
    "    matches = re.findall(date_regex, answer)\n",
    "    # if a match is found only the first element is returned\n",
    "    if len(matches) > 0:\n",
    "        return matches[0]\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1c966d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['answer_date'] = data['answer_text'].apply(get_answer_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e121247",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.answer_date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0384105",
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data.answer_date.isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92553783",
   "metadata": {},
   "source": [
    "## How long does it take to respond to a question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44b81a91",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['question_date'] = pd.to_datetime(data['document_date'])\n",
    "data['answer_date'] = pd.to_datetime(data['answer_date'], dayfirst=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "625c1061",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_date_difference(start_date, end_date):\n",
    "    return (end_date - start_date).days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9f12915",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['response_time'] = data.apply(lambda row: compute_date_difference(row['question_date'], row['answer_date']), axis=1)\n",
    "data['response_time']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e625859",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['response_time'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "436773b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Response time statistics per institution\n",
    "data.groupby('recipient')['response_time'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afba4c09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Response time statistics by month of questioning\n",
    "data['question_month'] = data['question_date'].dt.strftime('%Y-%m')\n",
    "data.groupby('question_month')['response_time'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6cf3b16",
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4949cfdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "sns.swarmplot(x='document_type', y='response_time', data=data)\n",
    "plt.xlabel('Month')\n",
    "plt.ylabel('Response Time (Days)')\n",
    "plt.title('Response Time Distribution by Category')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "659da8ef",
   "metadata": {},
   "source": [
    "## Named Entity Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "868e4e2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the spacy English model for Named Entity Recognition\n",
    "!python3 -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e411a230",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "# Load the English language model in spaCy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "def extract_named_entities(text):\n",
    "    doc = nlp(text)\n",
    "    named_entities = []\n",
    "\n",
    "    for entity in doc.ents:\n",
    "        if entity.label_ in [\"DATE\", \"PERSON\", \"ORG\"]:\n",
    "            named_entities.append((entity.text, entity.label_))\n",
    "\n",
    "    return named_entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28c8b14f",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59ed745a",
   "metadata": {},
   "outputs": [],
   "source": [
    "extract_named_entities(sample_answer)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}