Deep Dive - Content Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f30f9f46",
   "metadata": {},
   "source": [
    "# Content Analysis\n",
    "\n",
    "- What are the parliamentary questions about?\n",
    "- What are the most common recurring topcis?\n",
    "- Which documents talk about a specific document"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5da2de",
   "metadata": {},
   "source": [
    "# Reading in data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22774886",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "016d0b71",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = './data/parliamentary-questions_2023_sample.csv'\n",
    "data = pd.read_csv(path, index_col=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "161a37f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb63d97f",
   "metadata": {},
   "source": [
    "# Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60dcb53e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join answers and questions texts together to get all context\n",
    "data[['question_text', 'answer_text']] = data[['question_text', 'answer_text']].fillna(value='')\n",
    "data['text'] = data['question_text'].str.cat(data['answer_text'], sep=' -- ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b352fd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25f5946d",
   "metadata": {},
   "source": [
    "# Most frequent Words\n",
    "\n",
    "We are using the Natural Language Toolkit, also called `nltk`.\n",
    "\n",
    "More information: https://www.nltk.org/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cb306b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15f93cc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.probability import FreqDist\n",
    "\n",
    "FreqDist(['A', 'B', 'A', 'C', 'C', 'C', 'D'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb10eb9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a sample text\n",
    "sample = data.text.values[10]\n",
    "sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c89c323",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the words of a sample text\n",
    "sample.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9eccfefa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the most common words in the sample text\n",
    "fdist = FreqDist(sample.split())\n",
    "fdist.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b27255e",
   "metadata": {},
   "source": [
    "# Cleaning the text\n",
    "\n",
    "- write all words in lowercase\n",
    "- remove punctuation . , ( )\n",
    "- remove `s \n",
    "- remove stopwords (and, this, to, in, the)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39bd11a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "punctuation = ['.', ',', '?', ':', ';', '!', '-', '(', ')', '\"', \"“\", '„', '–']\n",
    "\n",
    "def clean_text(text):\n",
    "    clean_text = text.lower()\n",
    "    clean_text = clean_text.replace('’s', '')\n",
    "    clean_text = clean_text.replace('\\n', ' ')\n",
    "    for punct_char in punctuation:\n",
    "        clean_text = clean_text.replace(punct_char, '')\n",
    "    return clean_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf96fdf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_sample = clean_text(sample)\n",
    "clean_sample.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d936330",
   "metadata": {},
   "source": [
    "## Remove stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7bf2d49",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e92fae4b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "stopwords_en = stopwords.words('english')\n",
    "stopwords_en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7860b08c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stopwords in other languages\n",
    "print(stopwords.fileids())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95172251",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Funktion clean_words\n",
    "def remove_stopwords(text):\n",
    "    words = text.split()\n",
    "    clean = []\n",
    "    for word in words:\n",
    "        if word.lower() not in stopwords_en:\n",
    "            clean.append(word)\n",
    "    return ' '.join(clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a0f2c88",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_sample = remove_stopwords(sample)\n",
    "clean_sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95acb4fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get most common words in the sample text\n",
    "fdist = FreqDist(clean_sample.split())\n",
    "fdist.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7e3df7f",
   "metadata": {},
   "source": [
    "## Applied to all data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a515016b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['clean_text'] = data['text'].apply(clean_text)\n",
    "data['clean_text'] = data['clean_text'].apply(remove_stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "160be7d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_most_frequent_words(text):\n",
    "    fdist = FreqDist(text.split())\n",
    "    return fdist.most_common(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38cd2a32",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_most_frequent_words(data.clean_text.values[10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3be2789e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the most important keywords for all texts\n",
    "data['keywords'] = data['clean_text'].apply(get_most_frequent_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffdd2915",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.keywords"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c7a515",
   "metadata": {},
   "source": [
    "## Word Clouds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40f09522",
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def generate_word_clouds(freq_dict):\n",
    "    wordcloud = WordCloud(max_font_size=50, max_words=100, background_color=\"white\").generate_from_frequencies(freq_dict)\n",
    "    plt.figure()\n",
    "    plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "    plt.axis(\"off\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0de77ef9",
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_word_clouds(dict(data.keywords.values[20]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d0f84df",
   "metadata": {},
   "source": [
    "## Mehrere Wörter - Bigrams und Trigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "039bbca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.collocations import *\n",
    "bigrams = nltk.collocations.BigramAssocMeasures()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a8225a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "bigrams = list(nltk.bigrams(data.clean_text.values[10].split()))\n",
    "\n",
    "fdist = FreqDist(bigrams)\n",
    "fdist.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f0dd877",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.keywords"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "861ec6d2",
   "metadata": {},
   "source": [
    "# Search for documents by keyword"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "822499aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_related_documents(data, keyword, cols):\n",
    "    keyword = keyword.lower()\n",
    "    # Create an empty list to store the relevancy scores\n",
    "    relevancy_scores = []\n",
    "\n",
    "    # Iterate over each row in the dataframe\n",
    "    for index, row in data.iterrows():\n",
    "        document_keywords = dict(row['keywords'])\n",
    "\n",
    "        # Check if the keyword exists in the document's keywords\n",
    "        if keyword in document_keywords:\n",
    "            # Get the frequency of the keyword in the document\n",
    "            frequency = document_keywords[keyword]\n",
    "            \n",
    "            # Append the relevancy score (frequency) and document index to the list\n",
    "            relevancy_scores.append((frequency, index))\n",
    "    \n",
    "    # Sort the relevancy scores in descending order\n",
    "    relevancy_scores.sort(reverse=True)\n",
    "    \n",
    "    # Get the document indices from the sorted relevancy scores\n",
    "    document_indices = [score[1] for score in relevancy_scores]\n",
    "    \n",
    "    # Return the list of documents related to the keyword, sorted by relevancy\n",
    "    return data[cols].loc[document_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dc23391",
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = ['document_title','keywords', 'text']\n",
    "get_related_documents(data, 'education', cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f0e2f45",
   "metadata": {},
   "source": [
    "# Further Resources\n",
    "\n",
    "## TF-IDF\n",
    "\n",
    "TF-IDF (Term Frequency-Inverse Document Frequency) is a measure to identify keywords which are specific for a document given the context of the entire document collection. \n",
    "\n",
    "Tutorial by Melanie Walsh: https://melaniewalsh.github.io/Intro-Cultural-Analytics/05-Text-Analysis/03-TF-IDF-Scikit-Learn.html\n",
    "\n",
    "\n",
    "## Topic Modeling \n",
    "\n",
    "Topic Modeling is a statical approach to group documents based on their content\n",
    "\n",
    "Tutorial by Shashank Kapadia: https://towardsdatascience.com/end-to-end-topic-modeling-in-python-latent-dirichlet-allocation-lda-35ce4ed6b3e0"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}