Rapport.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [DK914] Context of difference\n",
    "\n",
    "The aim of this project is to propose a formal definition of the contextual difference relation between 2 URIs. \n",
    "A context of difference can be a subgraph of the instances description of the URIs. Then develop a tool that can extract these subgraphs for each pair of URIS.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors : Yohan Chalier, François Amat, Subhy Albakour, Luka Jakovljevic\n",
    "\n",
    "## Contact : {firstname}.{lastname}@telecom-paristech.fr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This project has been split in different parts:\n",
    "1. The first part is the parsing, which consist at getting the owl ontology from the IIMB_LARGE files into the python owl api.\n",
    "\n",
    "2. The second part is the brain of our algorithm. The computation of common_properties or differences between two individuals. \n",
    "\n",
    "3. The third part is optimisation of our algorithm; the usage of multithreading.\n",
    "\n",
    "4. The final part is testing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The algorithm described here is in the file parser.py under the class `Ontology`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "    def __init__(self, subfolder_id=\"000\", folder=\"IIMB_LARGE\",\n",
    "                 filename=\"onto.owl\"):\n",
    "        \"\"\" Load an ontology from a given folder.\n",
    "\n",
    "        By extracting the default folder IIMB_LARGE, subfolders are of the\n",
    "        form 000, 001, 002, ..., 080. Each contains a file onto.owl with an\n",
    "        ontology in XML format.\n",
    "\n",
    "        This method intializes a Ontology object by loading the content of a\n",
    "        given folder.\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        self.iri = os.path.join(\"file://\", sys.path[0],\n",
    "                                folder, subfolder_id, filename)\n",
    "        print(\"Loading ontology at\", self.iri)\n",
    "        owl.Ontology.__init__(self, owl.World(), base_iri=self.iri+\"#\")\n",
    "        self.load()\n",
    "\n",
    "    def select(self, query):\n",
    "        \"\"\" Selects one item based on its node name.\n",
    "\n",
    "        If none element is found, returns None.\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        search = self.search(iri=\"*\"+query)\n",
    "        if len(search) > 0:\n",
    "            return search[0]\n",
    "        return None\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the `class Ontology(owl.Ontology)` which use the python library `owlready2`. In order to adapt this class to our project we need to change the `init` and `select` methods.\n",
    "\n",
    "The init method takes for arguments the filename of the ontologies, the name of the folder and subfolder.\n",
    "For the select method, we only need the name of the element if it exists. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Get the differences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The algorithm described here is in the file context.py under the name `difference`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``` python\n",
    "def difference(idv_a, idv_b, depth=1, keep_values=False, verbose=False):\n",
    "    \"\"\" Computes the difference subgraph of two entities.\n",
    "\n",
    "    Returns a JSON-like dictionnary. Keys are properties, and values are either\n",
    "    nested elements with other entities, or a pair of \"real\" (string) values\n",
    "    that differed from the two elements.\n",
    "\n",
    "    The `depth` parameter controls the recursion depth of the search.\n",
    "\n",
    "    The parameters `idv_a` and `idv_b` are objects representing individuals from\n",
    "    the two ontologies considered.\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    if verbose:\n",
    "        print(\"Entering difference for\", idv_a,\n",
    "              \"and\", idv_b, \"width depth\", depth)\n",
    "\n",
    "    graph = {}\n",
    "\n",
    "    # prevents infinite recursion\n",
    "    if depth >= 0:\n",
    "\n",
    "        # only consider common properties\n",
    "        for ppt_a, ppt_b in common_properties(idv_a, idv_b):\n",
    "\n",
    "            if verbose:\n",
    "                print(\"Property:\", ppt_a)\n",
    "\n",
    "            # each property may have several values\n",
    "            # TODO: maybe product comparison is not the best\n",
    "            for value_a in ppt_a[idv_a]:\n",
    "                for value_b in ppt_b[idv_b]:\n",
    "\n",
    "                    if owl.ObjectProperty in ppt_a.is_a:\n",
    "                        # `value_a` and `value_b` are instances of classes from\n",
    "                        # the ontology, hence we go deeper in the graph\n",
    "                        sub_graph = difference(value_a, value_b, depth - 1)\n",
    "                        if len(sub_graph) > 0:\n",
    "                            graph[str(ppt_a)] = sub_graph\n",
    "\n",
    "                    elif owl.DataProperty in ppt_a.is_a:\n",
    "                        # `value_a` and `value_b` are simple strings, so we just\n",
    "                        # match them\n",
    "                        if value_a != value_b:\n",
    "                            if keep_values:\n",
    "                                graph[str(ppt_a)] = {\"a\": value_a, \"b\": value_b}\n",
    "                            else:\n",
    "                                graph[str(ppt_a)] = {}\n",
    "\n",
    "    if verbose:\n",
    "        print(\"Exiting difference.\")\n",
    "\n",
    "    return graph\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input of the difference algorithm is a pair of elements (idv_a, idv_b).\n",
    "We want to build the graph of their differences.\n",
    "\n",
    "First, we takes all the properties they have in commun, for instance if idv_a has an attribute 'createdAt',\n",
    "and idv_b has also an attribute 'createdAt', they have' createdAt' as a commun property.\n",
    "```python \n",
    "for ppt_a, ppt_b in common_properties(idv_a, idv_b):\n",
    "```\n",
    "\n",
    "Next, we takes all the values of the properties found, in most case it is a single value but in order to be general we consider them as an array of values.\n",
    "\n",
    "\n",
    "```python \n",
    "            for value_a in ppt_a[idv_a]:\n",
    "                for value_b in ppt_b[idv_b]:                \n",
    "```\n",
    "\n",
    "\n",
    "Then, we test if the property value is a node or not, if it is a node we call `difference` on the subgraph.\n",
    "If it is not a node, it is a name (we have designed our ontology class for that). We compare the name and if it is not the exact same string name we add the difference in a graph. \n",
    "\n",
    "Finaly, we return the graph. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Optimisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The algorithm described here is in the file process.py under the classes `worker` and `process`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os \n",
    "from IPython.display import clear_output\n",
    "from context import  difference, plot\n",
    "from parser import Ontology\n",
    "from process import process\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = os.listdir(\"IIMB_LARGE\")\n",
    "files.remove('ontBig.owl')\n",
    "files.remove('description.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For the qualitative evaluation of your tool, you may inject some random erroneous sameAs links for the class Film in the refalign and compute the contextual difference of each pair of URIs (no recall and precision are needed). You may evaluate there readability of the contexts and scalability of your tool.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "ONTOLOGIES = []\n",
    "tmp = []\n",
    "choices = np.random.choice(len(files) - 1, 3)\n",
    "for i in range(len(choices)):\n",
    "    tmp.append(files[choices[i]])\n",
    "files = tmp\n",
    "for file in files:\n",
    "    ONTOLOGIES.append(Ontology(file))\n",
    "#ONTOLOGIES  = tmp\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_time_whitout_multiprocessing(depth = 3,max_iter = 1000):\n",
    "    ONTOLOGIES = []\n",
    "    for file in files:\n",
    "        ONTOLOGIES.append(Ontology(file))\n",
    "    source  = Ontology(\"000\")\n",
    "    operation_limit, operation_count = max_iter, 0 \n",
    "    for target in ONTOLOGIES:\n",
    "        if(source != target):\n",
    "            for idv_a in source.individuals():\n",
    "                if(operation_count > operation_limit):\n",
    "                    break\n",
    "\n",
    "                for idv_b in target.individuals():\n",
    "                    operation_count+=1\n",
    "                    difference(idv_a, idv_b, depth)\n",
    "                    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.24 s ± 259 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "test_time_whitout_multiprocessing()\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_time_whith_multiprocessing(max_iter=1000,files=files):\n",
    "    source_file  = \"000\"\n",
    "    for target_file in files:\n",
    "        if(source_file != target_file):\n",
    "            process(target_file, max_iter=max_iter,source_folder_id=source_file)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26.4 s ± 3.67 s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "test_time_whith_multiprocessing()\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}