day5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Day 5: Advanced Processing, Bring Your Own Corpora\n",
    "\n",
    "Na-Rae Han (`naraehan@pitt.edu`) and David J. Birnbaum (`djbpitt@pitt.edu`) \n",
    "\n",
    "June 25-29, [NASSLLI 2018 at CMU](https://www.cmu.edu/nasslli2018/) \n",
    "\n",
    "This tutorial is found on https://github.com/naraehan/NASSLLI2018-Corpus-Linguistics. \n",
    "- Jump to: [Day 1](day1.ipynb), [Day 2](day2.ipynb), [Day 3](day3.ipynb), [Day 4](day4.ipynb), [Day 5](day5.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced processing: lemmatization\n",
    "- NLTK's WordNet lemmatizer \n",
    "- It works well for nouns. Verbs are tricky: default POS is set to 'noun', and verbs need to be specified as such. \n",
    "- For a better/knowlege-rich/context-aware solution, you might need to venture outside Python/NLTK and try full-scale NLP suites such as [Stanford's Core NLP](https://stanfordnlp.github.io/CoreNLP/). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "wnl = nltk.WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cat'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# try 'geese', 'walks', 'walked', 'walking' \n",
    "wnl.lemmatize('cats')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'walk'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 'n' (noun; default), 'v' (verb), 'a' (adjective), 'r' (adverb)\n",
    "wnl.lemmatize('walking', 'v')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# From this page: http://www.pitt.edu/~naraehan/python3/text-samples.txt\n",
    "moby = \"\"\"Call me Ishmael. Some years ago--never mind how long precisely--having\n",
    "little or no money in my purse, and nothing particular to interest me on\n",
    "shore, I thought I would sail about a little and see the watery part of\n",
    "the world. It is a way I have of driving off the spleen and regulating\n",
    "the circulation.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretty printing has been turned OFF\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Call', 'me', 'Ishmael', '.', 'Some', 'years', 'ago', '--', 'never', 'mind', 'how', 'long', 'precisely', '--', 'having', 'little', 'or', 'no', 'money', 'in', 'my', 'purse', ',', 'and', 'nothing', 'particular', 'to', 'interest', 'me', 'on', 'shore', ',', 'I', 'thought', 'I', 'would', 'sail', 'about', 'a', 'little', 'and', 'see', 'the', 'watery', 'part', 'of', 'the', 'world', '.', 'It', 'is', 'a', 'way', 'I', 'have', 'of', 'driving', 'off', 'the', 'spleen', 'and', 'regulating', 'the', 'circulation', '.']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%pprint\n",
    "nltk.word_tokenize(moby)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Call', 'me', 'Ishmael', '.', 'Some', 'year', 'ago', '--', 'never', 'mind', 'how', 'long', 'precisely', '--', 'having', 'little', 'or', 'no', 'money', 'in', 'my', 'purse', ',', 'and', 'nothing', 'particular', 'to', 'interest', 'me', 'on', 'shore', ',', 'I', 'thought', 'I', 'would', 'sail', 'about', 'a', 'little', 'and', 'see', 'the', 'watery', 'part', 'of', 'the', 'world', '.', 'It', 'is', 'a', 'way', 'I', 'have', 'of', 'driving', 'off', 'the', 'spleen', 'and', 'regulating', 'the', 'circulation', '.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[wnl.lemmatize(t) for t in nltk.word_tokenize(moby)]\n",
    "# Output isn't very intelligent without us supplying individual tokens with their correct POS \n",
    "# Any way to identify verbs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced processing: POS tagging\n",
    "- `nltk.pos_tag` is NLTK's default POS tagger.  \n",
    "- Default tagset is the [Penn Treebank ('wsj') tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html). \n",
    "- A word of warning: it is not state-of-the-art. (Built on limited data.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['colorless', 'green', 'ideas', 'sleep', 'furiously']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chom = 'colorless green ideas sleep furiously'.split()\n",
    "chom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('colorless', 'NN'), ('green', 'JJ'), ('ideas', 'NNS'), ('sleep', 'VBP'), ('furiously', 'RB')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.pos_tag(chom)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Call', 'VB'), ('me', 'PRP'), ('Ishmael', 'NNP'), ('.', '.'), ('Some', 'DT'), ('years', 'NNS'), ('ago', 'RB'), ('--', ':'), ('never', 'RB'), ('mind', 'VB'), ('how', 'WRB'), ('long', 'JJ'), ('precisely', 'RB'), ('--', ':'), ('having', 'VBG'), ('little', 'JJ'), ('or', 'CC'), ('no', 'DT'), ('money', 'NN'), ('in', 'IN'), ('my', 'PRP$'), ('purse', 'NN'), (',', ','), ('and', 'CC'), ('nothing', 'NN'), ('particular', 'JJ'), ('to', 'TO'), ('interest', 'NN'), ('me', 'PRP'), ('on', 'IN'), ('shore', 'NN'), (',', ','), ('I', 'PRP'), ('thought', 'VBD'), ('I', 'PRP'), ('would', 'MD'), ('sail', 'VB'), ('about', 'IN'), ('a', 'DT'), ('little', 'JJ'), ('and', 'CC'), ('see', 'VB'), ('the', 'DT'), ('watery', 'JJ'), ('part', 'NN'), ('of', 'IN'), ('the', 'DT'), ('world', 'NN'), ('.', '.'), ('It', 'PRP'), ('is', 'VBZ'), ('a', 'DT'), ('way', 'NN'), ('I', 'PRP'), ('have', 'VBP'), ('of', 'IN'), ('driving', 'VBG'), ('off', 'RP'), ('the', 'DT'), ('spleen', 'NN'), ('and', 'CC'), ('regulating', 'VBG'), ('the', 'DT'), ('circulation', 'NN'), ('.', '.')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.pos_tag(nltk.word_tokenize(moby))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function pos_tag in module nltk.tag:\n",
      "\n",
      "pos_tag(tokens, tagset=None, lang='eng')\n",
      "    Use NLTK's currently recommended part of speech tagger to\n",
      "    tag the given list of tokens.\n",
      "    \n",
      "        >>> from nltk.tag import pos_tag\n",
      "        >>> from nltk.tokenize import word_tokenize\n",
      "        >>> pos_tag(word_tokenize(\"John's big idea isn't all that bad.\"))\n",
      "        [('John', 'NNP'), (\"'s\", 'POS'), ('big', 'JJ'), ('idea', 'NN'), ('is', 'VBZ'),\n",
      "        (\"n't\", 'RB'), ('all', 'PDT'), ('that', 'DT'), ('bad', 'JJ'), ('.', '.')]\n",
      "        >>> pos_tag(word_tokenize(\"John's big idea isn't all that bad.\"), tagset='universal')\n",
      "        [('John', 'NOUN'), (\"'s\", 'PRT'), ('big', 'ADJ'), ('idea', 'NOUN'), ('is', 'VERB'),\n",
      "        (\"n't\", 'ADV'), ('all', 'DET'), ('that', 'DET'), ('bad', 'ADJ'), ('.', '.')]\n",
      "    \n",
      "    NB. Use `pos_tag_sents()` for efficient tagging of more than one sentence.\n",
      "    \n",
      "    :param tokens: Sequence of tokens to be tagged\n",
      "    :type tokens: list(str)\n",
      "    :param tagset: the tagset to be used, e.g. universal, wsj, brown\n",
      "    :type tagset: str\n",
      "    :param lang: the ISO 639 code of the language, e.g. 'eng' for English, 'rus' for Russian\n",
      "    :type lang: str\n",
      "    :return: The tagged tokens\n",
      "    :rtype: list(tuple(str, str))\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(nltk.pos_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bring Your Own Corpora (1): Treebanks\n",
    "- Treebanks are syntactically annotated sentences. \n",
    "- They are used in training POS-taggers and syntactic parsers. \n",
    "- NLTK includes a sample section of the Penn English Treebank (3914 sentences and about 10% of the entire corpus). \n",
    "- For more details on Treebanks and how to interact with tree structure, see [this NLTK book section](http://www.nltk.org/book/ch08.html#treebanks-and-grammars). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Pierre', 'Vinken', ',', '61', 'years', 'old', ',', ...]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import treebank\n",
    "treebank.words()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Pierre', 'Vinken', ',', '61', 'years', 'old', ',', 'will', 'join', 'the', 'board', 'as', 'a', 'nonexecutive', 'director', 'Nov.', '29', '.'], ['Mr.', 'Vinken', 'is', 'chairman', 'of', 'Elsevier', 'N.V.', ',', 'the', 'Dutch', 'publishing', 'group', '.'], ...]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treebank.sents()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ('61', 'CD'), ('years', 'NNS'), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), ('the', 'DT'), ('board', 'NN'), ('as', 'IN'), ('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ('Nov.', 'NNP'), ('29', 'CD'), ('.', '.')], [('Mr.', 'NNP'), ('Vinken', 'NNP'), ('is', 'VBZ'), ('chairman', 'NN'), ('of', 'IN'), ('Elsevier', 'NNP'), ('N.V.', 'NNP'), (',', ','), ('the', 'DT'), ('Dutch', 'NNP'), ('publishing', 'VBG'), ('group', 'NN'), ('.', '.')], ...]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treebank.tagged_sents()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Tree('S', [Tree('NP-SBJ', [Tree('NP', [Tree('NNP', ['Pierre']), Tree('NNP', ['Vinken'])]), Tree(',', [',']), Tree('ADJP', [Tree('NP', [Tree('CD', ['61']), Tree('NNS', ['years'])]), Tree('JJ', ['old'])]), Tree(',', [','])]), Tree('VP', [Tree('MD', ['will']), Tree('VP', [Tree('VB', ['join']), Tree('NP', [Tree('DT', ['the']), Tree('NN', ['board'])]), Tree('PP-CLR', [Tree('IN', ['as']), Tree('NP', [Tree('DT', ['a']), Tree('JJ', ['nonexecutive']), Tree('NN', ['director'])])]), Tree('NP-TMP', [Tree('NNP', ['Nov.']), Tree('CD', ['29'])])])]), Tree('.', ['.'])]), Tree('S', [Tree('NP-SBJ', [Tree('NNP', ['Mr.']), Tree('NNP', ['Vinken'])]), Tree('VP', [Tree('VBZ', ['is']), Tree('NP-PRD', [Tree('NP', [Tree('NN', ['chairman'])]), Tree('PP', [Tree('IN', ['of']), Tree('NP', [Tree('NP', [Tree('NNP', ['Elsevier']), Tree('NNP', ['N.V.'])]), Tree(',', [',']), Tree('NP', [Tree('DT', ['the']), Tree('NNP', ['Dutch']), Tree('VBG', ['publishing']), Tree('NN', ['group'])])])])])]), Tree('.', ['.'])]), ...]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treebank.parsed_sents()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Note: just flashing the first tree will give you an \"unable to find the gs file\" error. \n",
    "#    Saving it into t works, however. \n",
    "# https://stackoverflow.com/questions/36942270/nltk-was-unable-to-find-the-gs-file/37160385\n",
    "# In short: you need to install GhostScript and add it to your system's PATH. \n",
    "\n",
    "t = treebank.parsed_sents()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  (NP-SBJ\n",
      "    (NP (NNP Pierre) (NNP Vinken))\n",
      "    (, ,)\n",
      "    (ADJP (NP (CD 61) (NNS years)) (JJ old))\n",
      "    (, ,))\n",
      "  (VP\n",
      "    (MD will)\n",
      "    (VP\n",
      "      (VB join)\n",
      "      (NP (DT the) (NN board))\n",
      "      (PP-CLR (IN as) (NP (DT a) (JJ nonexecutive) (NN director)))\n",
      "      (NP-TMP (NNP Nov.) (CD 29))))\n",
      "  (. .))\n"
     ]
    }
   ],
   "source": [
    "# Trees are composed of subtrees, each of which itself is a Tree. \n",
    "print(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Opens up a new window. Close it before moving to next cell. \n",
    "t.draw()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  (NP-SBJ (DT A) (NNP Lorillard) (NN spokewoman))\n",
      "  (VP\n",
      "    (VBD said)\n",
      "    (, ,)\n",
      "    (`` ``)\n",
      "    (S\n",
      "      (NP-SBJ (DT This))\n",
      "      (VP (VBZ is) (NP-PRD (DT an) (JJ old) (NN story)))))\n",
      "  (. .))\n"
     ]
    }
   ],
   "source": [
    "# \"said\" is a verb (VBD) that takes a clausal complement (S). \n",
    "#   The nodes are children of a VP node. \n",
    "print(treebank.parsed_sents()[7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# myfilter: returns True/False on whether current Tree is a VP node with an S child. \n",
    "# You can define your own function through def keyword. \n",
    "\n",
    "def myfilter(tree):\n",
    "    child_nodes = [child.label() for child in tree if isinstance(child, nltk.Tree)]\n",
    "    return  (tree.label() == 'VP') and ('S' in child_nodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretty printing has been turned ON\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Tree('VP', [Tree('VBN', ['named']), Tree('S', [Tree('NP-SBJ', [Tree('-NONE-', ['*-1'])]), Tree('NP-PRD', [Tree('NP', [Tree('DT', ['a']), Tree('JJ', ['nonexecutive']), Tree('NN', ['director'])]), Tree('PP', [Tree('IN', ['of']), Tree('NP', [Tree('DT', ['this']), Tree('JJ', ['British']), Tree('JJ', ['industrial']), Tree('NN', ['conglomerate'])])])])])]),\n",
       " Tree('VP', [Tree('VBD', ['said']), Tree(',', [',']), Tree('``', ['``']), Tree('S', [Tree('NP-SBJ', [Tree('DT', ['This'])]), Tree('VP', [Tree('VBZ', ['is']), Tree('NP-PRD', [Tree('DT', ['an']), Tree('JJ', ['old']), Tree('NN', ['story'])])])])]),\n",
       " Tree('VP', [Tree('VBD', ['said']), Tree('S', [Tree('-NONE-', ['*T*-1'])])]),\n",
       " Tree('VP', [Tree('VBN', ['expected']), Tree('S', [Tree('-NONE-', ['*?*'])])]),\n",
       " Tree('VP', [Tree('VBD', ['said']), Tree('S', [Tree('-NONE-', ['*T*-1'])])]),\n",
       " Tree('VP', [Tree('VBZ', ['appears']), Tree('S', [Tree('NP-SBJ', [Tree('-NONE-', ['*-1'])]), Tree('VP', [Tree('TO', ['to']), Tree('VP', [Tree('VB', ['be']), Tree('NP-PRD', [Tree('NP', [Tree('DT', ['the']), Tree('JJS', ['highest'])]), Tree('PP', [Tree('IN', ['for']), Tree('NP', [Tree('NP', [Tree('DT', ['any']), Tree('NN', ['asbestos']), Tree('NNS', ['workers'])]), Tree('RRC', [Tree('VP', [Tree('VBN', ['studied']), Tree('NP', [Tree('-NONE-', ['*'])]), Tree('PP-LOC', [Tree('IN', ['in']), Tree('NP', [Tree('JJ', ['Western']), Tree('VBN', ['industrialized']), Tree('NNS', ['countries'])])])])])])])])])])])])]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For every full tree in the Treebank, recurse through its subtrees, \n",
    "#    filter in only those that meet the filter condition. \n",
    "# Searching through first 20 sentences only: remove [:20] for a full search. \n",
    "\n",
    "%pprint \n",
    "[subtree for tree in treebank.parsed_sents()[:20]\n",
    "             for subtree in tree.subtrees(myfilter)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "found = [subtree for tree in treebank.parsed_sents()[:20]\n",
    "             for subtree in tree.subtrees(myfilter)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(VP\n",
      "  (VBN named)\n",
      "  (S\n",
      "    (NP-SBJ (-NONE- *-1))\n",
      "    (NP-PRD\n",
      "      (NP (DT a) (JJ nonexecutive) (NN director))\n",
      "      (PP\n",
      "        (IN of)\n",
      "        (NP (DT this) (JJ British) (JJ industrial) (NN conglomerate))))))\n",
      "\n",
      "(VP\n",
      "  (VBD said)\n",
      "  (, ,)\n",
      "  (`` ``)\n",
      "  (S\n",
      "    (NP-SBJ (DT This))\n",
      "    (VP (VBZ is) (NP-PRD (DT an) (JJ old) (NN story)))))\n",
      "\n",
      "(VP (VBD said) (S (-NONE- *T*-1)))\n",
      "\n",
      "(VP (VBN expected) (S (-NONE- *?*)))\n",
      "\n",
      "(VP (VBD said) (S (-NONE- *T*-1)))\n",
      "\n",
      "(VP\n",
      "  (VBZ appears)\n",
      "  (S\n",
      "    (NP-SBJ (-NONE- *-1))\n",
      "    (VP\n",
      "      (TO to)\n",
      "      (VP\n",
      "        (VB be)\n",
      "        (NP-PRD\n",
      "          (NP (DT the) (JJS highest))\n",
      "          (PP\n",
      "            (IN for)\n",
      "            (NP\n",
      "              (NP (DT any) (NN asbestos) (NNS workers))\n",
      "              (RRC\n",
      "                (VP\n",
      "                  (VBN studied)\n",
      "                  (NP (-NONE- *))\n",
      "                  (PP-LOC\n",
      "                    (IN in)\n",
      "                    (NP\n",
      "                      (JJ Western)\n",
      "                      (VBN industrialized)\n",
      "                      (NNS countries))))))))))))\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for t in found:\n",
    "    print(t)\n",
    "    print()   # prints an empty line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Treebanks in Non-English\n",
    "- A sample of 'Sinica Treebank' (Chinese) is available as part of NLTK's data. \n",
    "- You should download it first. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package sinica_treebank to\n",
      "[nltk_data]     D:\\narae/nltk_data...\n",
      "[nltk_data]   Package sinica_treebank is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('sinica_treebank')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(VP\n",
      "  (Ndabe 同時)\n",
      "  (Dd 就)\n",
      "  (VC32 帶)\n",
      "  (Di 了)\n",
      "  (NP\n",
      "    (NP (DM 四張) (VH11 熟) (Nab 牛皮))\n",
      "    (Caa 和)\n",
      "    (NP (DM 十二頭) (VH16 肥) (Nab 牛))))\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import sinica_treebank as chtb\n",
    "print(chtb.parsed_sents()[3450])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chtb.parsed_sents()[3450].draw()    # Opens a new window"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bring Your Own Corpora (2): CHILDES\n",
    "**CHAT vs. XML**\n",
    "- CHILDES uses its own corpus format: CHAT. \n",
    "- Many data sets also come in XML format (https://childes.talkbank.org/data-xml/), which NLTK can read in.\n",
    "- If no XML version is provided, you can use a converter called Chatter: https://talkbank.org/software/chatter.html\n",
    "\n",
    "**Getting the data**\n",
    "1. Navigate to <https://childes.talkbank.org/data-xml/>.\n",
    "1. Click on the link to the language that interests you, e.g., `Eng-NA` (North American English). These directories hold `zip` archives of subcorpora in the designated language.\n",
    "1. Download one of more of the zip files, say `Valian.zip`. \n",
    "1. Create a new directory named `CHILDES` on your Desktop. Unzip the downloaded file into it. \n",
    "1. Now you should have a `Valian` directory inside `CHILDES`. \n",
    "\n",
    "**Starter code:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus.reader import CHILDESCorpusReader\n",
    "corpus_root = 'C:/Users/narae/Desktop/CHILDES/Valian' # change path as needed\n",
    "valian = CHILDESCorpusReader(corpus_root, '.*.xml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretty printing has been turned OFF\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['01a.xml', '01b.xml', '02a.xml', '02b.xml', '03a.xml', '03b.xml', '04a.xml', '04b.xml', '04c.xml', '05a.xml', '06a.xml', '06b.xml', '07a.xml', '08a.xml', '08b.xml', '09a.xml', '09b.xml', '09c.xml', '10a.xml', '10b.xml', '10c.xml', '11a.xml', '11b.xml', '12a.xml', '12b.xml', '13a.xml', '13b.xml', '14a.xml', '14b.xml', '15a.xml', '15b.xml', '16a.xml', '16b.xml', '17a.xml', '17b.xml', '18a.xml', '19a.xml', '19b.xml', '20a.xml', '20b.xml', '21a.xml', '21b.xml', '21c.xml']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%pprint\n",
    "valian.fileids()         # returns list of filenames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- If that all works, navigate to <http://www.nltk.org/howto/childes.html> and begin at the line that reads “Printing properties of the corpus files”.\n",
    "- More CHILDES & Python tutorials:\n",
    "  - http://ling-blogs.bu.edu/lx390f17/standoff-annotation-xml-and-more-childes/\n",
    "  - http://aaronstevenwhite.io/language-acquisition/working-with-childes-part1/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How about...?\n",
    "- Files in MS Word or PDF? (See [this NLTK book section](http://www.nltk.org/book/ch03.html#extracting-text-from-pdf-msword-and-other-binary-formats))\n",
    "- Non-English corpora? (See [this NLTK book section](http://www.nltk.org/book/ch02.html#corpora-in-other-languages))\n",
    "- Corpora in XML format? (See [this NLTK book section](http://www.nltk.org/book/ch11.html#working-with-xml))\n",
    "- Looking to load your own annotated corpus (POS-tagged, Treebanks, etc.)? NLTK provides specialized corpus loaders for such formats: see [this NLTK how-to page](http://www.nltk.org/howto/corpus.html).  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What next?\n",
    "Take a Python course! There are many online courses available on [Coursera](http://www.coursera.org), [EdX](https://www.edx.org/), [udemy](https://www.udemy.com/courses/), [DataCamp](https://www.datacamp.com/courses), and more.\n",
    "\n",
    "The NLTK book \"Natural Language Processing with Python\" is available here: http://www.nltk.org/book/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}