python_basic_2_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An introduction to solving biological problems with Python\n",
    "\n",
    "## Session 2.3: Files\n",
    "\n",
    "- [Using files](#Using-files)\n",
    "- [Reading from files](#Reading-from-files)\n",
    "- [Exercises 2.3.1](#Exercises-2.3.1)\n",
    "- [Writing to files](#Writing-to-files)\n",
    "- [Exercises 2.3.2](#Exercises-2.3.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data input and output (I/O)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, all that data we have been working with has been written by us into our scripts, and the results of out computation has just been displayed in the terminal output. In the real world data will be supplied by the user of our programs (who may be you!) by some means, and we will often want to save the results of some analysis somewhere more permanent than just printing it to the screen. In this session we cover the way of reading data into our programs by reading files from disk, we also discuss writing out data to files. \n",
    "\n",
    "There are, of course, many other ways of accessing data, such as querying a database or retrieving data from a network such as the internet. We don't cover these here, but python has excellent support for interacting with databases and networks either in the standard library or using external modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequently the data we want to operate on or analyse will be stored in files, so in our programs we need to be able to open files, read through them (perhaps all at once, perhaps not), and then close them. \n",
    "\n",
    "We will also frequently want to be able to print output to files rather than always printing out results to the terminal.\n",
    "\n",
    "Python supports all of these modes of operations on files, and provides a number of useful functions and syntax to make dealing with files straightforward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Opening files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To open a file, python provides the `open` function, which takes a filename as its first argument and returns a _file object_ which is python's internal representation of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"data/datafile.txt\"\n",
    "fileObj = open( path )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`open` takes an optional second argument specifying the _mode_ in which the file is opened, either for reading, writing or appending.\n",
    "\n",
    "It defaults to `'r'` which means open for reading in text mode. Other common values are `'w'` for writing (truncating the file if it already exists) and `'a'` for appending."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "open( \"data/myfile.txt\", \"r\" ) # open for reading, default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "open( \"data/myfile.txt\", \"w\" ) # open for writing (existing files will be overwritten)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "open( \"data/myfile.txt\", \"a\" ) # open for appending"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Closing files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To close a file once you finished with it, you can call the `.close` method on a file object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mode modifiers\n",
    "\n",
    "These mode strings can include some extra modifier characters to deal with issues with files across multiple platforms.\n",
    "\n",
    "`'b'`: binary mode, e.g. `'rb'`. No translation for end-of-line characters to platform specific setting value.\n",
    "\n",
    "|Character | Meaning |\n",
    "|----------|---------|\n",
    "|`'r'` |\topen for reading (default) |\n",
    "|`'w'` |\topen for writing, truncating the file first |\n",
    "|`'x'` |\topen for exclusive creation, failing if the file already exists |\n",
    "|`'a'` |\topen for writing, appending to the end of the file if it exists |\n",
    "|`'b'` |\tbinary mode |\n",
    "|`'t'` |\ttext mode (default) |\n",
    "|`'+'` |\topen a disk file for updating (reading and writing) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading from files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have opened a file for reading, file objects provide a number of methods for accessing the data in a file. The simplest of these is the `.read` method that reads the entire contents of the file into a string variable.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileObj = open( \"data/datafile.txt\" )\n",
    "print(fileObj.read()) # everything\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this means the entire file will be read into memory. If you are operating on a large file and don't actually need all the data at the same time this is rather inefficient.\n",
    "\n",
    "Frequently, we just need to operate on individual lines of the file, and you can use the `.readline` method to read a line from a file and return it as a python string.\n",
    "\n",
    "File objects internally keep track of your current location in a file, so to get following lines from the file you can call this method multiple times.\n",
    "\n",
    "It is important to note that the string representing each line will have a trailing newline `\"\\n\"` character, which you may want to remove with the `.rstrip` string method.\n",
    "\n",
    "Once the end of the file is reached, `.readline` will return an empty string `''`. This is different from an apparently empty line in a file, as even an empty line will contain a newline character. Recall that the empty string is considered as `False` in python, so you can readily check for this condition with an `if` statement etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one line at a time\n",
    "fileObj = open( \"data/datafile.txt\" )\n",
    "print(\"1st line:\", fileObj.readline())\n",
    "print(\"2nd line:\", fileObj.readline())\n",
    "print(\"3rd line:\", fileObj.readline())\n",
    "print(\"4th line:\", fileObj.readline())\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read in all lines from a file as a list of strings containing the data from each line, use the `.readlines` method (though note that this will again read all data into memory)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all lines\n",
    "fileObj = open( \"data/datafile.txt\" )\n",
    "\n",
    "lines = fileObj.readlines()\n",
    "\n",
    "print(\"The file has\", len(lines), \"lines\")\n",
    "\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looping over the lines in a file is a very common operation and python lets you iterate over a file using a `for` loop just as if it were an array of strings. This does not read all data into memory at once, and so is much more efficient that reading the file with `.readlines` and then looping over the resulting list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# as an iterable\n",
    "fileObj = open( \"data/datafile.txt\" )\n",
    "\n",
    "for line in fileObj:\n",
    "    print(line.rstrip().upper())\n",
    "\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The with statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important that files are closed when they are no longer required, but writing ``fileObj.close()`` is tedious (and more importantly, easy to forget). An alternative syntax is to open the files within a ``with`` statement, in which case the file will automatically be closed at the end of the `with` block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fileObj will be closed when leaving the block\n",
    "with open( \"data/datafile.txt\" ) as fileObj:\n",
    "    for ( i, line ) in enumerate( fileObj, start = 1 ):\n",
    "        print( i, line.strip() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises 2.3.1\n",
    "\n",
    "Write a script that reads a file containing many lines of nucleotide sequence. For each line in the file, print out the line number, the length of the sequence and the sequence (There is an example file <a href=\"http://www.ebi.ac.uk/~grsr/perl/dna.txt\">here</a> or in `data/dna.txt` from the course materials )."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing to files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a file has been opened for writing, you can use the `.write()` method on a file object to write data to the file.\n",
    "\n",
    "The argument to the `.write()` method must be a string, so if you want to write out numerical data to a file you will have to convert it to a string somehow beforehand."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-warning\">**Remember** to include a newline character `\\n` to separate lines of your output, unlike the `print()` statement, `.write()` does not include this by default.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_counts = {\n",
    "    'BRCA2': 43234,\n",
    "    'FOXP2': 3245,\n",
    "    'SORT1': 343792\n",
    "}\n",
    "\n",
    "with open( \"out.txt\", \"w\" ) as output:\n",
    "    output.write(\"GENE\\tREAD_COUNT\\n\")\n",
    "\n",
    "    for gene in read_counts:\n",
    "        line = \"\\t\".join( [ gene, str(read_counts[gene]) ] )\n",
    "        output.write(line + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the output file, open a terminal window, go to the directory where the file has been written, and print the content of the file using `cat` command or open it using your favourite editor:\n",
    "\n",
    "```bash\n",
    "cat out.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be cautious when opening a file for writing, as python will happily let you overwrite any existing data in the file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises 2.3.2\n",
    "\n",
    "Create a script that writes the values of a list of numbers to a file, with each number on a seperate line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next session\n",
    "\n",
    "Go to our next notebook: [python_basic_2_4](python_basic_2_4.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}