BADAS_texteditors_regex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Regular Expressions\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A regular expression (aka regex) is a sequence of characters that describe or match a pattern of text. For example, the string **aabb12** could be described as *aabb12*, two a's, *two'bs, then 1, then 2*, or *four letters followed by two numbers*. \n",
    "\n",
    "Keep in mind that regular expressions differ slightly per language but the theort and application remain the same. We will start with Python.\n",
    "\n",
    "## Metacharacters\n",
    "\n",
    "Metacharacters are characters that have an alternate meaning rather than a literal meaning. There are many of these and they can be found on the [Python regular expression documentation](https://docs.python.org/2/library/re.html#regular-expression-syntax).\n",
    "\n",
    "For example, we'll take the following characters to search a text file:\n",
    "- 'A'\n",
    "- '.'\n",
    "- '$'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "text = 'Alan is the coolest man. Ever!$$$'\n",
    "\n",
    "# compiling a regular expression allows you to reuse the regular expression.\n",
    "A = re.compile('A')\n",
    "dot = re.compile('.')\n",
    "dollar = re.compile('$')\n",
    "\n",
    "# Any pattern that is matches will be presented as an item within a list.\n",
    "A.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It matches 'A'! No surprise here. Now let's see what the '.' does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dot.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It matched everything?! That's because the '.' is a special character that we will get into later.\n",
    "\n",
    "Now let's see what the dollar sign finds us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dollar.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nope, not a bug: '$' did match something but it is also a special character."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Position Metacharacters\n",
    "\n",
    "This type of regular expression is used to match characters based on where they are located as opposed to what the character means.\n",
    "\n",
    "Let's take the `$` for example. I told you that it matched something, but nothing was returned in the output. That is because `$` is a special character that matches the end of a line. Let's try an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'Alan is the coolest man ever, man'\n",
    "\n",
    "# In regular expressions, backslashes tell the engine to interpret metacharacters as literals.\n",
    "pattern = re.compile('man')\n",
    "\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is no surprise, it found both instances of 'man'. Now, let's say we only want to capture 'man' at the end of a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern = re.compile('man$')\n",
    "\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trust me, this is grabbing the last man. Don't believe me?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text = 'Alan is the coolest man ever'\n",
    "\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Told ya. Now, what happens if we actually do want to match a dollar sign? Well, we can use an escape character, `\\` (backslash), before the dollar sign to tell the regex interpreter to use the literal meaning. Let's use the first example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text = 'Alan is the coolest man. Ever!$$$'\n",
    "\n",
    "pattern = re.compile('$')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nothing. Let's try to escape it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern = re.compile('\\$')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, what do you think this regular expression do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern = re.compile('\\$$')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Similarly, '^' (caret, but not the kind that rabbits eat) matches the beginning of a line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# IGNORECASE tells the intepreter match the meaning of the character, not character and case\n",
    "pattern = re.compile('a', re.IGNORECASE)\n",
    "\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now with the caret:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern = re.compile('^a', re.IGNORECASE)\n",
    "\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "There are also boundary metacharacters. For example, '\\b' matches a word that ends with 'ing'. Inversely, '\\B' matches a non-boundary word, so it would match the 'ing' in 'things' but not 'thing'. This is useful for specifying substrings or whole words exclusively.\n",
    "\n",
    "NOTE: some characters are treated as literals even when they are backslached. See here for more deltails: \n",
    "https://stackoverflow.com/questions/2241600/python-regex-r-prefix\n",
    "https://stackoverflow.com/questions/21104476/what-does-the-r-in-pythons-re-compiler-pattern-flags-mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'pistol'\n",
    "\n",
    "# 'is' is surrounded by non-blank characters, not boundaries\n",
    "pattern = re.compile(r'\\bis\\b')\n",
    "print('word boundary',pattern.findall(text))\n",
    "\n",
    "# However, it is surrounded by non-boundary characters, so it is found by non-word boundary searches\n",
    "pattern = re.compile('\\Bis\\B')\n",
    "print('non-word boundary',pattern.findall(text))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'is'\n",
    "\n",
    "pattern = re.compile(r'\\bis\\b')\n",
    "print('word boundary', pattern.findall(text))\n",
    "\n",
    "pattern = re.compile('\\Bis\\B')\n",
    "print('non-word boundary', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Single Metacharacters\n",
    "\n",
    "These metacharacters match specific types of charactes. For example, you can match all alphanumeric characters with '\\w' or any whitespace character with '\\s'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = '123 abc !@#'\n",
    "\n",
    "pattern = re.compile('\\d')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# any alphanumeric character\n",
    "pattern = re.compile('\\w')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# any non-word character\n",
    "pattern = re.compile('\\W')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# any non-newline character\n",
    "pattern = re.compile('.')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Quantifiers\n",
    "\n",
    "All examples before have been searching for individual characters. Quanitfiers allow you to match repeated patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'aa bb cdef 123'\n",
    "\n",
    "# this looks for every instance of a word character\n",
    "pattern = re.compile('\\w')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# the '+' tells the interpreter to look for one or more consecutive characters\n",
    "pattern = re.compile('\\w+')\n",
    "pattern.findall(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# '?' is looking for characters that appear once or not at all. It basically makes the preceding character optional.\n",
    "text = 'colour'\n",
    "\n",
    "pattern = re.compile('colou?r')\n",
    "print('colour',pattern.findall(text))\n",
    "\n",
    "text = 'color'\n",
    "print('color', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Asterisk is doesn't require the preceding character to be there, but if it is it will match it\n",
    "# will repeat the pattern as many times as is can.\n",
    "text = 'b'\n",
    "pattern = re.compile('bo*')\n",
    "print('b',pattern.findall(text))\n",
    "\n",
    "text = 'boo'\n",
    "print('boo',pattern.findall(text))\n",
    "\n",
    "text = 'boooo!'\n",
    "print('boooo!', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# curly brackets define a number of times in which the preceding pattern will be repeated\n",
    "pattern = re.compile('bo{2}')\n",
    "text = 'boo'\n",
    "print('boo',pattern.findall(text))\n",
    "\n",
    "text = 'boooo!'\n",
    "print('boooo!', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Character Classes\n",
    "\n",
    "A character class allows you to match a particular set of user defined characters as oppsed to the predefined metacharacters we went over previously. Think of searching for any vowel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'I like chocolate'\n",
    "\n",
    "pattern = re.compile('[AEIOUY]', re.IGNORECASE)\n",
    "print('vowels', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Alternatively, **when you use a caret `^` within the square brackets** it will match the **inverse** of those defined in the character class. In this case, it will match all consonants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "pattern = re.compile('[^AEIOUY]', re.IGNORECASE)\n",
    "print('consonants', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can also specify ranges of characters if they are naturally consecutive by using a hyphen `-` to separate the beginning and the end:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "pattern = re.compile('[a-d]', re.IGNORECASE)\n",
    "print('pattern', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Alterations\n",
    "\n",
    "This is essentially just an 'or' statement. An example would be if you are looking for whether a sentence says 'we have ten dollars.' or 'I have ten dollars.' Since the only varying piece of that sentence are the pronouns, you can try to match either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'I have ten dollars'\n",
    "\n",
    "pattern = re.compile('we|i|they', re.IGNORECASE)\n",
    "print('I', pattern.findall(text))\n",
    "\n",
    "text = 'They have ten dollars'\n",
    "print('They', pattern.findall(text))\n",
    "\n",
    "text = 'We have ten dollars'\n",
    "print('We', pattern.findall(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Backreferences\n",
    "\n",
    "Back references/captures allow you to reuse regular expressions and/or patterns that match that regular expression. \n",
    "\n",
    "Let's give a biological example:\n",
    "You're looking for a motif that has flanking restriction sites ACTG. The motif can be of any length and any composition of nucleotides but is always flanked by those cuts sites:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'ACTGTTTTTTTTTACTG'\n",
    "\n",
    "# the '\\1' is the refence to the first captured pattern.\n",
    "# All patterns are ennumerated but can also be named.\n",
    "print('matches:',re.search(r'(ACTG)([ACTG]+)\\1', text).groups())\n",
    "\n",
    "text = 'ATCGCAGCTACGACTGAAAAAAAAAAAAAAACTG'\n",
    "print('matches:',re.search(r'(ACTG)([ACTG]+)\\1', text).groups())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Substitution Mode\n",
    "\n",
    "There are several ways that one can use regular expressions, though the two main modes are searching and substituting.\n",
    "\n",
    "Searching/matching will only look for whether or not a pattern is matched. Substituting will replace any pattern that is matched with another pattern. Above we have used only searching, so below will only showcase an example of a substitution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = 'ACTGTTTTTTTTTACTG'\n",
    "re.sub(r'ACTG', 'AAAA', text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Command Line Examples\n",
    "\n",
    "\n",
    "## Constructing a Regular Expression\n",
    "\n",
    "Arguably the most difficult task with regular expressions is being able to construct one. Often in cases I am trying extract information that match multiple different patterns or the same pattern but in different lines of text. **In order to create a regular expression, you first need to identify the pattern**. This generally requires some brief understanding of the file format and the relevant information as well as finding a pattern that applies to information that you're interested in.\n",
    "\n",
    "To give you some insight on how to do this, the examples below are scenarios that I often find myself in that are a perfect fit for regular expressions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing a GFF file\n",
    "\n",
    "There are a few tools, such as sed, awk, and grep, that are used for text munging but I happen to use Perl, as it provides a lot of flexibility when doing complex regular expressions and other data munging tasks.\n",
    "\n",
    "grep can only match patterns (or the inverse of patterns) and cannot be used to replace or transform text. Though limited, it is a nice tool to have for quick searches in files or across filesystems. In this example, all I need is to match a pattern, so grep will suffice.\n",
    "\n",
    "A GFF file is a standard tab-delimited file format for genomic annotations. Most gff files contain every type of annotated feature for that organism (mRNA, exons, UTRs, motifs, etc.) which are not always relevant to the analysis and may sometimes may actuall interfere with the analysis.\n",
    "\n",
    "Let's say **I want to extract all annotated mRNAs** from this file. First thing is to understand the [gff file format](https://www.ensembl.org/info/website/upload/gff.html). Reading the documentation will tell you that it is tab delimited and the third column contains the feature type. So looking at the file will sho you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! head -n 20 caenorhabditis_elegans.PRJNA13758.WBPS9.annotations.gff3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Looking at the entire file is daunting and doesn't provide a list of features found within this file. So, we can use a few commands to get a full representation of each feature type described in the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! cut -f3 caenorhabditis_elegans.PRJNA13758.WBPS9.annotations.gff3 | sort | uniq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there are a lot of *RNA*s here, and there happen to be two different *mRNA* feature types. So, now that we know the feature type and the format we can construct our regular expression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The -P tells grep to use perl regular expressions which make things a bit easier otherwise you'll have\n",
    "# to escape some characters!\n",
    "\n",
    "# The pattern we're looking for is mRNA surrounded by tabs/white space\n",
    "! grep -P '\\tmRNA\\t' caenorhabditis_elegans.PRJNA13758.WBPS9.annotations.gff3 | head -n 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Renaming FastQ Files\n",
    "\n",
    "The filenames for sequences are generally too many characters an too much information, which can be annoying when working with those files. To extract only the relevant information and shorten the filename I often construct a regular expression, coupled with some bash commands, to rename the files how I deem fit.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ls /home/at120/badas/2015-03-11_AE52Y-redo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a lot of information here, some of which is intelligible only to Tubo. The only information that I would like to retain are the sample names, consisting of **one lowercase letter followed by one number** and the pairing information, which is **n0 and then a 1 or a 2**.\n",
    "\n",
    "What I usually do to get started is to work with one filename and construct the pattern from that. Regular expressions are read from left to right, so we should start by grabbing the first bit of information we want, which is the pair information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo \"000000000-AE52Y l01n02 flu mrt-pcr updated f3.34200000008543.fastq.gz\" | perl -pe 's/^.+n0([12]).+/$1/g'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The syntax for a perl substitution regular expression is as follows:\n",
    "\n",
    "`perl -pe '<mode>/<pattern>/<replacement text>/<modifier>'`\n",
    "\n",
    "*mode* can be:\n",
    "- substitution `s`\n",
    "- match `m`\n",
    "- translate `tr`\n",
    "\n",
    "*pattern* is your regular expression\n",
    "*replacement text* is the text to replace text that matches your regular expression\n",
    "\n",
    "*modifier* can be:\n",
    "- ignore case `i`\n",
    "- global `g`: will find all patterns, not just the first\n",
    "\n",
    "and some others. Take a look at the [quick reference](http://www.erudil.com/preqr.pdf) for perl regular expressions for more information.\n",
    "\n",
    "Next would be to grab the sample name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# It's good to be very specific with regular expressions for multiple reasons.\n",
    "# 1) It limits the scope of the regular expression\n",
    "# 2) It makes it easier to read later.\n",
    "# So I do not NEED to specify 'flu' but it makes for fewer instructions and is more human readable.\n",
    "\n",
    "echo \"000000000-AE52Y l01n02 flu mrt-pcr updated f3.34200000008543.fastq.gz\" \\\n",
    "| perl -pe 's/^.+n0([12])\\sflu.+\\s([a-z]\\d)\\..+/$1$2/g'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, the sample name and file extension and rearrange the caputred patterns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo \"000000000-AE52Y l01n02 flu mrt-pcr updated f3.34200000008543.fastq.gz\" \\\n",
    "| perl -pe 's/^.+n0([12])\\sflu.+\\s([a-z]\\d)\\..+(fastq.gz)$/$2.r$1.$3/g'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to wrap it up in a for loop and invoke a copy command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd /home/at120/badas/2015-03-11_AE52Y-redo\n",
    "for i in *gz; do cp \"$i\" `echo $i | perl -pe 's/^.+n0([12])\\sflu.+\\s([a-z]\\d)\\..+(fastq.gz)$/$2.r$1.$3/g'` ; done\n",
    "\n",
    "ls"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "latex_envs": {
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}