13-Advanced Python Modules/05-Regular Expressions - re.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regular Expressions\n",
    "\n",
    "Regular expressions are text-matching patterns described with a formal syntax. You'll often hear regular expressions referred to as 'regex' or 'regexp' in conversation. Regular expressions can include a variety of rules, from finding repetition, to text-matching, and much more. As you advance in Python you'll see that a lot of your parsing problems can be solved with regular expressions (they're also a common interview question!).\n",
    "\n",
    "\n",
    "If you're familiar with Perl, you'll notice that the syntax for regular expressions are very similar in Python. We will be using the <code>re</code> module with Python for this lecture.\n",
    "\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for Patterns in Text\n",
    "\n",
    "One of the most common uses for the re module is for finding patterns in text. Let's do a quick example of using the search method in the re module to find some text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching for \"term1\" in:\n",
      " \"This is a string with term1, but it does not have the other term.\"\n",
      "\n",
      "Match was found. \n",
      "\n",
      "Searching for \"term2\" in:\n",
      " \"This is a string with term1, but it does not have the other term.\"\n",
      "\n",
      "No Match was found.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "# List of patterns to search for\n",
    "patterns = ['term1', 'term2']\n",
    "\n",
    "# Text to parse\n",
    "text = 'This is a string with term1, but it does not have the other term.'\n",
    "\n",
    "for pattern in patterns:\n",
    "    print('Searching for \"%s\" in:\\n \"%s\"\\n' %(pattern,text))\n",
    "    \n",
    "    #Check for match\n",
    "    if re.search(pattern,text):\n",
    "        print('Match was found. \\n')\n",
    "    else:\n",
    "        print('No Match was found.\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we've seen that <code>re.search()</code> will take the pattern, scan the text, and then return a **Match** object. If no pattern is found, **None** is returned. To give a clearer picture of this match object, check out the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_sre.SRE_Match"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# List of patterns to search for\n",
    "pattern = 'term1'\n",
    "\n",
    "# Text to parse\n",
    "text = 'This is a string with term1, but it does not have the other term.'\n",
    "\n",
    "match = re.search(pattern,text)\n",
    "\n",
    "type(match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This **Match** object returned by the search() method is more than just a Boolean or None, it contains information about the match, including the original input string, the regular expression that was used, and the location of the match. Let's see the methods we can use on the match object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show start of match\n",
    "match.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "27"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show end\n",
    "match.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split with regular expressions\n",
    "\n",
    "Let's see how we can split with the re syntax. This should look similar to how you used the split() method with strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['What is the domain name of someone with the email: hello', 'gmail.com']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Term to split on\n",
    "split_term = '@'\n",
    "\n",
    "phrase = 'What is the domain name of someone with the email: hello@gmail.com'\n",
    "\n",
    "# Split the phrase\n",
    "re.split(split_term,phrase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how <code>re.split()</code> returns a list with the term to split on removed and the terms in the list are a split up version of the string. Create a couple of more examples for yourself to make sure you understand!\n",
    "\n",
    "## Finding all instances of a pattern\n",
    "\n",
    "You can use <code>re.findall()</code> to find all the instances of a pattern in a string. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['match']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Returns a list of all matches\n",
    "re.findall('match','test phrase match is in middle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## re Pattern Syntax\n",
    "\n",
    "This will be the bulk of this lecture on using re with Python. Regular expressions support a huge variety of patterns beyond just simply finding where a single string occurred. \n",
    "\n",
    "We can use *metacharacters* along with re to find specific types of patterns. \n",
    "\n",
    "Since we will be testing multiple re syntax forms, let's create a function that will print out results given a list of various regular expressions and a phrase to parse:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def multi_re_find(patterns,phrase):\n",
    "    '''\n",
    "    Takes in a list of regex patterns\n",
    "    Prints a list of all matches\n",
    "    '''\n",
    "    for pattern in patterns:\n",
    "        print('Searching the phrase using the re check: %r' %(pattern))\n",
    "        print(re.findall(pattern,phrase))\n",
    "        print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repetition Syntax\n",
    "\n",
    "There are five ways to express repetition in a pattern:\n",
    "\n",
    "   1. A pattern followed by the meta-character <code>*</code> is repeated zero or more times. \n",
    "   2. Replace the <code>*</code> with <code>+</code> and the pattern must appear at least once. \n",
    "   3. Using <code>?</code> means the pattern appears zero or one time. \n",
    "   4. For a specific number of occurrences, use <code>{m}</code> after the pattern, where **m** is replaced with the number of times the pattern should repeat. \n",
    "   5. Use <code>{m,n}</code> where **m** is the minimum number of repetitions and **n** is the maximum. Leaving out **n** <code>{m,}</code> means the value appears at least **m** times, with no maximum.\n",
    "    \n",
    "Now we will see an example of each of these using our multi_re_find function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching the phrase using the re check: 'sd*'\n",
      "['sd', 'sd', 's', 's', 'sddd', 'sddd', 'sddd', 'sd', 's', 's', 's', 's', 's', 's', 'sdddd']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: 'sd+'\n",
      "['sd', 'sd', 'sddd', 'sddd', 'sddd', 'sd', 'sdddd']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: 'sd?'\n",
      "['sd', 'sd', 's', 's', 'sd', 'sd', 'sd', 'sd', 's', 's', 's', 's', 's', 's', 'sd']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: 'sd{3}'\n",
      "['sddd', 'sddd', 'sddd', 'sddd']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: 'sd{2,3}'\n",
      "['sddd', 'sddd', 'sddd', 'sddd']\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_phrase = 'sdsd..sssddd...sdddsddd...dsds...dsssss...sdddd'\n",
    "\n",
    "test_patterns = [ 'sd*',     # s followed by zero or more d's\n",
    "                'sd+',          # s followed by one or more d's\n",
    "                'sd?',          # s followed by zero or one d's\n",
    "                'sd{3}',        # s followed by three d's\n",
    "                'sd{2,3}',      # s followed by two to three d's\n",
    "                ]\n",
    "\n",
    "multi_re_find(test_patterns,test_phrase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Character Sets\n",
    "\n",
    "Character sets are used when you wish to match any one of a group of characters at a point in the input. Brackets are used to construct character set inputs. For example: the input <code>[ab]</code> searches for occurrences of either **a** or **b**.\n",
    "Let's see some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching the phrase using the re check: '[sd]'\n",
      "['s', 'd', 's', 'd', 's', 's', 's', 'd', 'd', 'd', 's', 'd', 'd', 'd', 's', 'd', 'd', 'd', 'd', 's', 'd', 's', 'd', 's', 's', 's', 's', 's', 's', 'd', 'd', 'd', 'd']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: 's[sd]+'\n",
      "['sdsd', 'sssddd', 'sdddsddd', 'sds', 'sssss', 'sdddd']\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_phrase = 'sdsd..sssddd...sdddsddd...dsds...dsssss...sdddd'\n",
    "\n",
    "test_patterns = ['[sd]',    # either s or d\n",
    "                's[sd]+']   # s followed by one or more s or d\n",
    "\n",
    "multi_re_find(test_patterns,test_phrase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It makes sense that the first input <code>[sd]</code> returns every instance of s or d. Also, the second input <code>s[sd]+</code> returns any full strings that begin with an s and continue with s or d characters until another character is reached."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exclusion\n",
    "\n",
    "We can use <code>^</code> to exclude terms by incorporating it into the bracket syntax notation. For example: <code>[^...]</code> will match any single character not in the brackets. Let's see some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_phrase = 'This is a string! But it has punctuation. How can we remove it?'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use <code>[^!.? ]</code> to check for matches that are not a !,.,?, or space. Add a <code>+</code> to check that the match appears at least once. This basically translates into finding the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['This',\n",
       " 'is',\n",
       " 'a',\n",
       " 'string',\n",
       " 'But',\n",
       " 'it',\n",
       " 'has',\n",
       " 'punctuation',\n",
       " 'How',\n",
       " 'can',\n",
       " 'we',\n",
       " 'remove',\n",
       " 'it']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('[^!.? ]+',test_phrase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Character Ranges\n",
    "\n",
    "As character sets grow larger, typing every character that should (or should not) match could become very tedious. A more compact format using character ranges lets you define a character set to include all of the contiguous characters between a start and stop point. The format used is <code>[start-end]</code>.\n",
    "\n",
    "Common use cases are to search for a specific range of letters in the alphabet. For instance, <code>[a-f]</code> would return matches with any occurrence of letters between a and f. \n",
    "\n",
    "Let's walk through some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching the phrase using the re check: '[a-z]+'\n",
      "['his', 'is', 'an', 'example', 'sentence', 'ets', 'see', 'if', 'we', 'can', 'find', 'some', 'letters']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '[A-Z]+'\n",
      "['T', 'L']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '[a-zA-Z]+'\n",
      "['This', 'is', 'an', 'example', 'sentence', 'Lets', 'see', 'if', 'we', 'can', 'find', 'some', 'letters']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '[A-Z][a-z]+'\n",
      "['This', 'Lets']\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "test_phrase = 'This is an example sentence. Lets see if we can find some letters.'\n",
    "\n",
    "test_patterns=['[a-z]+',      # sequences of lower case letters\n",
    "               '[A-Z]+',      # sequences of upper case letters\n",
    "               '[a-zA-Z]+',   # sequences of lower or upper case letters\n",
    "               '[A-Z][a-z]+'] # one upper case letter followed by lower case letters\n",
    "                \n",
    "multi_re_find(test_patterns,test_phrase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Escape Codes\n",
    "\n",
    "You can use special escape codes to find specific types of patterns in your data, such as digits, non-digits, whitespace, and more. For example:\n",
    "\n",
    "<table border=\"1\" class=\"docutils\">\n",
    "<colgroup>\n",
    "<col width=\"14%\" />\n",
    "<col width=\"86%\" />\n",
    "</colgroup>\n",
    "<thead valign=\"bottom\">\n",
    "<tr class=\"row-odd\"><th class=\"head\">Code</th>\n",
    "<th class=\"head\">Meaning</th>\n",
    "</tr>\n",
    "</thead>\n",
    "<tbody valign=\"top\">\n",
    "<tr class=\"row-even\"><td><tt class=\"docutils literal\"><span class=\"pre\">\\d</span></tt></td>\n",
    "<td>a digit</td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><tt class=\"docutils literal\"><span class=\"pre\">\\D</span></tt></td>\n",
    "<td>a non-digit</td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><tt class=\"docutils literal\"><span class=\"pre\">\\s</span></tt></td>\n",
    "<td>whitespace (tab, space, newline, etc.)</td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><tt class=\"docutils literal\"><span class=\"pre\">\\S</span></tt></td>\n",
    "<td>non-whitespace</td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><tt class=\"docutils literal\"><span class=\"pre\">\\w</span></tt></td>\n",
    "<td>alphanumeric</td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><tt class=\"docutils literal\"><span class=\"pre\">\\W</span></tt></td>\n",
    "<td>non-alphanumeric</td>\n",
    "</tr>\n",
    "</tbody>\n",
    "</table>\n",
    "\n",
    "Escapes are indicated by prefixing the character with a backslash <code>\\</code>. Unfortunately, a backslash must itself be escaped in normal Python strings, and that results in expressions that are difficult to read. Using raw strings, created by prefixing the literal value with <code>r</code>, eliminates this problem and maintains readability.\n",
    "\n",
    "Personally, I think this use of <code>r</code> to escape a backslash is probably one of the things that block someone who is not familiar with regex in Python from being able to read regex code at first. Hopefully after seeing these examples this syntax will become clear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching the phrase using the re check: '\\\\d+'\n",
      "['1233']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '\\\\D+'\n",
      "['This is a string with some numbers ', ' and a symbol #hashtag']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '\\\\s+'\n",
      "[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '\\\\S+'\n",
      "['This', 'is', 'a', 'string', 'with', 'some', 'numbers', '1233', 'and', 'a', 'symbol', '#hashtag']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '\\\\w+'\n",
      "['This', 'is', 'a', 'string', 'with', 'some', 'numbers', '1233', 'and', 'a', 'symbol', 'hashtag']\n",
      "\n",
      "\n",
      "Searching the phrase using the re check: '\\\\W+'\n",
      "[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' #']\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_phrase = 'This is a string with some numbers 1233 and a symbol #hashtag'\n",
    "\n",
    "test_patterns=[ r'\\d+', # sequence of digits\n",
    "                r'\\D+', # sequence of non-digits\n",
    "                r'\\s+', # sequence of whitespace\n",
    "                r'\\S+', # sequence of non-whitespace\n",
    "                r'\\w+', # alphanumeric characters\n",
    "                r'\\W+', # non-alphanumeric\n",
    "                ]\n",
    "\n",
    "multi_re_find(test_patterns,test_phrase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "You should now have a solid understanding of how to use the regular expression module in Python. There are a ton of more special character instances, but it would be unreasonable to go through every single use case. Instead take a look at the full [documentation](https://docs.python.org/3/library/re.html#regular-expression-syntax) if you ever need to look up a particular pattern.\n",
    "\n",
    "You can also check out the nice summary tables at this [source](http://www.tutorialspoint.com/python/python_reg_expressions.htm).\n",
    "\n",
    "Good job!\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}