01-01-regex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Regular Expressions\n",
    "\n",
    "A regular expression, regex or regexp (sometimes called a rational expression) is, in theoretical computer science and formal language theory, a sequence of characters that define a search pattern. Usually this pattern is then used by `string` searching algorithms for \"find\" or \"find and replace\" operations on strings, or for input validation.\n",
    "\n",
    "The concept arose in the 1950s when the American mathematician Stephen Cole Kleene formalized the description of a regular language. The concept came into common use with `Unix` text-processing utilities. Since the 1980s, different syntaxes for writing regular expressions exist, one being the POSIX standard and another, widely used, being the Perl syntax."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regex in Python\n",
    "\n",
    "python provide module for regex called `re`\n",
    "\n",
    "we are going to use the `findall` and `sub` functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-07T11:21:00.622234Z",
     "start_time": "2021-11-07T11:21:00.609972Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my name is Ali'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "text = \"my name is Ahmed\"\n",
    "\n",
    "re.sub(\"Ahmed\", \"Ali\", text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-07T11:21:04.652769Z",
     "start_time": "2021-11-07T11:21:04.649010Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Ahmed']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"Ahmed\", text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "here we used normal text for substitue and search, we can use a regex though !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Special Characters (Regular Expressions)\n",
    "\n",
    "some characters have special functions and are not just character, for example the `\\n` which indicate a newline and the `\\t` which is a tap space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Basic patterns that match single chars\n",
    "\n",
    "| Character  | function |\n",
    "| ------------- | ------------- |\n",
    "| a-z, 0-9  | ordinary characters just match themselves exactly.|\n",
    "| . (dot)  | matches any single character except newline '\\n'  |\n",
    "| \\w | matches a \"word\" character: a letter or digit or underbar [a-zA-Z0-9_] |\n",
    "| \\W | matches any non-word character |\n",
    "| \\b | boundary between word and non-word |\n",
    "| \\s | matches a single whitespace character -- space, newline, return, tab |\n",
    "| \\S | matches any non-whitespace character |\n",
    "| \\t, \\n, \\r | tab, newline, return |\n",
    "| \\d | decimal digit [0-9] |\n",
    "| ^ | matches start of the string |\n",
    "| $ | match the end of the string |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Let's mix them with normal characters\n",
    "\n",
    "> note that we use `r` before the pattern string to let python know not to parse them, for example not to take \\n and replace it by newline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['regular']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"\"\"regular expression is a special sequence of characters \\\n",
    "that helps you match or find other strings or sets of strings, \\\n",
    "using regular expression pattern. regular expressions are widely used in UNIX world.\"\"\"\n",
    "\n",
    "re.findall(r\"^regular\", text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Replace the first regular to a title case one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Regular expression is a special sequence of characters that helps you match or find other strings or sets of strings, using regular expression pattern. regular expressions are widely used in UNIX world.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"\"\"regular expression is a special sequence of characters \\\n",
    "that helps you match or find other strings or sets of strings, \\\n",
    "using regular expression pattern. regular expressions are widely used in UNIX world.\"\"\"\n",
    "\n",
    "re.sub(r\"^regular\", \"Regular\", text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lab 1 : Ends with number\n",
    "\n",
    "using regular expression write a script that check if a string ends with a numbe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "enter a string:it's 2020\n",
      "string ends with a number\n"
     ]
    }
   ],
   "source": [
    "text = input(\"enter a string:\").strip()\n",
    "\n",
    "result = re.findall(r\"[0-9]$\", text)\n",
    "\n",
    "if result:\n",
    "    print(\"string ends with a number\")\n",
    "else:\n",
    "    print(\"it is not\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lab 2 : Only Valid text\n",
    "\n",
    "write a script using regular expression to check if user input consists of only **`alphabet letters`**, **`number`** and **`_`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-07T11:27:15.073066Z",
     "start_time": "2021-11-07T11:27:11.409811Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "enter your username:user_123\n",
      "username user_123 is valid\n"
     ]
    }
   ],
   "source": [
    "username = input(\"enter your username:\").strip()\n",
    "\n",
    "invalid_input = re.findall(r\"\\W\", username)\n",
    "\n",
    "if invalid_input:\n",
    "    print(\"invalid username\")\n",
    "else:\n",
    "    print(\"username {} is valid\".format(username))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Building a bigger regular expression\n",
    "\n",
    "you can mix and match multiple expressions and have more than one instances of them\n",
    "\n",
    "| Example  | description |\n",
    "| --- | --- |\n",
    "| [Pp]ython | Match \"Python\" or \"python\" |\n",
    "| rub[ye] | Match \"ruby\" or \"rube\" |\n",
    "| [aeiou] | Match any one lowercase vowel |\n",
    "| [0-9] | Match any digit; same as [0123456789] |\n",
    "| [a-z] | Match any lowercase ASCII letter |\n",
    "| [A-Z] | Match any uppercase ASCII letter |\n",
    "| [a-zA-Z0-9] | Match any of the defined |\n",
    "| [^aeiou] | Match anything other than a lowercase vowel |\n",
    "| [^0-9] | Match anything other than a digit |\n",
    "\n",
    "> we can use **OR** to use multiple regex together.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-07T11:30:13.589177Z",
     "start_time": "2021-11-07T11:30:13.576608Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "talking about python\n",
      "talking about python\n",
      "talking about python\n",
      "something else\n"
     ]
    }
   ],
   "source": [
    "texts = [\n",
    "  \"python is a great language\",\n",
    "  \"i lov to write in py\",\n",
    "  \"what a cool language Python is\",\n",
    "  \"the pyramids of giza are so huge!\"\n",
    "]\n",
    "\n",
    "\n",
    "for text in texts:\n",
    "    python_detected = re.findall(r\"[Pp]ython|\\b[Pp]y\\b\", text)\n",
    "    if python_detected:\n",
    "        print(\"talking about python\")\n",
    "    else:\n",
    "        print(\"something else\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Repetition Cases\n",
    "\n",
    "| Example | description |\n",
    "| --- | --- |\n",
    "| ruby? | Match \"rub\" or \"ruby\": the y is optional |\n",
    "| ruby* | Match \"rub\" plus 0 or more y(s) |\n",
    "| ruby+ | Match \"rub\" plus 1 or more y(s) |\n",
    "| \\d{3} | Match exactly 3 digits |\n",
    "| \\d{3,} | Match 3 or more digits |\n",
    "| \\d{3,5} | Match 3, 4, or 5 digits |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2018']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"it's 2018, happy new year!\"\n",
    "\n",
    "re.findall(r\"\\d{4}\", text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Let's replace numbers with NUM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is a text tweet that containsmultiple numbers  PHONE ,  NUM \n"
     ]
    }
   ],
   "source": [
    "text = \"this is a text tweet that contains\\\n",
    "multiple numbers 01112131411, 012121212\"\n",
    "\n",
    "new_text = re.sub(r\"\\d{11}\", \" PHONE \", text)\n",
    "new_text = re.sub(r\"\\d+\", \" NUM \", new_text)\n",
    "print(new_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Search groups\n",
    "\n",
    "you can create a search groups with regex and retrieve each one with `search()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-07T11:31:47.693666Z",
     "start_time": "2021-11-07T11:31:47.679953Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "support@datacamp.com\n",
      "support\n",
      "datacamp.com\n"
     ]
    }
   ],
   "source": [
    "email_address = 'Please contact us at: support@datacamp.com'\n",
    "match = re.search(r'([\\w\\.-]+)@([\\w\\.-]+)', email_address)\n",
    "if match:\n",
    "    print(match.group()) # The whole matched text\n",
    "    print(match.group(1)) # The username (group 1)\n",
    "    print(match.group(2)) # The host (group 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `match` vs `search`\n",
    "\n",
    "The `match()` function checks for a match only at the beginning of the string whereas the `search()` function checks for a match anywhere in the string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The `re.compile()`\n",
    "\n",
    "we can compile a regular expression instead of writing it multiple time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['example.name@company.org']\n",
      "['my-email55@yahoo.com']\n",
      "['shortmail@long-company.net']\n"
     ]
    }
   ],
   "source": [
    "mail_re = re.compile(r\"[\\w\\.-]+@[\\w\\.-]+\\.[\\w\\.-]+\")\n",
    "\n",
    "mails = [\n",
    "    \"this is a message with an email of:example.name@company.org\",\n",
    "    \"my-email55@yahoo.com is the email you would like to use\",\n",
    "    \"send me an email at:shortmail@long-company.net\"\n",
    "]\n",
    "\n",
    "for mail in mails:\n",
    "    print(mail_re.findall(mail))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lab 3 : Extract Hashtags\n",
    "\n",
    "Write a script to extract a hashtag from tweet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-07T11:34:39.185590Z",
     "start_time": "2021-11-07T11:34:39.182456Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['#HASHTAG', '#cool', '#هاشتاج']\n"
     ]
    }
   ],
   "source": [
    "tweet = \"a tweet with no hashtag, but a #HASHTAG and another #cool one #هاشتاج\"\n",
    "\n",
    "hashtags = re.findall(r\"#\\S+\", tweet)\n",
    "\n",
    "if hashtags:\n",
    "    print(hashtags)\n",
    "else:\n",
    "    print(\"no hashtags found\")"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}