Finding Text Patterns Without Regular Expressions

Say you want to find a US phone number in a string; you’re looking for three numbers, a hyphen, three numbers, a hyphen, and four numbers. Here’s an example: 415-555-4242.

Let’s write a function named is_phone_number() to check whether a string matches this pattern and return either True or False. Open a new file editor tab and enter the following code, then save the file as isPhoneNumber.py:

def is_phone_number(text):
  ❶ if len(text) != 12:  # Phone numbers have exactly 12 characters.
        return False
    for i in range(0, 3):  # The first three characters must be numbers.
      ❷ if not text[i].isdecimal():
            return False
  ❸ if text[3] != '-':  # The fourth character must be a dash.
        return False
    for i in range(4, 7): # The next three characters must be numbers.
      ❹ if not text[i].isdecimal():
            return False
  ❺ if text[7] != '-':  # The eighth character must be a dash.
        return False
    for i in range(8, 12):  # The next four characters must be numbers.
      ❻ if not text[i].isdecimal():
            return False
  ❼ return True

print('Is 415-555-4242 a phone number?', is_phone_number('415-555-4242'))
print(is_phone_number('415-555-4242'))
print('Is Moshi moshi a phone number?', is_phone_number('Moshi moshi'))
print(is_phone_number('Moshi moshi'))

When this program is run, the output looks like this:

Is 415-555-4242 a phone number?
True
Is Moshi moshi a phone number?
False

The is_phone_number() function has code that does several checks to determine whether the string in text is a valid phone number. If any of these checks fail, the function returns False. First, the code checks that the string is exactly 12 characters long ❶. Then, it checks that the area code (that is, the first three characters in text) consists of only numeric characters ❷ by calling the isdecimal() string method. The rest of the function checks that the string follows the pattern of a phone number: the number must have the first hyphen after the area code ❸, three more numeric characters ❹, another hyphen ❺, and finally, four more numeric characters ❻. If the program execution manages to get past all the checks, it returns True ❼.

Calling is_phone_number() with the argument '415-555-4242' will return True. Calling is_phone_number() with 'Moshi moshi' will return False; the first test fails because 'Moshi moshi' is not 12 characters long.

If you wanted to find a phone number within a larger string, you would have to add even more code to locate the pattern. Replace the last four print() function calls in isPhoneNumber.py with the following:

message = 'Call me at 415-555-1011 tomorrow. 415-555-9999 is my office.'
for i in range(len(message)):
  ❶ segment = message[i:i+12]
  ❷ if is_phone_number(segment):
        print('Phone number found: ' + segment)
print('Done')

When this program is run, the output will look like this:

Phone number found: 415-555-1011
Phone number found: 415-555-9999
Done

On each iteration of the for loop, a new segment of 12 characters from message is assigned to the variable segment ❶. For example, on the first iteration, i is 0, and segment is assigned message[0:12] (that is, the string 'Call me at 4'). On the next iteration, i is 1, and segment is assigned message[1:13] (the string 'all me at 41'). In other words, on each iteration of the for loop, segment takes on the following values

'Call me at 4'
'all me at 41'
'll me at 415'
'l me at 415-'

and so on, until its last value is 's my office.'

The loop’s code passes segment to is_phone_number() to check whether it matches the phone number pattern ❷, and if so, it prints the segment. Once it has finished going through message, we print Done.

While the string in message is short in this example, the program would run in less than a second even if it were millions of characters long. A similar program that finds phone numbers using regular expressions would also run in less than a second; however, regular expressions make writing these programs much quicker.

Finding Text Patterns with Regular Expressions

The previous phone number–finding program works, but it uses a lot of code to do something limited. The is_phone_number() function is 17 lines but can find only one phone number format. What about a phone number formatted like 415.555.4242 or (415) 555-4242? And what if the phone number had an extension, like 415-555-4242 x99? The is_phone_number() function would fail to find them. You could add yet more code for these additional patterns, but there is an easier way to tackle the problem.

Regular expressions, called regexes for short, are a sort of mini language that describes a pattern of text. For example, the characters \d in a regex stand for a decimal numeral between 0 and 9. Python uses the regex string r'\d\d\d-\d\d\d-\d\d\d\d' to match the same text pattern the previous is_phone_number() function did: a string of three numbers, a hyphen, three more numbers, another hyphen, and four numbers. Any other string would not match the r'\d\d\d-\d\d\d-\d\d\d\d' regex.

Regular expressions can be much more sophisticated than this one. For example, adding a numeral, such as 3, in curly brackets ({3}) after a pattern is like saying, “Match this pattern three times.” So the slightly shorter regex r'\d{3}-\d{3}-\d{4}' also matches the phone number pattern.

Note that we often write regex strings as raw strings, with the r prefix. This is useful, as regex strings often have backslashes. Without using raw strings, we would have to enter expressions such as '\\d'.

Before we cover all of the details of regular expression syntax, let’s go over how to use them in Python. We’ll stick with the example regular expression string r'\d{3}-\d{3}-\d{4}' used to find US phone numbers in a text string 'My number is 415-555-4242'. The general process of using regular expressions in Python involves four steps:

  1.  Import the re module.

  2.  Pass the regex string to re.compile() to get a Pattern object.

  3.  Pass the text string to the Pattern object’s search() method to get a Match object.

  4.  Call the Match object’s group() method to get the string of the matched text.

In the interactive shell, these steps look like this:

>>> import re
>>> phone_num_pattern_obj = re.compile(r'\d{3}-\d{3}-\d{4}')
>>> match_obj = phone_num_pattern_obj.search('My number is 415-555-4242.')
>>> match_obj.group()
'415-555-4242'

All regex functions in Python are in the re module. Most of the examples in this chapter will require the re module, so remember to import it at the beginning of the program. Otherwise, you’ll get a NameError: name 're' is not defined error message. As with importing any module, you need to import it only once per program or interactive shell session.

Passing the regular expression string to re.compile() returns a Pattern object. You only need to compile the Pattern object once; after that, you can call the Pattern object’s search() method for as many different text strings as you want.

A Pattern object’s search() method searches the string it is passed for any matches to the regex. The search() method will return None if the regex pattern isn’t found in the string. If the pattern is found, the search() method returns a Match object, which will have a group() method that returns a string of the matched text.

NOTE

While I encourage you to enter the example code into the interactive shell, you could also make use of web-based regular expression testers, which can show you exactly how a regex matches a piece of text that you enter. I recommend the testers at https://pythex.org and https://regex101.com. Different programming languages have slightly different regular expression syntax, so be sure to select the “Python” flavor on these websites.

The Syntax of Regular Expressions

Now that you know the basic steps for creating and finding regular expression objects using Python, you’re ready to learn the full range of regular expression syntax. In this section, you’ll learn how to group regular expression elements together with parentheses, escape special characters, match several alternative groups with the pipe character, and return all matches with the findall() method.

>>> import re
>>> phone_re = re.compile(r'(\d\d\d)-(\d\d\d-\d\d\d\d)')
>>> mo = phone_re.search('My number is 415-555-4242.')
>>> mo.group(1)  # Returns the first group of the matched text
'415'
>>> mo.group(2)  # Returns the second group of the matched text
'555-4242'
>>> mo.group(0)  # Returns the full matched text
'415-555-4242'
>>> mo.group()  # Also returns the full matched text
'415-555-4242'

>>> mo.groups()
('415', '555-4242')
>>> area_code, main_number = mo.groups()
>>> print(area_code)
415
>>> print(main_number)
555-4242

>>> pattern = re.compile(r'(\(\d\d\d\)) (\d\d\d-\d\d\d\d)')
>>> mo = pattern.search('My phone number is (415) 555-4242.')
>>> mo.group(1)
'(415)'
>>> mo.group(2)
'555-4242'

$ () * + - . ? [\] ^ {|}

\$ \(\) \* \+ \- \. \? \[\\ \] \^ \{\| \}

>>> import re
>>> re.compile(r'(\(Parentheses\)')
Traceback (most recent call last):
--snip--
re.error: missing), unterminated subpattern at position 0

>>> import re
>>> pattern = re.compile(r'Cat(erpillar|astrophe|ch|egory)')
>>> match = pattern.search('Catch me if you can.')
>>> match.group()
'Catch'
>>> match.group(1)
'ch'

>>> import re
>>> pattern = re.compile(r'\d{3}-\d{3}-\d{4}')  # This regex has no groups.
>>> pattern.findall('Cell: 415-555-9999 Work: 212-555-0000')
['415-555-9999', '212-555-0000']

>>> import re
>>> pattern = re.compile(r'(\d{3})-(\d{3})-(\d{4})')  # This regex has groups.
>>> pattern.findall('Cell: 415-555-9999 Work: 212-555-0000')
[('415', '555', '9999'), ('212', '555', '0000')]

>>> import re
>>> pattern = re.compile(r'\d{3}')
>>> pattern.findall('1234')
['123']
>>> pattern.findall('12345')
['123']
>>> pattern.findall('123456')
['123', '456']

Qualifier Syntax: What Characters to Match

Regular expressions are split into two parts: the qualifiers that dictate what characters you are trying to match followed by the quantifiers that dictate how many characters you are trying to match. In the r'\d{3}-\d{3}-\d{4}' phone number regex string example we’ve been using, the r'\d' and '-' parts are qualifiers and the '{3}' and '{4}' are quantifiers. Let’s now examine the syntax of qualifiers.

>>> import re
>>> vowel_pattern = re.compile(r'[aeiouAEIOU]')
>>> vowel_pattern.findall('RoboCop eats BABY FOOD.')
['o', 'o', 'o', 'e', 'a', 'A', 'O', 'O']

>>> import re
>>> consonant_pattern = re.compile(r'[^aeiouAEIOU]')
>>> consonant_pattern.findall('RoboCop eats BABY FOOD.')
['R', 'b', 'C', 'p', ' ', 't', 's', ' ', 'B', 'B', 'Y', ' ', 'F', 'D', '.']

>>> import re
>>> pattern = re.compile(r'\d+\s\w+')
>>> pattern.findall('12 drummers, 11 pipers, 10 lords, 9 ladies, 8 maids, 
7 swans, 6 geese, 5 rings, 4 birds, 3 hens, 2 doves, 1 partridge')
['12 drummers', '11 pipers', '10 lords', '9 ladies', '8 maids', '7 swans', '
6 geese', '5 rings', '4 birds', '3 hens', '2 doves', '1 partridge']

>>> import re
>>> at_re = re.compile(r'.at')
>>> at_re.findall('The cat in the hat sat on the flat mat.')
['cat', 'hat', 'sat', 'lat', 'mat']

Quantifier Syntax: How Many Qualifiers to Match

In a regular expression string, quantifiers follow qualifier characters to dictate how many of them to match. For example, in the phone number regex considered earlier, the {3} follows the \d to match exactly three digits. If there is no quantifier following a qualifier, the qualifier must appear exactly once: you can think of r'\d' as being the same as r'\d{1}'.

>>> import re
>>> pattern = re.compile(r'42!?')
>>> pattern.search('42!')
<re.Match object; span=(0, 3), match='42!'>
>>> pattern.search('42')
<re.Match object; span=(0, 2), match='42'>

>>> import re
>>> pattern = re.compile(r'42?!')
>>> pattern.search('42!')
<re.Match object; span=(0, 3), match='42!'>
>>> pattern.search('4!')
<re.Match object; span=(0, 2), match='4!'>
>>> pattern.search('42') == None  # No match
True

>>> pattern = re.compile(r'(\d{3}-)?\d{3}-\d{4}')
>>> match1 = pattern.search('My number is 415-555-4242')
>>> match1.group()
'415-555-4242'

>>> match2 = pattern.search('My number is 555-4242')
>>> match2.group()
'555-4242'

>>> import re
>>> pattern = re.compile('Eggs(and spam)*')
>>> pattern.search('Eggs')
<re.Match object; span=(0, 4), match='Eggs'>
>>> pattern.search('Eggs and spam')
<re.Match object; span=(0, 13), match='Eggs and spam'>
>>> pattern.search('Eggs and spam and spam')
<re.Match object; span=(0, 22), match='Eggs and spam and spam'>
>>> pattern.search('Eggs and spam and spam and spam')
<re.Match object; span=(0, 31), match='Eggs and spam and spam and spam'>

>>> pattern = re.compile('Eggs(and spam)+')
>>> pattern.search('Eggs and spam')
<re.Match object; span=(0, 13), match='Eggs and spam'>
>>> pattern.search('Eggs and spam and spam')
<re.Match object; span=(0, 22), match='Eggs and spam and spam'>
>>> pattern.search('Eggs and spam and spam and spam')
<re.Match object; span=(0, 31), match='Eggs and spam and spam and spam'>

(Ha){3}
HaHaHa

(Ha){3,5}
(HaHaHa)|(HaHaHaHa)|(HaHaHaHaHa)

>>> import re
>>> haRegex = re.compile(r'(Ha){3}')
>>> match1 = haRegex.search('HaHaHa')
>>> match1.group()
'HaHaHa'

>>> match = haRegex.search('HaHa')
>>> match == None
True

Greedy and Non-greedy Matching

Because (Ha){3,5} can match three, four, or five instances of Ha in the string 'HaHaHaHaHa', you may wonder why the Match object’s call to group() in the previous curly bracket example returns 'HaHaHaHaHa' instead of the shorter possibilities. After all, 'HaHaHa' and 'HaHaHaHa' are also valid matches of the regular expression (Ha){3,5}.

Python’s regular expressions are greedy by default, which means that in ambiguous situations, they will match the longest string possible. The non-greedy (also called lazy) version of the curly brackets, which matches the shortest string possible, must follow the closing curly bracket with a question mark.

Enter the following into the interactive shell, and notice the difference between the greedy and non-greedy forms of the curly brackets searching the same string:

>>> import re
>>> greedy_pattern = re.compile(r'(Ha){3,5}')
>>> match1 = greedy_pattern.search('HaHaHaHaHa')
>>> match1.group()
'HaHaHaHaHa'

>>> lazy_pattern = re.compile(r'(Ha){3,5}?')
>>> match2 = lazy_pattern.search('HaHaHaHaHa')
>>> match2.group()
'HaHaHa'

Note that the question mark can have two meanings in regular expressions: declaring a lazy match or declaring an optional qualifier. These meanings are entirely unrelated.

It’s worth pointing out that, technically, you could get by without using the optional ? quantifier, or even the * and + quantifiers:

• The ? quantifier is the same as {0,1}.
• The * quantifier is the same as {0,}.
• The + quantifier is the same as {1,}.

However, the ?, *, and + quantifiers are common shorthand.

>>> import re
>>> name_pattern = re.compile(r'First Name: (.*) Last Name: (.*)')
>>> name_match = name_pattern.search('First Name: Al Last Name: Sweigart')
>>> name_match.group(1)
'Al'
>>> name_match.group(2)
'Sweigart'

>>> import re
>>> lazy_pattern = re.compile(r'<.*?>')
>>> match1 = lazy_pattern.search('<To serve man> for dinner.>')
>>> match1.group()
'<To serve man>'

>>> greedy_re = re.compile(r'<.*>')
>>> match2 = greedy_re.search('<To serve man> for dinner.>')
>>> match2.group()
'<To serve man> for dinner.>'

>>> import re
>>> no_newline_re = re.compile('.*')
>>> no_newline_re.search('Serve the public trust.\nProtect the innocent. 
\nUphold the law.').group()
'Serve the public trust.'

>>> newline_re = re.compile('.*', re.DOTALL)
>>> newline_re.search('Serve the public trust.\nProtect the innocent. 
\nUphold the law.').group()
'Serve the public trust.\nProtect the innocent.\nUphold the law.'

Matching at the Start and End of a String

You can use the caret symbol (^) at the start of a regex to indicate that a match must occur at the beginning of the searched text. Likewise, you can put a dollar sign ($) at the end of the regex to indicate that the string must end with this regex pattern. And you can use the ^ and $ together to indicate that the entire string must match the regex—that is, it’s not enough for a match to be made on some subset of the string.

For example, the r'^Hello' regular expression string matches strings that begin with 'Hello'. Enter the following into the interactive shell:

>>> import re
>>> begins_with_hello = re.compile(r'^Hello')
>>> begins_with_hello.search('Hello, world!')
<re.Match object; span=(0, 5), match='Hello'>
>>> begins_with_hello.search('He said "Hello."') == None
True

The r'\d$' regular expression string matches strings that end with a numeric character between 0 and 9. Enter the following into the interactive shell:

>>> import re
>>> ends_with_number = re.compile(r'\d$')
>>> ends_with_number.search('Your number is 42')
<re.Match object; span=(16, 17), match='2'>
>>> ends_with_number.search('Your number is forty two.') == None
True

The r'^\d+$' regular expression string matches strings that both begin and end with one or more numeric characters. Enter the following into the interactive shell:

>>> import re
>>> whole_string_is_num = re.compile(r'^\d+$')
>>> whole_string_is_num.search('1234567890')
<re.Match object; span=(0, 10), match='1234567890'>
>>> whole_string_is_num.search('12345xyz67890') == None
True

The last two search() calls in the previous interactive shell example demonstrate how the entire string must match the regex if ^ and $ are used. (I always confuse the meanings of these two symbols, so I use the mnemonic “carrots cost dollars” to remind myself that the caret comes first and the dollar sign comes last.)

You can also use \b to make a regex pattern match only on a word boundary: the start of a word, end of a word, or both the start and end of a word. In this case, a “word” is a sequence of letters separated by non-letter characters. For example, r'\bcat.*?\b' matches a word that begins with 'cat' followed by any other characters up to the next word boundary:

>>> import re
>>> pattern = re.compile(r'\bcat.*?\b')
>>> pattern.findall('The cat found a catapult catalog in the catacombs.')
['cat', 'catapult', 'catalog', 'catacombs']

The \B syntax matches anything that is not a word boundary:

>>> import re
>>> pattern = re.compile(r'\Bcat\B')
>>> pattern.findall('certificate')  # Match
['cat']
>>> pattern.findall('catastrophe')  # No match
[]

It is useful for finding matches in the middle of a word.

Case-Insensitive Matching

Normally, regular expressions match text with the exact casing you specify. For example, the following regexes match completely different strings:

>>> import re
>>> pattern1 = re.compile('RoboCop')
>>> pattern2 = re.compile('ROBOCOP')
>>> pattern3 = re.compile('robOcop')
>>> pattern4 = re.compile('RobocOp')

But sometimes you care only about matching the letters, and aren’t worried about whether they’re uppercase or lowercase. To make your regex case-insensitive, you can pass re.IGNORECASE or re.I as a second argument to re.compile(). Enter the following into the interactive shell:

>>> import re
>>> pattern = re.compile(r'robocop', re.I)
>>> pattern.search('RoboCop is part man, part machine, all cop.').group()
'RoboCop'

>>> pattern.search('ROBOCOP protects the innocent.').group()
'ROBOCOP'

>>> pattern.search('Have you seen robocop?').group()
'robocop'

The regular expression now matches strings with any casing.

Substituting Strings

Regular expressions don’t merely find text patterns; they can also substitute new text in place of those patterns. The sub() method for Pattern objects accepts two arguments. The first is a string that should replace any matches. The second is the string of the regular expression. The sub() method returns a string with the substitutions applied.

For example, enter the following into the interactive shell to replace secret agents’ names with CENSORED:

>>> import re
>>> agent_pattern = re.compile(r'Agent \w+')
>>> agent_pattern.sub('CENSORED', 'Agent Alice contacted Agent Bob.')
'CENSORED contacted CENSORED.'

Sometimes you may need to use the matched text itself as part of the substitution. In the first argument to sub(), you can include \1, \2, \3, and so on, to mean “Enter the text of group 1, 2, 3, and so on, in the substitution.” This syntax is called a back reference.

For example, say you want to censor the names of the secret agents by showing just the first letters of their names. To do this, you could use the regex Agent (\w)\w* and pass r'\1****' as the first argument to sub():

>>> import re
>>> agent_pattern = re.compile(r'Agent (\w)\w*')
>>> agent_pattern.sub(r'\1****', 'Agent Alice contacted Agent Bob.')
'A**** contacted B****.'

The \1 in the regular expression string is replaced by whatever text was matched by group 1—that is, the (\w) group of the regular expression.

Managing Complex Regexes with Verbose Mode

Regular expressions are fine if the text pattern you need to match is simple. But matching complicated text patterns might require long, convoluted regular expressions. You can mitigate this complexity by telling the re.compile() function to ignore whitespace and comments inside the regular expression string. Enable this “verbose mode” by passing the variable re.VERBOSE as the second argument to re.compile().

Now, instead of a hard-to-read regular expression like this

pattern = re.compile(r'((\d{3}|\(\d{3}\))?(\s|-|\.)?\d{3}(\s|-
|\.)\d{4}(\s*(ext|x|ext\.)\s*\d{2,5})?)')

you can spread the regular expression over multiple lines and use comments to label its components, like this:

pattern = re.compile(r'''(
    (\d{3}|\(\d{3}\))?  # Area code
    (\s|-|\.)?  # Separator
    \d{3}  # First three digits
    (\s|-|\.)  # Separator
    \d{4}  # Last four digits
    (\s*(ext|x|ext\.)\s*\d{2,5})?  # Extension
    )''', re.VERBOSE)

Note how the previous example uses the triple-quote syntax (''') to create a multiline string so that you can spread the regular expression definition over many lines, making it much more legible.

The comment rules inside the regular expression string are the same as for regular Python code: the # symbol and everything after it until the end of the line are ignored. Also, the extra spaces inside the multiline string for the regular expression are not considered part of the text pattern to be matched. This lets you organize the regular expression so that it’s easier to read.

While verbose mode makes your regex strings more readable, I advise you to instead use the Humre module, covered later in this chapter, to improve the readability of your regular expressions.

Combining re.IGNORECASE, re.DOTALL, and re.VERBOSE

What if you want to use re.VERBOSE to write comments in your regular expression, but also want to use re.IGNORECASE to ignore capitalization? Unfortunately, the re.compile() function takes only a single value as its second argument.

You can get around this limitation by combining the re.IGNORECASE, re.DOTALL, and re.VERBOSE variables using the pipe character (|), which in this context is known as the bitwise or operator. For example, if you want a regular expression that is case-insensitive and includes newlines to match the dot character, you would form your re.compile() call like this:

>>> some_regex = re.compile('foo', re.IGNORECASE | re.DOTALL)

Including all three options in the second argument looks like this:

>>> some_regex = re.compile('foo', re.IGNORECASE | re.DOTALL | re.VERBOSE)

This syntax is a little old-fashioned and originates from early versions of Python. The details of the bitwise operators are beyond the scope of this book, but check out the resources at https://nostarch.com/automate-boring-stuff-python-3rd-edition for more information. You can also pass other options for the second argument; they’re uncommon, but you can read more about them in the resources too.

Project 3: Extract Contact Information from Large Documents

Say you’ve been given the boring task of finding every phone number and email address in a long web page or document. If you manually scroll through the page, you might end up searching for a long time. But if you had a program that could search the text in your clipboard for phone numbers and email addresses, you could simply press CTRL-A to select all the text, press CTRL-C to copy it to the clipboard, and then run your program. It could replace the text on the clipboard with just the phone numbers and email addresses it finds.

Whenever you’re tackling a new project, it can be tempting to dive right into writing code. But more often than not, it’s best to take a step back and consider the bigger picture. I recommend first drawing up a high-level plan for what your program needs to do. Don’t think about the actual code yet; you can worry about that later. Right now, stick to broad strokes.

For example, your phone number and email address extractor will need to do the following:

• Get the text from the clipboard.
• Find all phone numbers and email addresses in the text.
• Paste them onto the clipboard.

Now you can start thinking about how this might work in code. The code will need to do the following:

• Use the pyperclip module to copy and paste strings.
• Create two regexes, one for matching phone numbers and one for matching email addresses.
• Find all matches (not just the first match) of both regexes.
• Neatly format the matched strings into a single string to paste.
• Display some kind of message if no matches were found in the text.

This list is like a road map for the project. As you write the code, you can focus on each of these steps separately, and each step should seem fairly manageable. They’re also expressed in terms of things you already know how to do in Python.

import pyperclip, re

phone_re = re.compile(r'''(
    (\d{3}|\(\d{3}\))?  # Area code
    (\s|-|\.)?  # Separator
    (\d{3})  # First three digits
    (\s|-|\.)  # Separator
    (\d{4})  # Last four digits
    (\s*(ext|x|ext\.)\s*(\d{2,5}))?  # Extension
    )''', re.VERBOSE)

# TODO: Create email regex.

# TODO: Find matches in clipboard text.

# TODO: Copy results to the clipboard.

import pyperclip, re

phone_re = re.compile(r'''(
--snip--

# Create email regex.
email_re = re.compile(r'''(
  ❶ [a-zA-Z0-9._%+-]+  # Username
  ❷ @  # @ symbol
  ❸ [a-zA-Z0-9.-]+  # Domain name
    (\.[a-zA-Z]{2,4})  # Dot-something
    )''', re.VERBOSE)

# TODO: Find matches in clipboard text.

# TODO: Copy results to the clipboard.

import pyperclip, re

phone_re = re.compile(r'''(
--snip--

# Find matches in clipboard text.
text = str(pyperclip.paste())

❶ matches = []
❷ for groups in phone_re.findall(text):
    phone_num = '-'.join([groups[1], groups[3], groups[5]])
    if groups[6] != '':
        phone_num += ' x' + groups[6]
    matches.append(phone_num)
❸ for groups in email_re.findall(text):
    matches.append(groups)

# TODO: Copy results to the clipboard.

import pyperclip, re

phone_re = re.compile(r'''(
--snip--
for groups in email_re.findall(text):
    matches.append(groups[0])

# Copy results to the clipboard.
if len(matches) > 0:
    pyperclip.copy('\n'.join(matches))
    print('Copied to clipboard:')
    print('\n'.join(matches))
else:
    print('No phone numbers or email addresses found.')

Copied to clipboard:
800-555-7240
415-555-9900
415-555-9950
info@nostarch.com
media@nostarch.com
academic@nostarch.com
info@nostarch.com

Humre: A Module for Human-Readable Regexes

Code is read far more often than it’s written, so it’s important for your code to be readable. But the punctuation-dense syntax of regular expressions can be hard for even experienced programmers to read. To solve this, the third-party Humre Python module takes the good ideas of verbose mode even further by using human-readable, plain-English names to create readable regex code. You can install Humre by following the instructions in Appendix A.

Let’s go back to the r'\d{3}-\d{3}-\d{4}' phone number example from the beginning of this chapter. The functions and constants in Humre can produce the same regex string with plain English:

>>> from humre import *
>>> phone_regex = exactly(3, DIGIT) + '-' + exactly(3, DIGIT) + '-' + exactly(4, DIGIT)
>>> phone_regex
'\\d{3}-\\d{3}-\\d{4}'

Humre’s constants (like DIGIT) contain strings, and Humre’s functions (like exactly()) return strings. Humre doesn’t replace the re module. Rather, it produces regex strings that can be passed to re.compile():

>>> import re
>>> pattern = re.compile(phone_regex)
>>> pattern.search('My number is 415-555-4242')
<re.Match object; span=(13, 25), match='415-555-4242'>

Humre has constants and functions for each feature of regular expression syntax. You can then concatenate the constants and returned strings like any other string. For example, here are Humre’s constants for the shorthand character classes:

• DIGIT and NONDIGIT represent r'\d' and r'\D', respectively.
• WORD and NONWORD represent r'\w' and r'\W', respectively.
• WHITESPACE and NONWHITESPACE represent r'\s' and r'\S', respectively.

A common source of regex bugs is forgetting which characters need to be escaped. You can use Humre’s constants instead of typing the escaped character yourself. For example, say you want to match a single-digit floating-point number with one digit after the decimal point, like '0.9' or '4.5'. However, if you use the regex string r'\d.\d', you might not realize that the dot matches a period (as in '4.5') but also matches any other character (as in '4A5').

Instead, use Humre’s PERIOD constant, which contains the string r'\.'. The expression DIGIT + PERIOD + DIGIT evaluates to r'\d\.\d' and makes it much more obvious what the regex intends to match.

The following Humre constants exist for escaped characters:

PERIOD     OPEN_PAREN  OPEN_BRACKET    PIPE
DOLLAR_SIGN CLOSE_PAREN CLOSE_BRACKET   CARET
QUESTION_MARK   ASTERISK    OPEN_BRACE  TILDE
HASHTAG     PLUS        CLOSE_BRACE 
AMPERSAND   MINUS       BACKSLASH   

There are also constants for NEWLINE, TAB, QUOTE, and DOUBLE_QUOTE. Back references from r'\1' to r'\99' are represented as BACK_1 to BACK_99.

However, you’ll make the largest readability gains by using Humre’s functions. Table 9-2 shows these functions and their equivalent regular expression syntax.

Humre also has several convenience functions that combine common pairs of function calls. For example, instead of using optional(group('A')) to create '(A)?', you can simply call optional_group('A'). Table 9-3 has the full list of Humre convenience functions.

All of Humre’s functions except either() and group_either() allow you to pass multiple strings to automatically join them. This means that calling group(DIGIT, PERIOD, DIGIT) produces the same regex string as group(DIGIT + PERIOD + DIGIT). They both return the regex string r'(\d\.\d)'.

Finally, Humre has constants for common regex patterns:

ANY_SINGLE The . pattern that matches any single character (except newlines)

ANYTHING_LAZY The lazy .*? zero or more pattern

ANYTHING_GREEDY The greedy .* zero or more pattern

SOMETHING_LAZY The lazy .+? one or more pattern

SOMETHING_GREEDY The greedy .+ one or more pattern

The readability of regex written with Humre becomes more obvious when you consider large, complicated regular expressions. Let’s rewrite the phone number regex from the previous phone number extractor project using Humre:

import re
from humre import *
phone_regex = group(
    optional_group(either(exactly(3, DIGIT),  # Area code
                          OPEN_PAREN + exactly(3, DIGIT) + CLOSE_PAREN)),
    optional(group_either(WHITESPACE, '-', PERIOD)),  # Separator
    group(exactly(3, DIGIT)),  # First three digits
    group_either(WHITESPACE, '-', PERIOD),  # Separator
    group(exactly(4, DIGIT)),  # Last four digits
    optional_group(  # Extension
      zero_or_more(WHITESPACE),
      group_either('ext', 'x', r'ext\.'),
      zero_or_more(WHITESPACE),
      group(between(2, 5, DIGIT))
      )
    )

pattern = re.compile(phone_regex)
match = pattern.search('My number is 415-555-1212.')
print(match.group())

When you run this program, the output is this:

415-555-1212

This code is much more verbose than even the verbose mode regex. It helps to import Humre using the from humre import * syntax so that you don’t need to put humre. before every function and constant. But the length of the code doesn’t matter as much as the readability.

You can switch your existing regular expressions to Humre code by calling the humre.parse() function, which returns a string of Python source code:

>>> import humre
>>> humre.parse(r'\d{3}-\d{3}-\d{4}')
"exactly(3, DIGIT) + '-' + exactly(3, DIGIT) + '-' + exactly(4, DIGIT)"

When combined with a modern editor such as PyCharm or Visual Studio Code, Humre offers several further advantages:

• You can indent your code to make it obvious which parts of the regex contain which other parts.
• Your editor’s parentheses matching works.
• Your editor’s syntax highlighting works.
• Your editor’s linter and type hints tool picks up typos.
• Your editor’s autocomplete fills in the function and constant names.
• Humre handles raw strings and escaping for you.
• You can put Python comments alongside your Humre code.
• Typos cause more helpful error messages.

Many experienced programmers will object to using anything other than the standard, complicated, unreadable regular expression syntax. As programmer Peter Bhat Harkins once said, “One of the most irritating things programmers do regularly is feel so good about learning a hard thing that they don’t look for ways to make it easy, or even oppose things that would do so.”

However, if a co-worker objects to your use of Humre, you can simply print the underlying regex string that your Humre code generates and put it back into your source code. For example, the contents of the phone_regex variable in the phone number extractor project are as follows:

r'((\d{3}|\(\d{3}\))?(\s|-|\.)?(\d{3})(\s|-|\.)(\d{4})(\s*(ext|x|ext\.)\s*(\d{2,5}))?)'

Your co-worker is welcome to use this regular expression string if they feel it is more appropriate.

Summary

While a computer can search for text quickly, it must be told precisely what to look for. Regular expressions allow you to specify the pattern of characters you are looking for, rather than the exact text itself. In fact, some word processing and spreadsheet applications provide find-and-replace features that allow you to search using regular expressions. The punctuation-heavy syntax of regular expressions is composed of qualifiers that detail what to match and quantifiers that detail how many to match.

The re module that comes with Python lets you compile a regex string into a Pattern object. These objects have several methods: search(), to find a single match; findall(), to find all matching instances; and sub(), to do a find-and-replace substitution of text.

You can find out more in the official Python documentation at https://docs.python.org/3/library/re.html. Another useful resource is the tutorial website https://www.regular-expressions.info. The Humre page on the Python Package Index is https://pypi.org/project/Humre/.

Practice Questions

  1.  What is the function that returns Regex objects?

  2.  Why are raw strings often used when creating Regex objects?

  3.  What does the search() method return?

  4.  How do you get the actual strings that match the pattern from a Match object?

  5.  In the regex created from r'(\d\d\d)-(\d\d\d-\d\d\d\d)', what does group 0 cover? Group 1? Group 2?

  6.  Parentheses and periods have specific meanings in regular expression syntax. How would you specify that you want a regex to match actual parentheses and period characters?

  7.  The findall() method returns a list of strings or a list of tuples of strings. What makes it return one or the other?

  8.  What does the | character signify in regular expressions?

  9.  What two things does the ? character signify in regular expressions?

10.  What is the difference between the + and * characters in regular expressions?

11.  What is the difference between {3} and {3,5} in regular expressions?

12.  What do the \d, \w, and \s shorthand character classes signify in regular expressions?

13.  What do the \D, \W, and \S shorthand character classes signify in regular expressions?

14.  What is the difference between the .* and .*? regular expressions?

15.  What is the character class syntax to match all numbers and lowercase letters?

16.  How do you make a regular expression case-insensitive?

17.  What does the . character normally match? What does it match if re.DOTALL is passed as the second argument to re.compile()?

18.  If num_re = re.compile(r'\d+'), what will num_re.sub('X', '12 drummers, 11 pipers, five rings, 3 hens') return?

19.  What does passing re.VERBOSE as the second argument to re.compile() allow you to do?

Practice Programs

For practice, write programs to do the following tasks.