Explaining Regular Expressions

Unless you get to work with regular expressions everyday, the details can easily blur. You remember enough to know when a regex could be useful, but not enough to write it without reaching for a reference manual.

There are a few tools out there to help you write and understand regular expressions, including some IDEs that can provide assistance (like Komodo).

Sometimes you just want a quick explanation of a regular expression you might have seen in some code. The YAPE::Regex::Explain perl module is one tool that helps you do just that. Turning the module into a simple command-line tool is a simple one-liner:

#!/usr/bin/perl -w
print YAPE::Regex::Explain->new($ARGV[0])->explain;

I've also turned it into a simple CGI utility for those times when my regex memory fails me. You can use it here: regexplainr (sorry, you may find that site offline but here's the source code).

In a previous post, On Parsing CSV and other Delimited/Quoted Formats, I used the following regular expression to parse a whitespace-delimited string:

"([^"]+?)"\s?|([^\s]+)\s?|\s

Regexplainr produces the following commentary:

The regular expression:

(?-imsx:"([^"]+?)"\s?|([^\s]+)\s?|\s)

matches as follows:

NODE EXPLANATION
----------------------------------------------------------------------
(?-imsx: group, but do not capture (case-sensitive)
(with ^ and $ matching normally) (with . not
matching \n) (matching whitespace and #
normally):
----------------------------------------------------------------------
" '"'

----------------------------------------------------------------------
( group and capture to \1:
----------------------------------------------------------------------
[^"]+? any character except: '"' (1 or more
times (matching the least amount
possible))
----------------------------------------------------------------------
) end of \1
----------------------------------------------------------------------
" '"'
----------------------------------------------------------------------
\s? whitespace (\n, \r, \t, \f, and " ")
(optional (matching the most amount
possible))
----------------------------------------------------------------------
| OR
----------------------------------------------------------------------
( group and capture to \2:
----------------------------------------------------------------------
[^\s]+ any character except: whitespace (\n,
\r, \t, \f, and " ") (1 or more times
(matching the most amount possible))
----------------------------------------------------------------------
) end of \2
----------------------------------------------------------------------
\s? whitespace (\n, \r, \t, \f, and " ")
(optional (matching the most amount
possible))
----------------------------------------------------------------------
| OR
----------------------------------------------------------------------
\s whitespace (\n, \r, \t, \f, and " ")
----------------------------------------------------------------------
) end of grouping
----------------------------------------------------------------------

(Permalink to this regexplanation)

Book tip: (thanks to Tony) O'Reilly's Mastering Regular Expressions. Available on google books, and also from Amazon.

On Parsing CSV and other Delimited/Quoted Formats

Parsing delimited text that may have quoted elements is a perennial requirement. Quick-and-dirty parses can be achieved with regular expressions, but for more flexible and encapsulated parsing I've been checking out the opencsv java library. Hat tip to Jakub Pawlowski for highlighting the library on his blog

A Regular Expression Approach
Just recently I released and blogged about a JDeveloper Filter Add-in, and it contains a class called ExecShell [API, source] which needs to know how to break a command line into its component arguments. The command line is of course space-delimited, but may use quotes to group an argument with embedded spaces (so a simple split on spaces won't do).

The salient code below uses the REGEX to chop theCmdLine String into theCmdArray Vector of arguments:

Vector<String> theCmdArray = new Vector<String>(0);
String REGEX = "\"([^\"]+?)\"\\s?|([^\\s]+)\\s?|\\s";
Pattern p = Pattern.compile(REGEX);
Matcher m = p.matcher(theCmdLine);
while (m.find())
{
theCmdArray.add( m.group().trim() );
}

The regular expression bears a little explaining, and is inspired by this example. Here's how it breaks down:

\"([^\"]+?)\"\\s?	Matches a group within double-quotes. Group is a lazy match on one or more characters except double-quote. Optionally followed by some whitespace
|([^\\s]+)\\s?	or Matches a group delimited by whitespace, optionally followed by some whitespace
|\\s	Discards a pure whitespace match

In this case, we are using whitespace as the delimiter (appropriate for command lines). The regex can be adapted for other delimiters by replacing \\s with the delimiter. For example, to handle a comma-separated format:

String REGEX = "\"([^\"]+?)\",?|([^,]+),?|,";

Using OpenCSV
The same space-delimited parsing requirement can be met with a couple of lines and the opencsv library:

CSVReader reader = new CSVReader(new StringReader(theCmdArray), ' ');
String[] s = reader.readNext();

Simple, yet currently not so robust. Since we define the delimiter to be a single space (over-ridding the default comma), other whitespace characters (like a tab) will not be recognised. Further, repeated spaces will not be coalesced, but will each be treated as the delimiter for a new element.

Internally, CSVReader parses the input character-by-character and so adapting to handle repeated delimiters as one would be reasonably straight-forward.

Exercising Regular Expressions and Arrays in Javascript

I've been working a bit with AJAX, and that soon prompted me to brush up on my Javascript. In the past I'd never used it to do more than a little validation or form "glue". Now revisiting the language from a true programmer's perspective, I recognise that it is pretty capable!

Speak Good Singlish! is a sample that gives regular expression and multi-dimensional array handling a good workout. It's a Singlish translator implemented in Javascript. I can't remember what sparked the idea to use this as an example, but somehow I got thinking back to ole "Jive" and "Valley girl" translators that have been floating around the net for many years.

The RegExp object usage in the sample is fairly standard, but the array handling is not.

I'm using a 2D array for the "lexicon" - basically a list of regex matches and replacements. For example the following fragment. You'll note that the second dimension can contain either a simple string or an array (as the replacement element):

var slexicon = [
[ '\\benglish\\b', 'Singlish' ] ,
[ '\\bdo\\b', [ 'do', 'do until sian', 'do sure can one' ] ]
];

There's a core routine that iterates through the array and performs the substitutions. Where an array substitution is present, it uses a funtion inline to pick a random element from the array to perform the substitution:

// do the translation
for (var i = 0; i < slexicon.length ; i++) {
var slexiconRow = slexicon[i];
var theRegex = new RegExp( slexiconRow[0], "gim" );
var theReplacement = slexiconRow[1];
if (theReplacement instanceof Array ) {
dataOut = dataOut.replace( theRegex, function (match) {return randomElement(theReplacement)} );
} else {
dataOut = dataOut.replace( theRegex, theReplacement );
}
}

The randomElement function is simplicity itself:

// returns random element of an array
var randomElement = new Function("x", "return x[Math.floor(Math.random() * (x.length))]");

The HTML page contains the whole script inline, so it is easy to review in-place with a "view source". Enjoy!