RFC 3986 URL Validation

Have you ever submitted a url to a site just to have the site tell you it was invalid when you knew it wasn't. That happened to be recently and happened on one of my drupal sites. Drupal told me a flickr url containing an @ symbol was invalid. When I looked deeper into the issue I found the URL really was valid. When I looked to see what other software was doing I found many cases where there was no validation or what was present failed for many types of valid urls. So, let's take a look at how to do good url validation.

The Specifications

The specification for URIs, of which URLs is a subset, is RFC 3986. This is not only a great place to start but Appendix A provides a detailed guide to the makeup of a URI.

filter_var in PHP

In PHP 5 there is a function called filter_var which allows you to test a piece of data against a filter. One of the filters is FILTER_VALIDATE_URL and this seems like the obvious case to use. It's usage would be something like:

  $result = filter_var($url, FILTER_VALIDATE_URL, FILTER_FLAG_SCHEME_REQUIRED);

Simple, right? The issue is with how this works. FILTER_VALIDATE_URL attempts to pass the URL through parse_url and see if it can break the URL down. In the documentation of parse_url it says, "This function is not meant to validate the given URL".

Additionally, if you test this you'll find urls like "http://", http://...", and many other invalid URLs will pass because they have a structure parse_url can handle.

Using A Regular Expression

I searched for a regular expression that validated for RFC 3986 and couldn't find one. So, I wrote the following (for PHP).

# Start at the beginning of the text
/^
# The scheme
([a-z][a-z0-9\*\-\.]*):\/\/
# Userinfo (optional)
(?:                                                     
  (?:(?:[\w\.\-\+!$&'\(\)*\+,;=]|%[0-9a-f]{2})+:)*
  (?:[\w\.\-\+%!$&'\(\)*\+,;=]|%[0-9a-f]{2})+@
)?
# The domain
(?:
  # Domain name or IPv4
  (?:[a-z0-9\-\.]|%[0-9a-f]{2})+
  # or IPv6
  |(?:\[(?:[0-9a-f]{0,4}:)*(?:[0-9a-f]{0,4})\])
)
# Server port number (optional)
(?::[0-9]+)?
# The path (optional)
(?:[\/|\?]
  (?:[\w#!:\.\?\+=&@!$'~*,;\/\(\)\[\]\-]|%[0-9a-f]{2})  
*)?
$/xi

For drupal we are narrowing down the scheme to just allow http, https, and ftp and we aren't looking for reference details to be returned so the regex looks like

# Start at the beginning of the text
/^
# Look for ftp, http, or https
(?:ftp|https?):\/\/
# Userinfo (optional)
(?:                                                     
  (?:(?:[\w\.\-\+!$&'\(\)*\+,;=]|%[0-9a-f]{2})+:)*
  (?:[\w\.\-\+%!$&'\(\)*\+,;=]|%[0-9a-f]{2})+@
)?
# The domain
(?:
  # Domain name or IPv4
  (?:[a-z0-9\-\.]|%[0-9a-f]{2})+
  # or IPv6
  |(?:\[(?:[0-9a-f]{0,4}:)*(?:[0-9a-f]{0,4})\])
)
# Server port number (optional)
(?::[0-9]+)?
# The path (optional)
(?:[\/|\?]
  (?:[\w#!:\.\?\+=&@!$'~*,;\/\(\)\[\]\-]|%[0-9a-f]{2})  
*)?
$/xi

Drupals testing framework has made it easy to pass valid and invalid urls through this expression to test if it's working. Details on the issue are at http://drupal.org/node/124492.

Think you can break it of do better? I would love it if you could. Please feel free to try and let me know. If it works for you please share with your friends, enemies, and anyone else who could use it.

Update: The regular expressions were updated to allow % HEXDIGIT HEXDIGIT rather than % and to allow for IPv6 addresses for the domain.