Matt Farina - Development

My Bazaar Core Development Workflow

matt — Fri, 28 Aug 2009 13:08:18 +0000

My old Drupal core development workflow revolved around CVS. Anytime I'd want to work on a new feature I'd do a CVS checkout of Drupal core and start working on the new feature. If I, or someone else, was working on 2 features that overlapped in code I would have to deal with massive conflict resolving or just have to wait until the other feature was committed or abandoned. Oh, and I had to be connected to the Internet to grab a new CVS checkout. That all changed when I switched to Bazaar for my core development work.

Some Of What I Can Do With Bazaar

Bazaar, a distributed version control system, let's me do a lot of things that simply can't be done with CVS or SVN. Here's a short and incomplete list:

I don't need to be connected to the Internet to create new feature branches.
Merges are much better making them useful. That makes branches cheap, easy, and useful.
I can make branches of other feature branches allowing me to layer patches and issues.
Did I mention merges are better. This means a lot less conflicts to deal with.

Setting Up Bazaar

Setting up Bazaar in fairly simple. You can do it with installers, through your favorite distribution os Linux, and with Macports. Once you have Bazaar installed there are two great places to look for how to setup Bazaar. First, there is the drupal handbook pages on the topic. Then, I would recommend reading David Strauss blog post about setting up a Bazaar repo of drupal head.

But, what about Git?

Yes, you can do the same things in Git and Mercurial. They are great distributed version control systems as well. I'd suggest Bazaar as a first choice because that's the direction the drupal community is currently leaning.

Leading In The Face of Criticism

matt — Fri, 28 Aug 2009 12:37:09 +0000

How do you react when your pet project is criticized? How about when a project you use and love is criticized? When it's the same criticism you've heard over and over you and are tired of hearing it? That's happened yesterday to the Drupal Views project when a tweet sent out of frustration turned into a swarm of tweets, IRC conversations, and back room talk. For the most part, the situation could have been handled better by everyone involved.

What happened?

Let's just say quick reactions, emotions, and defense mechanisms ruled the day. In the end I don't think anyone was satisfied and I have yet to see anything constructive come to light. There was name calling, anger, people trying to defend themselves and others, and a mess of unproductively.

What We Can Do Better Next Time

This situation could have been handled much better. Instead of acting the way many did with passion, frustration, and emotion the criticism should or could have been molded into actionable tasks and ideas.

Here are a handful of things we can do that can help these situations:

Don't respond to your emotions. It may be hard to do. Sometimes I will walk away from the situation for hours or even a day before responding to something so I can do so with a clear head.
Turn the criticism into something constructive. This could be as simple as pointing someone to a place (like the issue queues) and asking them to describe the problem and what they see would be better. If you have more time it could be good to engage the conversation to see what they see.
Be a diplomat. For some reason the role of attaching and defending soldier comes so naturally. Instead, play the role of diplomat.
Ignore Them. Sometimes it's better to ignore criticism and move on than let it get to you. If it eats you up that can make you unhappy and can even cause health problems.

Critics Not Going To Stop

One of the responses I heard numers times was that critics need to act different. If we create something with any popularity it will have critics. They will be there no matter what we desire. And, the only people we can change are ourselves. So, asking them to go away or asking them to act differently just isn't going to get them to do it.

People Smarter Than Me

A while back there was a Google Tech Talk called "How Open Source Projects Survive Poisonous People" which touched on this subject. It's done by a couple guys who have lived and learned how to deal with these situations successfully. You can see the video below.

Disclaimer: This post isn't directed at anyone. I've learned a lot of these lessons the hard way and just want to share them in the spirit of community growth.

The Horror of Views Markup

matt — Thu, 27 Aug 2009 12:35:17 +0000

Every so often someone points to the markup generated by the Drupal Views modules and calls it ugly. This seems come up every so often in the forums, issue queues, blog posts, and on twitter. Most of the time this is where the conversation turns a bit ugly. Since this conversation has been had many times the people who are involved are a bit tired of explaining what's going on to someone new and to many Views is their baby and it was just called ugly. So, let me take a shot at explaining why Views markup is the way it is, why that's good for some, and what you can do about it if you don't like it.

The Deal With Views Markup

The markup from Views has to be very flexible out of the box. It's going to be used by a lot of people to do a lot of different things. So, the markup provides spans, divs, and classes for almost every case you'd want to style. If you are someone who lives in CSS and loves classes and separation of everything it's there for you. There is a certain group of designers that want this and they have been vocal and are, for the most part, happy with the markup.

Can't Make Everyone Happy

As the saying goes, you can't make everyone happy all the time. Having lots of spans, divs, classes and other forms of markup is directly opposed to small concise markup others love. You can't have both out of the box and what Views picked is actually easier for new people to pick up.

In Drupal to alter the output you need to know how the theming system works and how to override the default markup. Since Views provides lots of markup many people can learn to style the output without needing to know how to alter the markup output of views. It saves them from a layer of complexity.

Having Markup Your Way

If you are a markup purist and want to change the markup output by Views it's there for you to change. Views uses the templating system provided by Drupal and has a display plugin system of it's own. With the templating system you can override all of the markup provided by Views and replace it with your own. It's almost as simple as copying the Views template files into your theme and altering or replacing them.

Drupalcon Paris

As Morten does at the Drupalcons he attends, in Paris, there will be a session about overriding the default markup in Drupal to replace it with what you want. If you're not going to make it to Paris you can wait for the video from his session to be published or check out the video of his session from Drupalcon DC.

Drupalcamp Boston Wrap-up

matt — Tue, 16 Jun 2009 14:09:15 +0000

The Drupal Design Camp in Boston this past weekend was fantastic. Having the event at the MIT Stata Center, a building that looks like it's right our of a Dr. Seuss book, was a perfect place for a design event. With well over 150 people, loads of fantastic sessions, and ideas for improving the tools and community designers and themers have the camp was a roaring success.

Photo by Jeff Eaton. Everyone is pointing at Morten.

The People

There weren't just people attending from the east coast or midwest and Morten wasn't the person who traveled the furthest. There were people from Texas, the Bay Area, and Europe. From the looks of things the event was 1/4 women. Has there ever been a drupal event with this high percentage of women? There were new people and people with 2 digit drupal.org user ids. There were developers and there were a whole lot of people with the creative art gene.

The Sessions

More than one attendee said they thought the sessions here were better than at drupalcon. With keynotes by Jeff Robbins and Jay Batson and huge sessions like the one on the 960 grid by Nathan Smith, the creator of 960, and Todd Nienkerk it's hard not to feel this way.

The 40 session covered topics ranging from the basics of drupal to themeing techniques to radical new ideas in how to theme like skinr.

The New Ideas

In Jay Batson's keynote many of the back room chatter started to turn into things we can act on as a community. Some ideas that came out were to start a site for designers and themers, to have a way to deal with snippets (something themers seem to do well), to have a showcase of hot drupal sites (giving credit where due). These are just the tip of the ice berg.

The Stata Center

The Stata Center was the perfect place for an event like this. Every direction you look there's something inspiring and artistic about the building.

Photo by Michael Verdi

I want to give a special thanks to Susan MacPhee for organizing the event and MIT for hosting the event. It was fantastic.

An Overlooked Drupal Design Problem

matt — Thu, 21 May 2009 11:05:45 +0000

Most of the shared Drupal themes aren't very good and there aren't very many commercial ones. That's what a study shared by Jay Batson pointed out. Jay went on to propose some steps we could take to court better design in the Drupal community. While he made some important points, there is one important point that was overlooked. The issue of crediting designers for their work.

Hostile Land For Designers

Let's be honest. The Drupal community is a hostile land for designers. Drupal.org is nothing to look at (yes, I know that's changing). The drupal web tools, like api.drupal.org, are centered on developers. To contribute a theme you have to know and use CVS. At drupalcon dc designers and themers had to take over BoF rooms to get the space they need.

The Drupal community is anything but warm and inviting to designers. We tend to treat designers like we do fellow developers. We can't do that. They are different. It would be like buying my wife a vacuum for her birthday. Just a bad idea.

Crediting Developers

Right now drupal developers are credited in a variety of ways. If they contribute code there are commit messages. We even keep stats on that. Their contributed modules are tracked on their user pages and you can see how many sites are using their modules. If they contribute to core their usernames are in the commit messages and can get up on the big screen at drupalcon.

If you are lucky enough (is lucky the right word) you can get your name in the MAINTAINERS.txt file for Drupal so everyone can see your name on the project.

As you contribute you build up community cred with the other developers. You make friends. You gain influence.

Difference Between Design and Development

When a developer writes a sweet module, like Views, for a site they can share it back with the community for fame, reputation, and more. The case is quite different for a designer.

If a themer contributes back a base theme like zen, blueprint, or moshpit it's a reusable piece of code like a module. A full design is something you can't share back from a project. If I share a design from a site I've done and people use it my branding status is lowered. It hurts the effectiveness of a site.

So, designs typically can't be contributed back in the same way modules or base themes can be.

Difference between Designers and Drupal Developers

Drupal developers usually want to move up the Drupal Developer food chain. When they contribute back there is an ecosystem of Drupal developers for them to interact with.

This is not the case for designers. There is no flourishing design community. When they go to interact within the design world they do it with the design world at large. Designers aren't interested in having their name in a drupal commit message about a change in the code. They are interested in receiving credit for the design they worked on. Note, a design is different than a theme in this case.

One of the things the Drupal community needs to figure out is how to credit designers for their designs in a way that showcases it the larger design community. This is especially important for drupal core where the drupal 7 version has just 2 designs.

So, this is a call for ideas. How can we do this and do it in an effective way for the design community?

I want to give a special thanks to Leisa Reichelt and Morten for helping me understand this.

Why Inline Editing In Drupal Is Hard

matt — Thu, 09 Apr 2009 19:12:06 +0000

Mark Boulton and Leisa Reichelt have suggest drupal provide inline editing in the interface. This is a fantastic idea and one I would love to see in drupal. But, (I hate this part) inline editing in drupal is a difficult feature to add. So, before everyone wonders whey we don't offer it or why it's going to take some work let's look at why this is hard.

An example

I think a good way to see some of the problems in action is to look an example of what would happen. For example, let's look at the process of editing the body of a page inline.

A user clicks on something to convert the body as displayed into a field to edit.
The page calls back to drupal to get the content of this field as someone would edit it. The content drupal displays goes through a filter system before it's displayed. This cleans out XSS and other security measures and lets other drupal modules plug-in to alter content too. An example of this altering is Filtered HTML which strips out unwanted tags.
A user updates this text and submits it.
This updated text needs to go back to the server, run through the filters, and returned to the page for display.

We might think this is where the fun stops. But, it doesn't. Many of the fields displayed by drupal are computed fields. A simple example of that is the teaser for the body text. In many cases this is computed by a number of conditions. How will this be handled? How will other computed fields be handled in updating the live page?

Not The First Time

This is not the first time someone has tried to add inline editing into drupal. Some, like Steven Wittens, have tried to do it in the past. This is a difficult problem to solve and is a product of drupals loved and utilized power.

Let's Figure It Out

I think we should try to figure it out. I love the idea. It means we have to make decisions and changes will touch some complex code. But, I think understanding a problem is a good place to start solving it. So, who wants to start rolling this patch?

Update: This post is based on a conversation in #drupal between leisa, chx, myself, and others.

RFC 3986 URL Validation

matt — Thu, 08 Jan 2009 13:17:01 +0000

Have you ever submitted a url to a site just to have the site tell you it was invalid when you knew it wasn't. That happened to be recently and happened on one of my drupal sites. Drupal told me a flickr url containing an @ symbol was invalid. When I looked deeper into the issue I found the URL really was valid. When I looked to see what other software was doing I found many cases where there was no validation or what was present failed for many types of valid urls. So, let's take a look at how to do good url validation.

The Specifications

The specification for URIs, of which URLs is a subset, is RFC 3986. This is not only a great place to start but Appendix A provides a detailed guide to the makeup of a URI.

filter_var in PHP

In PHP 5 there is a function called filter_var which allows you to test a piece of data against a filter. One of the filters is FILTER_VALIDATE_URL and this seems like the obvious case to use. It's usage would be something like:

  $result = filter_var($url, FILTER_VALIDATE_URL, FILTER_FLAG_SCHEME_REQUIRED);

Simple, right? The issue is with how this works. FILTER_VALIDATE_URL attempts to pass the URL through parse_url and see if it can break the URL down. In the documentation of parse_url it says, "This function is not meant to validate the given URL".

Additionally, if you test this you'll find urls like "http://", http://...", and many other invalid URLs will pass because they have a structure parse_url can handle.

Using A Regular Expression

I searched for a regular expression that validated for RFC 3986 and couldn't find one. So, I wrote the following (for PHP).

# Start at the beginning of the text
/^
# The scheme
([a-z][a-z0-9\*\-\.]*):\/\/
# Userinfo (optional)
(?:                                                     
  (?:(?:[\w\.\-\+!$&'\(\)*\+,;=]|%[0-9a-f]{2})+:)*
  (?:[\w\.\-\+%!$&'\(\)*\+,;=]|%[0-9a-f]{2})+@
)?
# The domain
(?:
  # Domain name or IPv4
  (?:[a-z0-9\-\.]|%[0-9a-f]{2})+
  # or IPv6
  |(?:\[(?:[0-9a-f]{0,4}:)*(?:[0-9a-f]{0,4})\])
)
# Server port number (optional)
(?::[0-9]+)?
# The path (optional)
(?:[\/|\?]
  (?:[\w#!:\.\?\+=&@!$'~*,;\/\(\)\[\]\-]|%[0-9a-f]{2})  
*)?
$/xi

For drupal we are narrowing down the scheme to just allow http, https, and ftp and we aren't looking for reference details to be returned so the regex looks like

# Start at the beginning of the text
/^
# Look for ftp, http, or https
(?:ftp|https?):\/\/
# Userinfo (optional)
(?:                                                     
  (?:(?:[\w\.\-\+!$&'\(\)*\+,;=]|%[0-9a-f]{2})+:)*
  (?:[\w\.\-\+%!$&'\(\)*\+,;=]|%[0-9a-f]{2})+@
)?
# The domain
(?:
  # Domain name or IPv4
  (?:[a-z0-9\-\.]|%[0-9a-f]{2})+
  # or IPv6
  |(?:\[(?:[0-9a-f]{0,4}:)*(?:[0-9a-f]{0,4})\])
)
# Server port number (optional)
(?::[0-9]+)?
# The path (optional)
(?:[\/|\?]
  (?:[\w#!:\.\?\+=&@!$'~*,;\/\(\)\[\]\-]|%[0-9a-f]{2})  
*)?
$/xi

Drupals testing framework has made it easy to pass valid and invalid urls through this expression to test if it's working. Details on the issue are at http://drupal.org/node/124492.

Think you can break it of do better? I would love it if you could. Please feel free to try and let me know. If it works for you please share with your friends, enemies, and anyone else who could use it.

Update: The regular expressions were updated to allow % HEXDIGIT HEXDIGIT rather than % and to allow for IPv6 addresses for the domain.

Drupal Patch Testing For Anyone

matt — Thu, 08 Jan 2009 12:13:46 +0000

Most of the development to drupal core happens through a patch review process. Even bigger changes, like the new database system, where development happens elsewhere, eventually go through a patch review process. This process can be intimidating to anyone who hasn't done it before or doesn't feel like they have a high enough skill level to contribute anything to core. In reality, anyone can review patches. Even individuals who can't write software at all.

Reviewing patches isn't just about knowing how to program. Someone who doesn't know how to program can help out with patch reviews. Let's take a look at some good places to start and to learn where anyone can review patches.

Getting Started

If you're interested in reviewing patches to drupal a good place to start is by reading the Reviewing Patches page on drupal.org. This page covers how the process works, coding standards, duplicate issues, and so much more.

Comment Reviews

Are you good with grammar? Or, even decent? Drupal is full of code comments that document what's going on. These need to be reviewed. For example, a comment in the newly committed patch to allow users to cancel their accounts (node 8 on drupal.org) looks like:

- // When a user is deleted, any content they created remains but the
+ // When a user account is canceled, any content they created remains but the
  // uid = 0. Their blog entry shows as "'s blog" on the home page. Records
  // in the watchdog table related to that user have the uid set to zero.

The line starting with - is a line being removed and the one starting with + is an addition. These comments need to be reviewed and need to be consistent with the comment standard in drupals coding standard. Anyone can provide feedback on these.

Usability

Usability is one of the top priorities in drupal 7. If you visit the Improvements to Core dashboard you'll see a list of top usability patches right up front. Imporvements to usability are more than coding a change. They require attention to detail and understanding what makes something usable to someone who doesn't know the system. So, if you're new to drupal this is a great place to offer your insights. If you are a designer, this is a place you can provide mock-ups.

Reviewing Patches

If you are familiar with PHP or JavaScript patch testing is a great place to get involved, win friends, and get core drupal developers to love you. A great place to learn how to patch review is from the drupal 7 co-maintainer webchick. She recently wrote a blog post about her 6 Pass Patch Review process. You may not be comfortable doing all the passes. Just doing one of the passes and providing feedback is valuable.

If someone want's to go deeper they are more than welcome to and you will be welcomed into the patch reviewing and testing party(?).

Additional Resources

If you have any additional resources please feel free to share.

Tutorial: Converting From Audio Module To Filefield

matt — Tue, 09 Dec 2008 15:12:32 +0000

When we updated the Geeks and God podcast to drupal 6 we migrated away from using the audio module and are now using CCK and filefield to manage our audio. This has enabled us to use modules like filefield podcaster to generate our feeds and have views generate different feeds for our different file types.

The audio module has been good to us but we didn't need most of its added functionality and some things, like ID3 tag management, had turned into a little bit of a nuisance.

A number of other people have wanted to make the same conversion and I have been asking me how to do it more than a few times. So, I wrote up a tutorial called "Drupal: Converting From Audio Module To Filefield Module".

Drupal Case Study: GeeksandGod.com

matt — Tue, 09 Dec 2008 14:55:44 +0000

Last week, Bob and I launched a new look Geeks and God podcast and community site. When we had first launched the site two years ago, under drupal 4.7, it was a basic site where page nodes were our podcast and we used an outside software like feeder and feedburner to generate our feeds. Since then a community has grown around the podcast, the technology has come a long way, and the dynamic around the site has changed. To embrace this we rebuilt the site, added new features, and re-constructed our data model (we have a data model for the first time).

In Episode 102 of the podcast we walk through many of the technical changes, the modules we used, how and why we built what we built and more.