New Bamboo Web Development

Bamboo blog. Our thoughts on web technology.

Notes from the open source battlefield

by Mark

Open source has provided us developers with so many valuable tools that we use every day that we'd be years behind where we are without it. It provides one of the best ways to contribute back to the community and develop as a programmer. For the most part it's fun to be involved in, but there are times when it can be tedious.

This blog post is about lessons learnt doing open source.
Hopefully it will serve to provide some food for thought for those maintaining, contributing to and using open source projects (which probably includes you if you're reading this!), particularly larger ones.

The end of the honeymoon period

Many open source projects begin when a creative person (which, as a web developer, pretty much describes everyone in my field) comes across a problem or a technical "itch" they want to scratch, and they say "I know! I'm gonna create some open source software to solve this and put it out there for the world to use". It could be either an app or a library.

So you write your perfect library for printing pictures of cats in ASCII. You put it up on github with a super helpful README.
Then you name it after an animal or vegetable, or a random but strangely satisfying word like "flannel", and you release it to rubygems or npm.

And then you sit back and wait for the accolades to pour in.

Before long you receive a message on github. Is this a message of congratulations for solving the famous "ASCII cat printing" problem? Or an invitation to speak at the upcoming world feline appreciation conference in Hawaii?

Oh... No. It's an issue - it doesn't work with ruby 1.9.2.

That's annoying.

OK - you put it on your "to do" list and leave it for now. But before long you get another, then another, until there's a whole pile of issues tugging at your sleeve, demanding your attention.

And though you want to address them, you haven't managed to get round to it, and soon it gets to the point where even tackling one seems pointless, as it would just be making a small dent in a mountain. Much like the broken window idea, if there were only one issue you'd probably just fix it, but now that having open issues is the norm, you just let them fester.

So your beloved library has now become a millstone around your neck, each new issue meaning another chunk of your precious time, no longer doing "fun" work with visible results, but doing menial work with barely any visible change.

One of my libraries, Dragonfly, at one point had over 60 issues and 20 pull requests, some of which were over a year old (slap wrist!). It may not seem like that many, but if the average issue takes, say, half an hour to deal with, then that's already 30 hours of free time, not to mention documentation, general upkeep, etc., meaning a number of weeks' worth of work, let alone trying to "have a life".

Ouch!

Was it worth doing in the first place?

Keeping your sanity

Well I would say "yes", provided you take steps to manage it properly.

Work/life balance

Most open source is done in people's free time, so if it starts growing into a monster to maintain, you have to start thinking about whether you really want to be spending time in evenings and weekends doing something you're already doing all day (if development is your job), but for no money.

When friends/family see me "working" outside of work on open source they often say "are you getting paid for this?", and when I say "no", they look aghast and say "well why don't you sell it?", to which I reply "it doesn't really work like that".
And nor should it. If it weren't for all the free tools and libraries I've been using over the years I'd still be coding my own web server from scratch using Microsoft Word.

But if the project does start to grow, there comes a point when you probably need to decide whether to recruit helpers, or even stop maintaining it altogether and hand over to others, like requirejs-rails has done, or to hide under a rock and pretend the angry users aren't there (bad idea!), or to step up and do the required maintenance yourself, albeit with your personal time taking a significant hit.

One important thing I've learnt is to keep your code as simple as possible. Don't include features unless you're absolutely sure they're needed. Unneccessary features create more maintenance work for you, not to mention making it harder for the user to get their head round your library.

Issues and contributions

If your library gets popular, then soon after releasing it you will start getting pull requests, issues and general questions/suggestions. Some of them are things you never thought of, some are recurrent themes (which are a good pointer that something needs changing), some are simply a case of RTFM (which are particularly annoying) and some are significant contributions.

Issues may come in one of the following categories:

Small change

This is a standard type of issue, one which probably requires a small amount of work on your part, if it does need doing. It's best to try and get it done quickly if you can.

Big issue

These should hopefully be fewer and further between, but are more urgent and may benefit from others' help and collaboration.

Paradigm shift

Some requests detract from the vision you have for your library and will usually not be worth doing. It's still worth replying to explain why though.

Not an issue

Sometimes people use the wrong channels to discuss things that aren't issues. It could be that you haven't properly documented the correct place to go (e.g. google groups, IRC, etc.). It could also be that they're just being annoying.

What the???

Some issues require more explanation

It's easy to assume that it's not really an issue, but often it really is so make sure you get them to clarify!

Self closers

These are my favourite type of issue.

Because each issue/pull request is something you have to deal with and will take up your time, it's very tempting to start viewing them as they come in as unwelcome visitors, trampling their mud in your clean, tidy house. Especially grating are the visitors (to continue the tenuous analogy) who come in without any airs or graces, don't give you any clue as how to clear up the mud, and forget that they're reaping the benefits of your work for free.

The truth is, however, that most of the time issues and pull requests highlight things you've missed, or things which genuinely better your code. To look at the flip-side, you may be developing it for free, but hundreds of other people around the world are testing it, refining it and improving it for free! That's the sort of thing many companies would pay good money for!

When suggestions and pull requests detract significantly from your vision for the project you may need to exercise good judgement, which could mean sticking to your guns and not just allowing anything through the door. In these cases it's good to remember that the contributor may have spent time on it and not to rudely dismiss it without explaining why.

Conclusions

So is it all worth it after all?

As I mentioned earlier, my answer would be a resounding "yes!", as long as you know what you might be getting into and are prepared to deal with the consequences.

There are many things in it for you - you'll learn, become a better coder, scratch your creative "itch", have some great software which you could find useful in the future, you'll have the satisfaction of having created something and it will be good for your reputation.

As for others, they get to use some great software for free, and have the opportunity to contribute to something themselves.

Win-Win!

The caveat would be that you don't want to get to that place where you resent ever having started it.
For small projects, you probably never will.
For larger projects, I would recommend the following

  1. Don't let issues build up. This is the hard part, but it's better even to communicate that you won't fix something and close it than to leave it there never to come back to it.
  2. Keep it simple! Don't start introducing functionality into your library that no-one uses and yet is still sitting there like a steaming turd, requiring your precious maintenance time.
  3. Don't take on/publicise anything you aren't willing to maintain/own - or at least hand over / recruit help for.
  4. Keep your own vision - don't let suggestions pull your library around until it loses its identity.
  5. Appreciate contributions - give people the benefit of the doubt when they raise an issue - it may well make your library better! Either if it doesn't, be honest and communicate well.

Personally, I finally managed to demolish my pile of issues by simply knuckling down and doing it, though it took a very long time and I should never have let build up in the first place (I even had one issue that said "is your library dead?"). Others may choose to recruit extra help. As long as the original vision for the project is maintained, I would say either choice is valid.

I hope that at least some of that is useful.
It would be great to hear other people's experiences of open source. Please feel free to comment below, or post a question.