Software product development is hard. How hard is it?

A couple of years ago the Standish Group published a revision of their famous "chaos" report indicating that things in our fair profession had gotten much better between 1996 and 2007. The number of "successful" software projects had moved up to a stellar 35% from the earlier 16%, and outright "failures" had reduced by a similar margin, from 31% to 19%, leaving the rest, around half of all software projects, as "challenged" (1). I would guess that our tendency to think well of our works would exaggerate the positive numbers, so I'll bet the "real" numbers, if there were such things, were lower. More on that below... (And it looks like the numbers are worse again in 2009 according to this year's report. (2))

OK so 35% is not a very impressive number, so software development must be hard. But despite those numbers, there are plenty of us who consistently ship software projects that objectively could be considered "successful." (I.e. make big money, win awards, etc.) Successful software projects are not evenly distributed throughout the industry, so how does one get on the right side of those numbers?

That's a daunting question, one that I can hardly begin to touch in a blog entry. I'm guessing a Steve-McConnell-Book-Length is about the right size to cover it, and Steve does present a lot of helpful material. However, the bibliography for getting good commercial software out the door does presently seem to leave many important topics uncovered: how to tell if a bug is "stop-ship," how to know when that API is really ready to freeze & publish, how to tell if that nagging user is a gift from God or a time-sucking servant of Satan? (Hint: if it's me, remember I'm always a gift from God and your best friend besides. Now fix that damn UI the way I told you...)

I believe development is mostly a teachable skill to people near the top of the talent range. Having played this game a dozen times with a few hundred software developers, I believe the number of candidates who could contribute to a successful software product is around 15-20% of the overall talent pool, including what I call "high intermediates" and "advanced" candidates. Daunting as it is, there are ways to win the software project game and avoid the more common chaos.

However, there is another factor in software product development, and that is the product. Getting a successful software project completed is hard, but doable. However, creating a successful product is a true dark art. I think the distinction brings us back to the Standish Group and a discussion about how we measure success. Software is soft, so it allows us a lot of leeway in how we define the boundaries between success and failure. Shipping a successful software project involves getting a bunch of not-very-objective people involved with a project to agree on the criteria for success, then fulfilling that criteria. However, shipping a successful product involves getting a bunch of uninvolved and indifferent people to agree you were successful in the form of giving you their money. That is a much higher bar, and it's only to be expected that far fewer software projects become successful commercial products. You can define your own criteria for success in a project, but there's nothing forcing the rest of the world to agree with you that your product is any good.

(1) http://www.sdtimes.com/content/article.aspx?ArticleID=30247
(2) http://www1.standishgroup.com/newsroom/chaos_2009.php

Stonewalling - It's all fun and games until somebody has to work...

The principal coin of the realm in software development is developer's time.
Consequently, a majority of significant software management, whether it's admitted or not, is spent trying to control what developers work on.
Under this definition, developers do a lot of software management, in the form of fighting over what task they get to work on, how they do it, etc.

If a certain personality type of developer doesn't like a task they've been assigned, they can find other things to do, get someone to agree these things are more important, then they have license to stonewall the original task. You see this pattern often in many different contexts, and it seems to relate more to an individual person's working style more than culture or circumstance.
In my experience, regardless of their experience or overall technical abilities, developers who tend to stonewall are significantly less effective than those who don't, with a markedly lower contribution.

the factor you haven't thought of yet

Re: relativistic judgments on software development

So many judgments about software and software products are based on weighing relative strengths and weaknesses of the effect of a particular decision on very different factors. I.e. doing it this way will help me ship about a week sooner, but will hurt usability a lot.

Consequently a lot of disputes are centered around not only how much weight you give each factor, but that there are an infinite number of factors to consider and any one person can only think of so many. People can knock a project or task on its ear by introducing new, unconsidered factors into a decision. Depending on their ability to find & articulate what really is the most important factor, this can be good, or it can just be a way to create chaos & manifest a personality disorder.