Returning to thoughts on branching... Recently, we've been discussing sandboxes. Every developer using a modern source control system works for some period of time and gets their software to where it doesn't break the build before checking in. Following best practices, developers have to integrate all the most recent changes into their sandbox, run the tests, and make sure they run clean before they check in. That way the build is isolated from each developer's code before it is ready, but each developer is not isolated from everyone else's working code in the build. So in sandboxes, the isolation is one-way, and it doesn't interfere with continuous integration. Now, what if that period of sandboxing before checkin starts to get longer, and perhaps more than one developer is working on that code? Then you could create a sandbox branch that continuously integrates with the main development line, but keeps its own changes separate.
It's true in this situation that the rest of the team can't integrate with the sandboxed code, but there can be some overall benefits to the product of using these types of sandbox branches. You can take risks with prototypes, allow features that might not make the product deadline, or allow other types of instability and creative experimentation in the main code line without constantly breaking the committed product.
Another factor is there are big teams and then there are teams orders of magnitude bigger than that. If teams grow from thousands to tens of thousands, even if everyone is very careful, the statistics are just too high that someone will break the build every hour of every day. Isolation into independently-revving components is the main strategy here, but sandboxing of isolated features becomes more and more attractive on top of that to keep people from stepping on each other.
I will still rant about two-way isolation badness, but I think these one-way integration sandbox branches are worth adding to our bag of tricks.
Thursday, September 26, 2013
Friday, January 15, 2010
Project Management Tips from Lao Tsu
From the Tao Te Ching 64:
...
Prevent trouble before it arises.
Put things in order before they exist.
The giant pine tree
grows from a tiny sprout.
The journey of a thousand miles
starts from beneath your feet.
Rushing into action, you fail.
Trying to grasp things, you lose them.
Forcing a project to completion,
you ruin what was almost ripe.
...
...damn, this guy was good.
(from the online translation at: http://academic.brooklyn.cuny.edu/core9/phalsall/texts/taote-v3.html)
...
Prevent trouble before it arises.
Put things in order before they exist.
The giant pine tree
grows from a tiny sprout.
The journey of a thousand miles
starts from beneath your feet.
Rushing into action, you fail.
Trying to grasp things, you lose them.
Forcing a project to completion,
you ruin what was almost ripe.
...
...damn, this guy was good.
(from the online translation at: http://academic.brooklyn.cuny.edu/core9/phalsall/texts/taote-v3.html)
Tuesday, September 8, 2009
Branch to Ship, Don't Branch to Put Off Integration
Recently, one of my software teams got into a heated discussion about branching and I had that "here we go again" feeling. It's surprising how often this wheel seems to be re-invented, and how the issues around branching should be universally understood, but aren't.
In my last few gigs, coming into organizations with teams working on commercially released software where you think they'd know better, in the first couple weeks someone always hauls me into a room with a whiteboard, cracks a few colored markers and says, "let me explain our branching strategy." Before they start I know it's going to suck, and it always does. The reason this is so predictable is that configuration management is a straight and narrow path. If you need a "strategy" for branching you've already sunk to some circle of self-inflicted Hell. These are opportunities for easy management scores where you can lead people out of Hell onto the righteous path and make (almost) everyone happier.
Usually the sin is too many branches for no good reason with complex integration schemes between them and varying amounts of testing being done on each. Our discussion last month was a rare one about not branching at all, but just shipping straight out of a development trunk. A few of us had to patiently explain the rules: you need a branch to stabilize the release version while most of the team continues to work on the development trunk. If you don't branch, someone will add a new feature we don't want in this release. They will introduce a bunch of bugs that will stop-ship the release, and that will cause lots of people to be very angry with us when we can't release anything and we'll look like a bunch of idiots.
Surprising as it was to find ourselves explaining something that every software developer should have learned on their first day-job, I was feeling that deja vu from all the other different-but-equally-bad configuration management decisions I've seen recently. So if the two ways off the straight-and-narrow are too many branches or no branches at all, how do you know when you have the right number?
Obviously, there is no single right number, but there is a law of nature that always applies: there is only one good reason to branch, and that is to ship software. You branch to release, period. The rule of thumb for when to branch is: as late as possible before someone needs to add something that won't make the current release. This can have two flavors (that often amount to the same thing anyway):
- either the new branch will become a dedicated "release branch" where the software is in late stage testing and restrictions are put on changes going in, or
- the new branch will become the beginning of the development trunk for the next release, and someone is starting early development of a feature that will interfere with the current release. (Note in this case, the old branch could still be a development trunk if it's not yet time to branch it for release.)
How many branches is it valid to have at once? For every release in active development you should have a development trunk. You should split off a release branch for each release from that trunk. You should branch as late as possible in the release cycle, and if you do a minor (meaning bug-fix, sustaining) update of that version, the update release line should be a (new development trunk) branch off that branch. Each release branch should live until the last update/patch for a version, then it should die. You can have parallel development trunks for as many active versions that you are developing concurrently. One trunk per product version. Period. Any more or fewer branches than that and you are making trouble for yourself. It may seem like a good idea at the time, but the downsides will suck you dry.
In my last few gigs, coming into organizations with teams working on commercially released software where you think they'd know better, in the first couple weeks someone always hauls me into a room with a whiteboard, cracks a few colored markers and says, "let me explain our branching strategy." Before they start I know it's going to suck, and it always does. The reason this is so predictable is that configuration management is a straight and narrow path. If you need a "strategy" for branching you've already sunk to some circle of self-inflicted Hell. These are opportunities for easy management scores where you can lead people out of Hell onto the righteous path and make (almost) everyone happier.
Usually the sin is too many branches for no good reason with complex integration schemes between them and varying amounts of testing being done on each. Our discussion last month was a rare one about not branching at all, but just shipping straight out of a development trunk. A few of us had to patiently explain the rules: you need a branch to stabilize the release version while most of the team continues to work on the development trunk. If you don't branch, someone will add a new feature we don't want in this release. They will introduce a bunch of bugs that will stop-ship the release, and that will cause lots of people to be very angry with us when we can't release anything and we'll look like a bunch of idiots.
Surprising as it was to find ourselves explaining something that every software developer should have learned on their first day-job, I was feeling that deja vu from all the other different-but-equally-bad configuration management decisions I've seen recently. So if the two ways off the straight-and-narrow are too many branches or no branches at all, how do you know when you have the right number?
Obviously, there is no single right number, but there is a law of nature that always applies: there is only one good reason to branch, and that is to ship software. You branch to release, period. The rule of thumb for when to branch is: as late as possible before someone needs to add something that won't make the current release. This can have two flavors (that often amount to the same thing anyway):
- either the new branch will become a dedicated "release branch" where the software is in late stage testing and restrictions are put on changes going in, or
- the new branch will become the beginning of the development trunk for the next release, and someone is starting early development of a feature that will interfere with the current release. (Note in this case, the old branch could still be a development trunk if it's not yet time to branch it for release.)
How many branches is it valid to have at once? For every release in active development you should have a development trunk. You should split off a release branch for each release from that trunk. You should branch as late as possible in the release cycle, and if you do a minor (meaning bug-fix, sustaining) update of that version, the update release line should be a (new development trunk) branch off that branch. Each release branch should live until the last update/patch for a version, then it should die. You can have parallel development trunks for as many active versions that you are developing concurrently. One trunk per product version. Period. Any more or fewer branches than that and you are making trouble for yourself. It may seem like a good idea at the time, but the downsides will suck you dry.
For example, let’s say you have your first major 1.0 release coming out soon. Being a world-class development shop, you have frozen new feature development, your developers are fixing the last few major showstopper bugs, your QA department is going through a long, thorough final regression test cycle, and your product stakeholders have been triaging bugs like mad and making difficult calls to stick to your promised release date. In this example, you haven’t branched yet, but your best developers, who fixed their final bugs weeks ago, are getting anxious to start coding your next major release, 2.0. Also, your world-class sustaining development team has already prioritized and triaged the bugs that will go into the first update, 1.1 of your current 1.0 release, even before 1.0 is out. Here you should branch twice. Let’s call the main development trunk the D-xx branch, for development of any major release. You should branch a release branch off of the current development trunk for the major 1.0 release, the R-10 branch. Then you should branch a development branch immediately off that release branch that will become the development trunk of the first update 1.1 release, the D-1x branch. The original development trunk, the D-xx branch, will go on to become development trunk for version 2.0. Here you have 3 releases in active development, 1.0, 1.1, and 2.0. They all get their own branches. The cross-porting situation is that every change in a branch that’s shipping sooner has to be merged into the branches shipping later, but not vice-versa. The changes in 1.0 (R-10) have to be merged into 1.1 (D-1x) and 2.0 (D-xx). Every change in 1.1 has to be merged into 2.0.
When can you get away with not branching at all? This is what our developers hoped for in the above-mentioned argument. Well, if you don’t have any parallel development, or any concurrent development activities that overlap, you can use the same branch for everything. This can happen if you are doing research development and aren’t releasing anything to a user community that demands quality software. It can also happen in small startup situations where there isn’t enough of a team to do more than one activity at once and everyone works in lockstep. Everyone develops the first release, everyone stabilizes the release at the same time, and then everyone moves on to the next release after the first one is completely out the door. (Note even in this situation, you have to branch right when you ship if you ever want to fix a bug in the released software in a professional way.)
Why should you branch as late as possible? Because of all that cross-porting. Branching increases busy work. Unless you have super-slick auto merging software in place, and unless everyone stays away from changing the same files in different branches, you have a lot of merging to do for each change. As any developer who has shipped software will tell you, in the late stages of a release when everything is time-sensitive, new crises are erupting every day and everyone is cranky, you've just merrily doubled your developer's non-value-add workload. Even though it's essential to release, branching is a pain in the ass. And it's not just a nuisance, but true software badness supported by development process theory.
It goes something like this. Software should always work, at every stage of development. That means whatever code is in the product now should be usable, no matter how far off the ship date is. There have been varying names for this through the ages. In modern times, it is the most popular of the agile practices and is called continuous integration. It is accomplished with continuous builds and thorough automated tests, all the time, as many times a day as possible. Whenever a build or test breaks, you drop everything else and fix it. This is popular because it does wonders for a software project. The more and the earlier people use a piece of software, the faster the feedback loop is, and the better the software gets. Also, a clean build means that larger groups get to work together on the same code without constantly breaking and delaying each other.
Branching is bad because it is the enemy of continuous integration. It means you have two versions of the same software that are not integrated, and infinite chances for discrepancies between them to cause trouble. It's as simple as that.
Well, I wish it were as simple as that, but nothing in commercial software development ever has one absolute truth (except that one). The biggest complicating factor for branching has to do with a development team's size and working style- how effective the team is at working together in a continuous integration environment.
If the team, no matter what size, is keeping a continuous build mostly clean and everyone can work most of the time unblocked, branching to isolate a team or component is just plain self-defeating foolishness. Once people start breaking the build and stepping on each other, the pissed-off among them become tempted to split off into their own isolated environments where they can have the illusion that they are working more efficiently. This is usually just an illusion, no matter how much better they feel about their day-to-day local checkins and local test runs being clean.
Note that there are plenty of developers who hold the exact contrary opinion. They are loners who are happiest when they are working by themselves in an isolated environment with no other inferior developer to ever inconvenience their daily code typing. They will grudgingly let a handful of like-minded souls into their build, but they will be less and less happy as the team grows. These folks will pound the table the loudest whenever anyone breaks the build and demand their own branch where they can be protected from the rabble.
As I said above, this protection is an illusion, because eventually they will have to integrate, and when you put off integration, the integration becomes exponentially more horrific as it grows bigger. Continuous integration works because lots of little bad integrations are much better than a few horribly bad integrations. The exponentially increased badness of those integrations means they could explode into scary project-threatening and career-limiting delays any time.
Ultimately, there's no way to work with other developers without some degree of annoyance from putting up with each other's quirks and occasionally breaking each other. Walling yourself off behind team- or component-based branches within the same product gives a false sense of security, but you should see it for what it is: an admission of failure that the team doesn't have the discipline to keep a build clean and enjoy the benefits of continuous integration.
I have heard a lot of people say that agile practices in general, and continuous integration in particular, don't scale to larger team sizes. I think that idea comes from a lack of experience. I have worked on teams of hundreds all working on the same product suite in one code tree who kept the build clean and the product usable almost every day of an 18 month product cycle. It wasn't easy and the team had to work at it, but it was worth it. I have also witnessed teams of less than 5 who couldn't keep a continuous build clean on any given day if their lives depended on it. Many teams are somewhere in the middle, where their habits are too ingrained to reliably change and they have to make a close judgment call between increased integration risk and constantly broken builds.
What integration cadence would be optimal in your current project? Which is the lesser of the evils? Does having a clean interface between components help lessen the big integration delay risk? Does auto-merging lessen the code integration hassles? Are your components truly separate products on separate production schedules that should be allowed to rev independently? Is your IT department putting constraints about the number of branches or which branch gets backed up, gets servers for automated testing, etc? What if you have a distributed development environment and not everyone has access to the same source control depot? In some environments the right build/check-in/branching structure can be a legitimately tough decision. Remember the rule of thumb from the big boys is: if you ship together, you should continuously integrate together. Save the branches for releasing, and work other decisions about team make up and IT around that rule.
Why should you branch as late as possible? Because of all that cross-porting. Branching increases busy work. Unless you have super-slick auto merging software in place, and unless everyone stays away from changing the same files in different branches, you have a lot of merging to do for each change. As any developer who has shipped software will tell you, in the late stages of a release when everything is time-sensitive, new crises are erupting every day and everyone is cranky, you've just merrily doubled your developer's non-value-add workload. Even though it's essential to release, branching is a pain in the ass. And it's not just a nuisance, but true software badness supported by development process theory.
It goes something like this. Software should always work, at every stage of development. That means whatever code is in the product now should be usable, no matter how far off the ship date is. There have been varying names for this through the ages. In modern times, it is the most popular of the agile practices and is called continuous integration. It is accomplished with continuous builds and thorough automated tests, all the time, as many times a day as possible. Whenever a build or test breaks, you drop everything else and fix it. This is popular because it does wonders for a software project. The more and the earlier people use a piece of software, the faster the feedback loop is, and the better the software gets. Also, a clean build means that larger groups get to work together on the same code without constantly breaking and delaying each other.
Branching is bad because it is the enemy of continuous integration. It means you have two versions of the same software that are not integrated, and infinite chances for discrepancies between them to cause trouble. It's as simple as that.
Well, I wish it were as simple as that, but nothing in commercial software development ever has one absolute truth (except that one). The biggest complicating factor for branching has to do with a development team's size and working style- how effective the team is at working together in a continuous integration environment.
If the team, no matter what size, is keeping a continuous build mostly clean and everyone can work most of the time unblocked, branching to isolate a team or component is just plain self-defeating foolishness. Once people start breaking the build and stepping on each other, the pissed-off among them become tempted to split off into their own isolated environments where they can have the illusion that they are working more efficiently. This is usually just an illusion, no matter how much better they feel about their day-to-day local checkins and local test runs being clean.
Note that there are plenty of developers who hold the exact contrary opinion. They are loners who are happiest when they are working by themselves in an isolated environment with no other inferior developer to ever inconvenience their daily code typing. They will grudgingly let a handful of like-minded souls into their build, but they will be less and less happy as the team grows. These folks will pound the table the loudest whenever anyone breaks the build and demand their own branch where they can be protected from the rabble.
As I said above, this protection is an illusion, because eventually they will have to integrate, and when you put off integration, the integration becomes exponentially more horrific as it grows bigger. Continuous integration works because lots of little bad integrations are much better than a few horribly bad integrations. The exponentially increased badness of those integrations means they could explode into scary project-threatening and career-limiting delays any time.
Ultimately, there's no way to work with other developers without some degree of annoyance from putting up with each other's quirks and occasionally breaking each other. Walling yourself off behind team- or component-based branches within the same product gives a false sense of security, but you should see it for what it is: an admission of failure that the team doesn't have the discipline to keep a build clean and enjoy the benefits of continuous integration.
I have heard a lot of people say that agile practices in general, and continuous integration in particular, don't scale to larger team sizes. I think that idea comes from a lack of experience. I have worked on teams of hundreds all working on the same product suite in one code tree who kept the build clean and the product usable almost every day of an 18 month product cycle. It wasn't easy and the team had to work at it, but it was worth it. I have also witnessed teams of less than 5 who couldn't keep a continuous build clean on any given day if their lives depended on it. Many teams are somewhere in the middle, where their habits are too ingrained to reliably change and they have to make a close judgment call between increased integration risk and constantly broken builds.
What integration cadence would be optimal in your current project? Which is the lesser of the evils? Does having a clean interface between components help lessen the big integration delay risk? Does auto-merging lessen the code integration hassles? Are your components truly separate products on separate production schedules that should be allowed to rev independently? Is your IT department putting constraints about the number of branches or which branch gets backed up, gets servers for automated testing, etc? What if you have a distributed development environment and not everyone has access to the same source control depot? In some environments the right build/check-in/branching structure can be a legitimately tough decision. Remember the rule of thumb from the big boys is: if you ship together, you should continuously integrate together. Save the branches for releasing, and work other decisions about team make up and IT around that rule.
Saturday, July 25, 2009
Software product development is hard. How hard is it?
A couple of years ago the Standish Group published a revision of their famous "chaos" report indicating that things in our fair profession had gotten much better between 1996 and 2007. The number of "successful" software projects had moved up to a stellar 35% from the earlier 16%, and outright "failures" had reduced by a similar margin, from 31% to 19%, leaving the rest, around half of all software projects, as "challenged" (1). I would guess that our tendency to think well of our works would exaggerate the positive numbers, so I'll bet the "real" numbers, if there were such things, were lower. More on that below... (And it looks like the numbers are worse again in 2009 according to this year's report. (2))
OK so 35% is not a very impressive number, so software development must be hard. But despite those numbers, there are plenty of us who consistently ship software projects that objectively could be considered "successful." (I.e. make big money, win awards, etc.) Successful software projects are not evenly distributed throughout the industry, so how does one get on the right side of those numbers?
That's a daunting question, one that I can hardly begin to touch in a blog entry. I'm guessing a Steve-McConnell-Book-Length is about the right size to cover it, and Steve does present a lot of helpful material. However, the bibliography for getting good commercial software out the door does presently seem to leave many important topics uncovered: how to tell if a bug is "stop-ship," how to know when that API is really ready to freeze & publish, how to tell if that nagging user is a gift from God or a time-sucking servant of Satan? (Hint: if it's me, remember I'm always a gift from God and your best friend besides. Now fix that damn UI the way I told you...)
I believe development is mostly a teachable skill to people near the top of the talent range. Having played this game a dozen times with a few hundred software developers, I believe the number of candidates who could contribute to a successful software product is around 15-20% of the overall talent pool, including what I call "high intermediates" and "advanced" candidates. Daunting as it is, there are ways to win the software project game and avoid the more common chaos.
However, there is another factor in software product development, and that is the product. Getting a successful software project completed is hard, but doable. However, creating a successful product is a true dark art. I think the distinction brings us back to the Standish Group and a discussion about how we measure success. Software is soft, so it allows us a lot of leeway in how we define the boundaries between success and failure. Shipping a successful software project involves getting a bunch of not-very-objective people involved with a project to agree on the criteria for success, then fulfilling that criteria. However, shipping a successful product involves getting a bunch of uninvolved and indifferent people to agree you were successful in the form of giving you their money. That is a much higher bar, and it's only to be expected that far fewer software projects become successful commercial products. You can define your own criteria for success in a project, but there's nothing forcing the rest of the world to agree with you that your product is any good.
(1) http://www.sdtimes.com/content/article.aspx?ArticleID=30247
(2) http://www1.standishgroup.com/newsroom/chaos_2009.php
OK so 35% is not a very impressive number, so software development must be hard. But despite those numbers, there are plenty of us who consistently ship software projects that objectively could be considered "successful." (I.e. make big money, win awards, etc.) Successful software projects are not evenly distributed throughout the industry, so how does one get on the right side of those numbers?
That's a daunting question, one that I can hardly begin to touch in a blog entry. I'm guessing a Steve-McConnell-Book-Length is about the right size to cover it, and Steve does present a lot of helpful material. However, the bibliography for getting good commercial software out the door does presently seem to leave many important topics uncovered: how to tell if a bug is "stop-ship," how to know when that API is really ready to freeze & publish, how to tell if that nagging user is a gift from God or a time-sucking servant of Satan? (Hint: if it's me, remember I'm always a gift from God and your best friend besides. Now fix that damn UI the way I told you...)
I believe development is mostly a teachable skill to people near the top of the talent range. Having played this game a dozen times with a few hundred software developers, I believe the number of candidates who could contribute to a successful software product is around 15-20% of the overall talent pool, including what I call "high intermediates" and "advanced" candidates. Daunting as it is, there are ways to win the software project game and avoid the more common chaos.
However, there is another factor in software product development, and that is the product. Getting a successful software project completed is hard, but doable. However, creating a successful product is a true dark art. I think the distinction brings us back to the Standish Group and a discussion about how we measure success. Software is soft, so it allows us a lot of leeway in how we define the boundaries between success and failure. Shipping a successful software project involves getting a bunch of not-very-objective people involved with a project to agree on the criteria for success, then fulfilling that criteria. However, shipping a successful product involves getting a bunch of uninvolved and indifferent people to agree you were successful in the form of giving you their money. That is a much higher bar, and it's only to be expected that far fewer software projects become successful commercial products. You can define your own criteria for success in a project, but there's nothing forcing the rest of the world to agree with you that your product is any good.
(1) http://www.sdtimes.com/content/article.aspx?ArticleID=30247
(2) http://www1.standishgroup.com/newsroom/chaos_2009.php
Friday, July 24, 2009
Stonewalling - It's all fun and games until somebody has to work...
The principal coin of the realm in software development is developer's time.
Consequently, a majority of significant software management, whether it's admitted or not, is spent trying to control what developers work on.
Under this definition, developers do a lot of software management, in the form of fighting over what task they get to work on, how they do it, etc.
If a certain personality type of developer doesn't like a task they've been assigned, they can find other things to do, get someone to agree these things are more important, then they have license to stonewall the original task. You see this pattern often in many different contexts, and it seems to relate more to an individual person's working style more than culture or circumstance.
In my experience, regardless of their experience or overall technical abilities, developers who tend to stonewall are significantly less effective than those who don't, with a markedly lower contribution.
Consequently, a majority of significant software management, whether it's admitted or not, is spent trying to control what developers work on.
Under this definition, developers do a lot of software management, in the form of fighting over what task they get to work on, how they do it, etc.
If a certain personality type of developer doesn't like a task they've been assigned, they can find other things to do, get someone to agree these things are more important, then they have license to stonewall the original task. You see this pattern often in many different contexts, and it seems to relate more to an individual person's working style more than culture or circumstance.
In my experience, regardless of their experience or overall technical abilities, developers who tend to stonewall are significantly less effective than those who don't, with a markedly lower contribution.
Thursday, July 23, 2009
the factor you haven't thought of yet
Re: relativistic judgments on software development
So many judgments about software and software products are based on weighing relative strengths and weaknesses of the effect of a particular decision on very different factors. I.e. doing it this way will help me ship about a week sooner, but will hurt usability a lot.
Consequently a lot of disputes are centered around not only how much weight you give each factor, but that there are an infinite number of factors to consider and any one person can only think of so many. People can knock a project or task on its ear by introducing new, unconsidered factors into a decision. Depending on their ability to find & articulate what really is the most important factor, this can be good, or it can just be a way to create chaos & manifest a personality disorder.
So many judgments about software and software products are based on weighing relative strengths and weaknesses of the effect of a particular decision on very different factors. I.e. doing it this way will help me ship about a week sooner, but will hurt usability a lot.
Consequently a lot of disputes are centered around not only how much weight you give each factor, but that there are an infinite number of factors to consider and any one person can only think of so many. People can knock a project or task on its ear by introducing new, unconsidered factors into a decision. Depending on their ability to find & articulate what really is the most important factor, this can be good, or it can just be a way to create chaos & manifest a personality disorder.
Thursday, June 11, 2009
managers going against the grain
To be a good manager, you have to challenge assumptions and knock tasks out of ruts.
Reviewer/extra pair of eyes is a big responsibility of management, so you can't be afraid to stick your neck out and call people on things.
If everything is close enough/good enough and you don't want to go against the grain, you'll end up being a babysitter & won't add any value.
Reviewer/extra pair of eyes is a big responsibility of management, so you can't be afraid to stick your neck out and call people on things.
If everything is close enough/good enough and you don't want to go against the grain, you'll end up being a babysitter & won't add any value.
Subscribe to:
Posts (Atom)