Recently, one of my software teams got into a heated discussion about branching and I had that "here we go again" feeling. It's surprising how often this wheel seems to be re-invented, and how the issues around branching should be universally understood, but aren't.
In my last few gigs, coming into organizations with teams working on commercially released software where you think they'd know better, in the first couple weeks someone always hauls me into a room with a whiteboard, cracks a few colored markers and says, "let me explain our branching strategy." Before they start I know it's going to suck, and it always does. The reason this is so predictable is that configuration management is a straight and narrow path. If you need a "strategy" for branching you've already sunk to some circle of self-inflicted Hell. These are opportunities for easy management scores where you can lead people out of Hell onto the righteous path and make (almost) everyone happier.
Usually the sin is too many branches for no good reason with complex integration schemes between them and varying amounts of testing being done on each. Our discussion last month was a rare one about not branching at all, but just shipping straight out of a development trunk. A few of us had to patiently explain the rules: you need a branch to stabilize the release version while most of the team continues to work on the development trunk. If you don't branch, someone will add a new feature we don't want in this release. They will introduce a bunch of bugs that will stop-ship the release, and that will cause lots of people to be very angry with us when we can't release anything and we'll look like a bunch of idiots.
Surprising as it was to find ourselves explaining something that every software developer should have learned on their first day-job, I was feeling that deja vu from all the other different-but-equally-bad configuration management decisions I've seen recently. So if the two ways off the straight-and-narrow are too many branches or no branches at all, how do you know when you have the right number?
Obviously, there is no single right number, but there is a law of nature that always applies: there is only one good reason to branch, and that is to ship software. You branch to release, period. The rule of thumb for when to branch is: as late as possible before someone needs to add something that won't make the current release. This can have two flavors (that often amount to the same thing anyway):
- either the new branch will become a dedicated "release branch" where the software is in late stage testing and restrictions are put on changes going in, or
- the new branch will become the beginning of the development trunk for the next release, and someone is starting early development of a feature that will interfere with the current release. (Note in this case, the old branch could still be a development trunk if it's not yet time to branch it for release.)
How many branches is it valid to have at once? For every release in active development you should have a development trunk. You should split off a release branch for each release from that trunk. You should branch as late as possible in the release cycle, and if you do a minor (meaning bug-fix, sustaining) update of that version, the update release line should be a (new development trunk) branch off that branch. Each release branch should live until the last update/patch for a version, then it should die. You can have parallel development trunks for as many active versions that you are developing concurrently. One trunk per product version. Period. Any more or fewer branches than that and you are making trouble for yourself. It may seem like a good idea at the time, but the downsides will suck you dry.
In my last few gigs, coming into organizations with teams working on commercially released software where you think they'd know better, in the first couple weeks someone always hauls me into a room with a whiteboard, cracks a few colored markers and says, "let me explain our branching strategy." Before they start I know it's going to suck, and it always does. The reason this is so predictable is that configuration management is a straight and narrow path. If you need a "strategy" for branching you've already sunk to some circle of self-inflicted Hell. These are opportunities for easy management scores where you can lead people out of Hell onto the righteous path and make (almost) everyone happier.
Usually the sin is too many branches for no good reason with complex integration schemes between them and varying amounts of testing being done on each. Our discussion last month was a rare one about not branching at all, but just shipping straight out of a development trunk. A few of us had to patiently explain the rules: you need a branch to stabilize the release version while most of the team continues to work on the development trunk. If you don't branch, someone will add a new feature we don't want in this release. They will introduce a bunch of bugs that will stop-ship the release, and that will cause lots of people to be very angry with us when we can't release anything and we'll look like a bunch of idiots.
Surprising as it was to find ourselves explaining something that every software developer should have learned on their first day-job, I was feeling that deja vu from all the other different-but-equally-bad configuration management decisions I've seen recently. So if the two ways off the straight-and-narrow are too many branches or no branches at all, how do you know when you have the right number?
Obviously, there is no single right number, but there is a law of nature that always applies: there is only one good reason to branch, and that is to ship software. You branch to release, period. The rule of thumb for when to branch is: as late as possible before someone needs to add something that won't make the current release. This can have two flavors (that often amount to the same thing anyway):
- either the new branch will become a dedicated "release branch" where the software is in late stage testing and restrictions are put on changes going in, or
- the new branch will become the beginning of the development trunk for the next release, and someone is starting early development of a feature that will interfere with the current release. (Note in this case, the old branch could still be a development trunk if it's not yet time to branch it for release.)
How many branches is it valid to have at once? For every release in active development you should have a development trunk. You should split off a release branch for each release from that trunk. You should branch as late as possible in the release cycle, and if you do a minor (meaning bug-fix, sustaining) update of that version, the update release line should be a (new development trunk) branch off that branch. Each release branch should live until the last update/patch for a version, then it should die. You can have parallel development trunks for as many active versions that you are developing concurrently. One trunk per product version. Period. Any more or fewer branches than that and you are making trouble for yourself. It may seem like a good idea at the time, but the downsides will suck you dry.
For example, let’s say you have your first major 1.0 release coming out soon. Being a world-class development shop, you have frozen new feature development, your developers are fixing the last few major showstopper bugs, your QA department is going through a long, thorough final regression test cycle, and your product stakeholders have been triaging bugs like mad and making difficult calls to stick to your promised release date. In this example, you haven’t branched yet, but your best developers, who fixed their final bugs weeks ago, are getting anxious to start coding your next major release, 2.0. Also, your world-class sustaining development team has already prioritized and triaged the bugs that will go into the first update, 1.1 of your current 1.0 release, even before 1.0 is out. Here you should branch twice. Let’s call the main development trunk the D-xx branch, for development of any major release. You should branch a release branch off of the current development trunk for the major 1.0 release, the R-10 branch. Then you should branch a development branch immediately off that release branch that will become the development trunk of the first update 1.1 release, the D-1x branch. The original development trunk, the D-xx branch, will go on to become development trunk for version 2.0. Here you have 3 releases in active development, 1.0, 1.1, and 2.0. They all get their own branches. The cross-porting situation is that every change in a branch that’s shipping sooner has to be merged into the branches shipping later, but not vice-versa. The changes in 1.0 (R-10) have to be merged into 1.1 (D-1x) and 2.0 (D-xx). Every change in 1.1 has to be merged into 2.0.
When can you get away with not branching at all? This is what our developers hoped for in the above-mentioned argument. Well, if you don’t have any parallel development, or any concurrent development activities that overlap, you can use the same branch for everything. This can happen if you are doing research development and aren’t releasing anything to a user community that demands quality software. It can also happen in small startup situations where there isn’t enough of a team to do more than one activity at once and everyone works in lockstep. Everyone develops the first release, everyone stabilizes the release at the same time, and then everyone moves on to the next release after the first one is completely out the door. (Note even in this situation, you have to branch right when you ship if you ever want to fix a bug in the released software in a professional way.)
Why should you branch as late as possible? Because of all that cross-porting. Branching increases busy work. Unless you have super-slick auto merging software in place, and unless everyone stays away from changing the same files in different branches, you have a lot of merging to do for each change. As any developer who has shipped software will tell you, in the late stages of a release when everything is time-sensitive, new crises are erupting every day and everyone is cranky, you've just merrily doubled your developer's non-value-add workload. Even though it's essential to release, branching is a pain in the ass. And it's not just a nuisance, but true software badness supported by development process theory.
It goes something like this. Software should always work, at every stage of development. That means whatever code is in the product now should be usable, no matter how far off the ship date is. There have been varying names for this through the ages. In modern times, it is the most popular of the agile practices and is called continuous integration. It is accomplished with continuous builds and thorough automated tests, all the time, as many times a day as possible. Whenever a build or test breaks, you drop everything else and fix it. This is popular because it does wonders for a software project. The more and the earlier people use a piece of software, the faster the feedback loop is, and the better the software gets. Also, a clean build means that larger groups get to work together on the same code without constantly breaking and delaying each other.
Branching is bad because it is the enemy of continuous integration. It means you have two versions of the same software that are not integrated, and infinite chances for discrepancies between them to cause trouble. It's as simple as that.
Well, I wish it were as simple as that, but nothing in commercial software development ever has one absolute truth (except that one). The biggest complicating factor for branching has to do with a development team's size and working style- how effective the team is at working together in a continuous integration environment.
If the team, no matter what size, is keeping a continuous build mostly clean and everyone can work most of the time unblocked, branching to isolate a team or component is just plain self-defeating foolishness. Once people start breaking the build and stepping on each other, the pissed-off among them become tempted to split off into their own isolated environments where they can have the illusion that they are working more efficiently. This is usually just an illusion, no matter how much better they feel about their day-to-day local checkins and local test runs being clean.
Note that there are plenty of developers who hold the exact contrary opinion. They are loners who are happiest when they are working by themselves in an isolated environment with no other inferior developer to ever inconvenience their daily code typing. They will grudgingly let a handful of like-minded souls into their build, but they will be less and less happy as the team grows. These folks will pound the table the loudest whenever anyone breaks the build and demand their own branch where they can be protected from the rabble.
As I said above, this protection is an illusion, because eventually they will have to integrate, and when you put off integration, the integration becomes exponentially more horrific as it grows bigger. Continuous integration works because lots of little bad integrations are much better than a few horribly bad integrations. The exponentially increased badness of those integrations means they could explode into scary project-threatening and career-limiting delays any time.
Ultimately, there's no way to work with other developers without some degree of annoyance from putting up with each other's quirks and occasionally breaking each other. Walling yourself off behind team- or component-based branches within the same product gives a false sense of security, but you should see it for what it is: an admission of failure that the team doesn't have the discipline to keep a build clean and enjoy the benefits of continuous integration.
I have heard a lot of people say that agile practices in general, and continuous integration in particular, don't scale to larger team sizes. I think that idea comes from a lack of experience. I have worked on teams of hundreds all working on the same product suite in one code tree who kept the build clean and the product usable almost every day of an 18 month product cycle. It wasn't easy and the team had to work at it, but it was worth it. I have also witnessed teams of less than 5 who couldn't keep a continuous build clean on any given day if their lives depended on it. Many teams are somewhere in the middle, where their habits are too ingrained to reliably change and they have to make a close judgment call between increased integration risk and constantly broken builds.
What integration cadence would be optimal in your current project? Which is the lesser of the evils? Does having a clean interface between components help lessen the big integration delay risk? Does auto-merging lessen the code integration hassles? Are your components truly separate products on separate production schedules that should be allowed to rev independently? Is your IT department putting constraints about the number of branches or which branch gets backed up, gets servers for automated testing, etc? What if you have a distributed development environment and not everyone has access to the same source control depot? In some environments the right build/check-in/branching structure can be a legitimately tough decision. Remember the rule of thumb from the big boys is: if you ship together, you should continuously integrate together. Save the branches for releasing, and work other decisions about team make up and IT around that rule.
Why should you branch as late as possible? Because of all that cross-porting. Branching increases busy work. Unless you have super-slick auto merging software in place, and unless everyone stays away from changing the same files in different branches, you have a lot of merging to do for each change. As any developer who has shipped software will tell you, in the late stages of a release when everything is time-sensitive, new crises are erupting every day and everyone is cranky, you've just merrily doubled your developer's non-value-add workload. Even though it's essential to release, branching is a pain in the ass. And it's not just a nuisance, but true software badness supported by development process theory.
It goes something like this. Software should always work, at every stage of development. That means whatever code is in the product now should be usable, no matter how far off the ship date is. There have been varying names for this through the ages. In modern times, it is the most popular of the agile practices and is called continuous integration. It is accomplished with continuous builds and thorough automated tests, all the time, as many times a day as possible. Whenever a build or test breaks, you drop everything else and fix it. This is popular because it does wonders for a software project. The more and the earlier people use a piece of software, the faster the feedback loop is, and the better the software gets. Also, a clean build means that larger groups get to work together on the same code without constantly breaking and delaying each other.
Branching is bad because it is the enemy of continuous integration. It means you have two versions of the same software that are not integrated, and infinite chances for discrepancies between them to cause trouble. It's as simple as that.
Well, I wish it were as simple as that, but nothing in commercial software development ever has one absolute truth (except that one). The biggest complicating factor for branching has to do with a development team's size and working style- how effective the team is at working together in a continuous integration environment.
If the team, no matter what size, is keeping a continuous build mostly clean and everyone can work most of the time unblocked, branching to isolate a team or component is just plain self-defeating foolishness. Once people start breaking the build and stepping on each other, the pissed-off among them become tempted to split off into their own isolated environments where they can have the illusion that they are working more efficiently. This is usually just an illusion, no matter how much better they feel about their day-to-day local checkins and local test runs being clean.
Note that there are plenty of developers who hold the exact contrary opinion. They are loners who are happiest when they are working by themselves in an isolated environment with no other inferior developer to ever inconvenience their daily code typing. They will grudgingly let a handful of like-minded souls into their build, but they will be less and less happy as the team grows. These folks will pound the table the loudest whenever anyone breaks the build and demand their own branch where they can be protected from the rabble.
As I said above, this protection is an illusion, because eventually they will have to integrate, and when you put off integration, the integration becomes exponentially more horrific as it grows bigger. Continuous integration works because lots of little bad integrations are much better than a few horribly bad integrations. The exponentially increased badness of those integrations means they could explode into scary project-threatening and career-limiting delays any time.
Ultimately, there's no way to work with other developers without some degree of annoyance from putting up with each other's quirks and occasionally breaking each other. Walling yourself off behind team- or component-based branches within the same product gives a false sense of security, but you should see it for what it is: an admission of failure that the team doesn't have the discipline to keep a build clean and enjoy the benefits of continuous integration.
I have heard a lot of people say that agile practices in general, and continuous integration in particular, don't scale to larger team sizes. I think that idea comes from a lack of experience. I have worked on teams of hundreds all working on the same product suite in one code tree who kept the build clean and the product usable almost every day of an 18 month product cycle. It wasn't easy and the team had to work at it, but it was worth it. I have also witnessed teams of less than 5 who couldn't keep a continuous build clean on any given day if their lives depended on it. Many teams are somewhere in the middle, where their habits are too ingrained to reliably change and they have to make a close judgment call between increased integration risk and constantly broken builds.
What integration cadence would be optimal in your current project? Which is the lesser of the evils? Does having a clean interface between components help lessen the big integration delay risk? Does auto-merging lessen the code integration hassles? Are your components truly separate products on separate production schedules that should be allowed to rev independently? Is your IT department putting constraints about the number of branches or which branch gets backed up, gets servers for automated testing, etc? What if you have a distributed development environment and not everyone has access to the same source control depot? In some environments the right build/check-in/branching structure can be a legitimately tough decision. Remember the rule of thumb from the big boys is: if you ship together, you should continuously integrate together. Save the branches for releasing, and work other decisions about team make up and IT around that rule.
