Cat5bird Seat

Surviving in a World Without Betas

Community Technical Previews are here to stay -- if you're concerned about quality issues, keep a nice, high barrier between your test and production environments.

• By Mike Gunderloy
• 05/23/2006

"If I had my way, there’d never be another beta," That’s not some Linux zealot speaking; rather, it's Paul Flessner, Microsoft senior vice president of Server Applications, discussing changes to the way that the SQL Server team (and many other teams within Microsoft) had started building applications. If you've been hanging around the Microsoft world for a few years, you've probably grown used to the pulse of an application release: Alpha (which most people never got their hands on), Beta 1, Beta 2, Release Candidate, Release. Well, it seems to be time to get un-used to it. The CTP process is here to stay.

CTP, if you haven’t run across the term before, stands for "Community Technical Preview," a new type of unreleased software that isn't quite a beta build. Different teams at Microsoft seem to be treating CTPs in different ways, but overall the defining characteristic of a CTP build seems to be that there’s less process associated with it. Over the last decade or so, putting together an official beta build of a major Microsoft project has become a tremendous undertaking, requiring thousands of man-hours in order to produce something that’s nearly production quality. With CTP builds, teams pick a day, and if the build for that day isn’t horrendously broken, they throw it over the fence to testers. Thus CTP builds tend to be less polished than beta builds. But on the flip side, they can come out more frequently, and provide better insight as to the current thinking and progress of the product team, than beta builds possibly could.

Sounds great, but there’s a darker side to CTPs. With software escaping from Microsoft in a less-tested state, you have to be more vigilant about how you use it in your own organization. I know plenty of people who have rolled out late betas of Windows or SQL Server or Visual Studio on a fairly live scale, confident that those builds went through extensive QA at Microsoft. With CTP builds, you can’t count on that being the case. There’s no quality guarantee, and planning to roll out a particular CTP build could prove to be a wildly dangerous idea.

CTP builds have also not been well coordinated across groups in the past. During the 2005 beta cycle for development products, there were times when the latest SQL Server CTP and the latest Visual Studio CTP were incompatible, simply because the teams were on slightly different cycles. This sort of thing can be a problem when you want to test integration scenarios, of course; you need to plan on extra time to sort out the dependencies and find a set of builds that play nicely together if you’re involved with multiple pre-release products at the same time.

In general, with Microsoft changing the rules for how product testing works, it’s time to re-evaluate your own attitude towards pre-release Microsoft products. If you've tended to widely distribute betas, you'll want to rethink that policy when it comes to CTP builds. Now that there's not so much of a quality barrier for these builds on the way out of Redmond, most organizations will want to set up their own quality barrier on the way in. Think of accepting a CTP build in the same terms as installing a security patch: you need to decide whether this build is right for your organization. Does it offer enough potential benefit, in terms of moving your own projects forward and acquainting your developers and administrators with new stuff, to be worth the potential pain? That’s the question that you have to answer before people start installing CTPs.

Consider appointing a CTP Czar to monitor this new world of non-beta software. Such a person might be the sole point of contact for incoming CTP releases. They’d install each release in an environment that mimics your company’s standard hardware and software (virtual machines will likely be essential here), and make an initial quality judgment. Those within the organization who want to play with the new toys would have to demonstrate that their need warrants the risk, as determined by the Czar. This person could also have the tedious job of trying to keep the conflicts and dependencies between multiple CTP builds sorted out.

And remember, whatever you do, don’t go installing those CTPs on production servers. You might have gotten away with that with Beta 2 builds in the past, but in this new world, wait for the real thing. You’ll be glad you did.