Tales from the Trenches: Growing Pains

The next time your company plans a move, you just might want to consider a change in careers.

I generally consider myself a pretty lucky guy. I’ve got a great career that I’m good at, a beautiful wife, and two wonderful kids (OEM with the wife), and one more on the way! And until recently I was also lucky enough to have always worked on either a pre-existing network or one that was built from scratch for a new company. Hence, I’ve never had to go through the pains of moving a LAN (complete with a DMZ of www, ftp, dns, and smtp) from one location to another. Never, that is, until now.

If you’ve been through this, you know exactly whereof I speak (can I get an amen?). If you haven’t, heed closely my advice, lest ye be subjected to the flagellations I’ve been forced to endure these past few months. Indeed, no move will ever be completely glitch-free, due to reasons that shall soon be apparent. Nevertheless, it’s my sincere hope that you’ll find in these pages sage counsel upon which to construct a checklist of do’s and don’ts (or as a former math teacher of mine used to say: “Uh-Huhs and Uh-Ohs”), should relocation be in your future.

Let’s Begin at the Beginning

Several months ago we realized that our current accommodations were simply no longer adequate. We were “living” in an office space of approximately 1,200 square feet, which at one time (long, long ago, when our U.S. office consisted of only six people) was more than sufficient. Fortune has smiled on us though, and we’ve recently grown to more than 14 full-time employees, with more on the way. When you consider that we often have personnel from our U.K. office visit for days at a time as well, it’s easy to see why there are times when it’s elbow to elbow.

Whenever you move an office from one place to another, you inevitably face many challenges, not the least of which are IT considerations. Nevertheless, today we’re going to focus only on the IT (and related) Uh-Huhs and Uh-Ohs.

Challenge No.1: Move LAN and DMZ to New Location

Sounds pretty straightforward, right? Well, it is… in theory. Since we needed to keep our existing network up and running until the actual move, we acquired a new range of IP addresses for our new DMZ. (Our internal LAN uses private IP addresses, so at least we didn’t have to worry about allocating addresses for our internal hosts. Thank goodness for small favors!)

Consistent with our growth, we decided to have our ISP allocate a subnet of 128 addresses, which we would then subnet down to our DMZ and external “point-of-entry.” (With IPv4 still being stretched way beyond what was ever intended, we couldn’t really justify an entire Class C space. When-oh-when will the promised IPv6 arrive?) We also decided that (since, after all, we’re growing and everything) we should bump up our bandwidth. We turned what was only (only?!) a T1 line into a flex DS-3 (T3—45Mbps max!) with a shadow T1 for fail-over support… just in case. After all, increased bandwidth enhances the customer experience, right? Never mind that now when I download beta updates from Microsoft, the process is so fast that they arrive before they’ve left!

As I’ve said, in theory the process is simple: Get the new IP address range, configure the external network at the new office, and everything’s cool, right? Right?! Wrong!

Uh-Oh No. 1: Get Circuits Installed by ISP/Telco

If you think that the Department of Justice possibly breaking up Microsoft is a good thing, just remember what happened a few years ago when they broke up the phone company into all the little “baby bells.” Now, in order to get any data service installed, you must go through at least two and up to four or five different companies: the local telco, since it owns the “local loop”; your ISP, whose role is usually “middleman;” the network backbone providers, which may or may not be wholly-owned subsidiaries of the ISP (and even if they are, it doesn’t make the process any smoother!); and other “middle-men,” depending on how big your ISP is. The process usually works something like this:

  1. Meet with the ISP to discuss your data service requirements. The ISP may even offer combined data and voice services at a discounted price to make it “easy” to set up and maintain. It won’t be, but the savings may make it worthwhile to go with the package nonetheless.
  2. Have the ISP schedule the install. This includes scheduling the local loop install from the telco. This part actually seemed to go rather smoothly until I received a call from the telco technician stating that he was in our new office and the conduit hadn’t yet been run from the office space to the phone room! A quick check of the build-out schedule confirmed that, yep, no conduit was due to be installed until the following week, which is the date I gave the ISP. Somehow this got lost between the scheduling department of the ISP and the telco.
  3. Re-schedule the installation of the circuits. Even though you should (in a perfect world) be dealing only with your ISP for scheduling, you’ll end up having to talk to the telco in order to ensure that the installation takes place on the correct date. In our case, the correct date came and went without incident… at least for the shadow T1. When they came to install the DS-3, they informed us that the building didn’t have sufficient fiber coming in to support a T3. We were told that it would take 30 to 45 days for the telco (Oh, no! Not the telco!) to run new circuits to the building. After many threats, pleas, and “We’ll get a new ISP,” we decided to use the shadow T1 as our main data line until the DS-3 could be installed. (At this point, I was praising the IT gods that we had decided to put the shadow T1 in the budget in the first place—it had just become our lifeboat!)

Uh-Oh No. 2: Get LAN and DMZ Configured

Since we knew that we’d have to build our new network on site and have it up and running before we shut down the old one, we arranged sufficient time in the build-out schedule (one week) to allow for delivery, installation, and configuration of our IT infrastructure. On the pre-determined date, we arrived at our new office with several servers, routers, switches, hubs, and sundries, only to be told that the electronic door locks hadn’t arrived, and the office space wasn’t secure. Even though our office building wasn’t located in any sort of “high crime” area, we decided to err on the side of prudence and take everything back to our “old” office. This had the effect of making things even more crowded, but at least there was a lock on the door!

The door locks were installed the following day and we began to move equipment over and build our new domain. The new domain, of course, needed to co-exist with our current domains, so trusts were established, Exchange data was transferred, and user accounts were created for existing employees in the new domain. We also had to build our DMZ, which meant setting up www, ftp, dns, and smtp servers; copying existing data; and updating primary DNS servers to point to the new DMZ instead of the old one. Since it can take up to 72 hours for the DNS entries pointing to our old IP addresses to expire, and we absolutely had to vacate our “old” office within four days, getting the DMZ up was a priority. (Tip: Always leave a “buffer zone” between occupying a new office and vacating the old one, just in case!)

Challenge No. 2: Get Phone System Transferred

The reason I mention this is because a) our phone system is a PRI voice T1 and switch (enough to make a seasoned tech-head drool!); b) I’m betting that many of you out there work for companies that don’t have separate telecom geeks, and hence you are the de facto phone system experts in your organization (reason enough to be prepared for potential glitches); and c) it represents a potentially major Oh-Uh during any office move!

To be fair, the phone system move was probably one of the smoothest aspects of moving to our new office. Getting everything ready for the phone move was the nightmare. The most significant hurdle was keeping our existing “published” phone numbers. We went round-and-round with the phone company, including another bout of threats, pleas, and “We’ll just get another phone company.” Finally, the issue was resolved and we figured out a way to keep our existing numbers. (The actual process seems to involve routing them through NASA launch system computers and via top-secret military satellites. I think there’s a slight possibility that we may have a disruption of service during periods when the space shuttle is in orbit.)

As I said, moving the switch and turning up the circuit went surprisingly well. Better, in fact, than when we initially had the system installed in the “old” office several months earlier. We checked, rechecked, then did it all again. I left the office sometime after midnight on that Friday. Considering that I was leaving for the airport the following day at 5:20 a.m. for a previously scheduled vacation, I felt lucky that we were able to get everything done.

Uh-Oh No. 3: Get Users on New Domain

The good news was that we had finished the move on June 30, just in time for the four-day Independence Day weekend. And vacation or no, I knew I’d start getting calls Wednesday morning from frustrated users. Considering how big of an Uh-Oh this could have been, I was surprised by how smoothly it went. Make no mistake… I did get calls, but most dealt with getting users getting Windows 2000 Pro laptops (that they took home over the weekend and we didn’t get a chance to pre-configure) to join the new domain. Of course, once we successfully migrated them into the new domain, I was immediately besieged with calls of “What the heck happened to my emails, documents, and desktop settings?!.” (Tip: If any of your users use Windows NT or 2000, prepare them for the fact that there will be some reconfiguring. After all, they are joining a whole new domain!)

Uh-Oh No. 4: Clean Up the Mess

I know I promised to focus only on IT issues, but this last point really bears mentioning. Although we had pre-moved most of the computer equipment, the office furniture was another story. We decided to hire a moving company that promised to box everything up, move it to the new office, and lovingly unpack it. (Tip: Get a referral from someone you trust before hiring a moving company.) When I arrived back from my vacation, the office looked like the moving company had scooped everything up with bulldozers at the old office, delivered it to the new office in a truck with pogo sticks instead of wheels, and (literally) dropped it off at the new location! I still haven’t found the power supply for my KVM switch, so my desk looks like a monitor farm.

Moving into a larger office space can be great for morale. It means that the company is doing well. It also means that people get more personal space (at least for a while). And in spite of all the Uh-Ohs, I think the experience has been mostly positive. When the time comes for you to expand into larger accommodations, just remember to plan, plan, plan. Then make sure you remember that nothing ever goes as planned.

comments powered by Disqus
Most   Popular