• Test Engineer - IP Network (M4)
• Project Manager - Telco
• Project Manager - IT Relocation...
• Senior Project Manager - Multi...
• Wintel Systems Specialist

The IT worst case scenario survival guide

Dan Tynan, Information Age

14/02/2006 12:08:00

When you're that far up the proverbial creek, and neither baling out nor soldiering on seem like workable options, it's easy to imagine that no one else in IT has ever been in such an impossible situation.

You'd be wrong about that. According to research by The Standish Group, one out of five IT projects fail outright, and more than half come in late or over budget. Why? The standard answer from the business side, "It's IT's fault", conveniently ignores equally likely causes: bad requirements management, poor business planning, lousy communication, or the dreaded "scope creep".

Here, we've identified five of the most common scenarios in which projects fail and what IT can do to avoid them. The stories you are about to read are true, although company names have been obscured to protect the guilty. Some date from the technology-at-any-cost 90s, while others are ripped from today's headlines. But their lessons are timeless.

Scenario 1: Outsourcing run amok

Back in the mid-90s, a large vending machine services firm wanted to save money by centralising its operations, so it hired a Big Five consultant to implement a $US20 million ERP system. Big mistake.

"Turns out, the consultant's dream was to build a management information system from scratch because he thought he knew how to do it better than anybody else," says Bob Price, former CEO of Control Data and author of The Eye for Innovation: Recognizing Possibilities and Managing the Creative Enterprise (Yale University Press, 2005).

The result was a near total meltdown. Prices in the ERP system didn't match those in the catalogue, so customers refused to pay. The centralised system made it too expensive to collect from individual buyers, so revenues dropped. Midlevel managers who were never consulted about the system revolted and left in droves. The firm's president lost his job over the fiasco, and his boss, the CEO of the firm's parent company, ultimately resigned.

What do you do if the consultant you thought was a godsend turns into Godzilla?

1. Assess your internal talent. Nobody in the organisation really understood the project, leaving the consultant totally in charge. If you don't have the necessary expertise in house -- or are unable to hire it -- don't take on the project. "It's better to do business in a poor way," Price says, "than undertake something you don't understand and end up not doing business at all."

2. Match the solution to your business. In this case, the firm was trying to force fit a centralised structure on an extremely decentralised business model. It ultimately solved the problem by putting sales and inventory management back out in the field but handling customer data centrally.

3. Avoid proprietary technology. The ERP system was not only a custom job, it was written in an obscure version of ALGOL (Algorithmic Language) -- a programming language the consultant loved but nobody else knew how to use, according to Price.

4. Manage it closely. Even the most minutely detailed consulting agreements can't cover everything, especially as changes or clarifications are needed. Consultants can be very useful, Price says, but you need to state in no uncertain terms what you want them to do, and manage them very closely.

Scenario 2: The incredible expanding IT project

Three years ago, a major West Coast tech services company decided to roll out a Web-based content management system to handle its internal communications. But inexorably, the features list began to grow. Could they use the same system for customer support? Sure, said the systems integrator. How about selling research reports to clients? No problem. The budget for the project rapidly climbed to $100 million.

The catch? The systems integrator was also the software vendor who'd built the content management system. The vendor had never met a problem their software couldn't solve -- for millions in additional development fees, naturally.

"By the time they contacted us, the company had spent closer to $280 million, and the percentage of test cases that actually worked was zero," says George Kondrach, executive VP of Innodata Isogen, a content supply chain consultancy. Innodata recommended scaling down the project and bringing in third-party software to handle jobs the content management system wasn't designed to do.

Kondrach says Innodata could have fixed the problems for about $10 million, but that would have meant the client would have had to admit failure. Instead, the client continues to spend millions each year trying to make the system work.

How do you keep a project from spiralling into unfathomable depths?

1. Be vendor agnostic. Hiring an integrator with a vested interest or one whose expertise is limited to a single vendor's products is a recipe for disaster, Kondrach says. Any proprietary solution can limit options later.

2. Know thy software. Understand the difference between a software feature and a new product category. Don't try to use one system to do the work of five different applications.

3. Don't bury your mistakes. Admitting failure early is one the best things you can do. "Don't euphemise a project that's failing as 'making progress' or 'iterative development'," says Kondrach. "It's better to acknowledge that your initial design decisions are not workable."

4. Know when to give up. "Don't throw good money after bad," says Patrick Gray, president of Prevoyance Group, a New York-based consulting firm. "It's painful, but it's still better to waste $100 million than $200 million."

Scenario 3: The hacker from hell

Last year, the Web site of a large financial services firm was hacked. Acting quickly, the company took down its primary site and brought a duplicate one online with virtually no service hiccups. But what the firm didn't know was that the attacker had planted malicious code on the site months earlier -- enough time for his backdoor to be propagated into all the site's backup sets. So when the second site came online, the hacker continued accessing user accounts. The firm had to go completely offline for several hours while it identified and fixed the problem.

By relying on a single application or platform for core business functions, you create a single point of failure, says John Pironti, principal security consultant for Unisys. In this case, the same flawed code was used in the firm's primary and backup Web sites -- which the hacker happily exploited.

How do you keep hackers from finding your Achilles' heel?

1. Diversify platforms. Enterprises need to put their core business functions on multiple platforms and keep data synchronised among them. So, for example, if the Windows platform is compromised, the organisation can fire up the Linux implementation and operate with minimal interruption.

2. Check your backups. Organisations should verify their code before they back it up or restore it, says Pironti. The easiest way is to create a hash from compiled code before it's put into production, then do the same thing before each backup. If the hashes aren't identical, the code has been tampered with -- and the organisation will discover it within hours, not months.

3. Audit processes. Most organisations need to do a better job of logging system activities and correlating them into events. By establishing a range of what "normal" transactions look like, organisations can more easily detect and respond to anomalous behavior.

4. Plan for small disasters. Most enterprises have their own worst-case-scenario guidebook on how to handle huge disasters but don't have a clue what to do if just part of their system -- such as a Web site -- breaks down. They need a plan for every part of the puzzle, says Pironti.

5. Think business, not IT. Organisations should approach security and continuity with the idea of doing whatever it takes to keep the business going -- even if that means reverting to pen and paper, says Pironti. "If institutions looked at things from the perspective of business processes and not technology, they would develop much better vulnerability management plans," he says.

Scenario 4: Software of the damned

Back in the late 90s, a large maker of consumer products thought a new ERP system would be just the ticket. All the top executives signed off on it, and the IT department got busy. Many months and $40 million later, the project was finally done -- and users wouldn't go near it.

"When they finally flicked the switch, they had a near-total rebellion from their users," says Phil Bloodworth, US leader of the IT Effectiveness practice at PricewaterhouseCoopers (PwC). Nobody had bothered to talk to the people who were supposed to use the thing. Some modules were retrofitted for use in other departments, but the bulk of the system was abandoned.

How do you keep your expensive project from becoming a lifeless zombie?

1. Talk to users. The bigger the project, the more important communication becomes, says Joel Koppelman, CEO for Primavera Systems, a maker of project management software. "The project itself may be done perfectly, but if the people are not prepared for the change, it will fail."

2. "Socialise" the project. Have users help spec out the project and evaluate candidates, advises Bloodworth. "Get everyone to buy in by asking them what their needs are, what the current system does well and what it does poorly, and what's required for the job."

3. Avoid "scope creep". Keep the feature list in check by putting a price tag on every change request. "A user or manager's request for a 'critical' change may become less critical when it is determined the associated change would burn $1.7 million," Gray says.

4. Keep training. "You need ongoing training, both before and after you convert to the new system," Bloodworth says. "Tell users: 'That [old] report is now this new report; your new screen will look like this.' "

5. Break it down. "Multi-year, monolithic IT projects are a sure-fire recipe for failure," says Michael Hugos, a former CIO of an $8 billion distribution cooperative and author of Building the Real-Time Enterprise: An Executive Briefing (John Wiley & Sons, 2004).

"The key to success is breaking big projects into smaller, self-contained pieces that can be put into production within three to nine months." The business side can see what it's getting for its money -- and bail on the project if it fails to meet expectations.

Scenario 5: The IT department tied in knots

PwC's Bloodworth says he was once called in to consult with a large entertainment company where technology was viewed as a necessary evil, unconnected to the company's core business. Even seemingly simple changes required huge lead times. The backlog on pending projects was enormous. Nothing ever got done, and IT was viewed as a dismal failure.

But Bloodworth discovered the fault lay not with the tech department but with how the company managed -- or rather, mismanaged -- projects. There was no process for approving projects, prioritising them, or determining which projects were too old or marginal to pursue. "We found a portfolio hundreds of projects deep," he says. "IT just focused its resources on whatever group shouted the loudest."

How do you cut through the tangle of demands and make IT functional again?

1. Speak truth to power. "You need to have someone in the organisation with the ability to take a step back and say, 'This is out of control; we have a problem we need to fix,'" says Bloodworth. "If no one's willing to step up, you may need to bring in an outside consultant who can tell the CEO, 'We have a governance issue, and it starts with you.'"

2. Limit the number of decision makers. One of the first things PwC did was build an executive committee to approve IT projects, along with a second-tier steering committee for day-to-day management -- and made everyone accountable for cost. "Putting in a formal process really helped them," Bloodworth says.

3. Analyse the portfolio. Scrap anything that's old or not aligned with core business goals, then prioritise what's left.

4. Elevate IT's stature. IT has to be about more than just keeping all the systems glued together every night, says Scott Christian, a director in PwC's IT Effectiveness practice. To create an environment where tech is viewed as vital to success, organisations must empower techies to be strategic thinkers and not just order takers. They may also need to parachute in an enlightened CIO who's a businessperson first and a techie second.

SIDEBAR

When disaster really strikes

You can't be blamed for acts of God. At least, not if you've got a robust disaster recovery plan in place

A failed project may feel like a disaster, but it's nothing compared to what a hurricane, earthquake, fire or man-made calamity can do to your business. As an IT pro, you'll be expected both to protect your organisation's technology assets and help it recover from the event as soon as possible. Here's how to prepare and prioritise:

Priority 1: People. "It's a cliche, but it's true: Your employees are your most vital asset," says John Laye, author of "Avoiding Disaster: How to Keep Your Business Going When Catastrophe Strikes" (John Wiley & Sons, 2002). That means knowing where your employees are and how to reach them quickly.

Hurricane Katrina knocked out cell towers, making mobile phones useless, so consider alternative methods such as long-range walkie-talkies. Smart companies also distribute emergency guidebooks and train employees so people know what to do without waiting for instructions.

Priority 2: Data. "You can replace new hardware servers, communications wires, and even buildings, but you can't go out and buy new data," notes Sami Akbay, senior director of marketing for GoldenGate Software, a provider of disaster recovery services for Fortune 500 companies. Daily backups and off-site storage are a good start, but backup sets must also be safe and accessible -- especially when roads are washed out and airports are closed. Test the backups by periodically restoring them; if the data is corrupt, you'll want to know that before the tsunami hits.

Priority 3: Software. Identify the applications most vital to getting the business back up -- such as HR, payroll, or e-mail -- and give them top priority, says Roy Jackson, vice president of IT consulting for CAS Severn. Make sure more than one staffer can operate them if needed. "We find in many organisations that only a single administrator knows how to start, stop or run a critical application," he says. If that person is unavailable, your organisation could be dead in the water.

Priority 4: Hardware. If your datacentre is wiped out, you'll need another -- and fast. Companies like Agility Recovery Systems and SunGard Availability Services can have replacement systems available within a few hours. If you're in a business where timely transactions are critical, you'll need a duplicate datacentre on "hot standby" ready to take over when the first one fails.

Priority 5: Facilities. Having a backup datacentre or storage facility only 50km away won't do you any good if you suffer a Katrina-sized disaster. "Remember to have enough geographic distance between your primary and secondary environments," advises GoldenGate's Akbay. "Ideally it would be in another state entirely."

But none of this advice will do you any good unless your staff is trained and ready to go when the worst happens. The key is making the disaster a "rehearsed event", Laye says. "If you're prepared and your management team considers it a rehearsed event, then it doesn't have to be a catastrophe."

SIDEBAR

Infamous IT meltdowns

The US Government's track record with IT projects is not exactly stellar

When it comes to IT projects, nobody bungles things better than Uncle Sam.

The FBI's "Virtual Case File" Price tag: $170 million The replacement for the Fed's antiquated case management system fell victim to massive scope creep, caused in part by changes to the FBI's mission after September 11 (having five CIOs in four years probably didn't help). Last year the Feds scrapped the VCF project in favour of a new SOA-based plan called Sentinel.

The FAA's "Advanced Automation System" Price tag: $2.6 billion This early attempt to modernise the nation's air traffic control system crashed before it ever took off. In 1994, the FAA wrote off an estimated $1.5 billion of the expenses and tried to salvage the rest. The AAS itself was merely part of a 25-year, $45 billion overhaul that has been fraught with every ill known to IT.

The IRS's "Business Systems Modernization" Price tag: $8 billion and counting. After abandoning a $3.4 billion modernisation project in 1997, the IRS embarked on a 15-year, $8 billion plan that quickly grew even more troubled than its predecessor. After numerous personnel changes and some scathing GAO reports, the project might finally be getting on course, if not on budget. Then again, it's not like they're going to run out of money.

Department of Defense's "Business Systems Modernization" Price tag: $19 billion (fiscal year 2004) From the people who brought you the $600 toilet seat comes the modernisation project from hell. A May 2004 report by the GAO found the project to be "fundamentally flawed ... and vulnerable to fraud, waste and abuse". The DOD's IT roster includes over 200 systems for managing inventory and 450 for personnel, but all come in the same colour: green.