Neil Hennessy, vice president of IT engineering for PeopleSoft, learned about Pilot's collapse in a most unusual way. At the end of a weekly meeting with his Pilot rep, the man announced, "I have to go back to the office and get fired at 4:30." And in the week leading up to Pilot's bankruptcy filing, Hennessy says he was fending off a wake of vultures.

"One guy calls from Southern California, and he's telling me how he can offer me everything Pilot did for less money," Hennessy says. "So I ask him, 'How many employees do you have?' and he tells me 40. So I said to him, 'Pilot had 400. Why would I trust you?'" Truthfully, Hennessy had started to lose faith in Pilot beginning a year before its collapse. He was particularly worried about the company's scalability. "They just couldn't step up. Not that they didn't try. Their model was very secure, but we started looking at other options back then," he says.

Hennessy's favored alternative was to phase out his managed security contract and take the task back in-house. This, after all, was the year of uber-viruses and broad, destructive hacks. Hennessy decided that security was just too critical to outsource.

His transition plan meshed with his contingency plan. Hennessy already had a backup carrier with a "dark" data line, one that's not turned on but could be activated in an emergency. And he started building an internal security staff of five with five more to come.

So when his Pilot rep told Hennessy he was going to get fired that afternoon, Hennessy was able to set the plan in motion, and the transition to in-house 24/7 security was done in five hours. He credits the quick shift to his engineering team, whom he ranks somewhere between "real strong" and "the best in the world."

The cost of doing it all in-house was and will continue to be massive, of course. Hennessy won't deign to put a number on it, but he readily accepts the fact that he's paying a premium for in-house security. "It's definitely far more expensive doing it in-house," he says. "On the other hand, there's far less risk. I'm paying to sleep well."

Why is it more expensive? To begin with, recruiting talent is hard. There's little out there, and there are plenty of posers. Some experts put the ratio at about one real expert for every 10 claiming expertise. Certifications are partly to blame. A rA©sumA© with a dozen security certifications might look impressive, but it's misleading. Some certifications are simply for specific products and teach nothing about best practices or security policy. A firewall "expert" might know how to configure the box but have no knowledge of what policies should be enforced or even where the firewall ought to be placed in the context of a specific network.

Paying talent sufficiently is even harder than finding it. Stephen Northcutt, founder of the Global Incident Analysis Center and security consultant, says security contractors demand up to US$500 per hour. Salaries are 5 percent to 10 percent higher than what standard IT staff earn.

Keeping talent is the hardest task of all. Northcutt says many true security experts are hopping jobs six times a year, upping their salaries $5,000 at each post. Len Cibelli, a former sales executive at Pilot, expects to get a 20 percent raise from his next employer.

Even so, Hennessy is convinced of the rightness of his decision. "We know doing it in-house is more expensive, but we've just decided it's better than outsourcing," he says. While talent is thin, Hennessy says a few strong candidates have come his way due to the economy.

The Hartford Financial Services Group, which was not a Pilot customer, has taken many of the same steps as Hennessy. Hartford Assistant Vice President of IT Jack Stoddard outsources little security, only ceding tasks such as auditing and penetration assessments to outside vendors. He retains 30 full-time security staff members, tries to recruit the best he can find, pays premiums for them and trains his staff continually. He is adamant about the limitations of the outsourcing model.

"I don't see us ever outsourcing," Stoddard says. His CIO, David Annis, believes acquiring and grooming security expertise in-house is critical, even if it costs more. He calls outsourcing "throwing in the towel." But he understands why so many companies do it anyway. Security is so complex and demands such constant reassessments, he says, that doing it in-house requires a "fair amount of redundant due diligence."