You'd think we'd run out of them, but technology simply hasn't advanced enough to take boneheaded users out of the daily equation that is the IT admin's life.
Whether it's clueless users, evil admins, or just completely bad luck, Mr. Murphy has the IT department pinned in his sights -- and there's no escaping the heartache, headaches, hassles, and hilarity of cluelessness run amok.
[ For more real-world tales of brain fail, see "Stupid user tricks 5: IT's weakest link." | Find out which of our eight classic IT personality types best suit your temperament by taking the InfoWorld IT personality type quiz. | Get a $50 American Express gift cheque if we publish your tech tale from the trenches. Send it to firstname.lastname@example.org. ]
Below, we've compiled nine more shining examples of user stupidity for your amusement, and education.
Stupid user trick No. 1: The itchy clicking finger of fateSometimes you have to don Nigerian princely robes to know just how likely your network is to get hosed, learned one IT admin at a midsize financial company in the Midwest.
"We've spent well into six figures on perimeter security, antivirus, and antimalware software to keep customer data and get through audits. But even so, in the last year and a half we've had no fewer than six breaches with data being stolen or compromised," says the admin.
"Then over drinks one day, a buddy who is a security consultant casually mentioned that human compromises were just as common as technology vulnerabilities."
Keen to quantify this collective brain fail, the admin's team set up a test.
"We took the roster of employees of our two largest offices and checked their corporate email addresses to see which were accessible off the Web. Out of 178 employees, 138 corporate email addresses were easily discovered -- like two or three clicks off Google. That alone surprised me."
The team then set up a phishing email and sent it to all 138 employees.
"Now I know why those Nigerian princes keep bothering people," the admin says. "Our current malware technology caught only 58 percent of our home-brew phishing mails. On top of that, because we didn't use the usual Nigerian-prince or $1-million-estate-up-for-grabs schemes, we managed to get 64 out of 138 to click on our 'malware' link."
Needless to say, the results raised eyebrows in the corner office.
"For the past six months, we've rebuilt and toughened up our antimalware perimeter, but much more importantly we've given several seminars on Internet and corporate security, and we got our COO to make attending at least one of those seminars mandatory for every employee."
Moral: Educate your users about social engineering, because rich Nigerian royalty, or corporate data raiders, can get you no matter what kind of antimalware you have.
Stupid user trick No. 2: The dirty back endCircumventing IT for remote access can put you in a compromising position, as one IT manager at a software company in Florida tells it, especially if your company employs developers with dirty minds -- and who does that?
"A couple of years ago, our GM of sales had to demo our product to a potentially large customer. It was a rush meeting, so he had the head software architect on his team set up a remote connection to the dev servers back at headquarters," the IT manager says.
"The sales GM didn't tell the architect what he needed those connections for, and the other guy didn't think to ask. Neither thought to ask us," the IT manager adds. "Had they done so, we'd have stopped it."
Fast-forward to the middle of the GM's sales demo.
"The GM tags back to the dev environment to access a back-end database the app was using. But the database was full of junk data, like dev databases usually are. Junk data with first some weird names and then some downright nasty names -- like XXX nasty. All that flashed up on the screen when the GM ran his query."
Needless to say, they didn't get the sale.
Moral: Don't set up a remote demo without talking to everyone concerned. And maybe talk to your developers about not getting their dev data from Hustler.
Stupid user trick No. 3: Welcome to the thunderdomeIt pays to be civil to your coworkers, says one IT admin who witnessed firsthand the special kind of evil a workplace feud can wreak on your IT systems.
"We had an exec who was, simply put, a dyed-in-the-wool jerk. No question, just a jerk. He gave everyone a hard time no matter whether they were on his team or whether the quality of their work was good or bad. He just enjoyed being a jerk," the admin says.
Until he decided to give IT a hard time.
"This one time he went out of his way to blame the low productivity of two teams, including his own, on technology problems. The senior IT manager was new to the role, just got promoted into it, and was completely unprepared for this in a senior staff meeting with the CEO. So he was majorly embarrassed and almost got fired only three weeks after taking the new gig. He did get put on probation and already had a ding on his review," the admin says.
Then the new senior IT guy decided to get even.
"None of us can prove it, but from what we were able to figure out, he hacked into the jerk exec's desktop, notebook, and we think even his phone. Dropped all kinds of nasty scripts on there, including one that kept the machine asking for new NAT leases, somehow kept Windows asking for updates no matter how often the guy installed them, added some kind of white list that kept the guy off of all the sites that he needed to see and only let him onto weirdo porn or pirate sites, deleted the contact list off the phone whenever the guy hooked it up to his PC, and autotrashed a random percent of any new files that were saved each day. It was ugly," the admin says. "The guy couldn't even log on at Starbucks."
Herein lies the second source of idiocy in this sordid little fight-club tale: the senior IT manager with an ax to grind.
"The executive lost two trip itineraries and even lost a sales presentation right before he had to give it. I think that last one is where the evil IT guy figured he'd done enough -- that cost us a sale."
Moral: Support your local IT sheriff. Feuds with anyone are a bad idea, but especially the guy who controls your computer. And of course, if you feel you've been wronged in the workplace, take it to HR, not the company network.
Stupid user trick No. 4: Developers do the darnedest thingsLazy is one thing; stupid, another. Together they can leave you pantless at the side of the road -- especially when it comes to developing code.
"We had a situation that wound up costing us almost a month of software development time just doing bug tracking," says one IT pro, who sets the stage of his company's dev-based debacle: "A new developer was tasked with updating one product with some similar functionality from one of our other products."
Two weeks later, still a week ahead of schedule, the developer says he's all done -- until QA gets involved.
"QA takes this guy's code and starts getting a weird error message in response to a key use case. Just couldn't get around it. That got kicked back to the developer, who was also scratching his head. He spends another week 'bug fixing' and resubmits the code saying the bug is gone now. Pow! QA immediately gets the same error again," the IT pro says.
"Now we're past deadline, and the dev guy is scrambling to get a fix going. A couple of days later, he's saying that it's not his code; it's the difference between our dev servers and our QA servers. Which was crap, because it's a mirrored environment."
Perplexed, IT begins comparing its server environments -- a process that included a code scan of the application the developer was supposed to emulate in the update to his app.
"Turns out the little worm just copy-and-pasted the code from the first app into the second and did some lazy work aligning the variables and syntax," the IT pro says. "But he left a number of routines in there that were going nowhere, so the code worked in his cooked dev environment, but died as soon as QA started running it on a clean set of servers. By the time we figured that out, we were four weeks past deadline."
And what of the time it took to "code" and "bug fix" the update?
"The whole time he was really working on his own app that he wanted to sell himself later," the IT pro says. "Yeah, that guy got fired."
Moral: It's more work to get away with code plagiarism than to do your own code in the first place. Oh, and QA guys are sticklers.
Stupid user trick No. 5: Meatballs -- IT's revengeLow blood sugar can turn almost anyone's brain to mush. But a stolen lunch? For some it's a one-way ticket to pathological.
"This is one of those situations that is simply unreal until it's actually happening," says a one-man IT department who was working with two subcontractors at the time of this tale. One of the subcontractors was hardworking and friendly. The other -- let's say he had a quick temper and that he really, really liked meatball sandwiches.
"One day, we hear a door slam and then lock. I look out into the hall and I can see this guy had just locked himself in an unmanned office," the IT pro says. "He's snarling at his notebook screen and mouthing a rant to himself. I figure he's just upset again and make a mental note to complain about that behavior to our rep at the consulting company. Oh, how little I knew."
As it turns out, there was a meatball sandwich thief on the loose in the office. It must have been tasty because the meatball burglar pilfered the IT guy's sandwich two days in a row.
"So the accounting department calls me saying their machines are frozen except for an internal SMS message that says nobody's getting back on until this guy gets his sandwich back," the IT pro says. "Of course, I'm thinking accounting is making a joke, so I laugh and say, 'Nice one.' The guy on the other end gets really angry and tells me to get my s#!t together or else. Now I'm scratching my head. This can't be real, can it?"
The phone rings again and it's marketing -- same scenario: "This guy was using his admin creds to systematically shut down our whole network, segment by segment, trying to get his stolen sandwich back."
Our pro and the other consultant try to log on, and they too are locked out. "Even the guest admin creds are blocked. He's got solo control of every server, switch, router, and firewall."
Panic button! "I'm kicking this guy's door and yelling that I'm going to break it down and strangle him if he doesn't give us our network back. I must have looked like a psycho, except this nut job has his face up to his hall window and is screaming right back at me in Russian."
It certainly isn't our IT pro's finest moment, but in the end, he calms down enough to tell the rogue subcontractor that in all likelihood, "his sandwich is a digested memory."
"I ask him what he will take to give us our systems back," the IT pro says. "That's when he gets really quiet, probably having figured out that he's fired, but he's too pissed and proud to give in without saving face."
Our IT pro gets creative.
"I tell him I'm going to find the sandwich heister. He lets me on the network long enough to send another SMS telling everyone that I had a webcam installed in the lunchroom, so I can easily find out who stole the sandwich. But if the guilty party comes forward voluntarily, we can have a calm discussion before things go back to normal," the IT pro says.
"Unbeknownst to our nut job, I have his partner make a deal with one of the account execs, who cops to stealing the sandwich, and I make a big show out of firing him on the spot where I know the crazy person is watching," he says. "The account exec really played it up, too, yelling and pleading for his job. I kick him out and go back to the nut. 'Satisfied?' He's reluctant, but it's that or I call the cops. So he gives us the network back and stomps out of the building. Never heard from him again and the consulting company gave us three months of free service to make up for his crap."
And the account exec?
"He comes back the next day and we all have a chuckle."
And the sandwich thief?
"Right after this I really did install a webcam in the lunchroom, which is how I caught this new guy stealing someone else's lunch. He got axed real fast and I enjoyed myself doing it."
Moral: Don't mess with another man's meatballs.
Stupid user trick No. 6: Self-service IT gets stickySelf-service IT may seem like a dream -- until it ends in a police raid.
"We were managing a big database and dev shop. Almost every day we got complaints that there weren't enough servers for Project Whatever. So I talked our CEO into funding this giant wad of hardware, which we used to double our server capacity," says one IT admin. "We then converted the whole infrastructure into a virtualized environment running on a big hypervisor."
Each team was given open access to its own resource pool and could allocate those resources however it liked. There was nothing for IT to do except manage the infrastructure. Sounds great, right?
"Unfortunately, my guys interpret that to mean all they have to worry about is up-down green-light health on the physical servers," the admin says. "They don't check much on the virtualized servers."
Step one for exposing yourself to a bust.
"Seems one manager was effectively managing two teams," the admin continues. "He's got a real smart guy on one team and some problems on the other team, so he moves everyone out of the first team, except the genius, and uses them to bolster the second team. The genius is left to move his team's project forward on his own."
As luck would have it, Einstein resented his new responsibility, and with solo control over a whole bunch of servers, he decided to get stupid.
"He sets up his own porno site using our servers and an unused static external address that we bought to use in a project that had died shortly before," the admin says. "It also seems he didn't do a lot of research on how to run a porno site legally in the United States."
"A month later, three squad cars, two unmarked cars, and a big black van pull up to the front entrance," the admin says. "They confiscate most of the data center and almost everyone's personal machines. Even with some DR in place, we were completely shut down for almost a week. My team almost got fired for not managing our neato new virtualization environment the right way. And Mr. Genius, I believe, had to pay a huge fine. Which I don't know how he afforded since he no longer had a job."
Moral: Self-service virtualization doesn't mean unmonitored virtualization -- unless you want to host illegal porn.
Stupid user trick No. 7: Control freak-outSome tales offer a lesson for everyone.
"We'd just finished installing this new point-of-sale system in 40 remote retail locations. Everything was running fine, and we were monitoring each POS setup with desktop management and remote connections," one IT pro says. "We were using the remote connections to check certain data repositories on an almost daily basis. It's important data, after all.
"Except one of my techs didn't know the difference between a basic remote connection, which runs in the background from the user perspective, and a remote control connection, which just takes over the PC from the user," the IT pro says.
So, after accessing a store in Ohio, the team gets a call from the local police.
"Seems the store manager had freaked out because her computer was being 'taken over by thieves' who were trying to access her payment and credit card data. She had been crying while trying to get control of her mouse and keyboard back and finally just yanked the power cord out of the wall." Totally desperate, she called 911.
"From the description, we immediately knew what had happened. They put her on the phone with me and I was trying not to laugh while she was crying hysterically. It took some time to clean up, and our tech got a crash course in remote connection management."
Moral: User education is important, but technician education is even more important -- and user error is no excuse to call 911.
Stupid user trick No. 8: The amazing adventures of Stupid SQL ManHere's to hoping your IT superhero isn't a supervillain in disguise.
"About five years ago, our company had one on-site tech, hired from a local IT consulting company, and a really basic network," one line-of-business manager says. "And our tech was really lazy and really evil."
As it turns out, the tech liked having an easy gig and didn't want to be reassigned to something more taxing. So he did what any lazy, evil tech hoping to appear important would do: He kept the basic network running smoothly and started inventing problems.
"He'd take a printer down, and when the employee called to get it fixed, he'd do it from his cube and look really smart," the manager says. "He was doing this stuff every day, so he could spend most of his time either reading comic books or talking with his friends. We wouldn't have caught on, except one day he decides he really needs to be a hero."
The fatal flaw: "He thought he was really good at SQL, but he really wasn't," the manager says. "He tried to set up a problem on our financial database that he could then fix -- and he did it right as we were going through an audit, so he could look particularly heroic. He screwed up and corrupted the whole database. And because he was lazy, he hadn't been doing full backups for a while. We lost quite a bit of data, failed the audit, and had to spend a whole bunch of money getting everything fixed in a big hurry."
Moral: When your IT guy is reading the latest issue of Batman on the clock, you might want to double-check his work.
Stupid user trick No. 9: Let's get redundant in hereIt pays to do more than just read the box.
"Four years ago, a few buddies and I decided to try our hand at running our own small-biz IT support company. It didn't work well, partially because we let our customers get away with too much just to keep the business," says one IT pro, now with a larger IT shop.
"A perfect example was this guy running a small real estate management office. We sold him on Small Business Server and made some decent per-hour money setting that up. But he was really tight and the SBS install was more than he expected. So he turns us down when we offer to set up extended services, like unattended backup, shadow copy, and shared files."
Instead, our real estate maven heads off to Staples to solve all his backup needs with a 500GB home network drive.
"He starts backing anything he thinks is valuable up to that drive manually. On top of that, because his company is pretty small, he doesn't have a lot of critical data to back up. So he makes one folder for his critical backups and tells his two bookkeepers to use the rest of the drive for 'live' data -- this with another 200GB just sitting open for file shares on the SBS server he already paid for. But hey, he's the client."
Yeah, the thing crashed.
"All of a sudden he loses all his backups and everything his employees were doing for three days prior, since he doesn't do his magic backup operation more than once a week. We get the phone call demanding that we fix the problem because it was our outrageous fees that forced him to do this. It shouldn't be hard, he says, because the drive was redundant."
On the drive over, our IT crew begins wondering why the real estate guy's second drive never kicked in.
"Once we get there, we find out that the box says the drive could be used as a redundant drive to other drives already on the network. Whoops. Now the drive is dead and all his data is gone," the IT pro says. "We sent the thing out for a platter-level recovery but only got back 60 percent of the drive and that took two weeks. And cost him another $1K, for which he blamed us -- right before he fired us."
Moral: It's not easy, but if you don't want to be blamed for user error, sometimes you need to lay down the law or walk away. And maybe not charge for every little thing just to chisel as much money out of your customer as you can. What? Did I say that out loud?
- Read the Off the Record blog for stories from IT pros -- and share your own tech tale
- Stupid user tricks 5: IT's weakest link
- Jackass IT: Stunts, idiocy, and hero hacks
- Dirty IT jobs: Partners in slime
- 2011 geek IQ test
- IT admins gone rogue
- Stupid hacker tricks: Exploits gone bad
- IT inferno: The nine circles of IT hell
- IT personality types: 8 profiles in geekdom
- 7 IT superheroes -- and their fatal flaws
- The 7 dirtiest jobs in IT
- True IT confessions
- Programming IQ test: Round 2
- Linux admin IQ test
This story, "Stupid user tricks 6: IT idiocy loves company," was originally published at InfoWorld.com. Get a digest of the key stories each day in the InfoWorld Daily newsletter, and for the latest business technology news, follow InfoWorld.com on Twitter.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.