Hiring the unhireable: Embrace failure

Part 5 of 16

Airline pilots don’t learn how to fly. They learn how to safely crash.

Security folks learn how to fly the networks – not how to survive a crash. (See the disconnect?)

As a society, and I argue, especially in the I.T. and security industry, we have been terribly negligent by not embracing failure as a positive characteristic that should be valued and actively sought out by organizations everywhere.

Tons of people get hired because they can configure a firewall. Should you even care? Does that mean anything to you? Well, sure, a little. It is a skill, but it is a fairly rote skill, based upon a checklist of Yes/No and ACLs. We have certifications for such baseline knowledge and skills.

What I believe has much greater value, though, is knowing what to do when the IT Hits the Fan. When that router, firewall, server (etc.) fails. When the bad guys start pounding and beating the bejeezus out of it, do you have the skills and experience to know what to do?

With a billion-plus records breached annually; with DDoS attacks, with insider threats and endless assaults on our infrastructures, I believe that the skills needed should focus on dynamic detection, remediation and reaction to the threat. These skills represent your best chance at functional resiliency. The skills required to set up a bunch of tables in a benign environment is a rote, policy driven, albeit, necessary exercise.

Our networks, our companies and nations do not live in a benign environment. The internet is a hostile place. Let me give you an example from the olden days – analogue, of course.

When I was a recording engineer and producer in my former career, we were hired to stage and record a concert in Kingston, Jamaica with Stevie Wonder and Bob Marley. We had to set up and manage the technology for an entire football (soccer) stadium; the sound system, the lighting – the whole shebang; one big concert. All good. Long live rock’n’roll.

About a half hour into the concert however, disaster stuck.

The power died and now there were 120,000 really annoyed music fans. What do you do? We had complete, catastrophic audio/video/lighting failure. WTF, eh? We scrambled. We had riot cages. Talk about an adrenaline rush. One of the roadies, completely ripped on local ganja came up to me running… “I think I know what it is.” I said, “How the hell would you know what it is. You’re stoned out of your mind.” We’re in Jamaica, right? He says, “No, I really think I’ve got it.” And, he grabs a can of CO2 (standard in those days at every concert) and runs over to a nearby power pole. He starts shooting the CO2 on a slightly smoking, overheated transformer whose self-protection circuits had kicked in. We were drawing too much power for the lights and audio on a single feed. He got arrested. We got him un-arrested thanks to the local British Consulate who was biding his time in one of our riot cages, and in about ½ hour, the concert resumed, ending with an incredible 4 hour jam. Another story.

Point is: that is exactly what I want to see in IT. I don’t care what you can do in a benign environment. I want to see what you can do in a hostile environment because that, ultimately, is what matters. We need to embrace failure. We need to teach failure inside of our schools. We need to create these hostile environments and embrace failure as part of our curriculum so that when you get out of school, you’ll actually be useful on the front lines.

We’ve got to start it early. We should start teaching failure at the kindergarten level or earlier, along with coding. Yes, I mean it. Coding is a series of logical decisions even tots can relate to: If you sit quietly for 10 minutes, I will give you a star. A Yes/No decision box. Behavior. Result. They can handle it.

I wrote a book a number of years ago on cyber security ethics for kids. I’ve got a new version coming out, and it’s going toward the 5 year olds, the 6 year olds, the 8 year olds because that’s the age at which the brain is still hard-coding much of the behaviors and beliefs we will retain throughout our lives. “Don’t touch the hot stove, don’t cross the street in the middle, don’t click on that you idiot.” We’ve gotta start it really, really early.

When you’re hiring a geek, for certain positions, I advocate questions like, “tell me about your worst failure. What did you do? I want to know where you have made mistakes, what kind of mistakes and how you detected and reacted to them.”

Not much else matters when you’re under attack 24/7/365.

Part 4: Celsius or Fahrenheit?
Part 6: You’re not so special (Really, you’re not)

Winn Schwartau is the CEO of The Security Awareness Company, the author of Information Warfare, Pearl Harbor Dot Com (Die Hard IV), and the upcoming Analogue Network Security.