By: Jack Jones
Richard Stiennon recently posted a presentation on SlideShare called “Why Risk Management is Impossible” (http://www.slideshare.net/stiennon/risk-managementfalisitec13#). Because I respect Richard’s intellect and experience I tend to take his proclamations seriously. Consequently, I was surprised by the title and, frankly, concerned about what the presentation seemed to convey. This post will outline my observations, questions, and concerns regarding his presentation.
Let’s go through this slide-by-slide...
Title Slide - Risk Management: A Failed Strategy with Unachievable Goals
His claim of risk management being a “failed strategy” seems surprising to me given the results I and other risk-focused professionals I know have had. Certainly, there have been plenty of failures WRT “risk” in information security over the years (more failures than not), but most of those failures are because of the extremely superficial and inconsistent way people have approached the problem. Heck, I was visiting a large financial institution the other day and asked the four assembled infosec and ERM leaders to share their definition for “risk”. Each defined it differently. A couple of the definitions boiled down to loss likelihood and impact, another was all about impact but didn’t consider likelihood, and the third defined risk as uncertainty that could apply to gain as well as loss. It’s not hard to picture “failure” when the definition of the problem you’re trying to manage is inconsistent.
As for “unachievable goals” -- well, let’s deal with that in a minute...
Slide 2 - What is risk?
See my point above regarding inconsistent definition of the problem. Again, when an organization can agree on the definition of the problem they’re trying to manage, it’s remarkable what they’re able to achieve if they take it seriously.
Slide 3 - Risk Management 101
Here Richard claims that identifying all critical assets, scoring them by “value”, and discovering all vulnerabilities constitutes RM 101. He also says, quite rightly, that it’s impossible to discover ALL critical assets and ALL vulnerabilities. Of course, from a practical perspective it isn’t about discovering all assets and vulnerabilities, it’s about the process of discovering as many as you can given your resources, which provides better but not perfect visibility into your risk landscape. As for scoring assets by “value” -- yes, you should be able to characterize an asset’s value/liability. You won’t likely be able to do it with a high degree of precision, but you can certainly do it with a useful degree of accuracy.
So, although Richard’s statement of impossibility is accurate, it doesn’t seem to recognize that pragmatic risk management is about improving the quality of information being used in decision-making, which is very achievable.
Slides 4, 5 and 6
These slides appear to paint a picture of impossible complexity in order to support slide 3. Nobody’s arguing that the risk problem space isn’t complicated and challenging. If it were simple we wouldn’t be having this discussion. But complicated and challenging doesn’t equate to impossible.
Slides 7 thru 12
It seems like these examples of nasty situations and events are intended to support the notion that risk management inevitably fails. Although there’s no question that these kinds of events are challenging, a mature risk management program can help improve the odds of dealing with them. Certainly nothing (including a threat-based approach) can guarantee that events like these can’t happen or that if they do happen that the losses would be nominal, but making guarantees isn’t the point of risk management.
Slide 13 - Risk management is based on normal distribution of events
Although I know it’s common to use normal distributions when data are sparse (because the world has a tendency to behave in a Gaussian manner in general) I’m not familiar with anyone who believes it’s the only applicable distribution shape for risk management. Normal distributions can be a reasonable starting point, but shouldn’t be relied on solely.
Slide 14 - Targeted Attacks are Not Random
I’m not sure why Richard states that risk management arose only to address random attacks. Most of the analyses I’ve performed, and certainly much of what I had to worry about as a CISO for large financial institutions, dealt with non-random scenarios. After all, viruses, worms, and opportunistic hackers aren’t entirely random. All sorts of non-random factors are at play - e.g., asset type, surface area, threat actor intent (DoS, vandalism, theft, etc.).
As for targeted attacks being “Black Swans”, it seems as though Richard is using a different definition of Black Swan than I’m familiar with. Black Swans, by the definition I’m familiar with, are supposedly the unimagined or unimaginable event. Targeted attacks are anything but Black Swan in the vast majority of instances given this definition.
Slide 15 - So, if Risk Management is a failure what should be done?
See my previous points about “failure”...
Slide 16 - Some scenarios
Appear to be efforts to show scenarios where “risk management” would fall short. Without being there to hear his narrative, I can’t tell what his rationale would be or whether Richard offers a better option.
Slide 17 - Cyber kill chain
This slide cites examples of infosec tools/techniques within an “Attack Phase” and “Courses of Action” matrix. Cool stuff here. Really. This is an interesting and potentially useful reference when thinking thru how to address various threat scenarios. Within the risk management programs I’ve built and been around, this would be a useful element.
Slide 18 - Security Intelligence is the key to threat management
Actually, I agree that threat intelligence is one of the keys to effective threat management (which is one part of risk management). In fact, as a CISO I have always subscribed to premium threat intelligence providers for this very reason. And if I’m ever a CISO again I’ll continue to use those services. This is, however, only one part of what it takes to manage information security in an organization.
Slide 19 - The Cyber Defense Team
My gut reaction to this slide is that it’s intended to cater to people who want a cooler name for the organization and its leadership. I mean, how much cooler can you get than “Cyber Commander”. I need to think about adding that to my resume.
Slide 20 - Let’s be honest
Richard’s first point is that risk management was developed to better communicate with organization leadership. He’s at least partly right, although I would phrase it differently and broaden it some. From my point of view managing risk is the process of information gathering, analysis, communication, decision-making, and actions that are intended to affordably achieve and maintain an acceptable level of loss exposure.
His second point is that “Management understands threats not risks”. Hmmm, this has NOT been my experience at all. Certainly, management can be coached (sometimes easily) to fear threats, but understand them? Not so much. Leading an organization requires making decisions based on imperfect information, and these decisions invariably have an upside and a downside to them. The downside component of these decisions represents loss exposure (risk) and executives understand well-defined representations of these downsides very well. One of the things infosec has been horrible at over the years though, is representing this downside in meaningful and defensible terms, which is one of the big reasons for risk management’s challenges over time.
Richard’s last point is “Show them the threats and they will respond.” How very true. Up to a point. FUD (and this can be interpreted as FUD even if that’s not his intent) can work for a while depending on how convincing you are and how paranoid the stakeholders are. It has a shelf-life though, and eventually (in my experience) one or more curious stakeholders start asking questions like, “How much less risk will we have if we spend this money you’re asking for?” This happens sooner if you ask for a LOT of money, which it seems like a purely threat-based approach almost insists on. After all, if the threat-based approach isn’t making distinctions about which threats are most significant, which assets are most critical, or which vulnerabilities are most severe, then how does it prioritize? Because to think about these considerations means you’re doing risk management. But then again, I don’t believe for a moment that Richard advocates FUD or blind, unbounded security expenditures. Thus my confusion with the presentation.
I’ve had the opportunity to speak with Richard about his presentation to see whether I was truly getting the gist of his position and to discuss our different points of view. It turns out that he and I see most things very similarly. His strong critique of risk management is driven, at least in part, on the same stuff that drives me nuts -- superficiality, ambiguity, and a misguided belief that it provides discrete solutions (for example, check boxes) to what are ultimately a set of open-ended questions. He also believes that efforts to quantify to decimal precision are time and resources that could be better used to address obvious problems. I couldn’t agree more. When we train people in FAIR we always advocate beginning with triage-like quick-and-dirty analysis (which are good enough most of the time) and we don’t advocate trying to achieve decimal point precision because it can be resource intensive and it simply isn’t necessary in order to make better-informed decisions.
Oh, and I was pleased to hear Richard say that he remains an advocate of FAIR and believes that if someone is doing infosec risk management they should be using FAIR. Cool.