Highlights for How to Measure Anything in Cybersecurity Risk
What risks are acceptable is often not documented, and when they are, they are stated in soft, unquantified terms that cannot be used clearly in a calculation to determine if a given expenditure is justified or not.
Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations.
If a decision maker or analyst engages in what they believe to be measurement activities, but their estimates and decisions actually get worse or don’t at least improve, then they are not actually reducing their error and are not conducting a measurement according to the stated definition.
What you want to know is whether you have less uncertainty after considering some source of data and whether that reduction in uncertainty warrants some change in actions.
We can measure the value of art, free time, or reducing risk to your life by assessing how much people actually pay for these things.
If your concern is that upper management won’t understand this, we can say we have not observed this—even when we’ve been told that management wouldn’t understand it. In fact, upper management seems to understand having to determine which risks are acceptable at least as well as anyone in cybersecurity.
Since we know at least one (if not both) must be wrong, then we know qualifications and expertise in cybersecurity alone are not sufficient to determine if a given opinion on this topic is correct.
More fundamentally, does it even matter whether risk analysis works? And by “works,” do we really just mean whether it succeeds in putting on a show for compliance, or should we mean it actually improves the identification and management of risks?
They are all based in part on the idea that not knowing exact quantities is the same as knowing nothing of any value.
All of the operations just described require some source of an input. In this example, we will be using the calibrated estimates of the CISO. Since the CISO is using his previous experience and his calibrated probability-assessment skill to generate the inputs, we call them an “informative prior.”
Practically speaking, there are only so many models that can be run and maintained at any given time.