Decision Standards
Carl Sagan once said "Extraordinary claims require extraordinary evidence". For us, extraordinary decisions follow the same principle.
Leadership can require that a risk measurement meet certain requirements that defend it against negligence, cognitive error, corruption, imaginative failures, or gaps of evidence.
Here are some example thoughts on standardizing the level of expense expected for a measurement.
Level Zero: An individual, “water cooler” estimate.
Level One: A estimate from a single individual who has received risk training, has particular subject matter expertise, reasonable access to reference classes, or has specific historical data.
Level Two: In addition to previous levels, this estimation is from an individual who is calibration trained (See: calibration) and maintained a track record. (See keeping score)
Level Three: In addition to previous levels, a panel of three or more calibrated individuals were involved with this estimation.
Level Four: In addition to previous levels, an external expert was involved with the estimation.
You can, of course, stack and order these however you’d like. Each example simply increases the rigor involved, approaching greater scientific integrity. Some of these qualities of rigor are listed below:
Rigor
If a leveling approach is not desired, the following criteria can be requested as part of a study as needed:
- Risks are decomposed and their components are individually (or independently) studied.
- Forecasts are elicted by individuals who have been calibration trained.
- Forecasters have a determined track record with Brier scores meeting a specific threshold. (See: Keeping Score)
- Historical data was sought out and modeled with frequentist approaches to inform forecasts.
- Measurement of "unknown" factors were a part of forecasts.
- A panel of individuals were relied on instead of a single individual.
- Critique, feedback, and whistleblowing have formal process.
- Outside expertise is leveraged.
- Monte Carlos simulations are developed and studied.
Rigor is not required for all measurements. Rather, important decisions should clearly identify what amount of rigor is expected in any measurement effort. Measurements can happen verbally at a water cooler, or with concerted effort in order to inform an important decision.