Reading Material

This document contains reading material bolstering the estimation, expert elicitation, quantitative certainty, scenario, and forecasting subject matters that guide Risk Measurement.

Risk Language

This section includes reading that helps navigate the problems with "risk" language, miscommunications and so forth.

Defining Risk

The language of risk is used in a variety of ways and shows up with different intentions in practice.

Specific Scenarios

This section contains supporting reading for specific scenario building. The "Scenario" is frequently used language in modern approaches to risk analysis.

Hierarchy of Scenarios

The hierarchal relationship between specific future events. This is related by the third axiom of probility, that decomposition of a scenario should have “disjoint sets” or should be mutually exclusive from one another.

Measurement / Approximation

This section includes all references to, and arguments that measurements are estimates. Generally speaking, everything we do is some form of approximation, even when employing the use of measurement instruments.

Expert Estimation

This section generally appeals to how experts can be queried for quantitative data.

Combining Expert Estimations

This describes the practice of gathering up forecast material and, typically, averaging it together. Parimutual Betting, Simple Averages, Weighted Scores.

Calibration of Experts

Also see [How to measure anything](#How to measure anything).

Humorous Examples

RAND

RAND has been developing methods for expert estimation for decades, described as DELPHI and Futures Methodology.

Expert Groups

Also see Tetlock.

  • Stan Kaplan, ‘Expert information’ versus ‘expert opinions’. Another approach to the problem of eliciting/ combining/using expert knowledge in PRA, Reliability Engineering & System Safety, Volume 35, Issue 1, 1992, Pages 61-72,
  • R.L. Keeney ; D. von Winterfeldt. Eliciting probabilities from experts in complex technical problems, IEEE Transactions on Engineering Management ( Volume: 38 , Issue: 3 , Aug 1991 )

IARPA

IARPA invests in quite a bit of predictive research and publishes results often. They are also involved in forecasting tournaments.

Cooke's "Classical Method"

Often found in environmental risk (Volcanic, Earthquake) and others.

Constructive critique of Cooke's method can be found here:

  • Bolger, F. and Rowe, G. (2015), The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?. Risk Analysis, 35: 5-11. doi:10.1111/risa.12272

Forecasting

Philip Tetlock

Tetlock's research revolves around how experts who are untrained in prediction are worse than random. He has since isolated those who are stronger forecasters (Superforecasters) and is identifying their qualities, especially around how someone a better forecaster, and how to further improve them with teams.

Meteorology

Maybe the oldest area of forecasting. Understanding the industrial development of meteorology is a great rubric for how a predictive industry is built over time. First, the theory. Then the infrastructure. Then the operational practice of prediction, decision making, and learning.

Cognitive Error

Kahneman / Tversky

Daniel Kahneman and Amos Tversky offer observations into how fallible the human mind is in the most common of circumstances. The classification of System 1 and System 2 thinking is highly relevant to this area of critical thinking around risk.

  • Kahneman, D. (2015). Thinking, fast and slow.

Meehl / Dawes

Paul E. Meehl and Robyn Dawes work in prediction inspired a full fledged assault on the credibility of expert prediction. Comprehensive findings that mechanical statistical models beat experts at prediction.

N. Taleb

Taleb explores the limitations of our ability to understand randomness and the nature of randomness. Preparation for inevitable surprise, and the emergence of Black Swans, is Taleb's core message.

  • Taleb, N. N., Taleb, N. N., Taleb, N. N., Taleb, N. N., & Taleb, N. N. (2016). Incerto.

Intelligence Analysis

Sherman Kent

Sherman Kent is considered a pioneer of intelligence analysis, and brought probabilistic rigor into the National Intelligence Estimate.

His writing:

Canadian Intelligence

There is research around Canada's application of modern intelligence processing and its effectiveness. The basis of this is all probabilistic.

Industry Examples

Industry examples where probabilistic risk assessment is at play:

This paper has a specifically useful overview of many different industry approaches to safety.

← Performance
Strategy →