Reading Material

This document contains reading material bolstering the estimation, expert elicitation, quantitative certainty, scenario, and forecasting subject matters that guide Simple Risk Measurement.

Risk Language

This section includes reading that helps navigate the problems with “risk” language, miscommunications and so forth.

Defining Risk

The language of risk is used in a variety of ways and shows up with different intentions in practice.

[1]Hansson, Sven Ove, “Risk”, The Stanford Encyclopedia of Philosophy (Fall 2018 Edition), Edward N. Zalta (ed.), URL = <https://plato.stanford.edu/archives/fall2018/entries/risk/>.
[2]Boholm, M. , Möller, N. and Hansson, S. O. (2016), The Concepts of Risk, Safety, and Security: Applications in Everyday Language. Risk Analysis, 36: 320-338. doi:10.1111/risa.12464

Specific Scenarios

This section contains supporting reading for specific scenario building. The “Scenario” is frequently used language in modern approaches to risk analysis.

This form of the scenario is produced immediately from fault tree analyses and similar methods associated with engineering safety studies (i.e., nuclear reactor safety studies). Each scenario represents a unique concatenation of events.
[3]
    1. Wall, DRMI, Naval Postgraduate School, July 29, 2011, The Kaplan and Garrick Definition of Risk and its Application to Managerial Decision Problems https://my.nps.edu/documents/103424423/106950799/DRMI+Working+Paper+2011-3.pdf/bad99104-b54b-4646-9d92-45e16c2f80d8
[4]Wikipedia contributors. (2018, June 24). Clarity test. In Wikipedia, The Free Encyclopedia. Retrieved 17:49, November 8, 2018, from https://en.wikipedia.org/w/index.php?title=Clarity_test&oldid=847323771
[5]Decision Analysis: Practice and Promise. Ronald A. Howard. Management Science, Vol. 34, No. 6. (Jun., 1988), pp. 679-695.

Hierarchy of Scenarios

The hierarchal relationship between specific future events.

[6]Wikipedia contributors. “Fault tree analysis.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 16 Jul. 2018. Web. 8 Nov. 2018.
[7]Schneier on Security https://www.schneier.com/academic/archives/1999/12/attack_trees.html
[8]Fault Tree Handbook with AeroSpace Applications, NASA, v1.1, August, 2002, https://elibrary.gsfc.nasa.gov/_assets/doclibBidder/tech_docs/25.%20NASA_Fault_Tree_Handbook_with_Aerospace_Applications%20-%20Copy.pdf
[9]Amoroso, E. G. (1999). Fundamentals of computer security technology. Place of publication not identified: Diane Pub Co.
[10]Swiderski, F., & Snyder, W. (2009). Threat Modeling. New York: O’Reilly Media, Inc.
[11]Shostack, A. (2014). Threat modeling: Designing for security.

Measurement / Approximation

This section includes all references to, and arguments that measurements are estimates. Generally speaking, everything we do is some form of approximation, even when employing the use of measurement instruments.

[12]Measurement Uncertainty Antonio Possolo - https://www.nist.gov/itl/sed/topic-areas/measurement-uncertainty
[13]Tal, Eran, “Measurement in Science”, The Stanford Encyclopedia of Philosophy (Fall 2017 Edition), Edward N. Zalta (ed.), URL = <https://plato.stanford.edu/archives/fall2017/entries/measurement-science/>.
[14]Hubbard, Douglas W., and Richard Seiersen. How to measure anything in cybersecurity risk. Hoboken, New Jersey: Wiley, 2016. Print.
[15]A Turning Point for Humanity: Redefining the World’s Measurement System Robin Materese - https://www.nist.gov/si-redefinition/turning-point-humanity-redefining-worlds-measurement-system
[16]Keeping the Standard Kilogram From Gaining Weight Is a Constant Struggle Nadia Drake - https://www.wired.com/2013/01/keeping-kilogram-constant/
[17]Why scientists are redefining the kilogram https://www.engadget.com/2017/10/24/why-scientists-are-redefining-the-kilogram/
[18]Wikipedia contributors. (2018, November 6). Kibble balance. In Wikipedia, The Free Encyclopedia. Retrieved 18:15, November 8, 2018, from https://en.wikipedia.org/w/index.php?title=Kibble_balance&oldid=867616683
[19]Evaluation of measurement data – Guide to the expression of uncertainty in measurement JCGM 100:2008 (GUM 1995 with minor corrections)

Expert Estimation

This section generally appeals to how experts can be queried for quantitative data.

Combining Expert Estimations

This describes the practice of gathering up forecast material and, typically, averaging it together. Parimutual Betting, Simple Averages, Weighted Scores.

[20]Wikipedia contributors. (2018, October 1). Parimutuel betting. In Wikipedia, The Free Encyclopedia. Retrieved 18:01, November 8, 2018, from https://en.wikipedia.org/w/index.php?title=Parimutuel_betting&oldid=861927567
[21]Brown, Thomas A., Probabilistic Forecasts and Reproducing Scoring Systems. Santa Monica, CA: RAND Corporation, 1970. https://www.rand.org/pubs/research_memoranda/RM6299.html. Also available in print form.

Calibration of Experts

Also see How to measure anything.

[22]Calibration Of Probabilities: The State Of the Art To 1980 Lichtenstein- Sarah- Fischhoff- Baruch- Phillips-Lawrence D - http://www.dtic.mil/docs/citations/ADA101986
[23]Calibration Of Probabilities: The State Of the Art Lichtenstein- Sarah- Fischhoff- Baruch- Phillips - http://www.dtic.mil/docs/citations/ADA033248

Humorous Examples

[24]Miyamoto’s Secret Hobby https://kotaku.com/5381876/miyamotos-secret-hobby-measuring-stuff
[25]Shigeru Miyamoto Guesses The Size of Random Objects (Jimmy Fallon) https://youtu.be/DiUeuc7eOh0?t=16s

RAND

RAND has been developing methods for expert estimation for decades, described as DELPHI and Futures Methodology.

[26]Publications on Futures Methodologies: Delphi https://www.rand.org/pardee/pubs/futures_method/delphi.html
[27]Probabilistic Forecasts and Reproducing Scoring Systems https://www.rand.org/pubs/research_memoranda/RM6299.html
[28]On the Epistemology of the Inexact Sciences https://www.rand.org/pubs/reports/R353.html
[29]An Experimental Application of the Delphi Method to the Use of Experts https://www.rand.org/pubs/research_memoranda/RM727z1.html
[30]The Systematic Use of Expert Judgment in Operations Research https://www.rand.org/pubs/papers/P2795.html
[31]Convergence of Expert Consensus Through Feedback https://www.rand.org/pubs/papers/P2973.html
[32]Improving the Reliability of Estimates Obtained from a Consensus of Experts https://www.rand.org/pubs/papers/P2986.html
[33]The Use of the Delphi Technique in Problems of Educational Innovations https://www.rand.org/pubs/papers/P3499.html
[34]Analysis of the Future https://www.rand.org/pubs/papers/P3558.html
[35]Delphi https://www.rand.org/pubs/papers/P3704.html
[36]Systematic Use of Expert Opinions https://www.rand.org/pubs/papers/P3721.html
[37]Delphi Process https://www.rand.org/pubs/papers/P3925.html
[38]Experiments in Group Prediction https://www.rand.org/pubs/papers/P3820.html
[39]Predicting the Future https://www.rand.org/pubs/papers/P3948.html
[40]Delphi and Values https://www.rand.org/pubs/papers/P4182.html
[41]The Delphi Method https://www.rand.org/pubs/research_memoranda/RM5888.html
[42]The DELPHI Method, II https://www.rand.org/pubs/research_memoranda/RM5957.html
[43]The Delphi Method, III https://www.rand.org/pubs/research_memoranda/RM6115.html
[44]The Delphi Method, IV https://www.rand.org/pubs/research_memoranda/RM6118.html
[45]Experimental Assessment of Delphi Procedures with Group Value Judgments https://www.rand.org/pubs/reports/R0612.html
[46]Comparison of Group Judgment Techniques with Short-Range Predictions and Almanac Questions https://www.rand.org/pubs/reports/R0678.html
[47]Delphi Assessment https://www.rand.org/pubs/reports/R1283.html

Expert Groups

Also see Tetlock.

[48]Stan Kaplan, ‘Expert information’ versus ‘expert opinions’. Another approach to the problem of eliciting/ combining/using expert knowledge in PRA, Reliability Engineering & System Safety, Volume 35, Issue 1, 1992, Pages 61-72,
[49]R.L. Keeney ; D. von Winterfeldt. Eliciting probabilities from experts in complex technical problems, IEEE Transactions on Engineering Management ( Volume: 38 , Issue: 3 , Aug 1991 )

IARPA

IARPA invests in quite a bit of predictive research and publishes results often. They are also involved in forecasting tournaments.

[50]https://www.iarpa.gov/index.php/newsroom/iarpa-in-the-news/2015/419-teams-better-than-individuals-at-intelligence-analysis-research-finds
[51]Teams Better Than Individuals at Intelligence Analysis, Research Finds. American Psychological Association. https://www.apa.org/news/press/releases/2015/01/intelligence-analysis.aspx

Cooke’s “Classical Method”

Often found in environmental risk (Volcanic, Earthquake) and others.

[52]Roger Cooke, Max Mendel, Wim Thijs, Calibration and information in expert resolution; a classical approach, Automatica, Volume 24, Issue 1, 1988, Pages 87-93, ISSN 0005-1098, http://rogermcooke.net/rogermcooke_files/Automatica%201988%20Cal%20Inf.pdf
[53]Abigail R Colson, Roger M Cooke; Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments, Review of Environmental Economics and Policy, Volume 12, Issue 1, 1 February 2018, Pages 113–132, https://doi.org/10.1093/reep/rex022
[54]A route to more tractable expert advice, Willy Aspinall. http://www.rff.org/files/sharepoint/Documents/Features/294-295%20Opinion%20-%20Aspinall%20pr.pdf
[55]Usgs Expert Elicitation Report https://www.eeri.org/products-page/other-seminars-and-workshops/usgs-expert-elicitation-report/
[56]Workshop on the ground motion models applied in the National Seismic Hazard Maps https://earthquake.usgs.gov/static/lfs/nshm/workshops/GMPE2012/WED_AM01_Petersen.gmworkshop-berkeley-newsm.pdf

Constructive critique of Cooke’s method can be found here:

[57]Bolger, F. and Rowe, G. (2015), The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?. Risk Analysis, 35: 5-11. doi:10.1111/risa.12272

Forecasting

Philip Tetlock

Tetlock’s research revolves around how experts who are untrained in prediction are worse than random. He has since isolated those who are stronger forecasters (Superforecasters) and is identifying their qualities, especially around how someone a better forecaster, and how to further improve them with teams.

[58]Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know?. Princeton, N.J: Princeton University Press.
[59]Tetlock, P. E., Gardner, D., Tetlock, Philip, Gardner, Dan, & Richards, Joel. (2015). Superforecasting: The art and science of prediction.
[60]Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Welton Chang, Eva Chen, Barbara Mellers, Philip Tetlock, Judgment and Decision Making, Vol. 11, No. 5, September 2016, pp. 509-526 http://journal.sjdm.org/16/16511/jdm16511.html
[61]https://www.sas.upenn.edu/tetlock/
[62]https://www.youtube.com/watch?v=xBXDTQdmNyw
[63]Everybody’s an Expert, Louis Menand. https://www.newyorker.com/magazine/2005/12/05/everybodys-an-expert

Meteorology

Maybe the oldest area of forecasting. Understanding the industrial development of meteorology is a great rubric for how a predictive industry is built over time. First, the theory. Then the infrastructure. Then the operational practice of prediction, decision making, and learning.

[64]Weather Analysis and Forecasting https://www.ametsoc.org/ams/index.cfm/about-ams/ams-statements/statements-of-the-ams-in-force/weather-analysis-and-forecasting/
[65]Wikipedia contributors. “Timeline of meteorology.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 27 Sep. 2018. Web. 8 Nov. 2018.
[66]Bauer, Peter & Thorpe, Alan & Brunet, Gilbert. (2015). The quiet revolution of numerical weather prediction. Nature. 525. 47-55. 10.1038/nature14956. https://www.researchgate.net/publication/281516336_The_quiet_revolution_of_numerical_weather_prediction

Cognitive Error

Kahneman / Tversky

Daniel Kahneman and Amos Tversky offer observations into how fallible the human mind is in the most common of circumstances. The classification of System 1 and System 2 thinking is highly relevant to this area of critical thinking around risk.

[67]Kahneman, D. (2015). Thinking, fast and slow.

Meehl / Dawes

Paul E. Meehl and Robyn Dawes work in prediction inspired a scientific full fledged assault on the credibility of expert prediction. Comprehensive findings that mechanical statistical models beat experts at prediction.

[68]Meehl, P. E. (1966). Clinical versus Statistical Prediction. Place of publication not identified: University of Minnesota Press. http://meehl.umn.edu/sites/g/files/pua1696/f/155dfm1993_0.pdf)

N. Taleb

Taleb explores the limitations of our ability to understand randomness and the nature of randomness. Preparation for inevitable surprise, and the emergence of Black Swans, is Taleb’s core message.

[69]Taleb, N. N., Taleb, N. N., Taleb, N. N., Taleb, N. N., & Taleb, N. N. (2016). Incerto.

Intelligence Analysis

Canadian Intelligence

There is research around Canada’s application of modern intelligence processing and its effectiveness. The basis of this is all probabilistic.

[75]Canada Is Actually Pretty Good At Intelligence Forecasting, Ben Makuch - https://motherboard.vice.com/en_us/article/kbz7gn/canadian-intelligence-agencies-are-actually-pretty-good-at-strategic-forecasting
[76]Accuracy Of Forecasts in Strategic Intelligence, David Mandel-Alan Barnes - http://www.pnas.org/content/111/30/10984

Industry

Industry examples where probabilistic risk assessment is at play:

[77]NASA Risk Management Handbook https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120000033.pdf
[78]EPA: Risk Assessment Forum White Paper: Probabilistic Risk Assessment Methods and Case Studies https://www.epa.gov/sites/production/files/2014-12/documents/raf-pra-white-paper-final.pdf
[79]Probabilistic Risk Assessment Procedures Guide for Offshore Applications https://www.bsee.gov/sites/bsee.gov/files/ProbalisticRiskAssessment%20%28PRA%29/bsee_pra_procedures_guide_-_10-26-17.pdf
[80]Nuclear Probabilistic Risk Assessment https://www.nrc.gov/about-nrc/regulatory/risk-informed/pra.html

This paper has a specifically useful overview of many different industry approaches to safety.

[81]White Paper on Approaches to Safety Engineering http://sunnyday.mit.edu/caib/concepts.pdf