This document contains reading material bolstering the estimation, expert elicitation, quantitative certainty, scenario, and forecasting subject matters that guide Simple Risk Measurement.
This section includes reading that helps navigate the problems with “risk” language, miscommunications and so forth.
The language of risk is used in a variety of ways and shows up with different intentions in practice.
|||Hansson, Sven Ove, “Risk”, The Stanford Encyclopedia of Philosophy (Fall 2018 Edition), Edward N. Zalta (ed.), URL = <https://plato.stanford.edu/archives/fall2018/entries/risk/>.|
|||Boholm, M. , Möller, N. and Hansson, S. O. (2016), The Concepts of Risk, Safety, and Security: Applications in Everyday Language. Risk Analysis, 36: 320-338. doi:10.1111/risa.12464|
This section contains supporting reading for specific scenario building. The “Scenario” is frequently used language in modern approaches to risk analysis.
This form of the scenario is produced immediately from fault tree analyses and similar methods associated with engineering safety studies (i.e., nuclear reactor safety studies). Each scenario represents a unique concatenation of events.
|||Wikipedia contributors. (2018, June 24). Clarity test. In Wikipedia, The Free Encyclopedia. Retrieved 17:49, November 8, 2018, from https://en.wikipedia.org/w/index.php?title=Clarity_test&oldid=847323771|
|||Decision Analysis: Practice and Promise. Ronald A. Howard. Management Science, Vol. 34, No. 6. (Jun., 1988), pp. 679-695.|
Hierarchy of Scenarios¶
The hierarchal relationship between specific future events.
|||Wikipedia contributors. “Fault tree analysis.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 16 Jul. 2018. Web. 8 Nov. 2018.|
|||Schneier on Security https://www.schneier.com/academic/archives/1999/12/attack_trees.html|
|||Fault Tree Handbook with AeroSpace Applications, NASA, v1.1, August, 2002, https://elibrary.gsfc.nasa.gov/_assets/doclibBidder/tech_docs/25.%20NASA_Fault_Tree_Handbook_with_Aerospace_Applications%20-%20Copy.pdf|
|||Amoroso, E. G. (1999). Fundamentals of computer security technology. Place of publication not identified: Diane Pub Co.|
|||Swiderski, F., & Snyder, W. (2009). Threat Modeling. New York: O’Reilly Media, Inc.|
|||Shostack, A. (2014). Threat modeling: Designing for security.|
Measurement / Approximation¶
This section includes all references to, and arguments that measurements are estimates. Generally speaking, everything we do is some form of approximation, even when employing the use of measurement instruments.
|||Measurement Uncertainty Antonio Possolo - https://www.nist.gov/itl/sed/topic-areas/measurement-uncertainty|
|||Tal, Eran, “Measurement in Science”, The Stanford Encyclopedia of Philosophy (Fall 2017 Edition), Edward N. Zalta (ed.), URL = <https://plato.stanford.edu/archives/fall2017/entries/measurement-science/>.|
|||Hubbard, Douglas W., and Richard Seiersen. How to measure anything in cybersecurity risk. Hoboken, New Jersey: Wiley, 2016. Print.|
|||A Turning Point for Humanity: Redefining the World’s Measurement System Robin Materese - https://www.nist.gov/si-redefinition/turning-point-humanity-redefining-worlds-measurement-system|
|||Keeping the Standard Kilogram From Gaining Weight Is a Constant Struggle Nadia Drake - https://www.wired.com/2013/01/keeping-kilogram-constant/|
|||Why scientists are redefining the kilogram https://www.engadget.com/2017/10/24/why-scientists-are-redefining-the-kilogram/|
|||Wikipedia contributors. (2018, November 6). Kibble balance. In Wikipedia, The Free Encyclopedia. Retrieved 18:15, November 8, 2018, from https://en.wikipedia.org/w/index.php?title=Kibble_balance&oldid=867616683|
|||Evaluation of measurement data – Guide to the expression of uncertainty in measurement JCGM 100:2008 (GUM 1995 with minor corrections)|
This section generally appeals to how experts can be queried for quantitative data.
Combining Expert Estimations¶
This describes the practice of gathering up forecast material and, typically, averaging it together. Parimutual Betting, Simple Averages, Weighted Scores.
|||Wikipedia contributors. (2018, October 1). Parimutuel betting. In Wikipedia, The Free Encyclopedia. Retrieved 18:01, November 8, 2018, from https://en.wikipedia.org/w/index.php?title=Parimutuel_betting&oldid=861927567|
|||Brown, Thomas A., Probabilistic Forecasts and Reproducing Scoring Systems. Santa Monica, CA: RAND Corporation, 1970. https://www.rand.org/pubs/research_memoranda/RM6299.html. Also available in print form.|
Calibration of Experts¶
Also see How to measure anything.
|||Calibration Of Probabilities: The State Of the Art To 1980 Lichtenstein- Sarah- Fischhoff- Baruch- Phillips-Lawrence D - http://www.dtic.mil/docs/citations/ADA101986|
|||Calibration Of Probabilities: The State Of the Art Lichtenstein- Sarah- Fischhoff- Baruch- Phillips - http://www.dtic.mil/docs/citations/ADA033248|
|||Miyamoto’s Secret Hobby https://kotaku.com/5381876/miyamotos-secret-hobby-measuring-stuff|
|||Shigeru Miyamoto Guesses The Size of Random Objects (Jimmy Fallon) https://youtu.be/DiUeuc7eOh0?t=16s|
RAND has been developing methods for expert estimation for decades, described as DELPHI and Futures Methodology.
|||Publications on Futures Methodologies: Delphi https://www.rand.org/pardee/pubs/futures_method/delphi.html|
|||Probabilistic Forecasts and Reproducing Scoring Systems https://www.rand.org/pubs/research_memoranda/RM6299.html|
|||On the Epistemology of the Inexact Sciences https://www.rand.org/pubs/reports/R353.html|
|||An Experimental Application of the Delphi Method to the Use of Experts https://www.rand.org/pubs/research_memoranda/RM727z1.html|
|||The Systematic Use of Expert Judgment in Operations Research https://www.rand.org/pubs/papers/P2795.html|
|||Convergence of Expert Consensus Through Feedback https://www.rand.org/pubs/papers/P2973.html|
|||Improving the Reliability of Estimates Obtained from a Consensus of Experts https://www.rand.org/pubs/papers/P2986.html|
|||The Use of the Delphi Technique in Problems of Educational Innovations https://www.rand.org/pubs/papers/P3499.html|
|||Analysis of the Future https://www.rand.org/pubs/papers/P3558.html|
|||Systematic Use of Expert Opinions https://www.rand.org/pubs/papers/P3721.html|
|||Delphi Process https://www.rand.org/pubs/papers/P3925.html|
|||Experiments in Group Prediction https://www.rand.org/pubs/papers/P3820.html|
|||Predicting the Future https://www.rand.org/pubs/papers/P3948.html|
|||Delphi and Values https://www.rand.org/pubs/papers/P4182.html|
|||The Delphi Method https://www.rand.org/pubs/research_memoranda/RM5888.html|
|||The DELPHI Method, II https://www.rand.org/pubs/research_memoranda/RM5957.html|
|||The Delphi Method, III https://www.rand.org/pubs/research_memoranda/RM6115.html|
|||The Delphi Method, IV https://www.rand.org/pubs/research_memoranda/RM6118.html|
|||Experimental Assessment of Delphi Procedures with Group Value Judgments https://www.rand.org/pubs/reports/R0612.html|
|||Comparison of Group Judgment Techniques with Short-Range Predictions and Almanac Questions https://www.rand.org/pubs/reports/R0678.html|
|||Delphi Assessment https://www.rand.org/pubs/reports/R1283.html|
Also see Tetlock.
|||Stan Kaplan, ‘Expert information’ versus ‘expert opinions’. Another approach to the problem of eliciting/ combining/using expert knowledge in PRA, Reliability Engineering & System Safety, Volume 35, Issue 1, 1992, Pages 61-72,|
|||R.L. Keeney ; D. von Winterfeldt. Eliciting probabilities from experts in complex technical problems, IEEE Transactions on Engineering Management ( Volume: 38 , Issue: 3 , Aug 1991 )|
IARPA invests in quite a bit of predictive research and publishes results often. They are also involved in forecasting tournaments.
|||Teams Better Than Individuals at Intelligence Analysis, Research Finds. American Psychological Association. https://www.apa.org/news/press/releases/2015/01/intelligence-analysis.aspx|
Cooke’s “Classical Method”¶
Often found in environmental risk (Volcanic, Earthquake) and others.
|||Roger Cooke, Max Mendel, Wim Thijs, Calibration and information in expert resolution; a classical approach, Automatica, Volume 24, Issue 1, 1988, Pages 87-93, ISSN 0005-1098, http://rogermcooke.net/rogermcooke_files/Automatica%201988%20Cal%20Inf.pdf|
|||Abigail R Colson, Roger M Cooke; Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments, Review of Environmental Economics and Policy, Volume 12, Issue 1, 1 February 2018, Pages 113–132, https://doi.org/10.1093/reep/rex022|
|||A route to more tractable expert advice, Willy Aspinall. http://www.rff.org/files/sharepoint/Documents/Features/294-295%20Opinion%20-%20Aspinall%20pr.pdf|
|||Usgs Expert Elicitation Report https://www.eeri.org/products-page/other-seminars-and-workshops/usgs-expert-elicitation-report/|
|||Workshop on the ground motion models applied in the National Seismic Hazard Maps https://earthquake.usgs.gov/static/lfs/nshm/workshops/GMPE2012/WED_AM01_Petersen.gmworkshop-berkeley-newsm.pdf|
Constructive critique of Cooke’s method can be found here:
|||Bolger, F. and Rowe, G. (2015), The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?. Risk Analysis, 35: 5-11. doi:10.1111/risa.12272|
Tetlock’s research revolves around how experts who are untrained in prediction are worse than random. He has since isolated those who are stronger forecasters (Superforecasters) and is identifying their qualities, especially around how someone a better forecaster, and how to further improve them with teams.
|||Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know?. Princeton, N.J: Princeton University Press.|
|||Tetlock, P. E., Gardner, D., Tetlock, Philip, Gardner, Dan, & Richards, Joel. (2015). Superforecasting: The art and science of prediction.|
|||Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Welton Chang, Eva Chen, Barbara Mellers, Philip Tetlock, Judgment and Decision Making, Vol. 11, No. 5, September 2016, pp. 509-526 http://journal.sjdm.org/16/16511/jdm16511.html|
|||Everybody’s an Expert, Louis Menand. https://www.newyorker.com/magazine/2005/12/05/everybodys-an-expert|
Maybe the oldest area of forecasting. Understanding the industrial development of meteorology is a great rubric for how a predictive industry is built over time. First, the theory. Then the infrastructure. Then the operational practice of prediction, decision making, and learning.
|||Weather Analysis and Forecasting https://www.ametsoc.org/ams/index.cfm/about-ams/ams-statements/statements-of-the-ams-in-force/weather-analysis-and-forecasting/|
|||Wikipedia contributors. “Timeline of meteorology.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 27 Sep. 2018. Web. 8 Nov. 2018.|
|||Bauer, Peter & Thorpe, Alan & Brunet, Gilbert. (2015). The quiet revolution of numerical weather prediction. Nature. 525. 47-55. 10.1038/nature14956. https://www.researchgate.net/publication/281516336_The_quiet_revolution_of_numerical_weather_prediction|
Kahneman / Tversky¶
Daniel Kahneman and Amos Tversky offer observations into how fallible the human mind is in the most common of circumstances. The classification of System 1 and System 2 thinking is highly relevant to this area of critical thinking around risk.
|||Kahneman, D. (2015). Thinking, fast and slow.|
Meehl / Dawes¶
Paul E. Meehl and Robyn Dawes work in prediction inspired a full fledged assault on the credibility of expert prediction. Comprehensive findings that mechanical statistical models beat experts at prediction.
|||Meehl, P. E. (1966). Clinical versus Statistical Prediction. Place of publication not identified: University of Minnesota Press. http://meehl.umn.edu/sites/g/files/pua1696/f/155dfm1993_0.pdf)|
Taleb explores the limitations of our ability to understand randomness and the nature of randomness. Preparation for inevitable surprise, and the emergence of Black Swans, is Taleb’s core message.
|||Taleb, N. N., Taleb, N. N., Taleb, N. N., Taleb, N. N., & Taleb, N. N. (2016). Incerto.|
Sherman Kent is considered a pioneer of intelligence analysis, and brought probabilistic rigor into the National Intelligence Estimate.
|||Wikipedia contributors. “Sherman Kent.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 20 Jun. 2018. Web. 8 Nov. 2018. https://en.wikipedia.org/wiki/Sherman_Kent|
|||Words Of Estimative Probability https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kent-and-the-board-of-national-estimates-collected-essays/6words.html|
|||The Law and Custom of the National Intelligence Estimate https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kent-and-the-board-of-national-estimates-collected-essays/5law.html|
|||The Making of an NIE https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kent-and-the-board-of-national-estimates-collected-essays/making.html|
|||The Theory of Intelligence https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kent-and-the-board-of-national-estimates-collected-essays/the-theory-of-intelligence.html|
There is research around Canada’s application of modern intelligence processing and its effectiveness. The basis of this is all probabilistic.
|||Canada Is Actually Pretty Good At Intelligence Forecasting, Ben Makuch - https://motherboard.vice.com/en_us/article/kbz7gn/canadian-intelligence-agencies-are-actually-pretty-good-at-strategic-forecasting|
|||Accuracy Of Forecasts in Strategic Intelligence, David Mandel-Alan Barnes - http://www.pnas.org/content/111/30/10984|
Industry examples where probabilistic risk assessment is at play:
|||NASA Risk Management Handbook https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120000033.pdf|
|||EPA: Risk Assessment Forum White Paper: Probabilistic Risk Assessment Methods and Case Studies https://www.epa.gov/sites/production/files/2014-12/documents/raf-pra-white-paper-final.pdf|
|||Probabilistic Risk Assessment Procedures Guide for Offshore Applications https://www.bsee.gov/sites/bsee.gov/files/ProbalisticRiskAssessment%20%28PRA%29/bsee_pra_procedures_guide_-_10-26-17.pdf|
|||Nuclear Probabilistic Risk Assessment https://www.nrc.gov/about-nrc/regulatory/risk-informed/pra.html|
This paper has a specifically useful overview of many different industry approaches to safety.
|||White Paper on Approaches to Safety Engineering http://sunnyday.mit.edu/caib/concepts.pdf|