Expert Elicitation

We have to make decisions about risks… often with imperfect information. Data may not be available, may not be timely to gather, or measurement options may not yet exist. We may be left to use data with only a loose relationship with the target problem, or nothing at all.

The response to these situations may be to trust experts who have developed the most intuition around these risks and elicit numerical values from them. They may adjust reference class data, or provide a best estimate through a structured approach.

The elicitation of numerical values from experts has many known approaches. These values can be used for all sorts of risk measurement, and the approaches increase in structure depending on the need for rigor.

Rigor

Watercooler estimations are encouraged. Casual discussions can help build culture around estimation skill. Structured elicitation can increasingly include the following techniques to support high impact decision making or dangerous risks.

  • First, the reduction of bias is common in elicitation methods. Elicitation is a human, subjective process that can be guarded against harmful influence.
  • A panel of experts can have values combined to better represent a consensus view and eliminate bias from a single individual.
  • Questions and feedback is structured to promote learning before concluding.
  • The debate between experts is often managed closely to protect participants from unhealthy influence.
  • Experts are trained for probabilistic calibration and forecasting skill.
  • Data is carefully introduced as a reference class, or monte carlo simulations are developed to support judgement.
  • Forecasting platforms like the Good Judgement Open are examples of elicitation at scale.

As you may imagine, Elicitation that incorporates all of these factors would be time consuming, resource heavy, and expensive to organize.

Simple Elicitation

A simple, workplace approach to expert elicitation looks as follows:

  1. Clearly define the desired values or distribution.
  2. Train expert(s) for elicitation.
  3. Elicit a preliminary value from experts.
  4. If time allows, encourage discussion between experts (if any)
  5. If time allows, elicit a final value.

Other methods

These are formal methods that look to preserve scientific integrity or assist with defensible policy making. They are procedurally “heavy”.

  • Delphi (paper) is used by RAND and has a colorful history in many industries. Is known for heavily protecting participants and structuring Q&A between elicitations.
  • Cooke’s “classical” method is well known in environmental sciences and is used for applying stronger weighting to experts who perform better in calibration exercises.
  • The SHELF method has visible usage and journal references but is not yet understood by this documentation’s authors.
Introduction →