Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
Abstract
Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created—and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.
Get full access to this article
View all access and purchase options for this article.
References
Arthur W. Jr., Tubre T. C., Paul D. S., Sanchez-Ku M. L. (1999). College-sample psychometric and normative data on a short form of the Raven Advanced Progressive Matrices Test. Journal of Psychoeducational Assessment, 17, 354–361.
Atanasov P., Rescober P., Stone E., Servan-Schreiber E., Tetlock P., Ungar L., Mellers B. (2014). Distilling the wisdom of crowds: Prediction markets versus prediction polls. Manuscript under review.
Balboni G., Naglieri J. A., Cubelli R. (2010). Concurrent and predictive validity of the Raven Progressive Matrices and the Naglieri Nonverbal Ability Test. Journal of Psychoeducational Assessment, 28, 222–235.
Baron J. (2000). Thinking and deciding (3rd ed.). New York, NY: Cambridge University Press.
Baron J., Scott S. E., Fincher K., Metz S. E. (2014). Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory and Cognition, Special Issue on Modeling and Aiding Intuitions in Organizational Decision Making. Advance online publication.
Baron J., Spranca M. (1997). Protected values. Organizational Behavior and Human Decision Processes, 70, 1–16.
Betts J. R., Shkolnik J. L. (2000). The effects of ability grouping on student achievement and resource allocation in secondary schools. Economics of Education Review, 19, 1–15.
Birnbaum M. H., Mellers B. A. (1983). Bayesian inference: Combining base rates with opinions of sources who vary in credibility. Journal of Personality and Social Psychology, 45, 792–804.
Brier G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1–3.
Cacioppo J. T., Petty R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131.
Cacioppo J. T., Petty R. E., Kao C.-F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306–307.
Cattell R. B., Horn J. L. (1978). A check on the theory of fluid and crystallized intelligence with description of new subtest designs. Journal of Educational Measurement, 15, 139–164.
Christensen-Szalanski J., Wilhelm C. (1991). The hindsight bias: A meta-analysis. Organizational Behavior and Human Decision Processes, 48, 147–168.
Cooper A., Woo C., Dunkelberg W. (1988). Entrepreneurs’ perceived chances for success. Journal of Business Venturing, 3, 97–108.
Dawes R. M., Faust D., Meehl P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668–1673.
Desvouges W. H., Johnson F., Dunford R., Hudson S., Wilson K., Boyle K. (1993). Measuring resource damages with contingent valuation: Tests of validity and reliability. In Hausman J. A. (Ed.), Contingent valuation, A critical assessment (pp. 91–164). Amsterdam, The Netherlands: North Holland.
Duckworth A. L., Peterson C., Matthew M. D., Kelly D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087–1101.
Dweck C. S. (2006). Mindset: The new psychology of success. New York, NY: Ballantine Books.
Epple D., Romano R. (2011). Peer effects in education: A survey of the theory and evidence. Handbook of Social Economics, 1, 1053–1163.
Ericsson K. A., Krampe R. T., Romer C. T. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363–406.
Fiske A., Tetlock P. E. (1997). Taboo tradeoffs: Reactions to transgressions that transgress spheres of influence. Political Psychology, 18, 255–297.
Frederick S. (2005). Cognitive reflection and decision making. Journal of Economic Perspective, 19(4), 25–42.
Gilovich T., Griffin D., Kahneman D. (2002). Heuristics and biases: The psychology of intuitive judgment. New York, NY: Cambridge University Press.
Gottfredson L. (1997). Why g matters: The complexity of everyday life. Intelligence, 24, 79–132.
Haran U., Ritov I., Mellers B. (2013). The role of actively open-minded thinking in information acquisition, accuracy, and calibration. Journal of Decision Making, 8, 188–201.
Hartzmark M. L. (1991). Luck versus forecast ability: Determinants of trader performance in futures markets. The Journal of Business, 64, 49–74.
Heisenberg W. (1999). Physics and philosophy: The revolution in modern science. Amherst, NY: Prometheus Books. (Original work published 1958)
Hunter J. E., Schmidt F. L. (1996). Intelligence and job performance: Economic and social implications. Psychology, Public Policy, and the Law, 2, 447–472.
Jagacinski R. J., Isaac P. D., Burke M. W. (1977). Application of signal detection theory to perceptual-motor skills: Decision processes in basketball shooting. Journal of Motor Behavior, 9, 225–234.
Jervis R. (2010). Why intelligence fails. Ithaca, NY: Cornell University Press.
Kahneman D. (2011). Thinking, fast and slow. New York, NY: Cambridge University Press.
Kahneman D., Knetsch J. (1992). Valuing public goods: The purchase of moral satisfaction. Journal of Environmental Economics and Management, 22, 57–70.
Kahneman D., Tversky A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–292.
Kahneman D., Tversky A. (1985). Evidential impact of base rates. In Kahneman D., Slovic P., Tversky A. (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 153–160). New York, NY: Cambridge University Press.
Kray L. J., George L. G., Galinsky A. D., Roese N. J., Liljenquist K. A., Tetlock P. E. (2010). From what might have been to what must have been: Counterfactual thinking creates meaning. Journal of Personality and Social Psychology, 98, 106–118.
Levitt B., March J. G. (1988). Organizational learning. Annual Review of Sociology, 14, 319–340.
Lichtenstein S., Fischhoff B., Phillips L. D. (1982). Calibration of probabilities: The state of the art to 1980. In Kahneman D., Slovic P., Tversky A. (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 306–334). Cambridge, England: Cambridge University Press.
Lipkus I. M., Samsa G., Rimer B. K. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37–44.
Mellers B. A., Stone E., Atanasov P., Rohrbaugh N., Metz E. S., Ungar L., . . . Tetlock P. (2014). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Manuscript under review.
Mellers B. A., Ungar L., Baron J., Ramos J., Gurcay B., Fincher K., . . . Tetlock P. (2014). Psychological strategies for winning a geopolitical tournament. Psychological Science, 25, 1106–1115.
Milgram S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.
Murphy A. H., Winkler R. L. (1987). A general framework for forecast verification. Monthly Weather Review, 115, 1330–1338.
Ofir C. (1988). Pseudodiagnosticity in judgment under uncertainty. Organizational Behavior and Human Decision Making, 42, 343–363.
Peters E., Västfjäll D., Slovic P., Mertz C. K., Mazzocco K., Dickert S. (2006). Numeracy and decision making. Psychological Science, 17, 407–413.
Rosenthal R. (1966). Experimenter effects in behavioral research. East Norwalk, CT: Appleton-Century-Crofts.
Satopää V. A., Baron J., Foster D., Mellers B. A., Tetlock P. E., Ungar L. H. (2014). Combining multiple probability predictions using a simple logit model. International Journal of Forecasting, 30, 344–356.
Satopää V. A., Jensen S., Mellers B. A., Tetlock P. E., Ungar L. H. (2014). Probability aggregation in time series: Dynamic hierarchical modeling of sparse expert beliefs. Annuals of Applied Statistics, 8, 1256–1280.
Schkade D., Payne J. (1994). How people respond to contingent valuation questions: A verbal protocol analysis of willingness to pay for an environmental regulation. Journal of Environmental Economics and Management, 26, 88–109.
Shipley W. C. (2009). Shipley–2 manual. Torrance, CA: Western Psychological Services.
Svenson O. (1981). Are we all less risky and more skillful than our fellow drivers? Acta Psychologica, 47, 143–148.
Swets J. A., Dawes R., Monahan J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26.
Tetlock P. E., Mellers B. A. (2002). The great rationality debate. Psychological Science, 13, 94–99.
Wason P. C. (1968). Reasoning about a rule. Quarterly Journal of Experimental Psychology, 20, 273–281.
Yaniv I., Foster D. (1995). Graininess of judgment under uncertainty: An accuracy-informativeness tradeoff. Journal of Experimental Psychology: General, 4, 424–432.
Zimbardo P. (1973). On the ethics of intervention in human psychological research: With special reference to the Stanford Prison Experiment. Cognition, 2, 243–256.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: May 18, 2015
Issue published: May 2015
Keywords
PubMed: 25987508
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in Perspectives on Psychological Science.
VIEW ALL JOURNAL METRICSArticle usage*
Total views and downloads: 4512
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 117 view articles Opens in new tab
Crossref: 136
-
Measuring the ratio of true‐positive to false‐positive judgements made...
-
Increasing Trust in Climate Vulnerability Projections
-
Forecasting adversarial actions using judgment decomposition-recomposi...
-
Measuring probabilistic coherence to identify superior forecasters
-
Improving the accuracy of social work judgements: A proof‐of‐concept s...
-
Expert Judgment in the 2022 Aotearoa New Zealand National Seismic Haza...
-
Crowd prediction systems: Markets, polls, and elite forecasters
-
Majority rule can help solve difficult tasks even when confident membe...
-
Developing Decision-Making Expertise in Professional Sports Staff: Wha...
-
Can Social Workers Estimate the Likelihood of Future Actions and Event...
-
Forecasting with jury-based probabilistic argumentation
-
Machine learning augmentation reduces prediction error in collective f...
-
Human and Algorithmic Predictions in Geopolitical Forecasting: Quantif...
-
Predicting decision-makers’ algorithm use
-
Big STEM collaborations should include humanities and social science
-
Much more than a prediction: Expert-based software effort estimation a...
-
Key Science Goals for the Next-Generation Event Horizon Telescope
-
Predicting and reasoning about replicability using structured groups
-
False dichotomy alert: Improving subjective-probability estimates vs. ...
-
Predicting reliability through structured expert elicitation with the ...
-
Decomposing the effects of crowd-wisdom aggregators: The bias–informat...
-
Predicting the future with humans and AI
-
Talent Spotting in Crowd Prediction
-
The Wisdom of Timely Crowds
-
Incorporating External Factors into Time Series Forecasts
-
Transforming Conservation
-
Predictive validity in drug discovery: what it is, why it matters and ...
-
Review of approaches and methods for identifying experts in organizati...
-
Can Groups Improve Expert Economic and Financial Forecasts?
-
Stability of Experimental Results: Forecasts and Evidence
-
Crowd Prediction Systems: Markets, Polls, and Elite Forecasters
-
Using scenarios to forecast outcomes of a refugee crisis
-
Convex combinations in judgment aggregation
-
Communicating uncertainty using words and numbers
-
Information aggregation and collective intelligence beyond the wisdom ...
-
Estrangement, immersion, and the future: Designing the speculative env...
-
Aggregating human judgment probabilistic predictions of the safety, ef...
-
Policy for promoting analytic rigor in intelligence: professionals’ vi...
-
Principal investigators over-optimistically forecast scientific and op...
-
What is a Good Calibration Question?
-
Uncertainty Quantification with Experts: Present Status and Research N...
-
What Correlates with Accuracy: The Empirical Epistemology of Optimal C...
-
How Can We Get More Accurate: Recommendations About Human Judgment
-
Red hot frogs: identifying the Australian frogs most at risk of extinc...
-
Hidden Experts in the Crowd: Using Meta-Predictions to Leverage Expert...
-
False Dichotomy Alert: Cultivating Talent at Probability Estimation vs...
-
Getting more Wisdom out of the Crowd: The Case of Competence-Weighted ...
-
Towards Augmented Financial Intelligence
-
Can Social Workers Forecast Future Actions, Events, and Outcomes? A St...
-
Forecast Hedging and Calibration
-
Bias, Information, Noise: The BIN Model of Forecasting
-
Can you crowdsource expertise? Comparing expert and crowd‐based scorin...
-
Risk perception and communication
-
Emotionally Numb: Expertise Dulls Consumer Experience
-
Mathematically aggregating experts’ predictions of possible futures
-
Forecasting under Uncertainty: How Network Composition Shapes Future-O...
-
Tracking accuracy of strategic intelligence forecasts: Findings from a...
-
Effects of Choice Restriction on Accuracy and User Experience in an In...
-
A Prediction Tournament Paradox
-
Pooling decisions decreases variation in response bias and accuracy
-
Did you see it coming? Explaining the accuracy of voter expectations f...
-
Hybrid Predictive Ensembles: Synergies Between Human and Computational...
-
Developing a reference protocol for structured expert elicitation in h...
-
Hidden dangers in complex computational structures: A commentary on Lu...
-
Behavioral brittleness: the case for strategic behavioral public polic...
-
Hybrid Intelligence in Big Data Environment: Concepts, Architectures, ...
-
Forecasting forecaster accuracy: Contributions of past performance and...
-
Superforecasting reality check: Evidence from a small pool of experts ...
-
Forecasting chaotic events and the prediction of a rare cognitive abil...
-
Decision making can be improved through observational learning
-
Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Informat...
-
Human Forest vs. Random Forest in Time-Sensitive COVID-19 Clinical Tri...
-
Quantifying machine influence over human forecasters
-
Quantitative Foresighting as a Means of Improving Anticipatory Scienti...
-
Small steps to accuracy: Incremental belief updaters are better foreca...
-
The protective scale of the Armidilo‐S: The importance of forensic and...
-
Weighting and aggregating expert ecological judgments
-
Predicting the Future in Child and Family Social Work: Theoretical, Et...
-
Scoring in context
-
Computer-Supported Collaborative Information Search for Geopolitical F...
-
Scientific testimonial standards for microbial forensic evidence
-
Bias, Information, Noise: The BIN Model of Forecasting
-
The Contributions of Diversity, Accuracy, and Group Size on Collective...
-
Better beware: comparing metacognition for phishing and legitimate ema...
-
Collectively Intelligent Teams: Integrating Team Cognition, Collective...
-
How to detect high-performing individuals and groups: Decision similar...
-
What Makes Foreign Policy Teams Tick: Explaining Variation in Group Pe...
-
Debiasing Training Improves Decision Making in the Field
-
Forecasting tournaments, epistemic humility and attitude depolarizatio...
-
From madness to wisdom: intelligence and the digital crowd
-
Consilience, Global Socioeconomic Political Enlightenment, and Socioen...
-
Intelligence, Educational and Learning Capital, and Domain Impact Leve...
-
Predicting Experimental Results: Who Knows What?
-
The Value of Performance Weights and Discussion in Aggregated Expert J...
-
Bayesian Revision vs. Information Distortion
-
The role of “Prominent Numbers” in open numerical judgment: Strained d...
-
What Motivates Effort? Evidence and Expert Forecasts
-
The Value of Precision in Probability Assessment: Evidence from a Larg...
-
Ever closer or lost at sea? Scenarios for the future of transatlantic ...
-
The Evolution of User Experiences and Interfaces for Delivering Contex...
-
A Generalizable Method for Validating the Utility of Process Analytics...
-
LineChange: An Analytic Framework for Automated Moderation of Crowdsou...
-
Expert Elicitation: Using the Classical Model to Validate Experts’ Jud...
-
Public Understanding of Ebola Risks: Mastering an Unfamiliar Threat
-
Shoot the Horse and Build a Better Barn Door: Exploring the Potential ...
-
Geopolitical Forecasting Skill in Strategic Intelligence
-
Neural Networks Assist Crowd Predictions in Discerning the Veracity of...
-
Artificial Neural Networks Can Distinguish Genuine and Acted Anger by ...
-
Forecasting success via early adoptions analysis: A data-driven study
-
Chapter 11: How Do Managers Really Think? Using Think Aloud and fMRI t...
-
How Bayesian Rationality Fails and Critical Rationality Works
-
Forecasts or fortune-telling: When are expert judgements of safety ris...
-
Confidence Calibration in a Multiyear Geopolitical Forecasting Competi...
-
A neglected dimension of good forecasting judgment: The questions we c...
-
The circadian profile of epilepsy improves seizure forecasting
-
How generalizable is good judgment? A multi-task, multi-benchmark stud...
-
Superforecasting: the art and science of prediction
-
Lessons from near real-time forecasting of irregular leadership change...
-
An IRT forecasting model: linking proper scoring rules to item respons...
-
Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Pol...
-
Bringing probability judgments into policy debates via forecasting tou...
-
What Is Fluid Intelligence? Can It Be Improved?
-
Expertise, credibility of system forecasts and integration methods in ...
-
I nvestigate D iscuss E stimate A ggregate for structured expert judg...
-
Use of expert knowledge to anticipate the future: Issues, analysis and...
-
Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions
-
2016 and the Split Prior: An Investigation into the Effect of Social I...
-
Developing Statistics Cognitions
-
Fifty percent of anaesthetists are worse than average at understanding...
-
Explanation, updating, and accuracy
-
Rethinking the training of intelligence analysts
-
Developing expert political judgment: The impact of training and pract...
-
The “CAPS” Prediction System and Stock Market Returns*
-
Validating the Contribution-Weighted Model: Robustness and Cost-Benefi...
-
Improving Intelligence Analysis With Decision Science
Figures and tables
Figures & Media
Tables
View Options
Get access
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
loading institutional access options
APS members can access this journal content using society membership credentials.
APS members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
