We all need a Framework for Evidence. How to improve decision making trough science?

Even today at the end of 2023 with the amount of information and processing power that we have, we still struggle to differentiate causality from a spurious correlation. In the economic jargon, identification of the problem is complex and sometimes hidden. Betting everything on observational analysis typically means adding more information and/or model complexity, though this can increase the probability of misleading results (spurious correlations, overfitting…), reinforcing our cognitive biases.

RCT’s & Natural Experiments

Randomized Control Trials (RCT) together with Natural Experiments are great tools to reduce biases in our causality journey. RCT’s were developed first in health disciplines during the XIX and the gradually in other domains (agriculture, education…). But they were really exploited at the beginning of XXI after the publication of the essay of John Loannidis “Why Most Published Research Findings Are False”. In this paper Loannidis argued about the issues on replicability of vast majority of published medical research. Today we can find more than 1.7 million of medicine RCT’s documented at the Cochrane Library.

Google Books Ngrams containing “Randomized Controlled Trial” or “Natural Experiment”

In Economic science during last decades there has been some concern about the reliability of previously used methods or identifying causal relationships. In fact the two Nobel Economy laureates in 2019 Duflo and Banerjee have been the main pioneers of RCT’s with the development of Abdul Latif Jameel Poverty Action Lab (J-PAL) as it is today with more than 1071 RCT’s performed. Their work has impacted institutions like the World Bank deciding in 2005 to establish a dedicated impact evaluation unit.

Also, the supporters of Evidence-based policies have been growing during the beginning of XXI with institutions like Overseas Development Institute (ODI)promoting the use of RCT’s for poverty reduction.

But sometimes controlled experimentation is just not possible due to ethical or costs reasons. This is where Natural Experiments can also enlighten fact based decisions. Some nice examples are shared by Esther Duflo and Banerjee in their great book that I can just recommend Good Economics for Hard Times.

Meta-research or Evidence-based research, the last frontier of science

The disciple of meta-research focuses on increasing the quality and robustness of science. By the way, this should be a key topic for any decision maker. In areas like Health this is actually quite advanced with organizations like CONSORT with their EQUATOR NETWORK publishing the guidelines and methodology to increase research robustness.

In economy World bank developed DIME a framework for evidence-based policy making, bridging the gap between research and practice. Another interesting one is the RAPID Outcome Mapping Approach ROMA with an 8 step approach.

Recently last year was published in Nature Standards for evidence in policy decision-making by Kai Ruggeri and multidisciplinary team of scientists, policymakers, government officials where they propose the THEARI rating system to score the evidence for robust policy implementation. This is a simple but from my perspective very straight forward framework to enable evidence robustness and therefore learning.

Criticism

There are big challenges like the robust definition of RCT’s, their horizon of control, the assumptions made raised by Alexander Krauss in the 2017 paper “Why all randomised controlled trials produce biased results” by assessing the issues at found at the 10 most cited RCT studies worldwide.

Nevertheless, I just can hope that while the Natural Experiment we are living today with COVID-19 ends, key decision makers in society takes these great tools into consideration and applies with continuous open peer to peer revision and constructive criticism.