1 Selective publication bias in economics
Selective publication bias exists in economics, as the author assessed in his previous paper (Andrews and Kasy 2019). Camerer et al. (2016) showed evidences in economics lab experiments.
Selection also happens in different forms. It may be driven by p-value hacking (a threshold of 95% confidence implies a z-stat of 1.96), or maybe by the expected sign of coefficient. Policy researches may favor “surprising” results, while journal papers – in contrast – tends to rejects unconventional, less “plausible” results.
Figure 1: Some possible forms of selection
Why is this a problem, you ask? Should we be only reading publications where the reported results are more likely to be true? Is there any welfare loss for publication bias? Turns out absorbing biased results might distorts our interference. One can create a Bayesian model of updated belief and test this out.
Figure 2: Settings for an example model. Source: a 2018 talk by the author
2 Reform proposal
- De-emphising statistical significance: is being done (e.g. aea guideline forbid star-reporting)
- Pre-analysis plans: another practice that economists are learning from clinical research, after rct has been heavily applied.
- Pre-results journal review: conducting peer-reviews before results are registered
- Journals for null results and replication studies
- Achieving multiple objectives in a functionally differentiated publication system
3 References
This post is in the collection of my public reading notes.