Designing Studies With High Informational Value

Kurt Lewin Instituut
Heidelberglaan 1
Room Number H1.42
3584 CS UTRECHT
The Netherlands
T: +31 (0)30 - 253 3027

Designing Studies With High Informational Value

Teaching staff
Dr. Uri Simonsohn (The Wharton School, University of Pennsylvania), dr. Ellen Evers, MSc. (The Wharton School, University of Pennsylvania), dr. Daniel Lakens (Eindhoven University of Technology)

Type of course
Methodological and Practical Courses

Date
September 9 and September 10, 2015

Location
VU University Amsterdam: computer room 1G-28 (Hoofdgebouw, Filosofenhof)


Duration

2 days 10.00 am - 5.00 pm

Language

English

Content

In pursuit of knowledge, our field is faced with two big problems. The first problem is publication bias. Due to a focus on significant results (and with very few not significant findings being published) it is difficult to know which findings are likely to be true. If lines of research have evidential value, publication bias makes it very difficult to know what an accurate approximation of the true effect size is. The second problem is a lack of power in performed studies. Lack of power undermines the informational value of studies. Underpowered experiments are unlikely to observe an effect even when it is actually true, and thus hinder theoretical progress.

In this workshop, students will gain hands-on experience with novel statistical techniques that will allow them to meta-analytically evaluate lines of research. First, we will discuss challenges in interpreting meta-analytic findings published in the literature based on traditional meta-analysis of effect sizes. We will provide recommendations to increase the quality of meta-analyses, including open data and disclosure tables. Subsequently, we will discuss novel statistical techniques to estimate true effect sizes in the presence of meta-analysis. We will get hands-on experiences with p-curve analyses, and learn how p-curves can provide information about the true effect size. Furthermore, we will review alternative meta-analytic approaches, such as meta-regression.

In addition, we will explain what normal patterns of experimental findings look like. Recently, empirical papers have been criticized for containing too many true effects, which is unlikely in light of the power of the performed experiments. We will explain how such sets of studies can be identified, what the effect of publication bias can be on the effect size estimation, and how to write-up sets of studies containing statistically significant and non-significant findings.

Students are invited to send in questions before the workshop starts, and we will try to address their questions during the workshop.

This workshop builds on a previous KLI workshop "How we know what is likely to be true" organized on June 19, 2014 (KLI Teaching Program 2013-2014), and assumes some of the knowledge taught in that workshop to be known. Additional reading is provided for students who did not attend this earlier workshop, and a brief summary will be provided on the first morning.

Literature

To be announced


« back

News

KLI conference 2018: program
Read...

KLI conference 2018 for members
Read...

News archive...

Planned defences

All planned defences...

Recent PhD titles

November 2, 2018
Dalya Samur
Vrije Universiteit Amsterdam
More...

September 7, 2018
Caroline Schlinkert
Vrije Universiteit Amsterdam
More...

All PhD titles...