Over at Compound Interest they’ve put together a handy Rough Guide to critically reading science reporting.
Might have to invest in a poster for the wall of my office!
Anybody can make a ‘scientifically proven’ claim, but what does that really mean? In its essence, the scientific method aims to prove or disprove a hypothesis by providing rigorously-gathered and reproducible evidence. Though this provides a loose framework under which scientists can go about addressing a multitude of questions we must be careful to ensure that the results have come about as a result of good practice. Over the next few blog posts I will be exploring some of the key areas of scientific inquiry and highlighting the possible downfalls.
Causal links vs coincidence
Just because an experiment gives a certain set of results that doesn’t mean it’s been proven without a doubt; often times people will try and ‘spin’ a weak study to suggest outcomes that either aren’t strongly supported by the evidence, or have been justified by bad science. It’s key to make sure that the study is carried out in a way that goes beyond coincidence and proves the link between the treatment and the results. This means ensuring there is only one dependent variable being measured, and that the other variables are carefully controlled.
Look at the claims below. Have they been reliably proven, or are there other explanations you can think of?
- In the 1970s a correlation was drawn between people who drink high volumes of coffee and heart problems. Hence it was said that caffeine is bad for your heart! Highlight for possible explanations: (Other explanations: people who drank large amounts of coffee were also generally highly stressed, smoked cigarettes, had poor diets and did not exercise. Unfiltered coffee also contains oils which have been shown to raise cholesterol levels.)
- Social drinking and earnings – drinkers earn more money according to Bethany L. Peters & Edward Stringham (2006. “No Booze? You May Lose: Why Drinkers Earn More Money Than Nondrinkers,” Journal of Labor Research, Transaction Publishers, vol. 27(3), pages 411-421, June). (Other explanations: people who earn more money drink more either because they have a greater disposable income or due to stress?)
- What do you think is the reasoning behind this correlation?
- One of the most famous examples is the stunning correlation between ice cream sales and shark attacks – the latter rises sharply when the former increases. Is it because it’s harder to swim away from a shark when you’re holding an ice cream cone? Are sharks particularly fond of mint chocolate chip? (In what is perhaps a boring revelation, both ice cream sales and shark attacks tend to go up during the summer months, when more people are enjoying the surf and sun at the beach.)
In addition to the fact that these correlations are spurious at best and silly at worst, they assume that any result can be overwhelmingly attributed to one single cause. While scientific studies certainly can (and do) support a ‘one cause, one effect’ relationship in some circumstances it would be naive to assume that all changes can be traced back to one single cause. It may be reassuring to think that there’s one thing you can cut out of your diet that will help your heart health, but it’s rarely the case. More often there are multiple causes that go into effecting a change, and it’s here that a properly conducted study will begin to reveal interesting results.
Stay tuned for more scientific inquiry principles! Next up: Study size and population representation!