Every few weeks, a science news headline comes along – ‘Scientists prove what we’ve known all along!‘. This week’s offering is that men’s and women’s brains are wired differently, thus justifying stereotypes about women being more emotional while men can read maps.
It’s hard to read and interpret a scientific paper in a field you’re not familiar with, even if you are used to reading papers in your own field. There’s a whole heap of background literature that you’re unfamiliar with, conventions and experimental methods that differ from your own, and you lack the familiarity that allows you to place a paper in context and fully appreciate what contribution is actually being made. Hence, reading experts in the field debunk sensationalist claims can be an interesting way to gain a better understanding. For example, here’s some further reading on the male/female brain wiring from some scientists who know what they’re talking about.
Still, despite not always fully understanding the original research, here are a few reasons I’m wary of science stories that just seem too good to be true:
1. The science in the paper almost never justifies the bold conclusion that the news story claims, and can be far removed from what we experience in everyday life. For example, in the brain wiring example, it’s a huge jump to suggest that different connections seen in male and female brains are responsible for stereotypical personality characteristics.
2. Studies are often built on top of previous, sometimes contradictory, studies. Science is incremental and each paper provides a little bit of the jigsaw, never the complete picture like news stories suggest.
3. Statistically significant is not significant. A tiny difference between two values may be statistically significant in a science paper, but may not have any noticeable difference in real life.
4. Science news stories are often uncritical without any comment from other scientists. This makes me think that the journalists are essentially reprinting the press release without any further thought.
5. The story is never as simple as presented. There may be multiple interpretations of the results that the authors haven’t discussed, especially when taken together with other related research.