Counter to intuition: Basic methods of science to pull back nature’s façade, Part II

Today we’re posting about the role of the scientific community.  Last week, Adrian gave a really great overview of what sorts of things scientists think about and practice in their own research.  Each scientist is responsible for their own integrity, but science doesn’t advance in a bubble.  Scientists participate in the greater scientific community, and strong scientists and strong communities build really great science. The following post highlights some of the roles the scientific community plays in good science.

Adrian writes:

Peer-review and reproducibility

Scientific journals have a peer-review process, but the exact process can vary depending on the journal and the type of article that is being reviewed. During this process, a team can submit their manuscript to a journal, which then sends out copies to a few reviewers. These reviewers make suggestions by comparing the manuscript to their own knowledge of the field, as well as articles that have been published already. The research team can then integrate those suggestions into the final manuscript and then finally submit it for publication – as long as the journal decides that it is worth publishing.5 The basic intention of this process is to ensure that the manuscript is congruent with prior literature, or at least offers reasonable explanations for why the findings contradict other published literature. While findings in rapidly-developing areas of science can sometimes conflict with prior literature, conflicting with more foundational science will draw far more scrutiny and frustrated glares.

Replicating findings

One salient example that demonstrates the need for results to be reproduced comes from a notorious report in 1998 by Andrew Wakefield and 11 other co-authors that made claims of a link between the MMR vaccine, cancer, colon disease, and autism. Although their article made a tremendous impact in press reports, their findings were unable to be reproduced and their report has since been retracted.6-11 Furthermore, Wakefield had undeclared financial interests closely tied with his published results.12 This point also illustrates why it’s important to find out whether a study agrees with prior literature, or to follow the study so whether it is replicated before its findings can be claimed as fact; publication of an article is only the beginning. Thus one needs to be cautious, even when perusing peer-reviewed journals, because it can be difficult to tell whether a study is in agreement with the vast amounts of similar research.

Though this is a very dramatic example of why reproducibility is necessary in science, failures to replicate often don’t indicate fraud, or even sloppiness. Failures to replicate can easily come up from pure bad luck – such as a sample that disproportionately favors outliers. Hopefully by now, you realize that you can reduce these incidents to a minimum by expanding the sample size, or using methods to ensure a random sample. However a statistically significant finding of less than 0.05 leaves open the possibility that the results were due to chance events and that the null hypothesis was indeed true after all.

Potential conflicts of interest don’t always mean that a study is wrong. For example, a study at University of Bordeaux in France found that drinking very small amounts of wine reduced the risk of Alzheimer’s dementia. This study could have been influenced by the local wine industry but follow-up studies seem to confirm their findings.13-15 Those who would have dismissed these results the first time around – because of a conflict of interest – would have a much more difficult time doing so each time the results are replicated.

Separating results from interpretation

One of the most difficult aspects of presenting data is to separate the results of an experiment from our interpretation of them. Statistical tests can objectively verify that a predictive pattern exists in the data, yet it is a completely different task to explain why the trend exists. This distinction needs to be made because if a subjective interpretation ends up being incorrect, it’s important to be able to backtrack to the actual objective data.

One example of the importance of separating data from interpretations is shown in light of the mixed health effects of drinking coffee. A lot of research shows that drinking coffee can protect your brain against the degenerative effect of dementia.16,17 This can lead people into interpreting this as meaning that “Coffee is good for my health.” This interpretation can come into conflict when presented with other research that indicates that it may increase anxiety and depression – which are obviously not good for your health.18 In this case it would be helpful to refer back to the data to realize that the findings aren’t mutually exclusive – a substance can reduce risk for dementia and also induce acute anxiety. The data are also helpful to weigh their relative risks and benefits of these routines.

By convention, peer-reviewed articles have distinct “results” and “discussion” sections. This enables researchers to simply describe what they found, and then to describe separately what they think happened. Furthermore, discussion sections encourage the researchers to speculate about how their research fits into broader contexts, identify unanswered questions, and point out unexpected patterns in the data.

Results are arguably going to be more objective than discussions, because the results simply state the bare-bones facts of what they found. However that hardly means that the discussion is guesswork or unimportant; frequently the discussions are more interesting than the results themselves because the authors tend to make interesting connections with other work, and to point out patterns that aren’t obvious to a less-informed reader.

A modern scientist

The accepted role of a scientist is simply to observe, describe, and report the findings but not to issue recommendations on lifestyle or policy matters. If you read a news article about the health benefits or risks of a particular food or drug, you should be wary not to mistake the finding for a recommendation. As a rule, scientists don’t issue health advice; that task is left up to medical doctors or political advisors (especially when crafting general policies). Although scientists don’t normally dispense health guidance in an official capacity, they can have informed opinions on health matters. In a much broader sense, science describes facts involving nature and what moves it along – but not what is “good,” “bad,” or what “should” be happening in nature.

Most scientists prefer to describe their jobs as gathering evidence, rather than proving things because the concept of proving something is much more nebulous. The ways I outlined above helps us gather objective data and provide evidence for things. These are the basic principles in science that most instructors try to instill when training scientists. These are also basic aspects of science that operate against the instinctive way we see connections and assume causality. This may all seem a bit excessive, but since science tends to build on itself, no scientist wants to find out that key assumptions they were making were based on someone else’s flawed data; likewise no scientist wants to give out flawed results for others to build on. Flawed interpretations are more permissible because at least the data can be correct and re-interpreted with a fresh insight. Conventions therefore, attempt to ensure that the data are objective and uncorrupted.

Some of these principles have developed over time since the time of Mendel. For example, it’s expected in the modern era to do statistical testing to determine how highly correlated two variables are. However, Mendel would never have noticed the pattern if he hadn’t spent years collecting an enormous sample, and then tracking the ratios of plants with certain traits over multiple generations. He would have been drawn into making false conclusions if his experiments were not carefully controlled for. Mendel’s works have, of course, been replicated but not in the strict sense of repeating the same experiment over, but instead inferring the effects of heritable characteristics in other experiments. For example, Mendel’s prediction (of discrete units of heredity) can be found in the DNA of individual genes – which of course, are stable over generations and track with their respective traits. Therefore, we must be cautious about dismissing findings of a study, even when they seem to be blatantly true; the true power of the findings are reflected in the predictions that they make, which tend not to be very obvious and are often unspoken.

In Plato’s allegory of the cave, Plato described prisoners trapped in a cave with their view restricted to observing shadows cast along the cave’s wall. He challenged philosophers to gain the exceptional insight to understand the objects and circumstances that cast the shadows. The paramount responsibility of the scientist is to understand the nature of reality by searching beyond the deceptions of the shadows. The conventions of modern science help accomplish this duty.

Works Cited:

5 Golash-Boza, T. How to write a peer review for an academic journal: Six steps from start to finish by Tanya Golash-Boza.

6  Afzal, M. A., Armitage, E., Ghosh, S., Williams, L. C. & Minor, P. D. Further evidence of the absence of measles virus genome sequence in full thickness intestinal specimens from patients with Crohn’s disease. Journal of Medical Virology 62, 377-382, doi:10.1002/1096-9071(200011)62:3<377::AID-JMV10>3.0.CO;2-1 (2000).

7  Deer, B. How the case against the MMR vaccine was fixed. Vol. 342 (2011).

8  Madsen, K. M. et al. A Population-Based Study of Measles, Mumps, and Rubella Vaccination and Autism. New England Journal of Medicine 347, 1477-1482, doi:doi:10.1056/NEJMoa021134 (2002).

9  Taylor, B. et al. Autism and measles, mumps, and rubella vaccine: no epidemiological evidence for a causal association. The Lancet 353, 2026-2029, doi:10.1016/S0140-6736(99)01239-8.

10  The Editors of The, L. Retraction—Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. The Lancet 375, 445, doi:10.1016/S0140-6736(10)60175-4.

11  Wakefield, A. J. et al. RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. The Lancet 351, 637-641, doi:10.1016/S0140-6736(97)11096-0.

12  Staff, C. W. Vaccine study’s author held related patent, medical journal reports.

13  Peters, R., Peters, J., Warner, J., Beckett, N. & Bulpitt, C. Alcohol, dementia and cognitive decline in the elderly: a systematic review. Age and Ageing 37, 505-512, doi:10.1093/ageing/afn095 (2008).

14  Orgogozo, J. M. et al. Wine consumption and dementia in the elderly: a prospective community study in the Bordeaux area. Rev Neurol (Paris) 153, 185-192 (1997).

15  Letenneur, L. Risk of dementia and alcohol and wine consumption: a review of recent results. Biol Res 37, 189-193 (2004).

16  Arendash, G. W. & Cao, C. Caffeine and coffee as therapeutics against Alzheimer’s disease. J Alzheimers Dis 20 Suppl 1, S117-126, doi:10.3233/jad-2010-091249 (2010).

17  Arendash, G. W. et al. Caffeine reverses cognitive impairment and decreases brain amyloid-beta levels in aged Alzheimer’s disease mice. J Alzheimers Dis 17, 661-680, doi:10.3233/jad-2009-1087 (2009).

18  Veleber, D. M. & Templer, D. I. Vol. 93   120-122 (American Psychological Association, US, 1984).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s