“Butter is Bad.” “Butter is Back, Baby.” “Red Meat is Bad.” “Red Meat=Beneficial Nutrients.”You get the point.
In our 24-hour news cycle, anything can be touted for “clicks” and “likes.” From time to time, studies which support writers’ personal biases are sought out. And, perhaps most importantly, today we are overwhelmed with the magnitude of available information, leaving one to scratch his or her head while it spins in confusion.
So, how do we narrow down all this info to make it meaningful? Consider going back to the source of the information and evaluate the studies yourself. Not a scientist? You can still find useful needles from the haystack by keeping in mind a few ideas:
1. Am I willing to invest the necessary time? Plan to read the entire study, not just the abstract (summary) of the paper. It will take some time. Have a notepad handy to jot down any ideas or questions the study prompts.
2. What kind of study is it?
a. A double-blind, random, controlled study (where neither participant nor researcher knows to what group the participant is assigned) is considered the best experimental model.
b. Observational studies typically observe a controlled environment, but results can be influenced by the participants’ awareness of the groups to which they are assigned. Conclusions in this case may be inferred but not necessarily proven.
c. Metanalysis examines multiple studies to look for consistent results.
3. Who funded the study? If the Widget Council funded a positive study on widgets, it may be a good study…or, it may be one good study among several unpublished ones that did not support their product. What about a study sponsored by American Gadget Association? A look at the sponsors may reveal funding for the study from the Widget Council. (A list of any researcher’s conflicts of interest should be listed.)
4. Where was the study published? In some journals the author can pay to have a study included regardless of the quality. Look for academic, peer-reviewed journals.
5. What are the researchers’ credentials? A neurobiologist with several advanced degrees and 15 years’ work experience studying the microscopic changes of a nerve probably has a great deal of credibility. An English major’s biology paper written to satisfy a graduation requirement might not contain the best criteria upon which to base a life-changing decision.
6. What is the study trying to prove? This information should be found in the Abstract or Introduction. A recent news article cited a study in the fats vs. carbs weight loss debate. However, the study’s Abstract revealed that the study topic regarded metabolism not weight loss.
7. How large is the study? A study of 10,000 participants will control for many more variables than a study of 100. Keep in mind though, under rare conditions ten participants may be a large sample size.
8. How many participants did the study start and end with? Suppose a study started with (a very reasonable) 1,200 screened participants, yet only 300 eventually participated. What was the elimination criteria? Was this a drug trial and participants quit taking the medicine due to side effects? Was it a diet that was too hard to follow? Did the scientists only include results that support one viewpoint?
9. How long did a study last? Two years for a chronic disease drug trial rather than a week could provide considerably more useful information. Conversely, the deaths of 10% of the sample within 5 days of study commencement is very telling.
10. Was this a cell culture, an animal or human study?
11. Do the results answer the question the researchers set out to prove? Can you think of any alternate reasons for why these results were obtained? Do the authors look at alternate reasons for the results they obtain? Don’t forget reader bias; are you willing to change your mind if the evidence is strong enough?
12. Good at statistics? Run the numbers. If not, then look for study critiques. Many journals have other researchers that will evaluate the study and explain the numbers as well as the study design, sometimes with different conclusions. Compare to other similar studies. Look at the references. They can shed new light on how the scientist is approaching the problem.
Like any other skill, learning to read this type of material takes time and patience. But the more you read the more you will learn, and this may help you to sort the wheat from the chaff the next time a new blockbuster study is rolled out in the media. For a more detailed study plan with additional questions to guide you, click here.