Good Studies, Bad StudiesGood Studies, Bad Studies

Good Studies, Bad Studies
December 1999 -- Perspectives

By: Heidi Kreuzer
Senior Editor

  When we hear the words "studies show," and "scientific evidence suggests," most of us automatically give more credence to what follows than we might otherwise. After all, isn't science the way to knowledge, the way we learn about our world, and often, the way we tame it and control it? Even though we may not entirely understand what's involved in a study, once tests are run and numbers are assigned to an otherwise intangible phenomenon, we automatically feel that we have captured the essence of the matter, the nuts and bolts of the issue.  This is just as true for studies dealing with food, ingredients and health as it is for any other area of investigation - perhaps even more so, given the fundamental nature of our relationship to food. But are all studies created and carried out equally well? No. Some are better-designed and more credible than others. What then, makes a study "good" or "bad"? The method to the science   All good studies are, in principle, based on the scientific method, which consists of four basic steps - observation, hypothesis, prediction and testing. These steps are vital to a good study because they help minimize bias.  The process begins with an observation, around which a hypothesis, or premise, is formed to try to explain what was observed. To either confirm or refute a hypothesis, a prediction is made based on deductive reasoning. If the hypothesis is true, then a certain outcome should be expected from testing.   Naturally, as is the case for most specialized areas, the area of testing itself is very complex, and many theories have been developed surrounding effective testing methods. The fundamentals, though, include the following standards: controlling the variables; restricting testing to one variable at a time; repeating the test at least several, if not many more, times; gathering quantitative data that can be analyzed and compared statistically; and considering all data, including figures that fall outside the expected results.  Many times, a hypothesis has to be thrown out because the test evidence doesn't support the original premise. The tricky part comes in recognizing an incorrect hypothesis - think of how difficult it was to let go of the notion that the earth is the center of the universe. A hypothesis is never proven, in fact. Rather, we assume it to be valid unless other evidence suggests that it's incorrect. In this way, we establish scientific theories, and collect a body of evidence to support the theories.  The collective nature of scientific evidence is an important element to keep in mind. In practical terms, this means that the process of peer review is crucial for a study. Submitting the entire study to an impartial, yet knowledgeable, set of peers helps identify flaws in the testing method, and further ensures that bias is eliminated. (Some of the most outrageous news headlines have resulted from studies that sail right past peer review and leap straight into the evening news.)Junk-science alert   Stories centering around food grab attention immediately - at one time or another, most types of foods, including eggs, pasta, fruits, vegetables and meats, have been targeted. So what's a critical thinker to do when coming face to face with study results? Carefully consider the evidence, looking for key elements that indicate whether or not the study is based on the scientific method.  Junk science is "messy" science - it doesn't follow the rules. Badly designed and executed studies are the essence of junk science, which at times can be hard for the observer to identify, because the results "seem" so valid. To address this problem, the Food and Nutrition Science Alliance (FANSA), which represents members of the American Dietetic Association, American Society for Nutritional Sciences, American Society for Clinical Nutrition, and the Institute of Food Technologists, has come up with 10 "red flags" for spotting junk science:1. Recommendations that promise a quick fix.
2. Dire warnings of danger from a single product or regimen.
3. Claims that sound too good to be true.
4. Simplistic conclusions drawn from a complex study.
5. Recommendations based on a single study.
6. Dramatic statements that are refuted by reputable scientific organizations.
7. Lists of "good" and "bad" foods.
8. Recommendations made to help sell a product.
9. Recommendations based on studies published without peer review.
10. Recommendations from studies that ignore differences among individuals or groups.  The goal of these red flags is to prevent over-reaction to the headlines, soundbites and condensed blurbs that often convey the results of studies. It's understandable why we rely on such means; after all, no one has time to study in-depth the methodology of each and every study to verify its trustworthiness. This is why it's helpful to learn how to interpret the results of scientific studies and to become familiar with what to look for when reviewing published data.Interpretation skills   To help with interpretation of scientific studies, the Washington, D.C.-based International Food Information Council (IFIC), has published a booklet entitled "How to Understand and Interpret Food and Health-Related Scientific Studies." This guide, which is available online at www.ificinfo.health.org/brochure/ificrevu.htm, is perfect for those who want to get a good handle on understanding what makes a study good or bad, but who don't want to devote every moment of their spare time to deciphering test results. The guide explains how best to read the introduction, purpose, methodology, discussion and references sections of a study, and lists key questions to keep in mind. For example: "Does the research design fit the stated purpose of the study?"; "Are there any major design flaws in the study?"; "What is the real statistical significance of these results?"; and "Are the conclusions supported by the data?"  The importance of asking these types of questions was illustrated recently when the results of a preliminary laboratory study at Cornell University, Ithaca, NY, were widely over-interpreted. The study, published as scientific correspondence in the May 20, 1999 issue of the British journal Nature, looked at the effect that Bt-corn pollen has on Monarch butterflies. As part of the study, butterfly larvae ate milkweed leaves - their natural food source - that had been dusted with Bt-corn pollen, which, it was theorized, could potentially drift from corn plants to any milkweed plants growing nearby. Bt-corn contains genes from the Bacillus thuringiensis bacterium, which is toxic to the corn borer, a pest that can wreak havoc in corn fields.   The monarch larvae that ate the pollen-dusted leaves grew more slowly and had higher mortality rates than those that ate plain leaves or leaves dusted with regular corn pollen. Does this mean that Bt-corn is dangerous and should be discontinued? No - in this case, the appropriate question to ask is: "What is the real statistical significance of these results?" According to John Losey, Ph.D., assistant professor of entomology at Cornell, and lead investigator of the study, "Our study was conducted in the laboratory, and while it raises an important issue, it would be inappropriate to draw any conclusions about the risk to monarch populations in the field based solely on these initial results."  As a result of the initial reports, "groups opposed to genetic engineering adopted the monarch, perhaps the most recognizable American butterfly, as a symbol of the potential for trouble when science transplants genes from one species to another," according to an article entitled "Monarch Butterfly So Far Not Imperiled" in the November 2, 1999 Chicago Tribune. "Swarms of scientists headed out to cornfields across North America during the summer to see if the work done in the lab made sense in the field," continues the article. Results of these studies appear to indicate, however, that because corn pollen doesn't travel far, and because milkweed just 2 meters away from a Bt-corn field appears to be monarch-safe, that Bt-corn pollen indeed presents minimal risk to monarch butterflies.  What all this means is that the interpretation of a study can be just as important as its methodology. Given its scope and limitations, the Cornell butterfly study was not badly designed or executed. Due to the popularity of the Monarch butterfly, however, and the tendency on the part of certain groups to distrust genetic engineering, the results of the initial study were in fact over-interpreted.Taking action   As important as it is to identify good and bad studies and correctly interpret their results, it's also important to take action to ensure that good science is being carried out in the food industry. One group that is marshaling its resources to make sure that scientific standards are at the forefront of research is the Research-based Dietary Ingredient Association (RDIA), Washington, D.C. This group was established in 1998 to develop and promote baseline scientific standards to ensure the safety of functional food and dietary ingredients.  RDIA's membership is restricted to companies and organizations that sponsor or license clinical research; current members include Chicago-based Monsanto Co. and Minneapolis-based Cargill. According to Robert Hoerr, M.D., president, "Our goal is to create a regulatory environment for functional foods, ingredients and dietary supplements that rewards companies for investing in science."  In the end, it's a given that scientific studies and research are valuable. Studies become even more valuable, however, when they use sound science and valid and repeatable research methods. Maybe science can't provide all the answers, but at least with well-designed studies, the results contribute to a solid body of scientific knowledge that has great bearing on the way we live our lives, run our businesses and plan for the future.Back to top© 1999 by Weeks Publishing Company3400 Dundee Rd. Suite #100
Northbrook, IL 60062
Phone: 847-559-0385
Fax: 847-559-0389
E-mail: [email protected]
Website: www.foodproductdesign.com