“Nutrition deals with something very close to our hearts but often pretty far from our good judgement,” said LA Times journalist Mary MacVean, shortly after she humorously named 10 fad diet books that she received in the last few weeks. MacVean, who received the Science Media Award from ASN for her work in communicating science, introduced the session “Unscientific Beliefs about Scientific Topics in Nutrition” at Experimental Biology in San Diego on Sunday, chaired by Dr. David Allison. The session took a hard look at bias in the nutrition science field and what we can do to reduce it.
The first speaker was Dr. John Ioannidis, who Dr. Allison noted “has figured out ways to find little holes in the literature.” Ioannidis discussed his study where he and a colleague selected random ingredients from a popular cookbook and searched the literature to see how many were associated with increased or decreased cancer risk. 40 of 50 were associated, but when meta-analyses were performed, this was reduced to 9, and the effect sizes were greatly reduced compared to the median of single studies. Are there really causal ingredients, is there confounding, or is there reporting and other biases?
Financial bias or allegiance bias, which is an adherence to a particular academic theory, are two possibilities. “Creatively exploring” the data to make positive results when they should be negative, non-significant results not being reported, or fraud are possibilities of bias that Ioannidis discussed. With colleagues Patel and Burford, Ioannidis showed that it is quite easy to achieve statistical significance, or not, between various nutrients and mortality based on what statistical adjustments are made. Using a diet-wide association study, bias may also be reduced by subjecting all nutrients to analysis at the same time, instead of individually. However there is still the issue of confounding in nutritional epidemiology, as he demonstrated by showing complex nutrient correlations. To reduce the impact of some of these biases, Ioannidis proposes registration of studies that are planned for the future. Better planning could then be done prior to an analysis.
Dr. Andrew Brown started next by highlighting some of their recent papers that show how myths are maintained in the literature, on obesity myths and the breakfast review. The latter showed that 60% of papers that cite the original paper on breakfast and weight report its results incorrectly. It also matters how abstracts are written: “spin” in abstracts was associated with spin in press releases, and therefore results in media reporting that are not accurate. Brown shifted to to talking about polemics in popular culture like chemical scares (using “dihydrogen monoxide” as an example). According to Google Ngram Viewer that searches books over time for keywords, “bad food” is rising over time. Headlines like “The Toxic Truth About Sugar” contribute shock value but go beyond the evidence. Finally, Brown cited several examples of authors failing to disclose books that they wrote in papers, a potential source of bias that is often ignored.
Books can be a source of revenue, but there are other types of bias, and Dr. Mark Cope discussed these. In particular, he singled out personal, political, academic, ideological, allegiance, promotion, and confirmation bias as non-financial examples. Although financial bias is important, we don’t give enough attention to other types. Others have postulated that other biases cancel each other out on average but financial bias persists, but Cope and Allison showed in 2010 that “white hat bias,” or as they define it, “bias leading to the distortion of information in the service of what may be perceived to be righteous ends,” is apparent in obesity research. Some other ways bias is introduced include unbalanced citations, publication bias, inappropriate inclusion or exclusion criteria, miscommunication in press releases, and selective citation. Cope and Allison published in 2008 an example of publication bias that stemmed from non-industry funded research on breastfeeding.
Finally, Dr. Dennis Bier, the editor-in-chief of AJCN discussed strategies to improve publication reliability. Bier notes that we have a problem with too many papers being statistically significant, citing a paper by Ioannidis and colleagues that looked at epidemiological studies published in a 2 year timespan (389) and found that 89% of them contained a positive result. “We’re just not that good- it doesn’t pass the smell test.” Bier notes that asking a lot of research questions is a sure way to get to a positive results – asking 61 questions gives a 95% chance of a positive study. Bier cautions about common misleading tactics in the literature: that there is much inappropriate implied causality in nutritional epidemiology, there is misdirection (or focusing on positive results and downplaying negative, a “bait and switch” where primary and secondary endpoints have been switched after the study has begun, grouping data together in a certain way to get positive results (e.g. quintiles vs quartiles vs tertiles), and ignoring analyses that don’t work as expected or hoped.
How can we reduce bias? Dr. Bier emphasizes registration, and the AJCN requires registration of clinical studies but he says registration should be required for all study types along with a data analysis plan, and all endpoints should be a priori defined and explicit. The original protocol should also be available, and reporting requirements need to be widely available. All analyses should be reported, and making original data is critical to transparency and reliability. Primary endpoints should go into one manuscript. Results should be reported with additional information: confidence limits, absolute risk when there is relative risk, number needed to treat for benefit (or harm). Bier believes allegiance bias is probably greater than financial bias and are almost never disclosed, and conflicts of interest should be better thought out and perhaps maintained in a centralized system.