s the new year begins, millions of people are vowing to shape up their eating habits. This usually involves dividing foods into moralistic categories: good/bad, healthy/unhealthy, nutritious/indulgent, slimming/fattening — but which foods belong where depends on whom you ask.
The U.S. Dietary Guidelines Advisory Committee recently released its latest guidelines, which define a healthy diet as one that emphasizes vegetables, fruits, whole grains, low- or nonfat dairy products, seafood, legumes and nuts while reducing red and processed meat, refined grains, and sugary foods and beverages.1 Some cardiologists recommend a Mediterranean diet rich in olive oil, the American Diabetes Association gives the nod to both low-carbohydrate and low-fat diets, and the Physicians Committee for Responsible Medicine promotes a vegetarian diet. Ask a hard-bodied CrossFit aficionado, and she may champion a “Paleo” diet based on foods our Paleolithic ancestors (supposedly) ate. My colleague Walt Hickey swears by the keto diet.
Who’s right? It’s hard to say. When it comes to nutrition, everyone has an opinion. What no one has is an airtight case. The problem begins with a lack of consensus on what makes a diet healthy. Is the aim to make you slender? To build muscles? To keep your bones strong? Or to prevent heart attacks or cancer or keep dementia at bay? Whatever you’re worried about, there’s no shortage of diets or foods purported to help you. Linking dietary habits and individual foods to health factors is easy — ridiculously so — as you’ll soon see from the little experiment we conducted.Our foray into nutrition science demonstrated that studies examining how foods influence health are inherently fraught. To show you why, we’re going to take you behind the scenes to see how these studies are done. The first thing you need to know is that nutrition researchers are studying an incredibly difficult problem, because, short of locking people in a room and carefully measuring out all their meals, it’s hard to know exactly what people eat. So nearly all nutrition studies rely on measures of food consumption that require people to remember and report what they ate. The most common of these are food diaries, recall surveys and the food frequency questionnaire, or FFQ.
Several versions of the FFQ exist, but they all use a similar technique: Ask people how often they eat particular foods and what serving size they usually consume. But it’s not always easy to remember everything you ate, even what you ate yesterday. People are prone to underreport what they consume, and they may not fess up to eating certain foods or may miscalculate their serving sizes.
“The bottom line here is that doing dietary assessment is difficult,” said Torin Block, CEO of NutritionQuest, a company that conducts FFQs and was founded by his mother, Gladys Block, a pioneer in the field who began developing food frequency questionnaires at the National Cancer Institute. “You can’t get away from it — there’s error involved.” Still, there’s a pecking order in terms of completeness, he said. Food diaries rank high and so do 24-hour food recalls, in which an administrator sits the subject down for a guided interview to catalog everything eaten in the past 24 hours. But, Block said, “you really need to do multiple administrations to get an assessment of someone’s usual long-term dietary intake.” For study purposes, researchers are not usually interested just in what people ate yesterday or the day before, but in what they eat regularly. Studies that use 24-hour recalls tend to under- or overestimate nutrients people don’t eat every day, since they record only a small and perhaps unrepresentative snapshot.
When I tried keeping a seven-day food diary, I discovered how right Block was — it’s surprisingly difficult to capture a record that reflects normal eating patterns when you collect only a few days’ worth of data. It so happened that I was traveling to a conference during my diary week, so I ate packaged snacks and restaurant meals far different from the foods I usually eat from my garden at home. My diary showed that before dinner one day, I’d eaten only a doughnut and two snack packs of potato chips. And what did I have for dinner? I can tell you that it was a delicious Indonesian seafood curry, but I couldn’t possibly begin to list all its ingredients.
Another lesson from my short stint keeping a food diary is that the sheer act of keeping track can change what you eat. When I knew I had to write it down, I paid far greater attention to how much I ate, and that sometimes meant that I opted not to eat something because I felt too lazy to write it down or else realized, nah, I didn’t really want a second doughnut (or else didn’t want to admit to eating it).
It’s not easy to circumvent the human instinct to fib about what we eat, but the FFQ aims to overcome the unrepresentativeness of short-term food records by assessing what people consume over a longer period. When you read a headline saying something like “blueberries prevent memory loss,” the evidence usually comes from some version of the FFQ. The questionnaire typically asks about what the survey-taker ate during the last three, six or 12 months.
In order to get a sense of how these surveys work and how reliable they might be, we hired Block to administer his company’s six-month FFQ to me, my colleagues Anna Barry-Jester and Walt Hickey, and a group of reader volunteers.2
Some questions — how often do you drink coffee? — were straightforward. Others confounded us. Take tomatoes. How often do I eat those in a six-month period? In September, when my garden is overflowing with them, I eat cherry tomatoes like a child devours candy. I might also eat two or three big purple Cherokees drizzled with balsamic and olive oil per day. But I can go November until July without eating a single fresh tomato. So how do I answer the question?
Questions about serving sizes perplexed us all. In some cases, the survey provided weird but helpful guides — for example, it depicted what a half-cup, one cup or two cups of yogurt looked like with photographs of bowls filled with various amounts of wood chips. Other questions seemed absurd. “Who on this planet knows what a cup of salmon or two cups of ribs looks like?” Walt asked.
Although the questionnaire was meant simply to measure our food intake, at times it felt judgmental — did we take our milk full fat, low fat or fat free? I noticed that when I was offered three choices of serving sizes, my inclination was to pick the middle one, regardless of what my actual portion might be.
Despite these challenges, Anna, Walt and I did our best to answer completely and honestly. Afterward, we compared our results. The questionnaire deemed “cheese, full fat” and some version of alcohol as our top sources of calories.3
From there, our diets diverged. Walt has lost 50 pounds on a ketogenic diet, Anna eats relatively little protein and, according to the FFQ, I devour almost twice the calories as either of them.
Could these results be correct? Anna and I are virtually the same height and weight; we could probably share clothes. How could I eat more than twice the calories she does?4 Block acknowledged that it’s difficult to get an accurate count of calories, especially without a long-term food record, and when you start looking at individual nutrients it gets even trickier. He pointed me to a 1987 study concluding that to estimate a true average calorie count, it takes an average of 27 days of daily intake data for men and 35 days for women. Some nutrients required even longer — 474 days on average to measure vitamin A intake for women, for example. This suggests our reports might be correct, but they might also contain lots of errors.Sure, memory-based measures have limitations, said Brenda Davy, a professor of human nutrition at Virginia Tech, “but most of us in the nutrition world still believe they have value.” Calories are probably the trickiest thing to measure, she said, noting that there’s good evidence that people underreport foods deemed unhealthy, like high-fat foods or sugary snacks. “But that doesn’t mean that everything is underreported. It doesn’t mean that fiber intake or calcium intake is problematic.”
Developers of the surveys recognize that answers are imperfect, and they correct for this with validation studies that check FFQ results against those obtained via other methods, usually a 24-hour food recall or longer food diary. The results of such validation studies, Block said, allow researchers to account for variability in daily intake.
Critics of FFQs, such as Edward Archer, a computational physiologist at the University of Alabama’s Nutrition Obesity Research Center in Birmingham, say that these validations are nothing more than circular reasoning. “You’re taking one type of subjective report and validating it with another form of subjective report,” he said.
Recording what you eat is harder than it might seem, said Tamara Melton, a registered dietitian and spokesperson for the Academy of Nutrition and Dietetics in Atlanta. Among other things, it’s almost impossible to measure ingredients and portion sizes when you dine out. “It’s cumbersome. If you’re out at a business lunch, you can’t whip out your measuring cup.”
When Anna, Walt and I compared the caloric intakes that our FFQs had spit out with the ones that we calculated from our seven-day food diaries,5 they didn’t match up. We ran into trouble estimating portions in the FFQ, too, and who’s to say which was more accurate?