As you may know, I have mixed feelings about Precision Nutrition (PN). On the one hand, I'm certified with them (PN1). On the other hand, I'm not a big fan of the way they run their online coaching (which differs from what they tell you in their certification manual). And while they sometimes post useful and informative articles, they sometimes write misleading ones. This post is about the latest example I've seen, and will hopefully help you with evaluating results of studies.
The article to examine is titled "The Precision Nutrition Method now validated in 3 peer-reviewed research studies." (I guess they've abandoned writing proper titles.) The three studies are:
- A personalized, multi-platform nutrition, exercise, and lifestyle coaching program: A pilot in women.
- Commercially available lifestyle modification program: randomized controlled trial addressing heart and bone health in BRCA1/2+ breast cancer survivors after risk-reducing salpingo-oophorectomy.
- Evaluation of a Web-based Weight Loss Intervention in Overweight Cancer Survivors Aged 50 and Younger.
Fun stuff, right? Here, I'm going to look at only the first study since it's applicable to a wide-ranging group.
Whenever someone writes about a study, it's always best to look at the study itself rather than rely on the authors of an article who my be subjectively (or erroneously) interpreting results. In the PN article, there's a link to an abstract of the study but from there you can get the full text. In the "Conflicts of interest" section we get this:
Authors declare financial support for submitted work by Precision Nutrition. Authors MH and JMB declare receiving salary from Precision Nutrition.
So two of the four authors work for PN including John Berardi who is a co-founder. The "Acknowledgements" section notes only that "This work was supported by Precision Nutrition." While these circumstances don't mean the study is invalid, it does mean we should look at the results with more skepticism.
The main thing that bothers me about this study is the participation rate; it's also something that bothers me about PN (and many other mass-market fitness and nutrition programs that tout great results). In this study, 9 men were recruited but only 3 of them completed the study...so the men were excluded from the analysis. Of the 56 women in the study (ignoring those who didn't even make it to baseline), 28 completed the full 12-month program and provided follow-up data. That's 50%. Half of them didn't make it.
What does the study say about this? "Despite the high attrition rate, women who completed 12 months of this program lost between 5 and 10% of their initial body weight..." No, PN, not despite the high attrition rate, ignoring the high attrition rate. Only those who completed the program are considered, everyone else is ignored. Yeah, I have a problem with that.
The results for the "completers" (as the study calls them, vs. the "non-completers") are then presented as averages. It would be nice to see data for all completers. After all, some of these women may have experienced little or no benefit from the program while others who responded well to the format did better.
I don't see any data for the individual completers (am I missing something?) but there is a scatter plot (Figure 4 in the study) showing "A change in (Δ) weight and B Change in (Δ) waist circumference (WC) by individual participant over the 12-month Precision Nutrition Coaching Program." This doesn't give us precise numbers but gives us an indication. It shows that two women gained weight and two lost very little (what looks like 1-3 pounds). That's now 32 of 56 women (57%) who either dropped out or experienced no meaningful weight loss. (If we also consider the men who were excluded from the study, that would be 38 of 65 people or 58%.)
Meanwhile, one woman lost much more than any other (60 pounds while the next biggest weight loss was less than 40) and most are around the 10-15 pound mark.
The study tells us that completers lost an average of 16.52 pounds in 12 months. Let's take a simplified example with 10 women loosely based on the scatter plot from the study using these numbers:
3, 0, -5, -10, -11, -12, -13, -20, -30, -60
An average for these sample numbers would be -15.8 pounds (add them up and divide by 10). Yet 70% of the numbers represent less weight loss than the average. A better number to use is the median (the midpoint of the distribution, in this case it would be -10.5) which helps deal with outliers like the 60-pound loss.
Overall, a bunch of women lost some weight, a few lost quite a lot, and that's great! But most of the women were not success stories. I think that's important for people to know going in, and especially going out—if things didn't go well, they shouldn't feel like "failures" when they're more the norm.
I really don't want to bash PN...they surely do help lots of people. But it's likely that most of the people who join their coaching will not be helped. My own very limited experience with one of their coaching groups suggested an attrition rate much higher than 50% but that's purely anecdotal. (It would be great if PN would release the actual numbers, but that won't happen.)
It's not just PN, of course. Other mass-market diet programs like Weight Watchers® or Nutrisystem® surely deal with the same issue, as do fitness programs like P90X®. That's the way things are. Keep it in mind when you see any studies about this stuff, especially if the study is performed or supported by the company it's analyzing. And remember that if you join one of these programs and don't have success, you didn't fail the program...the program failed you.
Be seeing you.
-gary