The Inspector Generals Report: Fun with Numbers

Its all in how you spin the numbers. No, Im not referring to the recent presidential race; Im talking about the portrayal of the dietary supplement industry by the department that oversees FDA. In October, the Office of the Inspector General (OIG) of the Department of Health & Human Services (HHS) issued a report titled Dietary Supplements: Structure/Function Claims Fail To Meet Federal Requirements." That report generated negative headlines in the consumer press that dietary supplement claims are false," misleading" and dangerous." The report also provided new fodder for the industrys critics to allege once again that these products are not properly regulated.

The recommendations OIG made using the findings of its report are even more alarming. It recommended FDA seek legislation requiring approval for all structure/function claims in the labeling of dietary supplements." In other words, FDA should ask Congress for pre-market approval of structure/function claims on supplements. This is a notion the industry should never stand for; it is exactly what the Dietary Supplement Health and Education Act of 1994 (DSHEA) was intended to guard againstto prevent FDA from reviewing the claims for supplements using drug-like standards. Isnt that why supplement labels must carry the disclaimer that These statements have not been evaluated by FDA"?

Nevertheless, there is no denying three troubling statistics emerged from the report:

But lets dig into the facts of the report because, like most things, the numbers that made the headlines dont tell the whole story.

The OIG pre-selected only two categories of products for review: those making claims for weight loss (60) and immune function (67). Of the 127 products selected, about half of them (60) came from Internet purchases and the rest (67) from retail stores. On both counts, these are hardly a representative sample of the industry. The Nutrition Business Journal estimated weight loss products are only 14 percent of all supplements, and Internet sales, although growing rapidly, only account for about 5 percent of total supplement retail sales.

The report explains OIG contacted those respective companies and asked them to submit their substantiation on a completely voluntary basis to the OIG for evaluation. Not surprisingly, 39 percent of companies that were asked declined to participate. What good could possibly come of such an examination? But companies for 72 of the products did submit substantiation in response to the request.

For those 72 products, the OIG received 1,624 documents the companies believed were responsive to the request for substantiation of their claims. OIG acknowledged the submitted evidence ranged from one document to as many as 137 documents for a single product, with the median being seven substantiation documents per product. That doesnt sound so bad to me. But then, the OIG uses 1,624" as the denominator for its statistics. Perhaps companies would have been better off to only send their best information, and not their entire files.

For example, the OIG lamented that only 34 percent (557 of the 1,624 documents) were studies conducted on humans, and correctly noted human data is favored as the basis for a structure/function claim. But 557 documents: thats still an average of more than seven human studies per product. Like so many other data points that would be helpful, we cant determine specifics because OIG didnt release the names of companies or products, or even how many studies were actually submitted per product.

Next, OIG complained that of the 557 human studies, 85 percent were not randomized, double blind, parallel-group, placebo-controlled trialsor RCTs," the gold standard of science. But here the denominator is 557, so 15 percent still translates into 83 RCTs submitted to support claims for 72 products. By my count, that is still more than one RCT per product on average. What if the headline had been Industry Submits More than One RCT for Each Product Examined"?

The report also criticized that nine products (12.5 percent) did not submit documentation that would qualify as substantiation under the FDA guidance. I wont make excuses for those nine (FDA, a little enforcement is appropriate, dont you think?); but, that also means 87.5 percent did. Further, OIG indicated only 2 percent of the documents submitted studied the actual product in the sample (but remember, the denominator is still 557, not 72). In other words, OIG received 12 studies on the actual products out of the 72 it purchased. Note that FDAs substantiation guidance suggests, but does not require, companies conduct studies on their products. For instance, a general study on the effects of vitamin D will suffice for a host of combination products that include the nutrient, unless the manufacturer has a biologically plausible reason why vitamin D might act differently in combination with those particular ingredients. The headline could have read, Nearly 17 Percent of Supplement Manufacturers Study Their Products, Not Just the IngredientsEven Though FDA Doesnt Require It."

Perhaps one of the more disturbing conclusions is the allegation, none [of the studies submitted to OIG] met all of FDAs recommendations for competent and reliable evidence." Again, this doesnt tell the whole story. One of the four criteria for substantiation used by the OIG is that marketers evaluate their claims in light of the totality of the evidencebut the OIG only asked for documents that substantiate your claims." Is it surprising most companies would not send in the null" studies for such a request? On the one hand, the OIG chastised manufacturers because 20 of the 557 human studies submitted (4 percent) had results that failed to demonstrate the same benefit claimed for the product, but it also criticizes the responding manufacturers that did not submit more studies that were negative or null, thus failing to consider the totality of the evidence." This seems like a setup to me.

Both industry and FDA have some homework assignments from this report before anyone rushes out to give FDA more authority. In the meantime, this OIG report amply demonstrated why the industry should keep doing its own math.

Steve Mister is president and CEO at the Council for Responsible Nutrition (CRN).