Much has been written over the past two months about the New York Attorney General’s investigation against four retailers of mainstream herbal supplements. In the weeks since that announcement, a host of scientists have criticized the application of DNA barcode testing to finished herbal products, especially extracts, because the extraction process separates the beneficial phytochemicals in the botanical from the other plant matter including the DNA. 

Unlike the raw materials, the finished product would not be expected to contain DNA. Even some of the supplement industry’s most vocal critics have publicly acknowledged that the Attorney General’s results defy credibility that so many mainstream products would not contain any trace of the herbs. Contrary to the Attorney General’s assertions, follow-up testing by the retailers confirmed the products actually do contain the extracts from the plants on the label.  

However, with all the attention focused on the DNA tests, we may have overlooked a more fundamental flaw with the test method: the internal inconsistency of the test results that make the very method scientifically unreliable and invalid. I’m calling it “The Rule of Five." 

In the cease-and-desist letters to the four retailers, the Attorney General briefly details that it selected six popular herbal supplements from each of the retailers and then purchased those products from either three or four locations around New York. That provided the AG with either 18 or 24 samples from each retailer for its investigation. Then the Attorney General stated that each sample was “genetically tested five times." That’s important. So, for instance, the GNC ginseng product was purchased in four stores and tested five times, producing 20 test results; the Target brand ginkgo biloba was purchased in three stores and produced 15 results. 

So, if DNA testing that was performed on these products was reliably producing accurate outcomes, one would expect the results to come back in multiples of five. Remember your sixth-grade multiplication tables: 5, 10, 15, 20, etc.? If, however, a particular sample is producing some positive and some negative results within the same sample (resulting in anything other than outcomes in multiples of five), it raises substantial doubts as to how reproducible—and how reliable—the test methodology might be.  

Here are just a few of the examples of the seemingly random results the AG disclosed in its letters to the retailers: 

•For the GNC brand ginkgo biloba, “nine (of 20) of the tests revealed no plant DNA" and four (of 20) allegedly tested positive for rice. 

•For Target’s echinacea product, the AG reported “11 of 15 tests identified echinacea DNA" (meaning four did not) and “three tests (out of 15) located no genetic evidence…" 

•In Walgreens’ case, its ginseng product yielded two positive tests (out of 15) for the presence of allium (garlic) and six (out of 15) for DNA from oryza (rice). 

•The Walmart saw palmetto tests showed three positive outcomes (out of 15) for saw palmetto DNA, three supposedly identifying garlic, and six identifying the presence of oryza.  

If each sample was tested five times, reliable results should all have come back in multiples of five. The only scenario that could produce these kinds of results is where the same test performed on the same sample yielded different results each time. 

Moreover, the Attorney General summarized his investigation in the letter to GNC saying “22 percent of the tests yielded DNA matching the product label" out of 24 GNC products tested. Hmmm… 22 percent of 24 is 5.28 products. But that can’t be possible. I did the math. If five of the 24 products tested consistently yielded DNA, it would have been 21 percent (actually 20.833 percent rounded up); if six out of 24 yielded DNA, the number would be 25 percent. But 22 percent? That can only mean that individual samples were testing both positive and negative for DNA across the five tests. 

Can you imagine submitting a study for peer-review where a test performed five times sometimes produces a positive response and sometimes not, but claiming that the tests are a definitive measure of the identity of the ingredients? Or having an FDA inspector ask to see your own identity testing on your products and producing results that look like this? And yet, these are apparently the results that gave the New York Attorney General’s office enough concern that it demanded immediate recalls of 19 of the 24 products it tested. 

When the same test—any scientific test— performed five times on the sample test material produces different results, a reasonable scientist would acknowledge the methodology is unreliable for the purpose it is being used, regardless of the type of analysis. Study results should be reproducible in the same laboratory on the same materials. 

And yet, without providing the details of the testing methodology or its results to anyone for review, the New York Attorney General has now come to an agreement with at least one of the retailers on a settlement that requires redundant and expensive testing to rationalize its wrong-headed investigation. Perhaps we now know why the New York Attorney General hasn’t provided his tests to the companies he accuses, to FDA who regulates these products and mandates “appropriate" identity testing, to the Congress that he asks to launch hearings, to the other attorneys general he wants to cosign letters, or even to The New York Times that first reported his investigation. He doesn’t want them to see how variable and inconsistent these results actually are. The retailers involved should demand transparency of the test methods and those results before engaging in any agreements. Why “settle" for less? 

For more coverage of the New York attorney general investigation, visit INSIDER’s Supplement Law blog.

Steve Mister is the president and CEO for the Council for Responsible Nutrition (CRN), a trade association for the dietary supplement and functional food industry.