Cherry-picked Data can sometimes be used selectively when comparing one result to another
Most farmers would scoff at the notion of replacing their nitrogen fertilizer with maple syrup.
But Manitoba Agriculture soil fertility specialist John Heard was able to make a convincing argument using some creative interpretation of data.
In 2009, Heard conducted a trial comparing the impact of a special “growth enhancer” derived from Acer negundo on canola biomass. The study showed biomass gains where the enhancer was applied were just as high as where 60 and 120 pounds an acre of nitrogen were applied.
One might conclude this product, also known as maple syrup, could replace nitrogen. But the check yielded just as well. Why? Because the plots were high in residual nitrogen.
“Sometimes data can be cherry-picked,” Heard said in an interview.
Every year some questionable products, often originating in the United States, vie for farmers’ money, he said. Some predict that could escalate as the federal government eases out of its current role as efficacy enforcer over the next two years.
In another demonstration in 2002, University of Manitoba soil scientist Don Flaten and Rigas Karamanos, then with Western Co-operative Fertilizers, conducted the “Two Penny” experiment demonstrating how selecting positive results from a couple of sites can be misused.
Two pennies were planted with canola at nine sites across Western Canada. The test was replicated six times at each site. At one, the two-penny plots outyielded the check by 45 per cent. The two pennies had nothing to do with the higher yield. The cause was random variability, which one time out of 20 results in a statistically significant response, Flaten said.
“If we sort and sift our data and show only data from that one trial in 20 we can make nothing look like something,” he said. “Whether it’s two pennies added to a plot or an ounce of maple sugar added per acre we can do these things.
“These are the sleight-of-hand techniques that can be used by someone who might be less than scrupulous or naively optimistic about their product. Farmers just need to be aware of those methods and keep their eyes peeled for that kind of misleading information.”
Flaten said he’s disappointed the federal efficacy requirements are disappearing, but he also recognizes the previous system was difficult to enforce.
It’ll be up to farmers, individually or collectively through their associations, to test new products, he said, especially in the wake of government spending cuts to fertility research and extension.