There’s a hierarchy of evidence that the medical profession accepts.  A randomized controlled trial is better than an observational study, which is better than a case series, which is better than a single report, and so on.  But what are randomized controlled trials good for?  Good randomized controlled trials have admirable goals.  They control for lots of different variables. The best ones – double-blinded studies -don’t even let the people who are doing the experiment know if they are providing an active treatment or a placebo.

Which brings us to the behavioral health literature.  So often treatment is denied because there is a lack of randomized controlled trial evidence.  Is this the right approach?

Let’s think about the literature related to alcohol consumption.  Observational studies have shown that a little less than a drink a day is associated with positive health outcomes.  Yet when randomized controlled trials have been conducted these benefits magically disappear. How can we reconcile these disparate results?

Think about someone who drinks a total of four drinks spread out over the course of the week. That’s not a person who is drinking to get drunk.  That’s not someone who is drinking alone.  I would wager that most people who have that pattern of drinking are having that drink sitting at a table, over a meal with family or friends.  Why don’t the randomized controlled trials show a benefit for a glass of wine a day?  Because it’s probably a proxy for the context in which that small amount of alcohol is consumed.  

Similarly, the literature on salt is all over the place – it doesn’t matter at all or it’s slowly killing us.  It’s the way it’s consumed that probably means the most.  Two people might consume the same amount of salt a day, but which kind of consumption is likely to be problematic? Eating fast food on the run for breakfast and lunch and a frozen meal for dinner – highly processed foods that often have more than the daily recommended amount of salt in a single serving – or preparing a simple pasta and sauce and adding salt from a shaker at the table.

A good randomized controlled trial might look at the milligrams of sodium or number of alcoholic drinks per week, but even the best multivariate analysis likely would not capture the context.

Randomized controlled trials like things clean, and that means that their results can have a pre-determined quality.  They assume we engage in behaviors that have consequences, but that we aren’t aware of the possible impacts of what we do. Life is messy, and the context of our behaviors matter.

Schooner Strategies’ new outcomes consulting service helps define and improve the assessment of evidence, including the statistical significance between one or more independent variables and the dependent variable(s) which is being measured (such as the health impact of alcohol consumption or salt intake). 

 The Schooner team helps identify trustworthy evidence by analyzing whether or not a particular healthcare intervention is making a real difference for the targeted population.  We do this in part by evaluating the quality of the evidence and how the findings will be applied in the real world.  Sometimes a randomized controlled trial analysis is not enough by itself.  Contact us if you want to learn more about our approach to comparative effectiveness.