Don’t believe (almost) anything you read

Human beings really hate uncertainty in a way that impacts our lives in many profound ways. We will predictably choose to stay in a bad situation rather than make a choice of uncertain outcome. How many people have stayed in bad relationships, bad jobs, or bad living environments even going out on their own may make things better but not for certain. It is for this reason that a pandemic such as this is so maddening. This is a new virus that we do not know much about but the situation is so severe that we are desperate for tiny snippets of hope. People understandably expect the information from the scientific authorities to be clear, accurate, and unchanging. Unfortunately the current reality is not conducive to information that is always clear, accurate, or unchanging.

The goal of scientists is to design experiments that control variables and minimize biases to get at the truth. In stark contrast, the news media sadly has a different incentive. These large companies are in the business of getting you to click, watch, or listen as much as possible to improve cash flows and balance sheets. They are conditioned to sacrifice accuracy for profitability.

The problem, however, is that they report the information as if it is clear, accurate, and reliable when this is not always the case. Anyone who has ever read a scientific publication knows that it is mandatory to report the flaws and drawbacks of that particular study. Further, life is complex as is science. A single journal article can be analyzed for hours to fully understand in its totality. This is not information that can always reliable be stuffed into a 2 minute piece on the evening news or 1500 words on the front page.

the goal of scientists is to design experiments that control variables and minimize biases to get at the truth

To make things even more difficult, anyone with even a tersory understanding of any scientific discipline knows that the big picture is always more complicated than any one publication can encapsulate. Usually this is compensated for by multiple other studies under similar or slightly varied conditions replicating the same results. Science marches slowly through a process of adding more pieces to a puzzle until the picture comes into focus. It is not typically punctuated by huge “breakthroughs” or “game changers”. Silver bullets are rare. This is not something that is in line with what the public wants to hear so it is not what the news will report.

This creates a problem that is manifest in the following common paradigm:

Scientists publish a study with specific findings. Those findings are not completely certain due to avoidable or unavoidable research design flaws, inherent bias that could not be completely overcome, and problems of generalizing the results to situations outside of the study design.
The news media or social media sees the published report and latches on to the conclusions reporting them to the public in a way that grabs attention. They do not adequately report the flaws or uncertainties of the results. The information goes viral because it is made to sound so exciting.
Over time more studies are done which often contradict the first study. This is typically not surprising to anyone who read and understood the first study fully.
The news media then reports that new science has contradicted its earlier reports. This is not entirely true since uncertainty was built into the initial report.
The public gets confused and trust in science and public health institutions is compromised.
This leads to greater spread of disease and loss of life.

A great example of this was with hydroxychloroquine. Remember this one? I know, it’s so April. This antimalarial drug was touted as a “game changer” by so many on social media and even the President. The reality was that the benefits were seen only in small preliminary studies with major design flaws. The flaws with the study were acknowledged in the publication itself but these flaws were not emphasized when reported. We all know what happened next. The antiviral went viral. People went bonkers for the drug and it turned into a political issue. Why are doctors denying us access to this game changer drug? It’s a conspiracy! The deep state strikes again!

Then time passes and more studies come out showing no benefit and even a risk of harm with the drug. The news media reports this “reversal”. It then feels to the public that the scientific establishment cannot be trusted. What in fact happened was the initial reports, while promising, were very uncertain and even suspect. It was crazy to think hydroxychloroquine was a “game changer” based on the data that was available at the time. I know no respected physician who felt that it was. That’s not to say that it wasn’t prescribed. Many physicians felt that the risk was worth taking in light of the potential benefit. We are forced to do this regularly in medicine. But everyone knew all along that it was a risk and that there was significant uncertainty.

Another important example that you need to be aware of is the use of preprinted articles. In an a long ago era (i.e. way back in February), issues with research studies would have been addressed during the peer review process before anyone in the press would get word of the results. Peer review is when scientific work is assessed by other researchers before being published. During this process flaws in design and data evaluation can be addressed and corrected before publication. The publication can also be held if it the results are unreliable. Respectable news outlets would on report on studies after they were published.

The frenzy around COVID-19 changed everything. Studies are uploaded to the internet as a preprint—e.g. a paper that hasn’t yet run the peer-review gauntlet (the main sites are medRxiv.org or bioRxiv.org). Preprints allow scientists to share data quickly and can be useful since speed is vital in a pandemic. The problem is that preprints can allow questionable work to enter the public discourse directly. If they have faulty conclusions that are publicized in the news or on social media this could lead to significant public misinformation and the potential for great harm.

preprints can allow questionable work to enter the public discourse directly.

These preprinted articles are being publicized with alarming frequency. As the pandemic drags on, scientific journals have been flooded with very poor quality research that in other times would not have be submitted. Some even come from people with no training or understanding who have cobbled together unreliable mathematical models; others come from actual virologists who have suddenly pivoted to studying coronaviruses and are submitting work they never normally would in a rush to be first. These studies are making it on the preprint sites and then distributed to the public without proper vetting.

The truth is tricky. The coronavirus does not care how sick we are of isolation or how bad the economy is doing. It is a new virus and doing research on new viruses takes time. Rushed science is usually bad science. I think the best course of action is to do the following:

Take everything you read or see in the popular media with a grain of salt. Assume that it is oversold.
Be wary of any report that quotes a preprinted article (e.g. from medRxiv.org or bioRxiv.org).
Never assume the results of a single scientific study are conclusive of anything.
Never be surprised when later research contradicts and supersedes earlier research.
Don’t expect the scientific process to be perfect (to learn more about how expectations can lead to unhappiness read the post just before this one).

The good news is that the accumulation of data for this virus has been astonishingly fast and we will likely have clearer answers sooner than for any other contagious disease in history. It just takes patience and a bit of luck.