A new study found why people often think experiments are inappropriate. People could be naturally skeptical of experiments even if they agree to policies, which must be tested through experiments.
The tendency of people to be skeptical of experiments was suggested by researchers at Geisinger Health System and the University of Pennsylvania. Their research showed that people who would approve of policies or treatments could object to experiments, specifically designed to prove safety, effectiveness, and other relevant elements. However, the tendency of a person to object experiments could be warranted, such as in moral concerns and ethical violations. And for some people, learning would be preferred in certain situations. They published their findings in the Proceedings of the National Academy of Sciences.
The Pros and Cons of Randomized Controlled Trials
Scientists require experiments to prove or disprove a function of an object, policy, or treatment. They usually begin with safety assessments and move to efficiency. If safety and efficiency scores are excellent, the object, policy, or treatment may seek regulatory approval for commercialization. This is how products, policies, and medications are evaluated before public release. One way to determine safety and efficiency is via randomized controlled trials or RCTs, sometimes referred to as A/B tests.
According to George Washington University, a private research university in the US, RCT is a study design that assigns participants randomly to a subject or control group. RCTs are considered one of the best ways to either prove or disprove matters being studied. The format features advantages like good chances of washing out population biases, better to blind than observational studies, compatibility with analytical statistical tools, and clear identification of populations of participants.
Unfortunately, RCT is not a perfect study model no matter how robust it may be. Some of its downsides include economically expensive, time-consuming, bias to the volunteers due to unable to represent everyone, and potential loss to follow-up correlated to treatment. As such, the causes of the downsides must be inspected to ensure integrity. Oftentimes investigators have to clarify if randomization is truly defined, not just a number of populations. And when it comes to data, variables have to be correctly traced back to the participants. Uncovering confusing variables can help with the study's credibility.
Opposition Expressed against Experiments Investigated
Ethical and moral reasons are strong forces behind opposition against experiments. If studies cannot guarantee the safety of participants, then people will view them as violators of human rights. Historically, some scientific experiments in the past have violated rights. But today, strict regulations are enforced to protect both studies and individuals, especially young participants. Still, it is curious why many tend to be skeptical about scientific studies.
At Geisinger Health, researchers led an investigation to find the cause of opposition in experiments. They resolved a controversy in two competing hypotheses. First, people unsurprisingly only object to experiments when they disagree with a policy or treatment that the experiment contains. And second, people could paradoxically object to experiments even when they approve of implementing the subjected policy or treatment.
"Our research finds that, despite this rational and evidence-based logic, people tend to object to RCTs and A/B tests even when these same people approve of the two policies that are supposed to be tested. This result, which we call the 'A/B Effect,' occurs when people approve of two policies but object to experiments designed to compare them," wrote Dr. Patrick Heck, first author of the study and research scientist at Geisinger, quoted US magazine Psychology Today.
In the study, researchers analyzed five studies with 1,955 participants to determine the reason for disapproval. These studies were examined for preregistered experiments wherein participants evaluated three options: policy A, policy B, and A/B tests. Across five experiments, participants who objected to A/B tests were greater, compared to those who objected to either policy. About 50% of participants rated A/B tests as less appropriate than their average policy rating, while 37% rates A/B tests less appropriate than either policy A or B. Around 24% of participants strongly objected to AB tests but not objecting either policy.
When the options were ranked, the A/B tests were ranked worst among three with 44% overall grade across all experiments. Around 37% across all experiments, however, graded A/B tests as the best option among three. Researchers found no substantial connection between the ratings and participant demographics.
Hospital Setting Interpretation of the Results
Dr. Heck illustrated their findings. Three hospitals encountered a common problem: potentially life-threatening infections from the insertion of a catheter into a vein. Hospital leaders know the issue and aware of steps that could reduce the infection rate by almost 50%. Though, each hospital used different ways to resolve it.
Hospital 1 decided to give printed safety checklists to doctors that they could attach to their badge. When a catheterization procedure would be done, doctors could lift up and read their badge in case a step was forgotten. This represented policy A.
Hospital 2 deployed posters in all rooms that hosted the procedure. The posters contained the safety checklist to provide a simple way for doctors to quickly remind of the steps. This represented policy B.
Hospital 3 defined both policies as countermeasures to the issue. But hospital researchers also wanted to know which of the two policies is better. So, they performed an experiment wherein half of patients were randomly assigned to doctors using policy A. While the other half was subjected to policy B. A year later, researchers examined the data and identified which policy led to the lowest death rate.
If correlated to the submitted paper, data showed that only 13% agreed that policy A was inappropriate and 5% agreed that policy B was also inappropriate. But at least 30% found deploying and comparing both policies as inappropriate. Despite the staggering number of disapproving individuals, a significant number remained open in learning via RCTs.
In this pandemic, RCTs could reveal the efficacy of a vaccine or COVID-19 treatment. People might express aversion to these experiments and fuel friction to those who want to collect concrete evidence. But identifying concerns within situations involved in experiments could ease doubts and urge attitudes toward evidence-based research.