The use of hydroxychloroquine to treat COVID-19 has been highly controversial with conflicting studies showing the presence or absence of efficacy against COVID-19. Most of these studies are marred by flaws in their methodology and results which have not been fixed in the rush to quickly publish results. These mistakes are picked up after publication which have generated outrage among different people and led to the removal or retraction of some papers from academic journals. These indicate that peer review, a process that is supposed to pick up and fix flaws in studies before they are published, has not been carefully followed which is negatively affecting the science and medicine of COVID-19.
In this article, I will talk about peer review and how this process has faltered during the COVID-19 pandemic. In particular, I will talk about two papers: one paper has numerous flaws in its study but has not been retracted yet and the other paper had concerns which prompted the authors to retract the paper.
What is peer review?
Peer review describes the process of reviewers, known as peers, evaluating the work of other researchers and offering comments to improve its quality. Peer review is a cornerstone of modern research that screens out low quality work and selects high quality work that should be disseminated widely to other people. Peer-reviewed papers generate trust not only among researchers who are running further studies but also among health professionals, news agencies and the general public that want to know more about a particular topic.
The peer review process is split into three stages:
- Desk evaluation: an editor of the journal receives and reviews the paper. The editor can reject the paper outright if they think it does not meet the journal requirements or the journal’s scope or identifies huge fundamental flaws in the paper. If the paper passes the desk evaluation stage, it goes to:
- Blind review: the editor invites reviewers, researchers who are experts in the field, to review the paper and give comments. Once the reviewers accept the invitation, they review the paper, testing the importance and originality of the findings as well as the validity and rigour of the study. The reviews can be done:
- Single blind: the reviewers receive the names of the authors but the authors do not know who reviewed their paper; or
- Double blind: neither the authors nor the reviewers know the names of each other.
The reviewers then decide whether to reject the paper outright, accept it as it is or send comments to the authors asking them to revise and edit the paper. Their views are sent to the editor who makes a decision based on the reviewers’ comments and relays that decision to the authors.
- Revisions: if the paper is not rejected outright but requires revision, the authors either revise and modify the paper in light of the reviewers’ comments or respond to them, justifying their decisions. The revised paper along with the comments are sent back to the editor and reviewers. Once the reviewers are happy with the revised paper, the editor accepts the paper and it is published.
Peer review improves the quality of research by challenging the ideas and results of other researchers and judging them by their responses. If the research stands up to the reviewers’ criticisms, then the paper represents a rigorous, well-informed study that can be published and widely disseminated to other people. However, if the research falls apart because there are fundamental flaws in the paper, it is rejected, so it is not published.
However, there are some flaws in the peer review process:
- Bias can be introduced in some parts of peer review. In particular, if the editor and/or reviewers know the authors’ and institutions’ names (as in a single blind peer review), the paper may be passed due to the prestige of the authors and institutions and not because of the contents of the paper itself.
- Reviewers may not spot all the errors in a paper and may not call out research misconduct because peer review assumes an implicit trust that the authors have not made up the data and findings.
- Traditionally, the reviewers’ comments are not published along with the paper. Therefore, readers do not know the exact details of the peer review process, particularly what the reviewers were thinking and whether the authors have sufficiently responded to their comments.
- Reviewing papers is an essential but thankless job for researchers with no remuneration or recognition. Researchers can spend at least 7-8 hours reviewing and making comments to each paper. This takes time away from their research and experiments which is a premium in the COVID-19 pandemic.
Nevertheless, peer review is still widely supported by researchers. In a 2015 survey on peer review, 82% researchers agreed that …without peer review there is no control in scientific communication… while 74% researchers agreed that peer review improves the quality of the published paper. However, researchers admitted that peer review is less able to detect fraud and plagiarism. These indicate gaps in research integrity that are only made worse by the rush to quickly publish results during the COVID-19 pandemic.
The paper with numerous flaws, and why it should be retracted
In mid-March, French researchers published a paper in the International Journal of Antimicrobial Agents where they tested the efficacy of hydroxychloroquine in treating hospitalised COVID-19 patients in Marseille. They purportedly found that patients who received hydroxychloroquine cleared the virus more quickly than control patients. This was accelerated when azithromycin, an antibiotic, was also given to patients.
This study has widespread implications with Trump endorsing hydroxychloroquine as a COVID-19 treatment and many countries, including Australia, stockpiling hydroxychloroquine. However, many scientists and medical professionals have discredited the study due to its many flaws. I describe some of the flaws in a separate article. In summary, most of the problems relate to how the researchers selected patients in the study and how the control and treatment groups were treated and measured very differently. These flaws are so extreme that not only should people not trust the results of the study but it should also be retracted.
This paper was only published due to the journal’s flawed peer review process. First, the journal operates a single blind peer review process which is conducted by an editor and at least one reviewer. It is possible that they may have accepted the paper outright due to the authors’ prestige and affiliation without critically reviewing its flawed contents. The peer review process was also done too quickly. The paper was submitted on 16 March, accepted within 1 day on 17 March and published online on 20 March. This makes it impossible for a critical peer review to be conducted to identify the many flaws in the paper which would lead to its rejection. Lastly, the paper presents a huge conflict of interest that was not declared despite one of its authors being the journal’s editor-in-chief and another being an editor of the journal. This not only raises the possibility that they may have influenced the paper’s peer review but it also makes it difficult to retract the paper due to their competing interests in the paper and journal.
In April, Elsevier and the International Society of Antimicrobial Chemotherapy (ISAC) that run the journal published two statements. In these statements, they announced their concerns about the scientific integrity of the study and ordered an independent peer review of the paper. Since then, Elsevier responded in an email I sent that they are still reviewing the paper. Given the continuing impacts of this flawed study as shown by the paper being the most downloaded in the journal, the peer review process should be expedited with the intention of retracting the paper so that it does not further impact the decisions of medical professionals in how they treat COVID-19 patients.
Looking at the paper that has been retracted
In late May, American researchers published an observational study in The Lancet where they collected clinical data of hospitalised COVID-19 patients from a database. While most of them acted as a negative control, some patients received chloroquine or hydroxychloroquine with or without an antibiotic to test their efficacy against COVID-19. They purportedly found that COVID-19 patients who took chloroquine or hydroxychloroquine were not only more likely to die but to also have cardiac arrhythmias. These are serious heart conditions caused by rapid or irregular heartbeats that can lead to cardiac arrest and death.
This study has seriously impacted the running of clinical trials on hydroxychloroquine such as WHO having to suspend the hydroxychloroquine arm of the Solidarity trial to review clinical and safety data. However, a lot of researchers expressed concerns on the study, particularly on:
- Data integrity with errors in the raw data. For example, one Asian hospital was misattributed as an Australian hospital which inflated the number of cases and deaths in Australia;
- Data analysis as it was insufficiently adjusted to account for confounding variables such as disease severity and side effects;
- The absence of an ethics process such as informed consent to access patient data from a database (which is quite involved and entails many factors);
- Implausible doses and rates of use of chloroquine and hydroxychloroquine globally such as chloroquine not being readily available in Australia; and
- How trustworthy and open Surgisphere, the data providers, are in providing raw data to other users to verify the data integrity and results.
Some of the study authors later launched an independent peer review of Surgisphere. However, as the peer reviewers could not access the full dataset to test the findings themselves, the authors were not confident on the integrity of the data and requested that the paper be retracted. This raises serious questions on how the paper was peer reviewed in the first place.
The peer review process of The Lancet is unclear as it does not specify whether the peer review is single or double blinded. It is possible that the paper was passed due to the primary author working in Harvard which may have lowered the reviewers’ scepticism of the paper. The timeline of the peer review process is also unknown other than that the whole process would have taken four weeks at most and that the paper was published online on 22 May 2020. Hence, we do not know how quickly the paper was peer reviewed. Lastly, the peer review process in The Lancet would have been conducted by one or more editors and at least three reviewers. However, it is unclear how these reviewers did not spot the flaws in the study that were identified after publication.
The authors took some responsibility for not being careful with their studies which have led to the paper’s retraction. However, this example also exposes the flaws of peer review, particularly in the lack of transparency of the process and the possible bias towards the authors’ prestige.
Conclusion
The research world is rushing to test treatments against COVID-19, but this pursuit cannot be sped up at the expense of research integrity. Given the speed in which legitimate and flawed papers are published and widely disseminated to other people and the criticisms that are widely publicised, conducting peer review after the paper is published is not the solution. Instead, the paper must be peer reviewed behind closed doors before it is published so that errors and flaws are picked up and fixed right away before they become widely known. After the paper is published, the reviewers’ comments should be published so that everyone, from the general public to seasoned researchers, can see what reviewers thought of the paper and how the authors responded to their comments. If reviewers choose to name themselves, they can also be recognised for reviewing and improving on the work of other researchers. Many journals are taking the initiative of making peer review more open and transparent to other people.
Ultimately, peer review is not broken, but it is not foolproof either. Improvements can be made to the process itself to reject any dubious studies and to fix flaws in the study before they are widely known. In a world where information and misinformation on COVID-19 spreads rapidly in many channels, it is important that peer review is done correctly to maintain the integrity of COVID-19 research. This will provide a credible source of information that will help people respond appropriately to COVID-19.