Erick Turner, member of our OpenTrialsFDA team, recently talked to Jeppe Schroll about the importance of including data from regulatory agencies in systematic reviews. Jeppe Schroll has been interested in research methodology for 8 years and wrote his PhD on unpublished data and harms at the Nordic Cochrane Centre in Denmark. He has examined publicly available data from regulatory agencies as well as data obtained through the Freedom of Information act. Currently he works as clinician in obstetrics and gynecology at the Hvidovre Hospital in Denmark. OpenTrialsFDA aims to make the information hidden in the user-unfriendly drug approval packages of the FDA (the US Food and Drug Administration) more easily accessible and searchable, and link these to documents and data related to clinical trials.
You and Lisa Bero co-wrote an editorial entitled Regulatory agencies hold the key to improving Cochrane reviews of drugs [1]. Cochrane reviews are generally considered to be the gold standard in systematic reviews, but are you saying there is room for improvement? What has been learned from regulatory data that could not be learned from a rigorous literature search?
A recent systematic review by Golder [2] found that 43% to 100% of adverse events are missed if you only consider published documents. At best Cochrane reviews are only summarizing 57% of the available data on harms and their results and conclusions therefore might be misleading. If reviewers assume that there are no adverse events when none are reported they will exaggerate their confidence in “no harms”. Even though publicly available regulatory data is not always, complete it provides lots of data on harms, comprehensive list of trials conducted at the time of approval, considerations and worries among regulators and sometimes rejected indications and additional analysis.
In the study in which you surveyed Cochrane reviewers, did you find that they commonly made use of data from regulatory agencies?
Among the Cochrane authors that got access to unpublished data only 3% got data from regulatory agencies. Considering that it is freely available it was surprisingly low.
Why do you think that number is so low? I have heard systematic reviewers, when referring to regulatory data, use the phrase “low yield”. Why do you think they might say that?
There is a conception that getting data from regulatory agencies is “not worth the effort”. In some situations the published articles might be a fair representation of the conducted trials but in other situations it might not be. Summary reports are freely available at the FDA and EMA (European Medicines Agency) websites and are easier to obtain than published articles hidden behind paywalls. The reports are not as structured as published articles but they often summarize many trials. At the EMA there is the possibility through the Freedom of Information act to apply for clinical study reports, which are more detailed reports. Another reason for the low percentage of reviewers that use the data is that the Cochrane Handbook has been vague on guidance concerning regulatory data.
You wrote a paper in which you compared reports from both the FDA and the EMA. [3] How would you compare and contrast them?
The EMA prepares their reports for the public whereas the FDA reports are internal reports. The EMA reports are therefore easier to read but the FDA reports are more comprehensive, especially about harms. The FDA also publishes their communication with the sponsor which the EMA does not. The EMA, however, publishes rejected indications as well as the reason why. Lastly, the EMA are more accommodating towards access to clinical study reports through the Freedom of Information act.
What do you think about the usability of FDA review documents?
FDA documents contain lots of important information. Unfortunately they often contain different documents lumped together without a table of contents or clear separation. The documents are often scanned from paper, which makes them unsearchable, and they are often structured differently.
As you know, OpenTrialsFDA is aimed at making FDA data much more usable. Do you believe that, if this is done well, it will lead to a change in the habits of researchers doing systematic reviewers?
It should. The FDA reports are essentially a systematic review of all the available trials at the stage of approval. They contain much more information about harms than articles in medical journals. When there are discrepancies between published papers and FDA reports the researchers will have to use additional time to resolve those.
And if that happens, what impact, if any, do you see that having on the public health?
The conclusion of the FDA reports, in my opinion, seems to contain less “spin”. When reading FDA reports one gets the impression that the balance between benefit and harms are often delicate. Many redundant biased systematic reviews are produced [4] and regulatory reports could help limiting that problem.
References:
- Schroll JB, Bero L. Regulatory agencies hold the key to improving Cochrane reviews of drugs. Cochrane Database Syst Rev 2015;4:ED000098. doi:10.1002/14651858.ED000098
- Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLoS Med. 2016 Sep 20;13(9):e1002127. doi: 10.1371/journal.pmed.1002127
- Schroll JB, Abdel-Sattar M, Bero L. The Food and Drug Administration reports provided more data but were more difficult to use than the European Medicines Agency reports. J Clin Epidemiol Published Online First: 19 August 2014. doi:10.1016/j.jclinepi.2014.06.019
- Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016 Sep;94(3):485-514
We will be sharing future progress on OpenTrialsFDA through this blog and hrough https://fda.opentrials.net as the work develops: the final prototype will be presented on 1 December at the BD2K Open Data Science Symposium, as one of the six finalists of the Open Science Prize. We welcome any questions and comments via [email protected].