Erick Turner, M.D., is a psychiatrist-researcher and transparency advocate who is a member of the team behind OpenTrialsFDA, which aims to make the drug efficacy and safety data in the Food and Drug Administration’s (FDA) Drug Approval Packages much more readily usable. In this interview, he shares some of the background behind this work, as well as his motivation for wanting to open up this type of data.
What is an FDA Drug Approval Package?
FDA Drug Approval Packages consist of the reviews conducted by FDA employees of the “new drug applications” (NDAs) submitted by pharmaceutical companies seeking approval to market their drugs in the US. These NDAs consist primarily of clinical study reports (CSRs). Some drugs have more than one approval package, because approval is granted not for a drug per se, but for a drug-indication combination. If you look in the product labeling (e.g. https://dailymed.nlm.nih.gov), you’ll see the FDA-approved indications (uses) listed — you may see one, or you may see several. For each of these approved indications, there should be one FDA Drug Approval Package.
Why should I care about this specific date when I can find out everything I need to know from the published literature?
Because you can’t! The published literature won’t tell you about (1) clinical trials that weren’t published and (2) trials that underwent outcome switching and were thus “spun” from negative to positive.
Is there more than one type of review? Who authors FDA reviews?
The FDA employees who conduct these reviews hail from various disciplines. There is a medical review authored by someone with a medical background, usually a physician. He or she reviews and summarizes the data on the drug’s efficacy and safety. The statistical review is carried out by — you guessed it — a statistician, who tries to replicate the sponsor’s results using the raw patient-level data submitted electronically. There are other review disciplines, e.g. chemistry, pharmacology-toxicology, but their reviews pertain more to the preclinical phases of drug development.
How does the FDA review process differ from the peer review process used for manuscripts submitted to journals?
The key is the before-versus-after-the-trial aspect. Before the sponsor can begin a trial—at least a trial in the US—it has to submit the trial protocol to the FDA. By means of a “pre-trial” review of the protocol, the FDA learns (1) that that trial is to be conducted and (2) the nitty-gritty methodological details. That way, a couple of years later, when the sponsor submits its NDA, the reviewer compares the clinical study report (CSR) to the original protocol to determine (1) whether any studies have been omitted and (2) whether any outcomes have been switched. By contrast, in the world of peer-reviewed journal articles, there is no “pre-trial” review, allowing both (1) and (2) to take place.
Do you need a “secret handshake” to gain access to FDA reviews?
Not at all, because they are publicly available! They are posted on the [email protected] website, and anyone anywhere in the world can download them. All we are doing with OpenTrialsFDA is making them easier to access and use.
What prompted you to start writing about them?
Back in 2004, in the wake of the lawsuit brought by the State of New York against GSK, there was an increased appreciation that reporting bias was corrupting the reliability of our evidence base. Most of the discussion had to do with registering trials, esp. in ClinicalTrials.gov. Meanwhile, it occurred to me that the FDA had, for decades, been running a semi-public registry and a results database, but it seemed to me that there was very little awareness of it.
That led me to write this essay, the purpose of which was simply to create some awareness of [email protected] as a resource. The essay also provided two examples in which the FDA reviews revealed unfavorable study data that could not be found in a narrative review article or a Cochrane systematic review.
Did you feel that anyone was paying attention at that time?
I felt like I was a voice in the wilderness. For what it’s worth, the essay appeared in only the second issue of PLoS Medicine, which wasn’t yet on the radar screen of many people.
Can you tell us about the first full study you published in which you use of them?
In 2008, my colleagues and I published a paper in the NEJM, which contrasted the efficacy of 12 antidepressants according to the published literature with their efficacy according to the FDA. The interested reader can read the full text at this link, but suffice it to say that the two data sources told a very different story.
Have you looked at other drug groups using FDA reviews?
Yes, we’ve also published papers using a similar approach on antipsychotics and drugs for anxiety disorders. While these papers focus on one drug class or indication, one could also look more broadly across indications. Lisa Bero has done some excellent work using that approach. Two papers are available here and here.
Why did you write the BMJ paper “How to access and process FDA drug approval packages for use in research”?
I wrote this because, despite all the above work, it seemed that a lot of researchers were unaware, or only vaguely aware, of FDA drug approval packages, so I wanted to draw attention once again to them as a resource for research on drug efficacy and safety. And then many researchers found themselves daunted by the user-unfriendliness of navigating (1) to the reviews and (2) within them. I wanted this BMJ paper to act as something of a “cookbook” that other researchers could follow.
Did that paper have the impact you intended?
It’s hard to know how what the impact was, but regardless, this article was just an attempt to cope with a problem that, in my opinion, shouldn’t exist. There should be no reason why researchers shouldn’t have easier access to the wealth of unbiased drug data hiding within FDA drug approval packages.
When did you first consider doing something like OpenTrialsFDA?
Sometime in the last 2000s, I contacted the people at Google about doing something like a Google Books project on FDA reviews. They responded, but didn’t seem interested at that time.
How did the OpenTrialsFDA project come about?
Ben Goldacre and the OpenTrials team at Open Knowledge learned about the Open Science Prize, and they thought it would be a great entry to make FDA drug approval packages much more easily accessible to researchers.
Have there been other researchers who have advocated for wider use of FDA Drug Approval Packages?
Yes — here are several references:
- Bennett DA, Jull A. FDA: untapped source of unpublished trials. The Lancet 2003;361:1402–3. doi:10.1016/S0140-6736(03)13147-9
- Lurie P, Zieve A. Sometimes the silence can be like the thunder: access to pharmaceutical data at the FDA. Law & Contemp Probs 2006;68:85
- O’Connor AB. The need for improved access to FDA reviews. JAMA 2009;302:191–3. doi:10.1001/jama.2009.973
- Cook GE, Madden MM. The Significance of Information Posted on the U.S. Food and Drug Administration’s Website. J Manag Care Pharm 2012;19:72–3
- Marchand HC, Ros BJ, Fine AM, et al. The U.S. Food and Drug Administration: drug information resource for formulary recommendations. J Manag Care Pharm 2012;18:713–8
- Navarro RP. Rediscovering FDA websites. J Manag Care Pharm 2012;18:719–20
- McDonagh MS, Peterson K, Balshem H, et al. US Food and Drug Administration documents can provide unpublished evidence relevant to systematic reviews. J Clin Epidemiol Published Online First: 12 July 2013. doi:10.1016/j.jclinepi.2013.05.006
- Schroll JB, Abdel-Sattar M, Bero L. The Food and Drug Administration reports provided more data but were more difficult to use than the European Medicines Agency reports. J Clin Epidemiol Published Online First: 19 August 2014. doi:10.1016/j.jclinepi.2014.06.019
There are several detailed points that we could not get into here, and other questions will surely arise, so we welcome questions and comments via [email protected]. We plan to follow up with blog pieces on FAQ and/or troubleshooting in the near future through fda.opentrials.net. The OpenTrialsFDA team will present the final prototype in early December at the Open Science Prize Showcase event.