After reaching the final three of the Open Science Prize, and being judged by an expert panel, we can announce that our project, OpenTrialsFDA was placed as a runner-up alongside MyGene2, with Real-Time Evolutionary Tracking for Pathogen Surveillance and Epidemiological Investigation being declared the winner of the grand prize of $230,000. As you may know, OpenTrialsFDA is designed to make clinical trial data from the Food and Drug Administration (FDA) more easily accessible and searchable. Congratulations to all the finalists – it’s been great to see such a range of projects dedicated to innovating and advancing areas of biomedicine and health through open science, content, and data.

Although OpenTrialsFDA didn’t win the grand prize, we were very grateful for all the public votes which took us from the original shortlist of six to the final three. A number of the public also chose to give comments along with their vote – there were many fantastic ones and we wanted to share a selection:

“I made OpenTrialsFDA my first choice because patients have suffered and died because information from research which should be available to health professionals and patients has not been accessible. This project addresses this important problem.”

“I am particularly keen on projects that get data out in there open where it can be subjected to independent analysis.”

“As a professional technologist, I have found the APIs produced by the folks at OpenTrials are of extremely high quality; making integrating with and using the data straightforward yet powerful. If all open science projects had as technically competent and sophisticated people involved then huge opportunities for knowledge growth would open up to us all very quickly.”

“OpenTrialsFDA project will be relevant to a wide range of clinical research fields, with important potential impacts on patients, clinicians, researchers, and health decision makers.”

“It’s great to see a clinical science entry in the Open Science Award. Easier access to the full FDA data for approved indications for approved drugs will assist clinical practitioners, biomedical scientists and educated citizens in making more informed choices on medical and research choices.”

“Improving transparency of clinical data will literally save and transform lives”

“Drugs affect peoples’ lives and all too frequently patients and physicians do not have the data and evidence they need to make truly informed decisions about treatments. Access to data on effectiveness and harms is becoming even more critical with rapidly rising drug prices. Shining a light on what we know and don’t know about treatments will hopefully move us in a direction of making better individual and population-based decisions about treatments.”

Thanks so much for all the kind words – it means a lot to the entire team and we strongly believe in the power of the project to facilitate positive change!

What’s next for OpenTrialsFDA?

The search engine will continue to exist at fda.opentrials.net – it currently indexes over 55,000 FDA approval documents and where possible links to clinical trials on OpenTrials (a few examples). While development on OpenTrialsFDA has stopped for the moment (we’re focusing on the main OpenTrials explorer) there are a number of ways you can help support the project:

• Help get access to important drug information by asking the FDA for specific documents
• Help improve the code that runs OpenTrialsFDA – all the code is in our public repository
  • We need help updating our FDA collector – details here
• Financial support – get in touch if you’d like to talk to us about a grant to continue improving OpenTrialsFDA
• Support in kind – are you working on a similar project and/or have expertise in processing unstructured documents/data (e.g. NLP) to extract structured data? Please get in touch

The OpenTrialsFDA prototype makes clinical trial data from the FDA more easily accessible and searchable through a user-friendly web interface that you can now explore through fda.opentrials.net. Winning the prize would enable us to develop this prototype further in 2017, improving matching documents to corresponding trials on OpenTrials, along with extracting other valuable information from them. In the meantime, you can help us to get more FDA data on clinical trials available by rescuing ‘missing-in-action’ drug approval packages.

How to rescue “MIA” Drug Approval Packages

When OpenTrialsFDA carried out an automated download of all documents posted on Drugs@FDA, this not only captured Drug Approval Packages (DAPs), but also letters written by the FDA notifying drug companies that their drug had been approved for a new indication. Interestingly, this allowed us to identify drug-indication combinations for which DAPs were absent. Nevertheless, those DAPs do exist, and the FDA is obliged to make them available via the Freedom of Information Act (FOIA), for example in response to a FOI request.

In this blog by Dr. Erick Turner you can find out more on which DAPs are missing and how to make a FOIA request to the FDA to get them online. If and when you do make a FOIA request, please let us know by completing this form so that we can keep track of what DAPs have been requested.

Many thanks to everyone for their support during the voting process!

Nevertheless, those DAPs do exist, and the FDA is obliged to make them available via the Freedom of Information Act (FOIA), if not electronically, then in response to a FOI request. In this blog we explain how you can help to get this missing information online.

The OpenTrialsFDA project began with an automated download of all documents posted on Drugs@FDA. The download captured not only DAPs, but also letters written by the FDA notifying drug companies that their drug had been approved for a new indication, allowing us to identify drug-indication combinations for which DAPs were absent. A list of these missing-in-action (MIA) DAPs is shown below and on this page.

Our invitation to you

If you, the reader, are in possession of any of these “MIA” DAPs, we invite you to send them to us so that we can make them available to the research community and others via OpenTrialsFDA.

If there is a DAP you do not have but think it should be shared publicly, please check Drugs@FDA and verify that our automated process did not somehow miss it. You may use the procedure in this BMJ article², though be aware that the FDA recently made slight modifications to the procedure.

Making a FOIA request

Once you have verified that the DAP of interest is indeed not posted on Drugs@FDA, please consider making a FOIA request for it. The usual way to place a FOIA request is via the FDA’s FOI website. We invite you to use this template. Your request should contain the following information (part of which can be found in our list above):

1. Drug brand name
2. Drug generic name
3. Indication approved (consider consulting the product labeling to verify that this is an FDA-approved, not an off-label, indication/use)
4. Date of approval for that indication, aka “Action Date” (often this information can be obtained via Google, but you we suggest you confirm this by looking at that drug’s approval history on Drugs@FDA, where you should find a letter of that date documenting the approval)
5. Types of reviews requested (e.g. medical, statistical)

Just be aware that it may take several months, perhaps more than a year, for the FDA to complete the request. A major reason for this delay is (a) the FDA’s FOIA office is chronically short-staffed (as noted above) and (b) they must go through the DAP and identify and redact any information they deem to be “commercial confidential” or “trade secrets”, as they are exempt from disclosure.

Making a FOIA request for previously released information (possible shortcut)

Before making such a FOIA request, we want to make you aware of a possible shortcut—a request for previously released information. That is, perhaps someone else has recently requested the same information and obtained it, in which case the FOIA office has already completed the time-consuming redaction process, which means there is no need for them to do it all over again. Therefore, in this type of FOIA request, you essentially say to the FDA’s FOIA office, “Whatever you gave to that previous requester, I’d like that, too.”

So now you might ask, how do you know whether the DAP you’re after has, in fact, been requested and released? FOIA case logs! A case log lists the completed FOIA requests, who made the request, the information requested (e.g. drug name, review types), and—most importantly—the request number (“control #”).

And how do you get a case log? You make a FOIA request for it! Even easier, feel free to search the one Dr Turner obtained recently, which covers the period January 2012 – September 2016.

This process might well sound circular, but it can save lots of time, because requests for previously released information often get completed within weeks rather than months or longer.

Tracking your contribution

If and when you do make a FOIA request, please let us know by completing this form so that we can keep track of what DAPs have been requested. Please also keep us informed on what the FDA’s response was (usually some variation of “we’ll get to it and send you the information when it’s ready”).

Please vote!

Finally, if you think OpenTrialsFDA is an important contribution to the public health, and if you’d like to support our effort to pry open this vault of priceless clinical trial data, please help us win the Open Science Prize by casting your vote before 6 January 2017. Thank you!

References

1. Turner EH. A taxpayer-funded clinical trials registry and results database. PLoS Med. 2004;1(3):e60. doi:10.1371/journal.pmed.0010060.
2. Turner EH. How to access and process FDA drug approval packages for use in research. BMJ. 2013;347(oct14 2):f5992-f5992. doi:10.1136/bmj.f5992.

OpenTrialsFDA will be featured in the OHSU (Oregon Health & Science University) Open Science Data Jamboree & Hackathon. The event will focus on the power of open content and data to advance biomedical research and human health.  Dr. Erick Turner, Associate Professor at OHSU and Staff Psychiatrist at the Portland VA Medical Center, will introduce OpenTrialsFDA, which is a finalist for the highly competitive Open Science Prize (voting for the finalists is open until 6 January 2017 here).

OpenTrialsFDA aims to increase access to and the utility of FDA drug approval packages. These review packages often contain information on clinical trials that have never been published.  When these “inconvenient truths” are integrated into a systematic review or meta-analysis, it can sometimes dramatically shift a drug’s risk-benefit ratio. Although the FDA makes the reviews available via its web portal Drugs@FDA, the documents are notoriously difficult to search and aggregate. As a consequence, they are rarely used by clinicians and researchers, despite their immense value.  OpenTrialsFDA intends to change this by allowing researchers, clinicians, the public, developers, and third party platforms to access, search, and build upon this data.

Dr. Turner’s keynote talk will be followed by an overview of the NIH’s new clinical trial data sharing requirements and two concurrent hands-on sessions:

• The first will be a deep dive on searching the OpenTrialsFDA website, understanding the contents of an FDA drug approval package (including the statistical content), and how to use the data for different use cases, such as meta analysis and systematic reviews.  Dr. Turner will team teach the breakout with an OHSU biostatistician.
• The second will be a OpenTrialsFDA hackathon using the OpenTrials API.  Attendees will have the opportunity to use their research, development, clinical, and visualization expertise (to name a few) to explore, present, and build upon the OpenTrials data and the OpenTrialsFDA application.  Please note, no technical expertise is required, as there will be multiple ways of contributing.

The Data Jamboree is presented by Computational Biology and the OHSU Library.  All are welcome to attend, including OHSU faculty, students, and staff as well as Portland area science enthusiasts and developers!

• Date: (to be announced)
• Time 2:30 – 5:00 PM (hackathon will continue past 5:00)
• Location: Collaborative Life Sciences Building (CLSB), 3A003, OHSU, Portland, Oregon
• Food & Drink provided

Contact:  Robin Champieux, Scholarly Communication Librarian, [email protected]

OpenTrialsFDA is a collaboration between Dr. Erick Turner (a psychiatrist-researcher and transparency advocate), Dr. Ben Goldacre (Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford) and the team behind OpenTrials at Open Knowledge International.  

OpenTrialsFDA works on making clinical trial data from the FDA (the US Food and Drug Administration) more easily accessible and searchable. Until now, this information has been hidden in the user-unfriendly Drug Approval Packages that the FDA publishes via its dataportal Drugs@FDA. These are often just images of pages, so you cannot even search for a text phrase in them. OpenTrialsFDA scrapes all the relevant data and documents from the FDA documents, runs Optical Character Recognition across all documents, links this information to other clinical trial data, and now presents it through a new user-friendly web interface at fda.opentrials.net.

Explore the OpenTrialsFDA search interface

Any user can type in a drug name, and see all the places where this drug is mentioned in an FDA document. Users can also access, search and present this information through the application programming interfaces (APIs) the team will produce. In addition, the information has been integrated into the OpenTrials database, so that the FDA reports are linked to reports from other sources, such as ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed.

The prototype will provide the academic research world with important information on clinical trials in general, improving the quality of research, and helping evidence-based treatment decisions to be properly informed. Interestingly, the FDA data is unbiased, compared to reports of clinical trials in academic journals. Dr. Erick Turner explains: “With journal articles everything takes place after a study has finished, but with FDA reviews, there is a protocol that is submitted to the FDA before the study has even started. So the FDA learns first of all that the study is to be done, which means it can’t be hidden later. Secondly it learns all the little details, methodological details about how the study is going to be done and how it is going to be analyzed, and that guards against outcome switching.”

Dr Ben Goldacre says: “These FDA documents are hugely valuable, but at the moment they’re hardly ever used. That’s because – although they’re publicly accessible in the most literal sense of that phrase – they are almost impossible to search, and navigate. We are working to make this data accessible, so that it has the impact it deserves.”

Voting for the Open Science Prize finalists is possible through http://event.capconcorp.com/wp/osp: more information on OpenTrialsFDA is available from fda.opentrials.net/about and from the team’s video below.

Editor’s notes

Dr. Ben Goldacre
Ben is a doctor, academic, writer, and broadcaster, and currently a Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford. His blog is at www.badscience.net and he is @bengoldacre on twitter. Read more here. His academic and policy work is in epidemiology and evidence based medicine, where he works on various problems including variation in care, better uses of routinely collected electronic health data, access to clinical trial data, efficient trial design, and retracted papers. In policy work, he co-authored this influential Cabinet Office paper, advocating for randomised trials in government, and setting out mechanisms to drive this forwards. He is the co-founder of the AllTrials campaign. He engages with policy makers. Alongside this he also works in public engagement, writing and broadcasting for a general audience on problems in evidence based medicine. His books have sold over 600,000 copies.

Dr. Erick Turner
Dr. Erick Turner is a psychiatrist-researcher and transparency advocate. Following a clinical research fellowship at the NIH, he worked for the US Food and Drug Administration (FDA), acting as gatekeeper for new psychotropic drugs seeking to enter the US market. In 2004 he published a paper drawing researchers’ attention to the Drugs@FDA website as a valuable but underutilized source of unbiased clinical trial data. Dissatisfied with the continuing underutilization of Drugs@FDA, he published a paper in the BMJ in order to encourage wider use of this trove of clinical trial data.

Open Knowledge International
https://okfn.org   
Open Knowledge International is a global non-profit organisation focussing on realising open data’s value to society by helping civil society groups access and use data to take action on social problems. Open Knowledge International addresses this in three steps: 1) we show the value of open data for the work of civil society organizations; 2) we provide organisations with the tools and skills to effectively use open data; and 3) we make government information systems responsive to civil society.

Open Science Prize
https://www.openscienceprize.org/res/p/finalists/
The Open Science Prize  is a collaboration between the National Institutes of Health and the Wellcome Trust, with additional funding provided by the Howard Hughes Medical Institute of Chevy Chase, Maryland.  The Open Data Science Symposium will feature discussions with the leaders in big data, open science, and biomedical research while also showcasing the finalists of the Open Data Science Prize, a worldwide competition to harness the innovative power of open data.

General questions

Background

The OpenTrialsFDA team is one of the six finalists for the Open Science Prize: a global science competition to make both the outputs from science and the research process broadly accessible to the public. This is a collaboration between Dr. Erick Turner, Dr. Ben Goldacre and the OpenTrials team at Open Knowledge International. Competing for the $230,000 prize that will be awarded to the winner in February 2017, OpenTrialsFDA is working on a prototype that will make the information hidden in the user-unfriendly drug approval packages of the FDA (the US Food and Drug Administration) more easily accessible and searchable, and link these to documents and data related to clinical trials.

The work is closely related to the OpenTrials project, which aims to locate, match, and share all publicly accessible data and documents, on all trials conducted, on all medicines and other treatments, globally. OpenTrials is aggregating this information from a wide variety of existing sources and aims to provide a comprehensive picture of the data and documents on clinical trials conducted on medicines and other treatments around the world.

What is the scope of OpenTrialsFDA?

OpenTrialsFDA works on making clinical trial data from the FDA (the US Food and Drug Administration) more easily accessible and searchable. Until now, this information has been hidden in the user-unfriendly Drug Approval Packages that the FDA publishes via its dataportal Drugs@FDA. These documents are notoriously difficult to access, aggregate, and search, since the Drugs@FDA website itself is not intuitive to navigate, and the reviews are stored as individual PDFs which, except for relatively new drugs, cannot be searched.  OpenTrialsFDA will extract the relevant data from the FDA documents, link it to other clinical trial data and present it through a new user-friendly web interface. Any user will be able to type in a drug name, and see all the places where this drug is mentioned in an FDA document. Users will also be able to access, search and present this information through the application programming interfaces (APIs) the team will produce.

Who is behind OpenTrialsFDA?

OpenTrialsFDA is a collaboration between Dr. Erick Turner (a psychiatrist-researcher and transparency advocate), Dr. Ben Goldacre (Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford) and the team behind OpenTrials at Open Knowledge International.  

What is the Drugs@FDA portal?

Drugs@FDA is a searchable catalog of the US Food and Drug Administration (FDA) approved drug products, both prescription and over the counter, with links to documents relating to marketing approval. More information is available from their FAQ section.

What are FDA drug approval packages?

FDA Drug Approval Packages consist of FDA employees’ reviews of new drug applications (NDAs) submitted by pharmaceutical companies seeking approval to market their drugs in the US. These NDAs consist primarily of clinical study reports (CSRs). Some drugs have more than one approval package, because approval is granted not for a drug per se, but for each drug-indication combination.

How will OpenTrialsFDA improve access to the FDA data?

The team will scrape the FDA website and extract the relevant information from the PDFs through a process of OCR (optical character recognition). Through the new OpenTrialsFDA interface, users will be able to explore and discover the FDA data. In addition, the information will be integrated into the OpenTrials database, so that the FDA report can be linked to reports from other sources, such as ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed.

How does the FDA review process differ from the peer review process used for manuscripts submitted to journals?

The key is the before-versus-after-the-trial aspect. Before the sponsor can begin a trial in the US, it must submit the trial protocol to the FDA. By means of this “pre-trial” review of the protocol, the FDA learns (1) that that trial is to be conducted and (2) the nitty-gritty methodological details. That way, a few years later, when the sponsor submits its NDA, the reviewer compares the clinical study report (CSR) to the original protocol to determine (1) whether any studies have been omitted and (2) whether any outcomes have been switched. By contrast, in the world of peer-reviewed journal articles, there is no “pre-trial” review, allowing both (1) and (2) to take place.

Who authors FDA reviews?

The FDA review team is multidisciplinary, yielding different types of reviews. Medical reviews are usually authored by physicians who summarize and evaluates the data on drug efficacy and safety. Statistical reviews are carried out by statisticians who test whether they can replicate the sponsor’s results using the patient-level data submitted electronically. There are other review disciplines, e.g. chemistry, pharmacology-toxicology, whose reviews pertain primarily to the preclinical phase and early clinical (human) phases of drug development.

What is the Open Science Prize?

The Open Science Prize is a collaboration between the Wellcome Trust, the US National Institutes of Health (NIH) and the Howard Hughes Medical Institute. Its goal is to unleash the power of open content and data to advance biomedical research and its application for health benefit. The OpenTrialsFDA team is one of the six finalist teams that were selected in May 2016. All teams will showcase their prototypes at the BD2K Open Data Science Symposium on 1 December 2016, when public voting will begin. The public is asked to help select the most promising, innovative and impactful prototypes from among the six finalists – among which one will receive the grand prize of $230,000.  

Troubleshooting

In case you’re having difficulty finding the right information in the Drugs@FDA portal, here are some troubleshooting questions to help you:

Was the drug approved before or after 1997?
The FDA started posting reviews (Drug Approval Packages) in 1997. If the drug in question was approved in 1996 or earlier, you probably will not find it posted.

Are you sure that the use you’re interested in is not off-label, ie. that the FDA approved it for the indication you’re interested in?
Oftentimes drugs are used widely for a certain condition (indication) for which the FDA has not granted approval (off-label use). If you check the product label (e.g. at https://dailymed.nlm.nih.gov/dailymed/index.cfm) and look within the section on “Indications and Usage”, you will see a list of the approved indications.

Are you looking for indication #1 or one of the subsequent indications?
A given drug might be FDA-approved for multiple indications. The FDA is careful about posting the Drug Approval Package for the first indication (the one that allowed that drug to enter the market), but for subsequent indications (#2, #3, etc.), it’s more hit-or-miss.

If you are motivated, one way to get such reviews is to file a Freedom of Information request here; but be warned—the FDA might take months or longer to fulfill your request. If you do get such reviews, we would appreciate your sharing it with us so we can add to OpenTrialsFDA and make it readily available to others.

Did you use the generic name or brand name?
If you use the generic name, you will probably get lots of hits: one for when the molecule was first being introduced to the market plus hits for subsequent generic versions. The FDA does its review for efficacy and safety when the molecule is first introduced to the market. Years later, when the patent expires, generic equivalents get approved, but what the FDA cares about at this stage is bioequivalence (in terms of blood levels) to the original brand name version. Thus generics do not have to demonstrate efficacy and safety all over again, which is why these approval packages generally contain little or no such data.

Is the review an NDA or an ANDA?
Please see the question and answer immediately above. NDAs (new drug applications) correspond to brand name versions, while ANDAs (abbreviated new drug applications) correspond to generic equivalents. They are abbreviated because less is required of the sponsor (see above). Assuming you are interested in drug efficacy and/or safety (and not bioequivalence), you will want to focus on NDAs and ignore ANDAs.

Is this the original or a subsequent formulation of a drug?
This is similar to the issue brand name versions vs. generic versions (see above). The burden of proof regarding efficacy and safety is highest when a molecule is first entering the market. A given drug will often enter the market as an immediate release (IR) formulation, before which the FDA will do its most comprehensive review of drug efficacy and safety. Some years later, the sponsor may seek approval for a sustained-release (SR) formulation, but by that time, millions of patients have already been exposed to the IR formulation. Because it is relatively unlikely that the efficacy and/or safety profile will differ a great deal between the IR and the SR, the SR may have to do fewer clinical trials compared to its IR version.

Want to get involved? Have further questions?

Please email us at [email protected]. This email is sent to Emma Beer, the Project Manager for OpenTrials and Ben Goldacre, its Principal Investigator.  

For further project updates, check the OpenTrialsFDA page at https://fda.opentrials.net or follow us on Twitter: @opentrials

You and Lisa Bero co-wrote an editorial entitled Regulatory agencies hold the key to improving Cochrane reviews of drugs [1]. Cochrane reviews are generally considered to be the gold standard in systematic reviews, but are you saying there is room for improvement? What has been learned from regulatory data that could not be learned from a rigorous literature search?
A recent systematic review by Golder [2] found that 43% to 100% of adverse events are missed if you only consider published documents. At best Cochrane reviews are only summarizing 57% of the available data on harms and their results and conclusions therefore might be misleading. If reviewers assume that there are no adverse events when none are reported they will exaggerate their confidence in “no harms”. Even though publicly available regulatory data is not always, complete it provides lots of data on harms, comprehensive list of trials conducted at the time of approval, considerations and worries among regulators and sometimes rejected indications and additional analysis.

In the study in which you surveyed Cochrane reviewers, did you find that they commonly made use of data from regulatory agencies?
Among the Cochrane authors that got access to unpublished data only 3% got data from regulatory agencies. Considering that it is freely available it was surprisingly low.

Why do you think that number is so low? I have heard systematic reviewers, when referring to regulatory data, use the phrase “low yield”. Why do you think they might say that?
There is a conception that getting data from regulatory agencies is “not worth the effort”. In some situations the published articles might be a fair representation of the conducted trials but in other situations it might not be. Summary reports are freely available at the FDA and EMA (European Medicines Agency) websites and are easier to obtain than published articles hidden behind paywalls. The reports are not as structured as published articles but they often summarize many trials. At the EMA there is the possibility through the Freedom of Information act to apply for clinical study reports, which are more detailed reports. Another reason for the low percentage of reviewers that use the data is that the Cochrane Handbook has been vague on guidance concerning regulatory data.

You wrote a paper in which you compared reports from both the FDA and the EMA. [3] How would you compare and contrast them?
The EMA prepares their reports for the public whereas the FDA reports are internal reports. The EMA reports are therefore easier to read but the FDA reports are more comprehensive, especially about harms. The FDA also publishes their communication with the sponsor which the EMA does not. The EMA, however, publishes rejected indications as well as the reason why. Lastly, the EMA are more accommodating towards access to clinical study reports through the Freedom of Information act.

What do you think about the usability of FDA review documents?
FDA documents contain lots of important information. Unfortunately they often contain different documents lumped together without a table of contents or clear separation. The documents are often scanned from paper, which makes them unsearchable, and they are often structured differently.

As you know, OpenTrialsFDA is aimed at making FDA data much more usable. Do you believe that, if this is done well, it will lead to a change in the habits of researchers doing systematic reviewers?
It should. The FDA reports are essentially a systematic review of all the available trials at the stage of approval. They contain much more information about harms than articles in medical journals. When there are discrepancies between published papers and FDA reports the researchers will have to use additional time to resolve those.

And if that happens, what impact, if any, do you see that having on the public health?
The conclusion of the FDA reports, in my opinion, seems to contain less “spin”. When reading FDA reports one gets the impression that the balance between benefit and harms are often delicate. Many redundant biased systematic reviews are produced [4] and regulatory reports could help limiting that problem.

References:

1. Schroll JB, Bero L. Regulatory agencies hold the key to improving Cochrane reviews of drugs. Cochrane Database Syst Rev 2015;4:ED000098. doi:10.1002/14651858.ED000098
2. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLoS Med. 2016 Sep 20;13(9):e1002127. doi: 10.1371/journal.pmed.1002127
3. Schroll JB, Abdel-Sattar M, Bero L. The Food and Drug Administration reports provided more data but were more difficult to use than the European Medicines Agency reports. J Clin Epidemiol Published Online First: 19 August 2014. doi:10.1016/j.jclinepi.2014.06.019
4. Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016 Sep;94(3):485-514

We will be sharing future progress on OpenTrialsFDA through this blog and hrough https://fda.opentrials.net as the work develops: the final prototype will be presented on 1 December at the BD2K Open Data Science Symposium, as one of the six finalists of the Open Science Prize. We welcome any questions and comments via [email protected].

What is an FDA Drug Approval Package?
FDA Drug Approval Packages consist of the reviews conducted by FDA employees of the “new drug applications” (NDAs) submitted by pharmaceutical companies seeking approval to market their drugs in the US. These NDAs consist primarily of clinical study reports (CSRs). Some drugs have more than one approval package, because approval is granted not for a drug per se, but for a drug-indication combination. If you look in the product labeling (e.g. https://dailymed.nlm.nih.gov), you’ll see the FDA-approved indications (uses) listed — you may see one, or you may see several. For each of these approved indications, there should be one FDA Drug Approval Package.

Why should I care about this specific date when I can find out everything I need to know from the published literature?
Because you can’t! The published literature won’t tell you about (1) clinical trials that weren’t published and (2) trials that underwent outcome switching and were thus “spun” from negative to positive.

Is there more than one type of review? Who authors FDA reviews?
The FDA employees who conduct these reviews hail from various disciplines. There is a medical review authored by someone with a medical background, usually a physician. He or she reviews and summarizes the data on the drug’s efficacy and safety. The statistical review is carried out by — you guessed it — a statistician, who tries to replicate the sponsor’s results using the raw patient-level data submitted electronically. There are other review disciplines, e.g. chemistry, pharmacology-toxicology, but their reviews pertain more to the preclinical phases of drug development.

How does the FDA review process differ from the peer review process used for manuscripts submitted to journals?
The key is the before-versus-after-the-trial aspect. Before the sponsor can begin a trial—at least a trial in the US—it has to submit the trial protocol to the FDA. By means of a “pre-trial” review of the protocol, the FDA learns (1) that that trial is to be conducted and (2) the nitty-gritty methodological details. That way, a couple of years later, when the sponsor submits its NDA, the reviewer compares the clinical study report (CSR) to the original protocol to determine (1) whether any studies have been omitted and (2) whether any outcomes have been switched. By contrast, in the world of peer-reviewed journal articles, there is no “pre-trial” review, allowing both (1) and (2) to take place.

Do you need a “secret handshake” to gain access to FDA reviews?
Not at all, because they are publicly available! They are posted on the Drugs@FDA website, and anyone anywhere in the world can download them. All we are doing with OpenTrialsFDA is making them easier to access and use.

What prompted you to start writing about them?
Back in 2004, in the wake of the lawsuit brought by the State of New York against GSK, there was an increased appreciation that reporting bias was corrupting the reliability of our evidence base. Most of the discussion had to do with registering trials, esp. in ClinicalTrials.gov. Meanwhile, it occurred to me that the FDA had, for decades, been running a semi-public registry and a results database, but it seemed to me that there was very little awareness of it.

That led me to write this essay, the purpose of which was simply to create some awareness of Drugs@FDA as a resource. The essay also provided two examples in which the FDA reviews revealed unfavorable study data that could not be found in a narrative review article or a Cochrane systematic review.

Did you feel that anyone was paying attention at that time?
I felt like I was a voice in the wilderness. For what it’s worth, the essay appeared in only the second issue of PLoS Medicine, which wasn’t yet on the radar screen of many people.

Can you tell us about the first full study you published in which you use of them?
In 2008, my colleagues and I published a paper in the NEJM, which contrasted the efficacy of 12 antidepressants according to the published literature with their efficacy according to the FDA. The interested reader can read the full text at this link, but suffice it to say that the two data sources told a very different story.

Have you looked at other drug groups using FDA reviews?
Yes, we’ve also published papers using a similar approach on antipsychotics and drugs for anxiety disorders. While these papers focus on one drug class or indication, one could also look more broadly across indications. Lisa Bero has done some excellent work using that approach. Two papers are available here and here.

Why did you write the BMJ paper “How to access and process FDA drug approval packages for use in research”?
I wrote this because, despite all the above work, it seemed that a lot of researchers were unaware, or only vaguely aware, of FDA drug approval packages, so I wanted to draw attention once again to them as a resource for research on drug efficacy and safety. And then many researchers found themselves daunted by the user-unfriendliness of navigating (1) to the reviews and (2) within them. I wanted this BMJ paper to act as something of a “cookbook” that other researchers could follow.

Did that paper have the impact you intended?
It’s hard to know how what the impact was, but regardless, this article was just an attempt to cope with a problem that, in my opinion, shouldn’t exist. There should be no reason why researchers shouldn’t have easier access to the wealth of unbiased drug data hiding within FDA drug approval packages.

When did you first consider doing something like OpenTrialsFDA?
Sometime in the last 2000s, I contacted the people at Google about doing something like a Google Books project on FDA reviews. They responded, but didn’t seem interested at that time.

How did the OpenTrialsFDA project come about?
Ben Goldacre and the OpenTrials team at Open Knowledge learned about the Open Science Prize, and they thought it would be a great entry to make FDA drug approval packages much more easily accessible to researchers.

Have there been other researchers who have advocated for wider use of FDA Drug Approval Packages?

Yes — here are several references:

• Bennett DA, Jull A. FDA: untapped source of unpublished trials. The Lancet 2003;361:1402–3. doi:10.1016/S0140-6736(03)13147-9
• Lurie P, Zieve A. Sometimes the silence can be like the thunder: access to pharmaceutical data at the FDA. Law & Contemp Probs 2006;68:85
• O’Connor AB. The need for improved access to FDA reviews. JAMA 2009;302:191–3. doi:10.1001/jama.2009.973
• Cook GE, Madden MM. The Significance of Information Posted on the U.S. Food and Drug Administration’s Website. J Manag Care Pharm 2012;19:72–3
• Marchand HC, Ros BJ, Fine AM, et al. The U.S. Food and Drug Administration: drug information resource for formulary recommendations. J Manag Care Pharm 2012;18:713–8
• Navarro RP. Rediscovering FDA websites. J Manag Care Pharm 2012;18:719–20
• McDonagh MS, Peterson K, Balshem H, et al. US Food and Drug Administration documents can provide unpublished evidence relevant to systematic reviews. J Clin Epidemiol Published Online First: 12 July 2013. doi:10.1016/j.jclinepi.2013.05.006
• Schroll JB, Abdel-Sattar M, Bero L. The Food and Drug Administration reports provided more data but were more difficult to use than the European Medicines Agency reports. J Clin Epidemiol Published Online First: 19 August 2014. doi:10.1016/j.jclinepi.2014.06.019

There are several detailed points that we could not get into here, and other questions will surely arise, so we welcome questions and comments via [email protected]. We plan to follow up with blog pieces on FAQ and/or troubleshooting in the near future through fda.opentrials.net. The OpenTrialsFDA team will present the final prototype in early December at the Open Science Prize Showcase event.

The Food and Drug Administration (FDA) publishes DAPs as part of the general information on drugs via its data portal known as Drugs@FDA. These documents contain detailed information about the methods and results of clinical trials, and are unbiased, compared to reports of clinical trials in academic journals. This is because FDA reviewers require adherence to the outcomes and analytic methods prespecified in the original trial protocols, so, in contrast to most journal editors, they are unforgiving of practices such as post hoc switching of outcomes and changes to the planned statistical analyses. These review packages also often report on clinical trials that have never been published.

However, despite their high value, these FDA documents are notoriously difficult to access, aggregate, and search. The website itself is not easy to navigate, and much of the information is stored in PDFs or non-searchable image files for older drugs. As a consequence, they are rarely used by clinicians and researchers. OpenTrialsFDA will work on improving this situation, so that valuable information that is currently hidden away can be discovered, presented, and used to properly inform evidence-based treatment decisions.

The team has started to scrape the FDA website, extracting the relevant information from the PDFs through a process of OCR (optical character recognition). A new OpenTrialsFDA interface will be developed to explore and discover the FDA data. In addition, the information will be integrated into the OpenTrials database, so that for any trial for which a match exists, users can see the corresponding FDA data.

We will be sharing future progress through this blog as the work develops: the final prototype will be presented in early December at the Open Science Prize Showcase.

Contact: [email protected]
@opentrials

More information about the Open Science Prize: https://www.openscienceprize.org/res/p/finalists/