The OpenTrialsFDA prototype makes clinical trial data from the FDA more easily accessible and searchable through a user-friendly web interface that you can now explore through fda.opentrials.net. Winning the prize would enable us to develop this prototype further in 2017, improving matching documents to corresponding trials on OpenTrials, along with extracting other valuable information from them. In the meantime, you can help us to get more FDA data on clinical trials available by rescuing ‘missing-in-action’ drug approval packages.

How to rescue “MIA” Drug Approval Packages

When OpenTrialsFDA carried out an automated download of all documents posted on Drugs@FDA, this not only captured Drug Approval Packages (DAPs), but also letters written by the FDA notifying drug companies that their drug had been approved for a new indication. Interestingly, this allowed us to identify drug-indication combinations for which DAPs were absent. Nevertheless, those DAPs do exist, and the FDA is obliged to make them available via the Freedom of Information Act (FOIA), for example in response to a FOI request.

In this blog by Dr. Erick Turner you can find out more on which DAPs are missing and how to make a FOIA request to the FDA to get them online. If and when you do make a FOIA request, please let us know by completing this form so that we can keep track of what DAPs have been requested.

Many thanks to everyone for their support during the voting process!

Nevertheless, those DAPs do exist, and the FDA is obliged to make them available via the Freedom of Information Act (FOIA), if not electronically, then in response to a FOI request. In this blog we explain how you can help to get this missing information online.

The OpenTrialsFDA project began with an automated download of all documents posted on Drugs@FDA. The download captured not only DAPs, but also letters written by the FDA notifying drug companies that their drug had been approved for a new indication, allowing us to identify drug-indication combinations for which DAPs were absent. A list of these missing-in-action (MIA) DAPs is shown below and on this page.

Our invitation to you

If you, the reader, are in possession of any of these “MIA” DAPs, we invite you to send them to us so that we can make them available to the research community and others via OpenTrialsFDA.

If there is a DAP you do not have but think it should be shared publicly, please check Drugs@FDA and verify that our automated process did not somehow miss it. You may use the procedure in this BMJ article², though be aware that the FDA recently made slight modifications to the procedure.

Making a FOIA request

Once you have verified that the DAP of interest is indeed not posted on Drugs@FDA, please consider making a FOIA request for it. The usual way to place a FOIA request is via the FDA’s FOI website. We invite you to use this template. Your request should contain the following information (part of which can be found in our list above):

1. Drug brand name
2. Drug generic name
3. Indication approved (consider consulting the product labeling to verify that this is an FDA-approved, not an off-label, indication/use)
4. Date of approval for that indication, aka “Action Date” (often this information can be obtained via Google, but you we suggest you confirm this by looking at that drug’s approval history on Drugs@FDA, where you should find a letter of that date documenting the approval)
5. Types of reviews requested (e.g. medical, statistical)

Just be aware that it may take several months, perhaps more than a year, for the FDA to complete the request. A major reason for this delay is (a) the FDA’s FOIA office is chronically short-staffed (as noted above) and (b) they must go through the DAP and identify and redact any information they deem to be “commercial confidential” or “trade secrets”, as they are exempt from disclosure.

Making a FOIA request for previously released information (possible shortcut)

Before making such a FOIA request, we want to make you aware of a possible shortcut—a request for previously released information. That is, perhaps someone else has recently requested the same information and obtained it, in which case the FOIA office has already completed the time-consuming redaction process, which means there is no need for them to do it all over again. Therefore, in this type of FOIA request, you essentially say to the FDA’s FOIA office, “Whatever you gave to that previous requester, I’d like that, too.”

So now you might ask, how do you know whether the DAP you’re after has, in fact, been requested and released? FOIA case logs! A case log lists the completed FOIA requests, who made the request, the information requested (e.g. drug name, review types), and—most importantly—the request number (“control #”).

And how do you get a case log? You make a FOIA request for it! Even easier, feel free to search the one Dr Turner obtained recently, which covers the period January 2012 – September 2016.

This process might well sound circular, but it can save lots of time, because requests for previously released information often get completed within weeks rather than months or longer.

Tracking your contribution

If and when you do make a FOIA request, please let us know by completing this form so that we can keep track of what DAPs have been requested. Please also keep us informed on what the FDA’s response was (usually some variation of “we’ll get to it and send you the information when it’s ready”).

Please vote!

Finally, if you think OpenTrialsFDA is an important contribution to the public health, and if you’d like to support our effort to pry open this vault of priceless clinical trial data, please help us win the Open Science Prize by casting your vote before 6 January 2017. Thank you!

References

1. Turner EH. A taxpayer-funded clinical trials registry and results database. PLoS Med. 2004;1(3):e60. doi:10.1371/journal.pmed.0010060.
2. Turner EH. How to access and process FDA drug approval packages for use in research. BMJ. 2013;347(oct14 2):f5992-f5992. doi:10.1136/bmj.f5992.

OpenTrialsFDA will be featured in the OHSU (Oregon Health & Science University) Open Science Data Jamboree & Hackathon. The event will focus on the power of open content and data to advance biomedical research and human health.  Dr. Erick Turner, Associate Professor at OHSU and Staff Psychiatrist at the Portland VA Medical Center, will introduce OpenTrialsFDA, which is a finalist for the highly competitive Open Science Prize (voting for the finalists is open until 6 January 2017 here).

OpenTrialsFDA aims to increase access to and the utility of FDA drug approval packages. These review packages often contain information on clinical trials that have never been published.  When these “inconvenient truths” are integrated into a systematic review or meta-analysis, it can sometimes dramatically shift a drug’s risk-benefit ratio. Although the FDA makes the reviews available via its web portal Drugs@FDA, the documents are notoriously difficult to search and aggregate. As a consequence, they are rarely used by clinicians and researchers, despite their immense value.  OpenTrialsFDA intends to change this by allowing researchers, clinicians, the public, developers, and third party platforms to access, search, and build upon this data.

Dr. Turner’s keynote talk will be followed by an overview of the NIH’s new clinical trial data sharing requirements and two concurrent hands-on sessions:

• The first will be a deep dive on searching the OpenTrialsFDA website, understanding the contents of an FDA drug approval package (including the statistical content), and how to use the data for different use cases, such as meta analysis and systematic reviews.  Dr. Turner will team teach the breakout with an OHSU biostatistician.
• The second will be a OpenTrialsFDA hackathon using the OpenTrials API.  Attendees will have the opportunity to use their research, development, clinical, and visualization expertise (to name a few) to explore, present, and build upon the OpenTrials data and the OpenTrialsFDA application.  Please note, no technical expertise is required, as there will be multiple ways of contributing.

The Data Jamboree is presented by Computational Biology and the OHSU Library.  All are welcome to attend, including OHSU faculty, students, and staff as well as Portland area science enthusiasts and developers!

• Date: (to be announced)
• Time 2:30 – 5:00 PM (hackathon will continue past 5:00)
• Location: Collaborative Life Sciences Building (CLSB), 3A003, OHSU, Portland, Oregon
• Food & Drink provided

Contact:  Robin Champieux, Scholarly Communication Librarian, [email protected]

OpenTrialsFDA is a collaboration between Dr. Erick Turner (a psychiatrist-researcher and transparency advocate), Dr. Ben Goldacre (Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford) and the team behind OpenTrials at Open Knowledge International.  

OpenTrialsFDA works on making clinical trial data from the FDA (the US Food and Drug Administration) more easily accessible and searchable. Until now, this information has been hidden in the user-unfriendly Drug Approval Packages that the FDA publishes via its dataportal Drugs@FDA. These are often just images of pages, so you cannot even search for a text phrase in them. OpenTrialsFDA scrapes all the relevant data and documents from the FDA documents, runs Optical Character Recognition across all documents, links this information to other clinical trial data, and now presents it through a new user-friendly web interface at fda.opentrials.net.

Explore the OpenTrialsFDA search interface

Any user can type in a drug name, and see all the places where this drug is mentioned in an FDA document. Users can also access, search and present this information through the application programming interfaces (APIs) the team will produce. In addition, the information has been integrated into the OpenTrials database, so that the FDA reports are linked to reports from other sources, such as ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed.

The prototype will provide the academic research world with important information on clinical trials in general, improving the quality of research, and helping evidence-based treatment decisions to be properly informed. Interestingly, the FDA data is unbiased, compared to reports of clinical trials in academic journals. Dr. Erick Turner explains: “With journal articles everything takes place after a study has finished, but with FDA reviews, there is a protocol that is submitted to the FDA before the study has even started. So the FDA learns first of all that the study is to be done, which means it can’t be hidden later. Secondly it learns all the little details, methodological details about how the study is going to be done and how it is going to be analyzed, and that guards against outcome switching.”

Dr Ben Goldacre says: “These FDA documents are hugely valuable, but at the moment they’re hardly ever used. That’s because – although they’re publicly accessible in the most literal sense of that phrase – they are almost impossible to search, and navigate. We are working to make this data accessible, so that it has the impact it deserves.”

Voting for the Open Science Prize finalists is possible through http://event.capconcorp.com/wp/osp: more information on OpenTrialsFDA is available from fda.opentrials.net/about and from the team’s video below.

Editor’s notes

Dr. Ben Goldacre
Ben is a doctor, academic, writer, and broadcaster, and currently a Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford. His blog is at www.badscience.net and he is @bengoldacre on twitter. Read more here. His academic and policy work is in epidemiology and evidence based medicine, where he works on various problems including variation in care, better uses of routinely collected electronic health data, access to clinical trial data, efficient trial design, and retracted papers. In policy work, he co-authored this influential Cabinet Office paper, advocating for randomised trials in government, and setting out mechanisms to drive this forwards. He is the co-founder of the AllTrials campaign. He engages with policy makers. Alongside this he also works in public engagement, writing and broadcasting for a general audience on problems in evidence based medicine. His books have sold over 600,000 copies.

Dr. Erick Turner
Dr. Erick Turner is a psychiatrist-researcher and transparency advocate. Following a clinical research fellowship at the NIH, he worked for the US Food and Drug Administration (FDA), acting as gatekeeper for new psychotropic drugs seeking to enter the US market. In 2004 he published a paper drawing researchers’ attention to the Drugs@FDA website as a valuable but underutilized source of unbiased clinical trial data. Dissatisfied with the continuing underutilization of Drugs@FDA, he published a paper in the BMJ in order to encourage wider use of this trove of clinical trial data.

Open Knowledge International
https://okfn.org   
Open Knowledge International is a global non-profit organisation focussing on realising open data’s value to society by helping civil society groups access and use data to take action on social problems. Open Knowledge International addresses this in three steps: 1) we show the value of open data for the work of civil society organizations; 2) we provide organisations with the tools and skills to effectively use open data; and 3) we make government information systems responsive to civil society.

Open Science Prize
https://www.openscienceprize.org/res/p/finalists/
The Open Science Prize  is a collaboration between the National Institutes of Health and the Wellcome Trust, with additional funding provided by the Howard Hughes Medical Institute of Chevy Chase, Maryland.  The Open Data Science Symposium will feature discussions with the leaders in big data, open science, and biomedical research while also showcasing the finalists of the Open Data Science Prize, a worldwide competition to harness the innovative power of open data.

General questions

Background

The OpenTrialsFDA team is one of the six finalists for the Open Science Prize: a global science competition to make both the outputs from science and the research process broadly accessible to the public. This is a collaboration between Dr. Erick Turner, Dr. Ben Goldacre and the OpenTrials team at Open Knowledge International. Competing for the $230,000 prize that will be awarded to the winner in February 2017, OpenTrialsFDA is working on a prototype that will make the information hidden in the user-unfriendly drug approval packages of the FDA (the US Food and Drug Administration) more easily accessible and searchable, and link these to documents and data related to clinical trials.

The work is closely related to the OpenTrials project, which aims to locate, match, and share all publicly accessible data and documents, on all trials conducted, on all medicines and other treatments, globally. OpenTrials is aggregating this information from a wide variety of existing sources and aims to provide a comprehensive picture of the data and documents on clinical trials conducted on medicines and other treatments around the world.

What is the scope of OpenTrialsFDA?

OpenTrialsFDA works on making clinical trial data from the FDA (the US Food and Drug Administration) more easily accessible and searchable. Until now, this information has been hidden in the user-unfriendly Drug Approval Packages that the FDA publishes via its dataportal Drugs@FDA. These documents are notoriously difficult to access, aggregate, and search, since the Drugs@FDA website itself is not intuitive to navigate, and the reviews are stored as individual PDFs which, except for relatively new drugs, cannot be searched.  OpenTrialsFDA will extract the relevant data from the FDA documents, link it to other clinical trial data and present it through a new user-friendly web interface. Any user will be able to type in a drug name, and see all the places where this drug is mentioned in an FDA document. Users will also be able to access, search and present this information through the application programming interfaces (APIs) the team will produce.

Who is behind OpenTrialsFDA?

OpenTrialsFDA is a collaboration between Dr. Erick Turner (a psychiatrist-researcher and transparency advocate), Dr. Ben Goldacre (Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford) and the team behind OpenTrials at Open Knowledge International.  

What is the Drugs@FDA portal?

Drugs@FDA is a searchable catalog of the US Food and Drug Administration (FDA) approved drug products, both prescription and over the counter, with links to documents relating to marketing approval. More information is available from their FAQ section.

What are FDA drug approval packages?

FDA Drug Approval Packages consist of FDA employees’ reviews of new drug applications (NDAs) submitted by pharmaceutical companies seeking approval to market their drugs in the US. These NDAs consist primarily of clinical study reports (CSRs). Some drugs have more than one approval package, because approval is granted not for a drug per se, but for each drug-indication combination.

How will OpenTrialsFDA improve access to the FDA data?

The team will scrape the FDA website and extract the relevant information from the PDFs through a process of OCR (optical character recognition). Through the new OpenTrialsFDA interface, users will be able to explore and discover the FDA data. In addition, the information will be integrated into the OpenTrials database, so that the FDA report can be linked to reports from other sources, such as ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed.

How does the FDA review process differ from the peer review process used for manuscripts submitted to journals?

The key is the before-versus-after-the-trial aspect. Before the sponsor can begin a trial in the US, it must submit the trial protocol to the FDA. By means of this “pre-trial” review of the protocol, the FDA learns (1) that that trial is to be conducted and (2) the nitty-gritty methodological details. That way, a few years later, when the sponsor submits its NDA, the reviewer compares the clinical study report (CSR) to the original protocol to determine (1) whether any studies have been omitted and (2) whether any outcomes have been switched. By contrast, in the world of peer-reviewed journal articles, there is no “pre-trial” review, allowing both (1) and (2) to take place.

Who authors FDA reviews?

The FDA review team is multidisciplinary, yielding different types of reviews. Medical reviews are usually authored by physicians who summarize and evaluates the data on drug efficacy and safety. Statistical reviews are carried out by statisticians who test whether they can replicate the sponsor’s results using the patient-level data submitted electronically. There are other review disciplines, e.g. chemistry, pharmacology-toxicology, whose reviews pertain primarily to the preclinical phase and early clinical (human) phases of drug development.

What is the Open Science Prize?

The Open Science Prize is a collaboration between the Wellcome Trust, the US National Institutes of Health (NIH) and the Howard Hughes Medical Institute. Its goal is to unleash the power of open content and data to advance biomedical research and its application for health benefit. The OpenTrialsFDA team is one of the six finalist teams that were selected in May 2016. All teams will showcase their prototypes at the BD2K Open Data Science Symposium on 1 December 2016, when public voting will begin. The public is asked to help select the most promising, innovative and impactful prototypes from among the six finalists – among which one will receive the grand prize of $230,000.  

Troubleshooting

In case you’re having difficulty finding the right information in the Drugs@FDA portal, here are some troubleshooting questions to help you:

Was the drug approved before or after 1997?
The FDA started posting reviews (Drug Approval Packages) in 1997. If the drug in question was approved in 1996 or earlier, you probably will not find it posted.

Are you sure that the use you’re interested in is not off-label, ie. that the FDA approved it for the indication you’re interested in?
Oftentimes drugs are used widely for a certain condition (indication) for which the FDA has not granted approval (off-label use). If you check the product label (e.g. at https://dailymed.nlm.nih.gov/dailymed/index.cfm) and look within the section on “Indications and Usage”, you will see a list of the approved indications.

Are you looking for indication #1 or one of the subsequent indications?
A given drug might be FDA-approved for multiple indications. The FDA is careful about posting the Drug Approval Package for the first indication (the one that allowed that drug to enter the market), but for subsequent indications (#2, #3, etc.), it’s more hit-or-miss.

If you are motivated, one way to get such reviews is to file a Freedom of Information request here; but be warned—the FDA might take months or longer to fulfill your request. If you do get such reviews, we would appreciate your sharing it with us so we can add to OpenTrialsFDA and make it readily available to others.

Did you use the generic name or brand name?
If you use the generic name, you will probably get lots of hits: one for when the molecule was first being introduced to the market plus hits for subsequent generic versions. The FDA does its review for efficacy and safety when the molecule is first introduced to the market. Years later, when the patent expires, generic equivalents get approved, but what the FDA cares about at this stage is bioequivalence (in terms of blood levels) to the original brand name version. Thus generics do not have to demonstrate efficacy and safety all over again, which is why these approval packages generally contain little or no such data.

Is the review an NDA or an ANDA?
Please see the question and answer immediately above. NDAs (new drug applications) correspond to brand name versions, while ANDAs (abbreviated new drug applications) correspond to generic equivalents. They are abbreviated because less is required of the sponsor (see above). Assuming you are interested in drug efficacy and/or safety (and not bioequivalence), you will want to focus on NDAs and ignore ANDAs.

Is this the original or a subsequent formulation of a drug?
This is similar to the issue brand name versions vs. generic versions (see above). The burden of proof regarding efficacy and safety is highest when a molecule is first entering the market. A given drug will often enter the market as an immediate release (IR) formulation, before which the FDA will do its most comprehensive review of drug efficacy and safety. Some years later, the sponsor may seek approval for a sustained-release (SR) formulation, but by that time, millions of patients have already been exposed to the IR formulation. Because it is relatively unlikely that the efficacy and/or safety profile will differ a great deal between the IR and the SR, the SR may have to do fewer clinical trials compared to its IR version.

Want to get involved? Have further questions?

Please email us at [email protected]. This email is sent to Emma Beer, the Project Manager for OpenTrials and Ben Goldacre, its Principal Investigator.  

For further project updates, check the OpenTrialsFDA page at https://fda.opentrials.net or follow us on Twitter: @opentrials

Who do you expect to use your data, and for what ends?
Ben Meghreblian: We expect a range of users to use OpenTrials, including researchers, doctors, and patients. A researcher could find out more about a range of trials on a drug, searching by various different features such as inclusion and exclusion criteria to match a specific population. A doctor interested in critical appraisal of research papers could see if sources of bias for specific trials have already been assessed by experts. A patient interested in participating in a trial for their condition could identify trials in their geographical area which are enrolling. We’re also interested to see how policy makers and regulators may use OpenTrials to inform their work, and how data journalists and developers will use the data to write interesting stories.

How easy will the platform be to use?
OpenTrials works like a search engine, with advanced search for filtering results by criteria such as drug and disease area. From our user testing so far we haven’t come across any major usability issues, but we’re taking all feedback into account – we want to make it easy to use for a wide range of people. No special software or IT skills will be needed.

Looking at the data, what discoveries have surprised you most so far?
So far we’ve found two interesting things. Firstly, we found a number of publications on PubMed which have an incorrect trial registry ID associated with them. We have used PubMed Commons to comment on these trials, and have already received positive responses from some authors. Secondly, we have found interesting discrepancies and problems with the data on registries, and elsewhere. Some of the errors we’ve found are widespread, and concerning. Keep an eye on the OpenTrials blog for details.

What have been the biggest challenges in developing OpenTrials?
Getting the data that we need and cleaning it. We currently extract data from several different sources, each with its own structure. For example, one source might use the location name “United States of America”, another might use “USA”, and so on. We have to keep the names consistent so the user can easily find trials. This problem gets more complicated when we consider things like drug names, condition names, company names, etc.

Is there an automated function to detect inconsistencies and poor research practices such as primary outcome switching?
For a given trial, we automatically list discrepancies across different registries – for example trial status and number of trial participants. For poor research practices such as primary outcome switching, this is currently too difficult to do automatically, but the COMPare Trials project is doing a great job of manually assessing outcome switching in trials published in academic journals. RobotReviewer are doing interesting work on assessing these flaws using software alone.

You plan to score the methodological rigour of trials. How will that work?
We have been given risk of bias data from the Cochrane Schizophrenia group, which grades trials on issues such as blinding and selective reporting. For those trials, we will display this information on the corresponding OpenTrials page along with other trial information (see an example here). We hope that showing this data integrated will encourage other groups and companies to share their datasets with OpenTrials.

How will your data set differ from similar ones such as the Good Pharma Scorecard or that generated by the 2015 STAT investigation?
The Good Pharma Scorecard is an excellent window onto a small number of trials where results have been manually searched for. Charles Piller’s 2015 STAT investigation was a valuable static snapshot of registry data showing which institutions are best and worst for overdue trials. OpenTrials advanced search will allow users to conduct similar searches, across all clinical trials conducted. As the population of the database becomes more complete, it will facilitate similar audits, but where the results update live as trials are published (or not).

What data will be included?
We currently extract and display data from ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed, and include risk of bias assessments from the Cochrane Schizophrenia group. After the launch, we plan to integrate systematic review data from Epistemonikos and other sources. There are seven additional sources of data that we’ve extracted, but can’t display because of licensing issues – we’re working with them to get permission to publish. We’ll let users know when they become available via the OpenTrials blog.

You plan to populate the database manually, but only for a small number of trials. What value will that add?
It will showcase what a perfect database would look like, and the value it can give to patients, researchers, and doctors. Additionally it will allow us to establish the amount of manual effort necessary to manually perfect the entire database.

You plan to allow third parties to submit data online. Is there a danger of players with vested financial interests submitting partial or tailored data to influence perceptions of drugs’ effectiveness?
Any submitted data will be manually approved by a researcher in the first instance. While it will be impossible to stop those with vested interests submitting altered data (something we don’t think will happen often), we hope the community of OpenTrials users will help flag any anomalies in the contributions we will host, which can then be reviewed by our team. As more information is contributed and data sets donated, we would expect outliers to become easier to spot by having many eyes on the data and triangulating information.

Can you explain what your transparency scoreboard will do?
We aren’t releasing the transparency scoreboard as part of our beta launch. We will release this at a later date, but meanwhile all our data will be open and accessible, along with an interface: we encourage others to run their own analyses, build applications, and find interesting patterns and stories in the data.

Does it make sense for several groups to build separate platforms? Wouldn’t it be better to agree on a universal data sharing standard first and then develop one definitive global platform?
Getting widespread agreement on standards may be ideal, but is notoriously hard to achieve. It is very unlikely that they would be imposed on all of the hundreds of thousands of trials already conducted. We can start linking, matching, and building, right now, and there is no sense in further delay. We would rather take the initiative and build a functional platform supported by the community. We’ve spoken to a number of groups in related fields and we are always keen to discuss shared interests and ways of collaborating in order to not reinvent the wheel!

Which is your favourite clinical trials registry, and why?
ClinicalTrials.gov is a leading registry both technically and in terms of the number of trials it contains. Their data is well-structured and well accessible. Currently there are quite a few registries with poor support for data collection, so this stands out amongst them.

What can existing clinical trials registries do to make their data more accessible and useful for researchers?
Registries could provide API access to their data, along with making sure that the trial metadata is well-structured and follows recommended standards (e.g. the WHO Trial Registration Data Set). Ideally, the metadata would follow known standards, for example using ISO codes for countries instead of their names and using a database such as MeSH to define terms like conditions names. This would make the different registries’ databases comparable. While API access is very useful, the best way a registry can offer its entire database is as a regular download, similar to what the FDA does with its OpenFDA website. This makes it much simpler for researchers who need a local copy of the database.

What can other players do to make their data more accessible and useful for researchers?
Beyond adopting standards and guidelines, we’d encourage a range of players to embrace being open by default. We’re keen to talk to any organisations or companies who want to make their data more accessible via OpenTrials.

Beyond OpenTrials, ten years from now, what information do you expect researchers to be able to access? And what information do you think will remain elusive?
There is a clear need for a step change. We need structured data about all trials, describing the methods and results, instead of only free text reports. That’s the big horizon. Basic knowledge management has been sorely lacking on clinical trials, and that makes no sense. We spend millions on individual trials. We need to spend a little on making sure the information is discoverable, machine readable, and impactful.

This interview was conducted via email by AllTrials campaign manager Dr Till Bruckner and has been reposted from the AllTrials blog.

You and Lisa Bero co-wrote an editorial entitled Regulatory agencies hold the key to improving Cochrane reviews of drugs [1]. Cochrane reviews are generally considered to be the gold standard in systematic reviews, but are you saying there is room for improvement? What has been learned from regulatory data that could not be learned from a rigorous literature search?
A recent systematic review by Golder [2] found that 43% to 100% of adverse events are missed if you only consider published documents. At best Cochrane reviews are only summarizing 57% of the available data on harms and their results and conclusions therefore might be misleading. If reviewers assume that there are no adverse events when none are reported they will exaggerate their confidence in “no harms”. Even though publicly available regulatory data is not always, complete it provides lots of data on harms, comprehensive list of trials conducted at the time of approval, considerations and worries among regulators and sometimes rejected indications and additional analysis.

In the study in which you surveyed Cochrane reviewers, did you find that they commonly made use of data from regulatory agencies?
Among the Cochrane authors that got access to unpublished data only 3% got data from regulatory agencies. Considering that it is freely available it was surprisingly low.

Why do you think that number is so low? I have heard systematic reviewers, when referring to regulatory data, use the phrase “low yield”. Why do you think they might say that?
There is a conception that getting data from regulatory agencies is “not worth the effort”. In some situations the published articles might be a fair representation of the conducted trials but in other situations it might not be. Summary reports are freely available at the FDA and EMA (European Medicines Agency) websites and are easier to obtain than published articles hidden behind paywalls. The reports are not as structured as published articles but they often summarize many trials. At the EMA there is the possibility through the Freedom of Information act to apply for clinical study reports, which are more detailed reports. Another reason for the low percentage of reviewers that use the data is that the Cochrane Handbook has been vague on guidance concerning regulatory data.

You wrote a paper in which you compared reports from both the FDA and the EMA. [3] How would you compare and contrast them?
The EMA prepares their reports for the public whereas the FDA reports are internal reports. The EMA reports are therefore easier to read but the FDA reports are more comprehensive, especially about harms. The FDA also publishes their communication with the sponsor which the EMA does not. The EMA, however, publishes rejected indications as well as the reason why. Lastly, the EMA are more accommodating towards access to clinical study reports through the Freedom of Information act.

What do you think about the usability of FDA review documents?
FDA documents contain lots of important information. Unfortunately they often contain different documents lumped together without a table of contents or clear separation. The documents are often scanned from paper, which makes them unsearchable, and they are often structured differently.

As you know, OpenTrialsFDA is aimed at making FDA data much more usable. Do you believe that, if this is done well, it will lead to a change in the habits of researchers doing systematic reviewers?
It should. The FDA reports are essentially a systematic review of all the available trials at the stage of approval. They contain much more information about harms than articles in medical journals. When there are discrepancies between published papers and FDA reports the researchers will have to use additional time to resolve those.

And if that happens, what impact, if any, do you see that having on the public health?
The conclusion of the FDA reports, in my opinion, seems to contain less “spin”. When reading FDA reports one gets the impression that the balance between benefit and harms are often delicate. Many redundant biased systematic reviews are produced [4] and regulatory reports could help limiting that problem.

References:

1. Schroll JB, Bero L. Regulatory agencies hold the key to improving Cochrane reviews of drugs. Cochrane Database Syst Rev 2015;4:ED000098. doi:10.1002/14651858.ED000098
2. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLoS Med. 2016 Sep 20;13(9):e1002127. doi: 10.1371/journal.pmed.1002127
3. Schroll JB, Abdel-Sattar M, Bero L. The Food and Drug Administration reports provided more data but were more difficult to use than the European Medicines Agency reports. J Clin Epidemiol Published Online First: 19 August 2014. doi:10.1016/j.jclinepi.2014.06.019
4. Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016 Sep;94(3):485-514

We will be sharing future progress on OpenTrialsFDA through this blog and hrough https://fda.opentrials.net as the work develops: the final prototype will be presented on 1 December at the BD2K Open Data Science Symposium, as one of the six finalists of the Open Science Prize. We welcome any questions and comments via [email protected].

OpenTrials works like a search engine, with advanced search options for filtering results by criteria such as drug and disease area. All data and documents for each trial included are “threaded” together and presented alongside each other. At the World Health Summit, the team will be demonstrating how the OpenTrials interface works, including how to explore trials and filter results by criteria such as drug and disease area. They will also demonstrate the power of linking clinical trial information together, showing how it can be used to highlight important discrepancies in the data.

We want the information provided on OpenTrials to inform decision-making and lead to better medical services worldwide. We expect a range of potential uses for the platform:

• A public health researcher could find out more about the range of trials on a drug, searching by various criteria to match a specific population.
• A doctor interested in critical appraisal of research papers could see if sources of bias for specific trials have already been assessed by experts.
• A researcher could see if the same trial reports somewhat different methods or results in different places.
• A patient interested in participating in a trial for their condition could identify trials in their geographical area which are enrolling.

A crowdsourcing functionality allows users to contribute data and documents and to provide feedback on the accuracy of trial information.

OpenTrials currently extracts and displays data from ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed, and risk of bias assessments from the Cochrane Schizophrenia group. After the beta launch, we plan to integrate systematic review data from Epistemonikos and other sources. There are seven additional sources of data that have been extracted, but can’t currently be displayed because of licensing issues – we are working with these sources of data to get permission to publish. We’ll keep updating the OpenTrials blog as they become available.

“There have been numerous positive statements about the need for greater transparency on information about clinical trials, over many years, but it has been almost impossible to track and audit exactly what is missing, or easily identify discrepancies in information about trials” explained Dr. Goldacre, the project’s Chief Investigator and a Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford. “This project aims to draw together everything that is known around each clinical trial. The end product will provide valuable information for patients, doctors, researchers, and policymakers—not just on individual trials, but also on how whole sectors, researchers, companies, and funders are performing. It has the potential to show who is failing to share information appropriately, who is doing well, and how standards can be improved.”

“OpenTrials is an important step towards ensuring researchers, journalists, and patient groups have access to the medical information they need,” said Pavel Richter, CEO of Open Knowledge International. “Through the OpenTrials platform, researchers can advance science more quickly, doctors can easily find the latest evidence to improve services, and patients can locate information about pressing public health issues. OpenTrials is a great example of the work we are doing at Open Knowledge International to equip civil society organisations with the tools and information they need to address social problems and improve people’s lives.”

The first phase of the Open Trials project is scheduled for completion in March 2017. For project updates, please follow @opentrials on twitter or get in touch with us at [email protected].  A Hack Day (a World Health Summit Satellite event) took place on 8 October in Berlin.  For more details, see https://opentrials.net/hackday

Further information on speakers and topics of the World Health Summit 2016:

www.worldhealthsummit.org/the-summit/program

www.worldhealthsummit.org/the-summit/speakers

The World Health Summit is open to media representatives:

www.worldhealthsummit.org/press-media/accreditation

Editor’s notes:

Ben Goldacre
Ben is a doctor, academic, writer, and broadcaster, and currently a Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford. His blog is at www.badscience.net and he is @bengoldacre on twitter. Read more here. His academic and policy work is in epidemiology and evidence based medicine, where he works on various problems including variation in care, better uses of routinely collected electronic health data, access to clinical trial data, efficient trial design, and retracted papers. In policy work, he co-authored this influential Cabinet Office paper, advocating for randomised trials in government, and setting out mechanisms to drive this forwards. He is the co-founder of the AllTrials campaign. He engages with policy makers. Alongside this he also works in public engagement, writing and broadcasting for a general audience on problems in evidence based medicine. His books have sold over 600,000 copies.

Open Knowledge International
https://okfn.org
https://opentrials.net
Open Knowledge International is a global non-profit organisation focussing on realising open data’s value to society by helping civil society groups access and use data to take action on social problems. Open Knowledge International addresses this in three steps: 1) we show the value of open data for the work of civil society organizations; 2) we provide organisations with the tools and skills to effectively use open data; and 3) we make government information systems responsive to civil society.

The Laura and John Arnold Foundation
www.arnoldfoundation.org
LJAF is a private foundation committed to producing substantial, widespread, and lasting reforms that will maximize opportunities and minimize injustice in our society. Its strategic investments are currently focused on criminal justice, education, public accountability, evidence-based policy, and research integrity. LJAF has offices in Houston, New York City and Washington D.C.

COS
http://centerforopenscience.org
COS is a non-profit technology company providing free, open source software and services to increase inclusivity and transparency of research. COS supports shifting incentives and practices to align more closely with scientific values.  COS develops the Open Science Framework as an infrastructure to enable a more open and transparent research workflow across all of the sciences.

World Health Summit
www.worldhealthsummit.org
Under the high patronage of German Chancellor Angela Merkel, French President François Hollande and European Commission President Jean-Claude Juncker, the WHS attracts about 1,800 participants from more than 80 countries. It is the premiere international platform for  exploring strategic developments and decisions in the area of healthcare.

What is an FDA Drug Approval Package?
FDA Drug Approval Packages consist of the reviews conducted by FDA employees of the “new drug applications” (NDAs) submitted by pharmaceutical companies seeking approval to market their drugs in the US. These NDAs consist primarily of clinical study reports (CSRs). Some drugs have more than one approval package, because approval is granted not for a drug per se, but for a drug-indication combination. If you look in the product labeling (e.g. https://dailymed.nlm.nih.gov), you’ll see the FDA-approved indications (uses) listed — you may see one, or you may see several. For each of these approved indications, there should be one FDA Drug Approval Package.

Why should I care about this specific date when I can find out everything I need to know from the published literature?
Because you can’t! The published literature won’t tell you about (1) clinical trials that weren’t published and (2) trials that underwent outcome switching and were thus “spun” from negative to positive.

Is there more than one type of review? Who authors FDA reviews?
The FDA employees who conduct these reviews hail from various disciplines. There is a medical review authored by someone with a medical background, usually a physician. He or she reviews and summarizes the data on the drug’s efficacy and safety. The statistical review is carried out by — you guessed it — a statistician, who tries to replicate the sponsor’s results using the raw patient-level data submitted electronically. There are other review disciplines, e.g. chemistry, pharmacology-toxicology, but their reviews pertain more to the preclinical phases of drug development.

How does the FDA review process differ from the peer review process used for manuscripts submitted to journals?
The key is the before-versus-after-the-trial aspect. Before the sponsor can begin a trial—at least a trial in the US—it has to submit the trial protocol to the FDA. By means of a “pre-trial” review of the protocol, the FDA learns (1) that that trial is to be conducted and (2) the nitty-gritty methodological details. That way, a couple of years later, when the sponsor submits its NDA, the reviewer compares the clinical study report (CSR) to the original protocol to determine (1) whether any studies have been omitted and (2) whether any outcomes have been switched. By contrast, in the world of peer-reviewed journal articles, there is no “pre-trial” review, allowing both (1) and (2) to take place.

Do you need a “secret handshake” to gain access to FDA reviews?
Not at all, because they are publicly available! They are posted on the Drugs@FDA website, and anyone anywhere in the world can download them. All we are doing with OpenTrialsFDA is making them easier to access and use.

What prompted you to start writing about them?
Back in 2004, in the wake of the lawsuit brought by the State of New York against GSK, there was an increased appreciation that reporting bias was corrupting the reliability of our evidence base. Most of the discussion had to do with registering trials, esp. in ClinicalTrials.gov. Meanwhile, it occurred to me that the FDA had, for decades, been running a semi-public registry and a results database, but it seemed to me that there was very little awareness of it.

That led me to write this essay, the purpose of which was simply to create some awareness of Drugs@FDA as a resource. The essay also provided two examples in which the FDA reviews revealed unfavorable study data that could not be found in a narrative review article or a Cochrane systematic review.

Did you feel that anyone was paying attention at that time?
I felt like I was a voice in the wilderness. For what it’s worth, the essay appeared in only the second issue of PLoS Medicine, which wasn’t yet on the radar screen of many people.

Can you tell us about the first full study you published in which you use of them?
In 2008, my colleagues and I published a paper in the NEJM, which contrasted the efficacy of 12 antidepressants according to the published literature with their efficacy according to the FDA. The interested reader can read the full text at this link, but suffice it to say that the two data sources told a very different story.

Have you looked at other drug groups using FDA reviews?
Yes, we’ve also published papers using a similar approach on antipsychotics and drugs for anxiety disorders. While these papers focus on one drug class or indication, one could also look more broadly across indications. Lisa Bero has done some excellent work using that approach. Two papers are available here and here.

Why did you write the BMJ paper “How to access and process FDA drug approval packages for use in research”?
I wrote this because, despite all the above work, it seemed that a lot of researchers were unaware, or only vaguely aware, of FDA drug approval packages, so I wanted to draw attention once again to them as a resource for research on drug efficacy and safety. And then many researchers found themselves daunted by the user-unfriendliness of navigating (1) to the reviews and (2) within them. I wanted this BMJ paper to act as something of a “cookbook” that other researchers could follow.

Did that paper have the impact you intended?
It’s hard to know how what the impact was, but regardless, this article was just an attempt to cope with a problem that, in my opinion, shouldn’t exist. There should be no reason why researchers shouldn’t have easier access to the wealth of unbiased drug data hiding within FDA drug approval packages.

When did you first consider doing something like OpenTrialsFDA?
Sometime in the last 2000s, I contacted the people at Google about doing something like a Google Books project on FDA reviews. They responded, but didn’t seem interested at that time.

How did the OpenTrialsFDA project come about?
Ben Goldacre and the OpenTrials team at Open Knowledge learned about the Open Science Prize, and they thought it would be a great entry to make FDA drug approval packages much more easily accessible to researchers.

Have there been other researchers who have advocated for wider use of FDA Drug Approval Packages?

Yes — here are several references:

• Bennett DA, Jull A. FDA: untapped source of unpublished trials. The Lancet 2003;361:1402–3. doi:10.1016/S0140-6736(03)13147-9
• Lurie P, Zieve A. Sometimes the silence can be like the thunder: access to pharmaceutical data at the FDA. Law & Contemp Probs 2006;68:85
• O’Connor AB. The need for improved access to FDA reviews. JAMA 2009;302:191–3. doi:10.1001/jama.2009.973
• Cook GE, Madden MM. The Significance of Information Posted on the U.S. Food and Drug Administration’s Website. J Manag Care Pharm 2012;19:72–3
• Marchand HC, Ros BJ, Fine AM, et al. The U.S. Food and Drug Administration: drug information resource for formulary recommendations. J Manag Care Pharm 2012;18:713–8
• Navarro RP. Rediscovering FDA websites. J Manag Care Pharm 2012;18:719–20
• McDonagh MS, Peterson K, Balshem H, et al. US Food and Drug Administration documents can provide unpublished evidence relevant to systematic reviews. J Clin Epidemiol Published Online First: 12 July 2013. doi:10.1016/j.jclinepi.2013.05.006
• Schroll JB, Abdel-Sattar M, Bero L. The Food and Drug Administration reports provided more data but were more difficult to use than the European Medicines Agency reports. J Clin Epidemiol Published Online First: 19 August 2014. doi:10.1016/j.jclinepi.2014.06.019

There are several detailed points that we could not get into here, and other questions will surely arise, so we welcome questions and comments via [email protected]. We plan to follow up with blog pieces on FAQ and/or troubleshooting in the near future through fda.opentrials.net. The OpenTrialsFDA team will present the final prototype in early December at the Open Science Prize Showcase event.

The Food and Drug Administration (FDA) publishes DAPs as part of the general information on drugs via its data portal known as Drugs@FDA. These documents contain detailed information about the methods and results of clinical trials, and are unbiased, compared to reports of clinical trials in academic journals. This is because FDA reviewers require adherence to the outcomes and analytic methods prespecified in the original trial protocols, so, in contrast to most journal editors, they are unforgiving of practices such as post hoc switching of outcomes and changes to the planned statistical analyses. These review packages also often report on clinical trials that have never been published.

However, despite their high value, these FDA documents are notoriously difficult to access, aggregate, and search. The website itself is not easy to navigate, and much of the information is stored in PDFs or non-searchable image files for older drugs. As a consequence, they are rarely used by clinicians and researchers. OpenTrialsFDA will work on improving this situation, so that valuable information that is currently hidden away can be discovered, presented, and used to properly inform evidence-based treatment decisions.

The team has started to scrape the FDA website, extracting the relevant information from the PDFs through a process of OCR (optical character recognition). A new OpenTrialsFDA interface will be developed to explore and discover the FDA data. In addition, the information will be integrated into the OpenTrials database, so that for any trial for which a match exists, users can see the corresponding FDA data.

We will be sharing future progress through this blog as the work develops: the final prototype will be presented in early December at the Open Science Prize Showcase.

Contact: [email protected]
@opentrials

More information about the Open Science Prize: https://www.openscienceprize.org/res/p/finalists/