It’s been a very interesting journey over the past couple of years and we’re really proud of what we’ve achieved with a small team in a short amount of time. This project couldn’t have got as far as it has without many of you, who’ve helped in a number of ways, from pledging assistance, helping with user testing or giving us feedback at conferences, and those of you we’ve met to discuss aspects of the project. And of course those individuals and organisations who have offered/donated data to the project. So, thank you – we’re making OpenTrials for you, and want to make it as useful as possible, so all your contributions have been invaluable.

What follows is a summary of what we’ve built, events we’ve been involved in, some challenges and successes of the project so far, how to get involved in the meantime, and lastly some interesting articles we’ve been reading recently.

What have we built?

OpenTrials

Based on Ben Goldacre and Jonathan Gray’s 2016 paper outlining the need for a linked database of clinical trials (and related documents and data), we’ve built an early (beta) version of OpenTrials, bringing together data from multiple clinical trial registries, deduplicated those trials, automatically matched them to publications, and integrated third-party datasets. We’ve also implemented a contribution feature, meaning that users can submit a wide range of documents and data relating to trials, further enhancing the potential information available.

Here’s a summary of what data we currently have in OpenTrials:

351,851 trials from five clinical trial registries:

• ClinicalTrials.gov
• EMA EUCTR
• WHO ICTRP
• ISRCTN (new)
• GSK (new)

Data integrations/linkages:

• PubMed – journal articles (example)
• Cochrane Schizophrenia Group – Risk of Bias data (example)
• Food and Drug Administration (FDA) – drug approval documents (example)
• Health Research Authority – lay summaries (example)

From a technical perspective, we’ve created an Application Programming Interface (API) which allows programmers, data scientists, and others who want to use the data in their own tools, research, or analyses to make live queries against the site using code. Alternatively, for those who want to play with the entire database, we’ve just made our database dumps available for download.

OpenTrialsFDA

Working with Drs. Erick Turner and Ben Goldacre, we were selected as one of six finalists of the Open Science Prize (ultimately being placed as a runner-up) to build a prototype solution to make the information contained in the FDA’s Drug Approval Packages more easily accessible and searchable. The result can be found at fda.opentrials.net and enables full text searching of over 55,000 FDA drug approval documents – something that was not possible before. For more information on OpenTrialsFDA here’s a more detailed blog post + 3min video.

OpenTrials walkthrough

If you’ve not seen it yet, here’s a 10min walkthrough of the site from Jan 2017 – you’ll notice we’ve added some more features since then, but it’s a helpful overview:

Events

Over the past year the OpenTrials team has presented at Evidence Live in Oxford (June 2016), the International Open Data Conference in Madrid (October 2016), the OpenTrials hack day in Berlin (October 2016), the World Health Summit in Berlin (October 2016), the Cochrane Colloquium in Seoul (October 2016), Bioinformatics Meetup in London (December 2016), Clinical Innovation & Partnering World in London (March 2017), and the Cochrane UK & Ireland Symposium in Oxford (March 2017). It’s been great to meet so many people who are as excited as we are about what OpenTrials is trying to achieve; we’ve had some great conversations with users and potential data partners, leading to some interesting ideas for collaborations and development in the next phase of the project.

We’ve also run a number of user testing sessions with domain experts, including medical librarians – this has been invaluable. Thank you again to all those who volunteered their time to help us understand what works, what doesn’t, what’s not obvious, and what features would be useful – we’ve added everything you’ve pointed out to our GitHub issues – have a look, and feel free to get involved in the conversations (or contribute code if you’re more technical).

Challenges and successes

Website layout/content changing, causing data scrapers to fail

The majority of the data we currently import is using scrapers, grabbing the data directly from a source’s website. This relies on two things:

• that the order/location of the data stays in the same place (i.e. the layout/design of the website does not change)
• the structure of the data itself stays the same (e.g. one data source changed the text it used to represent both sexes from ‘both’ to ‘all’)

If either or both of these change, our scraper can no longer retrieve data until it is rewritten to accommodate the changes. An example which has impacted us is the Drugs@FDA website which we’ve used as the data source for our OpenTrialsFDA project. The site was redesigned after our initial scrape, meaning that to keep the documents on OpenTrialsFDA up-to-date our scraper needed rewriting.

Suggestion: we encourage those offering searchable databases on their website to also provide the option to retrieve the data via either an API or bulk download (preferred).

Data heterogeneity

When combining or grouping data from multiple sources, we’ve encountered issues where the same elements are represented in different ways. This is due to a combination of sources allowing free-text input and not using standards.

A good example of this is geographical location – for instance ‘United Kingdom’ may be entered as ‘United Kingdom’, ‘Great Britain’, ‘UK’, or ‘GB’.

The effect of this on the project is that we’ve spent a lot of time processing and normalising data to make it more usable, and the task is ongoing and potentially unending.

Suggestion: trial registries should use known standards for metadata – in the case of countries, ISO country codes, and for fields such as condition names, a controlled vocabulary such as MeSH.

N.B. In Phase 2 we plan to deploy a controlled vocabulary/ontology such as MeSH or SNOMED CT.

Licensing

In order for us to use a dataset in OpenTrials it must be offered with a suitably permissive licence. Ideally, a dataset would be licenced as open data, meaning that the data can be “freely used, modified, and shared by anyone for any purpose”, for example under a Creative Commons licence such as Attribution 4.0 International (CC BY 4.0) or even better as a Public Domain Dedication (CC0).

Currently, the majority of datasets we see are not usable due to restrictive terms and conditions on their websites. This may be due to organisations using boilerplate terms and conditions with built in restrictions, erring on the side of perceived risk, or wanting to protect information they perceive as having value as intellectual property to the organisation.

Over the past six months, with the help of an intellectual property lawyer, we’ve been in discussions with a number of organisations which have restrictive data licenses. We’ve explained how we’d like to use their data on OpenTrials, how their current licence prevents that use, and how different parts of the licence (e.g. non-commercial, personal use only, no redistribution) are problematic/ambiguous, and have suggested more open, permissive alternatives.

Suggestion: if you’re a data provider, we’d encourage you to follow the example of one of the organisations below and make your data licences more open – we’re happy to talk to you about the issues: [email protected]

N.B. We’re planning a detailed blog post about licensing – watch this space!

Licensing successes

We’re pleased to announce that two organisations (ISRCTN and GSK) have already changed their terms and conditions to allow far greater use of their data.

In the case of ISRCTN, this covers their trial metadata (e.g. condition, intervention, trial title, phase etc) under a CC BY 4.0 licence. In the case of GSK, this covers both their trial metadata and their collection of documents relating to trials (Protocol Summaries, Scientific Results Summaries, Protocols, and Clinical Study Reports).

We’ve just added these new sources to OpenTrials, meaning that even more trials and documents are now listed – many thanks to both organisations for showing great leadership on this issue!

Commenting on this progress, Andrew Freeman, Head of Medical Policy at GSK said:

“We can and do publicly disclose information about our clinical trials on our register. Disclosure is important but it’s not enough. The value of disclosing information can be significantly limited if the information is not readily accessible and usable. To that end, we recently clarified that the use of information on our register is unrestricted provided that it may not be used in applications by others for regulatory approval of a product.”

How to get involved in the meantime

If you’re keen to get involved in discussions with other OpenTrials users and open data fans, there are a number of ways you can do that while our core team is having a rest.

If you’re interested in looking at the bugs and feature requests we have (or want to file new ones!), take a look at our GitHub issues and feel free to comment on any with your insights. We also have a forum where you can discuss issues relating to OpenTrials, for instance if you know of a data source we might be interested in using, or a way of improving matches or cleaning the data. For the more technical amongst you, feel free to also contribute code and get involved in our Gitter chat room.

We’d also like to hear about any problems you have with the OpenTrials explorer and OpenTrialsFDA – use the ‘Flag an error’ link at the bottom of any page.

And lastly, use our API and/or database downloads to create tools, visualisations, and analyses + let us know what you’ve made!

What we’ve been reading

And lastly, as it’s going to be a while until we’re in touch again, here’s a bumper crop of articles from the last few months:

Trial data transparency

• Journal editor calls for ‘culture change’ around clinical trial data
• EMA: Release of documents on two medicines temporarily halted
• Enhancing Transparency at the US Food and Drug Administration: Moving Beyond the 21st Century Cures Act
• Data Escrow and Clinical Trial Transparency

Technical

• Systematic drug repositioning through mining adverse event data in ClinicalTrials.gov
• Improving data transparency in clinical trials using blockchain smart contracts
• A metadata schema for data objects in clinical research

Reporting bias

• Bias in the reporting of harms in clinical trials of second-generation antidepressants for depression and anxiety: A meta-analysis
• Spin in RCTs of anxiety medication with a positive primary outcome: a comparison of concerns expressed by the US FDA and in the published literature

Other

• Why clinical trial outcomes fail to translate into benefits for patients
• An open source pharma roadmap
• Storm Warning: The Trial Disclosure Forecast for 2017

As always, for the latest updates make sure you’re subscribed to our newsletter and follow us on Twitter @opentrials

Signing out for now from the OpenTrials team at Open Knowledge International – see you on the other side!

After reaching the final three of the Open Science Prize, and being judged by an expert panel, we can announce that our project, OpenTrialsFDA was placed as a runner-up alongside MyGene2, with Real-Time Evolutionary Tracking for Pathogen Surveillance and Epidemiological Investigation being declared the winner of the grand prize of $230,000. As you may know, OpenTrialsFDA is designed to make clinical trial data from the Food and Drug Administration (FDA) more easily accessible and searchable. Congratulations to all the finalists – it’s been great to see such a range of projects dedicated to innovating and advancing areas of biomedicine and health through open science, content, and data.

Although OpenTrialsFDA didn’t win the grand prize, we were very grateful for all the public votes which took us from the original shortlist of six to the final three. A number of the public also chose to give comments along with their vote – there were many fantastic ones and we wanted to share a selection:

“I made OpenTrialsFDA my first choice because patients have suffered and died because information from research which should be available to health professionals and patients has not been accessible. This project addresses this important problem.”

“I am particularly keen on projects that get data out in there open where it can be subjected to independent analysis.”

“As a professional technologist, I have found the APIs produced by the folks at OpenTrials are of extremely high quality; making integrating with and using the data straightforward yet powerful. If all open science projects had as technically competent and sophisticated people involved then huge opportunities for knowledge growth would open up to us all very quickly.”

“OpenTrialsFDA project will be relevant to a wide range of clinical research fields, with important potential impacts on patients, clinicians, researchers, and health decision makers.”

“It’s great to see a clinical science entry in the Open Science Award. Easier access to the full FDA data for approved indications for approved drugs will assist clinical practitioners, biomedical scientists and educated citizens in making more informed choices on medical and research choices.”

“Improving transparency of clinical data will literally save and transform lives”

“Drugs affect peoples’ lives and all too frequently patients and physicians do not have the data and evidence they need to make truly informed decisions about treatments. Access to data on effectiveness and harms is becoming even more critical with rapidly rising drug prices. Shining a light on what we know and don’t know about treatments will hopefully move us in a direction of making better individual and population-based decisions about treatments.”

Thanks so much for all the kind words – it means a lot to the entire team and we strongly believe in the power of the project to facilitate positive change!

What’s next for OpenTrialsFDA?

The search engine will continue to exist at fda.opentrials.net – it currently indexes over 55,000 FDA approval documents and where possible links to clinical trials on OpenTrials (a few examples). While development on OpenTrialsFDA has stopped for the moment (we’re focusing on the main OpenTrials explorer) there are a number of ways you can help support the project:

• Help get access to important drug information by asking the FDA for specific documents
• Help improve the code that runs OpenTrialsFDA – all the code is in our public repository
  • We need help updating our FDA collector – details here
• Financial support – get in touch if you’d like to talk to us about a grant to continue improving OpenTrialsFDA
• Support in kind – are you working on a similar project and/or have expertise in processing unstructured documents/data (e.g. NLP) to extract structured data? Please get in touch

On 9th March, our Community Manager,  Ben Meghreblian, will participate in two sessions at Clinical Innovation and Partnering World in London. The event is a great opportunity for OpenTrials as it brings together leading experts in the fields of clinical innovation, outsourcing, alliance management and strategic partnering, with attendees from big pharma, biotechnology/technology firms, and contract research organisations (CROs). The focus of the event is on disruptors in clinical outsourcing and innovation, so we’re pleased to have been invited.

Ben will be giving a plenary talk, introducing attendees to OpenTrials, and will also run a roundtable titled ‘Aligning external with internal values’ which will explore some of the issues relating to clinical data sharing, including benefits to patients and researchers, concerns, solutions, and licensing issues (session details). We’re really looking forward to having some useful discussions, hearing industry views, and demonstrating the power of combining datasets in OpenTrials.

After presenting OpenTrials at the annual Cochrane Colloquium in Seoul last year, we’ll be giving an updated talk at the Cochrane UK & Ireland Symposium in Oxford on 15th March (session details). One of our researchers, Jessica Fleminger, will join Ben Meghreblian in giving an overview of the project, the latest progress, and ask for your feedback on the OpenTrials explorer.

We encourage anyone coming along (hello systematic reviewers!) to bring a laptop and make yourself familiar with OpenTrials so that we can have some fruitful discussions about what’s good, what’s missing/broken, and generally how we can make it a better tool for users.

User testing – medical librarians

We’ll also be conducting some user testing sessions with medical librarians, so if that’s you and you’re interested in helping us improve OpenTrials, fill in this form to let us know.

As always, if you have feedback about OpenTrials you can email us your thoughts, or file a GitHub issue if you’re familiar with it. We’ll keep you updated with any developments on the blog, but please make sure you subscribe to our newsletter and follow us on Twitter!

The OpenTrialsFDA prototype makes clinical trial data from the FDA more easily accessible and searchable through a user-friendly web interface that you can now explore through fda.opentrials.net. Winning the prize would enable us to develop this prototype further in 2017, improving matching documents to corresponding trials on OpenTrials, along with extracting other valuable information from them. In the meantime, you can help us to get more FDA data on clinical trials available by rescuing ‘missing-in-action’ drug approval packages.

How to rescue “MIA” Drug Approval Packages

When OpenTrialsFDA carried out an automated download of all documents posted on Drugs@FDA, this not only captured Drug Approval Packages (DAPs), but also letters written by the FDA notifying drug companies that their drug had been approved for a new indication. Interestingly, this allowed us to identify drug-indication combinations for which DAPs were absent. Nevertheless, those DAPs do exist, and the FDA is obliged to make them available via the Freedom of Information Act (FOIA), for example in response to a FOI request.

In this blog by Dr. Erick Turner you can find out more on which DAPs are missing and how to make a FOIA request to the FDA to get them online. If and when you do make a FOIA request, please let us know by completing this form so that we can keep track of what DAPs have been requested.

Many thanks to everyone for their support during the voting process!

Nevertheless, those DAPs do exist, and the FDA is obliged to make them available via the Freedom of Information Act (FOIA), if not electronically, then in response to a FOI request. In this blog we explain how you can help to get this missing information online.

The OpenTrialsFDA project began with an automated download of all documents posted on Drugs@FDA. The download captured not only DAPs, but also letters written by the FDA notifying drug companies that their drug had been approved for a new indication, allowing us to identify drug-indication combinations for which DAPs were absent. A list of these missing-in-action (MIA) DAPs is shown below and on this page.

Our invitation to you

If you, the reader, are in possession of any of these “MIA” DAPs, we invite you to send them to us so that we can make them available to the research community and others via OpenTrialsFDA.

If there is a DAP you do not have but think it should be shared publicly, please check Drugs@FDA and verify that our automated process did not somehow miss it. You may use the procedure in this BMJ article², though be aware that the FDA recently made slight modifications to the procedure.

Making a FOIA request

Once you have verified that the DAP of interest is indeed not posted on Drugs@FDA, please consider making a FOIA request for it. The usual way to place a FOIA request is via the FDA’s FOI website. We invite you to use this template. Your request should contain the following information (part of which can be found in our list above):

1. Drug brand name
2. Drug generic name
3. Indication approved (consider consulting the product labeling to verify that this is an FDA-approved, not an off-label, indication/use)
4. Date of approval for that indication, aka “Action Date” (often this information can be obtained via Google, but you we suggest you confirm this by looking at that drug’s approval history on Drugs@FDA, where you should find a letter of that date documenting the approval)
5. Types of reviews requested (e.g. medical, statistical)

Just be aware that it may take several months, perhaps more than a year, for the FDA to complete the request. A major reason for this delay is (a) the FDA’s FOIA office is chronically short-staffed (as noted above) and (b) they must go through the DAP and identify and redact any information they deem to be “commercial confidential” or “trade secrets”, as they are exempt from disclosure.

Making a FOIA request for previously released information (possible shortcut)

Before making such a FOIA request, we want to make you aware of a possible shortcut—a request for previously released information. That is, perhaps someone else has recently requested the same information and obtained it, in which case the FOIA office has already completed the time-consuming redaction process, which means there is no need for them to do it all over again. Therefore, in this type of FOIA request, you essentially say to the FDA’s FOIA office, “Whatever you gave to that previous requester, I’d like that, too.”

So now you might ask, how do you know whether the DAP you’re after has, in fact, been requested and released? FOIA case logs! A case log lists the completed FOIA requests, who made the request, the information requested (e.g. drug name, review types), and—most importantly—the request number (“control #”).

And how do you get a case log? You make a FOIA request for it! Even easier, feel free to search the one Dr Turner obtained recently, which covers the period January 2012 – September 2016.

This process might well sound circular, but it can save lots of time, because requests for previously released information often get completed within weeks rather than months or longer.

Tracking your contribution

If and when you do make a FOIA request, please let us know by completing this form so that we can keep track of what DAPs have been requested. Please also keep us informed on what the FDA’s response was (usually some variation of “we’ll get to it and send you the information when it’s ready”).

Please vote!

Finally, if you think OpenTrialsFDA is an important contribution to the public health, and if you’d like to support our effort to pry open this vault of priceless clinical trial data, please help us win the Open Science Prize by casting your vote before 6 January 2017. Thank you!

References

1. Turner EH. A taxpayer-funded clinical trials registry and results database. PLoS Med. 2004;1(3):e60. doi:10.1371/journal.pmed.0010060.
2. Turner EH. How to access and process FDA drug approval packages for use in research. BMJ. 2013;347(oct14 2):f5992-f5992. doi:10.1136/bmj.f5992.

Firstly, we were invited to talk to the members of the London Containing Bioinformatics & Data Analytics Meetup – they’re a mix of bio/health/medical informaticians, machine learners, software developers, engineers, and data scientists, so it was a great opportunity to talk about OpenTrials to a slightly different crowd from usual. There were lots of interesting questions, some potential offers of help from those who supervise students, and the conversations continued long after the session ended. Thanks to Paul A for organising and Ben vZ for hosting!

Secondly, we took part in a Data Journalism Hack Day at King’s College London (KCL), in collaboration with Open Knowledge International and SoBigData. The day was organised for students of the KCL Data Journalism Course, and saw them working in groups on issues from international taxation, clinical trials, human rights violations, and natural resource extraction in the Global South. The students had already extracted data from various sources and their aim for the day was to work with data and domain experts to build a story around the data which they would subsequently write up.

The students working with the OpenTrials data had previously decided to focus on migraine trials and had already used the OpenTrials API to extract the relevant trials, importing them into Excel where they spent most of their time cleaning the data, analysing it, and visualising it.

It was an enjoyable day helping the students better understand how clinical trials work, what sort of issues may be worth considering for their story, and how the data can inform it. It was also interesting to see the OpenTrials data being used in a real, hands-on way, how powerful it can be to answer specific questions, along with challenges and limitations relating to missing data from some registries and non-normalised company names – issues we are aware of and want to address.

We’re looking forward to reading the finished clinical trials ‘exclusive’ by the students – meanwhile we don’t want to steal their journalistic thunder, so we won’t post any of the cool visualisations they’ve created yet, but once the story is live, we’ll link to it here.

Our talk covered some of the problems with the information architecture of evidence-based medicine, how OpenTrials aims to help fix them, a technical overview of the platform, how we import data, licensing issues, user examples, and a demo – here are the slides:

Along with some good questions and suggestions for OpenTrials, we also had a number of meetings, both planned and spontaneous, to discuss potential collaborations to improve OpenTrials functionality and integrate others’ data into our system.

A great example of this is the Risk of Bias data which the Cochrane Schizophrenia Group previously kindly gave to us (thanks a lot!). This is structured data produced by researchers, who have graded schizophrenia trials on issues such as blinding and selective reporting. Here are some examples of how we’ve integrated this data onto individual trial pages (hint: have a look at the ‘Methodological rigour’ section at the bottom of each page).

As always, if you have feedback about OpenTrials you can email us your thoughts, or file a GitHub issue if you’re familiar with it. We’ll keep you updated with any developments on the blog, but please make sure you subscribe to our newsletter and follow us on Twitter!

OpenTrialsFDA will be featured in the OHSU (Oregon Health & Science University) Open Science Data Jamboree & Hackathon. The event will focus on the power of open content and data to advance biomedical research and human health.  Dr. Erick Turner, Associate Professor at OHSU and Staff Psychiatrist at the Portland VA Medical Center, will introduce OpenTrialsFDA, which is a finalist for the highly competitive Open Science Prize (voting for the finalists is open until 6 January 2017 here).

OpenTrialsFDA aims to increase access to and the utility of FDA drug approval packages. These review packages often contain information on clinical trials that have never been published.  When these “inconvenient truths” are integrated into a systematic review or meta-analysis, it can sometimes dramatically shift a drug’s risk-benefit ratio. Although the FDA makes the reviews available via its web portal Drugs@FDA, the documents are notoriously difficult to search and aggregate. As a consequence, they are rarely used by clinicians and researchers, despite their immense value.  OpenTrialsFDA intends to change this by allowing researchers, clinicians, the public, developers, and third party platforms to access, search, and build upon this data.

Dr. Turner’s keynote talk will be followed by an overview of the NIH’s new clinical trial data sharing requirements and two concurrent hands-on sessions:

• The first will be a deep dive on searching the OpenTrialsFDA website, understanding the contents of an FDA drug approval package (including the statistical content), and how to use the data for different use cases, such as meta analysis and systematic reviews.  Dr. Turner will team teach the breakout with an OHSU biostatistician.
• The second will be a OpenTrialsFDA hackathon using the OpenTrials API.  Attendees will have the opportunity to use their research, development, clinical, and visualization expertise (to name a few) to explore, present, and build upon the OpenTrials data and the OpenTrialsFDA application.  Please note, no technical expertise is required, as there will be multiple ways of contributing.

The Data Jamboree is presented by Computational Biology and the OHSU Library.  All are welcome to attend, including OHSU faculty, students, and staff as well as Portland area science enthusiasts and developers!

• Date: (to be announced)
• Time 2:30 – 5:00 PM (hackathon will continue past 5:00)
• Location: Collaborative Life Sciences Building (CLSB), 3A003, OHSU, Portland, Oregon
• Food & Drink provided

Contact:  Robin Champieux, Scholarly Communication Librarian, [email protected]

OpenTrialsFDA is a collaboration between Dr. Erick Turner (a psychiatrist-researcher and transparency advocate), Dr. Ben Goldacre (Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford) and the team behind OpenTrials at Open Knowledge International.  

OpenTrialsFDA works on making clinical trial data from the FDA (the US Food and Drug Administration) more easily accessible and searchable. Until now, this information has been hidden in the user-unfriendly Drug Approval Packages that the FDA publishes via its dataportal Drugs@FDA. These are often just images of pages, so you cannot even search for a text phrase in them. OpenTrialsFDA scrapes all the relevant data and documents from the FDA documents, runs Optical Character Recognition across all documents, links this information to other clinical trial data, and now presents it through a new user-friendly web interface at fda.opentrials.net.

Explore the OpenTrialsFDA search interface

Any user can type in a drug name, and see all the places where this drug is mentioned in an FDA document. Users can also access, search and present this information through the application programming interfaces (APIs) the team will produce. In addition, the information has been integrated into the OpenTrials database, so that the FDA reports are linked to reports from other sources, such as ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed.

The prototype will provide the academic research world with important information on clinical trials in general, improving the quality of research, and helping evidence-based treatment decisions to be properly informed. Interestingly, the FDA data is unbiased, compared to reports of clinical trials in academic journals. Dr. Erick Turner explains: “With journal articles everything takes place after a study has finished, but with FDA reviews, there is a protocol that is submitted to the FDA before the study has even started. So the FDA learns first of all that the study is to be done, which means it can’t be hidden later. Secondly it learns all the little details, methodological details about how the study is going to be done and how it is going to be analyzed, and that guards against outcome switching.”

Dr Ben Goldacre says: “These FDA documents are hugely valuable, but at the moment they’re hardly ever used. That’s because – although they’re publicly accessible in the most literal sense of that phrase – they are almost impossible to search, and navigate. We are working to make this data accessible, so that it has the impact it deserves.”

Voting for the Open Science Prize finalists is possible through http://event.capconcorp.com/wp/osp: more information on OpenTrialsFDA is available from fda.opentrials.net/about and from the team’s video below.

Editor’s notes

Dr. Ben Goldacre
Ben is a doctor, academic, writer, and broadcaster, and currently a Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford. His blog is at www.badscience.net and he is @bengoldacre on twitter. Read more here. His academic and policy work is in epidemiology and evidence based medicine, where he works on various problems including variation in care, better uses of routinely collected electronic health data, access to clinical trial data, efficient trial design, and retracted papers. In policy work, he co-authored this influential Cabinet Office paper, advocating for randomised trials in government, and setting out mechanisms to drive this forwards. He is the co-founder of the AllTrials campaign. He engages with policy makers. Alongside this he also works in public engagement, writing and broadcasting for a general audience on problems in evidence based medicine. His books have sold over 600,000 copies.

Dr. Erick Turner
Dr. Erick Turner is a psychiatrist-researcher and transparency advocate. Following a clinical research fellowship at the NIH, he worked for the US Food and Drug Administration (FDA), acting as gatekeeper for new psychotropic drugs seeking to enter the US market. In 2004 he published a paper drawing researchers’ attention to the Drugs@FDA website as a valuable but underutilized source of unbiased clinical trial data. Dissatisfied with the continuing underutilization of Drugs@FDA, he published a paper in the BMJ in order to encourage wider use of this trove of clinical trial data.

Open Knowledge International
https://okfn.org   
Open Knowledge International is a global non-profit organisation focussing on realising open data’s value to society by helping civil society groups access and use data to take action on social problems. Open Knowledge International addresses this in three steps: 1) we show the value of open data for the work of civil society organizations; 2) we provide organisations with the tools and skills to effectively use open data; and 3) we make government information systems responsive to civil society.

Open Science Prize
https://www.openscienceprize.org/res/p/finalists/
The Open Science Prize  is a collaboration between the National Institutes of Health and the Wellcome Trust, with additional funding provided by the Howard Hughes Medical Institute of Chevy Chase, Maryland.  The Open Data Science Symposium will feature discussions with the leaders in big data, open science, and biomedical research while also showcasing the finalists of the Open Data Science Prize, a worldwide competition to harness the innovative power of open data.

General questions

Background

The OpenTrialsFDA team is one of the six finalists for the Open Science Prize: a global science competition to make both the outputs from science and the research process broadly accessible to the public. This is a collaboration between Dr. Erick Turner, Dr. Ben Goldacre and the OpenTrials team at Open Knowledge International. Competing for the $230,000 prize that will be awarded to the winner in February 2017, OpenTrialsFDA is working on a prototype that will make the information hidden in the user-unfriendly drug approval packages of the FDA (the US Food and Drug Administration) more easily accessible and searchable, and link these to documents and data related to clinical trials.

The work is closely related to the OpenTrials project, which aims to locate, match, and share all publicly accessible data and documents, on all trials conducted, on all medicines and other treatments, globally. OpenTrials is aggregating this information from a wide variety of existing sources and aims to provide a comprehensive picture of the data and documents on clinical trials conducted on medicines and other treatments around the world.

What is the scope of OpenTrialsFDA?

OpenTrialsFDA works on making clinical trial data from the FDA (the US Food and Drug Administration) more easily accessible and searchable. Until now, this information has been hidden in the user-unfriendly Drug Approval Packages that the FDA publishes via its dataportal Drugs@FDA. These documents are notoriously difficult to access, aggregate, and search, since the Drugs@FDA website itself is not intuitive to navigate, and the reviews are stored as individual PDFs which, except for relatively new drugs, cannot be searched.  OpenTrialsFDA will extract the relevant data from the FDA documents, link it to other clinical trial data and present it through a new user-friendly web interface. Any user will be able to type in a drug name, and see all the places where this drug is mentioned in an FDA document. Users will also be able to access, search and present this information through the application programming interfaces (APIs) the team will produce.

Who is behind OpenTrialsFDA?

OpenTrialsFDA is a collaboration between Dr. Erick Turner (a psychiatrist-researcher and transparency advocate), Dr. Ben Goldacre (Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford) and the team behind OpenTrials at Open Knowledge International.  

What is the Drugs@FDA portal?

Drugs@FDA is a searchable catalog of the US Food and Drug Administration (FDA) approved drug products, both prescription and over the counter, with links to documents relating to marketing approval. More information is available from their FAQ section.

What are FDA drug approval packages?

FDA Drug Approval Packages consist of FDA employees’ reviews of new drug applications (NDAs) submitted by pharmaceutical companies seeking approval to market their drugs in the US. These NDAs consist primarily of clinical study reports (CSRs). Some drugs have more than one approval package, because approval is granted not for a drug per se, but for each drug-indication combination.

How will OpenTrialsFDA improve access to the FDA data?

The team will scrape the FDA website and extract the relevant information from the PDFs through a process of OCR (optical character recognition). Through the new OpenTrialsFDA interface, users will be able to explore and discover the FDA data. In addition, the information will be integrated into the OpenTrials database, so that the FDA report can be linked to reports from other sources, such as ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed.

How does the FDA review process differ from the peer review process used for manuscripts submitted to journals?

The key is the before-versus-after-the-trial aspect. Before the sponsor can begin a trial in the US, it must submit the trial protocol to the FDA. By means of this “pre-trial” review of the protocol, the FDA learns (1) that that trial is to be conducted and (2) the nitty-gritty methodological details. That way, a few years later, when the sponsor submits its NDA, the reviewer compares the clinical study report (CSR) to the original protocol to determine (1) whether any studies have been omitted and (2) whether any outcomes have been switched. By contrast, in the world of peer-reviewed journal articles, there is no “pre-trial” review, allowing both (1) and (2) to take place.

Who authors FDA reviews?

The FDA review team is multidisciplinary, yielding different types of reviews. Medical reviews are usually authored by physicians who summarize and evaluates the data on drug efficacy and safety. Statistical reviews are carried out by statisticians who test whether they can replicate the sponsor’s results using the patient-level data submitted electronically. There are other review disciplines, e.g. chemistry, pharmacology-toxicology, whose reviews pertain primarily to the preclinical phase and early clinical (human) phases of drug development.

What is the Open Science Prize?

The Open Science Prize is a collaboration between the Wellcome Trust, the US National Institutes of Health (NIH) and the Howard Hughes Medical Institute. Its goal is to unleash the power of open content and data to advance biomedical research and its application for health benefit. The OpenTrialsFDA team is one of the six finalist teams that were selected in May 2016. All teams will showcase their prototypes at the BD2K Open Data Science Symposium on 1 December 2016, when public voting will begin. The public is asked to help select the most promising, innovative and impactful prototypes from among the six finalists – among which one will receive the grand prize of $230,000.  

Troubleshooting

In case you’re having difficulty finding the right information in the Drugs@FDA portal, here are some troubleshooting questions to help you:

Was the drug approved before or after 1997?
The FDA started posting reviews (Drug Approval Packages) in 1997. If the drug in question was approved in 1996 or earlier, you probably will not find it posted.

Are you sure that the use you’re interested in is not off-label, ie. that the FDA approved it for the indication you’re interested in?
Oftentimes drugs are used widely for a certain condition (indication) for which the FDA has not granted approval (off-label use). If you check the product label (e.g. at https://dailymed.nlm.nih.gov/dailymed/index.cfm) and look within the section on “Indications and Usage”, you will see a list of the approved indications.

Are you looking for indication #1 or one of the subsequent indications?
A given drug might be FDA-approved for multiple indications. The FDA is careful about posting the Drug Approval Package for the first indication (the one that allowed that drug to enter the market), but for subsequent indications (#2, #3, etc.), it’s more hit-or-miss.

If you are motivated, one way to get such reviews is to file a Freedom of Information request here; but be warned—the FDA might take months or longer to fulfill your request. If you do get such reviews, we would appreciate your sharing it with us so we can add to OpenTrialsFDA and make it readily available to others.

Did you use the generic name or brand name?
If you use the generic name, you will probably get lots of hits: one for when the molecule was first being introduced to the market plus hits for subsequent generic versions. The FDA does its review for efficacy and safety when the molecule is first introduced to the market. Years later, when the patent expires, generic equivalents get approved, but what the FDA cares about at this stage is bioequivalence (in terms of blood levels) to the original brand name version. Thus generics do not have to demonstrate efficacy and safety all over again, which is why these approval packages generally contain little or no such data.

Is the review an NDA or an ANDA?
Please see the question and answer immediately above. NDAs (new drug applications) correspond to brand name versions, while ANDAs (abbreviated new drug applications) correspond to generic equivalents. They are abbreviated because less is required of the sponsor (see above). Assuming you are interested in drug efficacy and/or safety (and not bioequivalence), you will want to focus on NDAs and ignore ANDAs.

Is this the original or a subsequent formulation of a drug?
This is similar to the issue brand name versions vs. generic versions (see above). The burden of proof regarding efficacy and safety is highest when a molecule is first entering the market. A given drug will often enter the market as an immediate release (IR) formulation, before which the FDA will do its most comprehensive review of drug efficacy and safety. Some years later, the sponsor may seek approval for a sustained-release (SR) formulation, but by that time, millions of patients have already been exposed to the IR formulation. Because it is relatively unlikely that the efficacy and/or safety profile will differ a great deal between the IR and the SR, the SR may have to do fewer clinical trials compared to its IR version.

Want to get involved? Have further questions?

Please email us at [email protected]. This email is sent to Emma Beer, the Project Manager for OpenTrials and Ben Goldacre, its Principal Investigator.  

For further project updates, check the OpenTrialsFDA page at https://fda.opentrials.net or follow us on Twitter: @opentrials