On 9th March, our Community Manager,  Ben Meghreblian, will participate in two sessions at Clinical Innovation and Partnering World in London. The event is a great opportunity for OpenTrials as it brings together leading experts in the fields of clinical innovation, outsourcing, alliance management and strategic partnering, with attendees from big pharma, biotechnology/technology firms, and contract research organisations (CROs). The focus of the event is on disruptors in clinical outsourcing and innovation, so we’re pleased to have been invited.

Ben will be giving a plenary talk, introducing attendees to OpenTrials, and will also run a roundtable titled ‘Aligning external with internal values’ which will explore some of the issues relating to clinical data sharing, including benefits to patients and researchers, concerns, solutions, and licensing issues (session details). We’re really looking forward to having some useful discussions, hearing industry views, and demonstrating the power of combining datasets in OpenTrials.

After presenting OpenTrials at the annual Cochrane Colloquium in Seoul last year, we’ll be giving an updated talk at the Cochrane UK & Ireland Symposium in Oxford on 15th March (session details). One of our researchers, Jessica Fleminger, will join Ben Meghreblian in giving an overview of the project, the latest progress, and ask for your feedback on the OpenTrials explorer.

We encourage anyone coming along (hello systematic reviewers!) to bring a laptop and make yourself familiar with OpenTrials so that we can have some fruitful discussions about what’s good, what’s missing/broken, and generally how we can make it a better tool for users.

User testing – medical librarians

We’ll also be conducting some user testing sessions with medical librarians, so if that’s you and you’re interested in helping us improve OpenTrials, fill in this form to let us know.

As always, if you have feedback about OpenTrials you can email us your thoughts, or file a GitHub issue if you’re familiar with it. We’ll keep you updated with any developments on the blog, but please make sure you subscribe to our newsletter and follow us on Twitter!

Firstly, we were invited to talk to the members of the London Containing Bioinformatics & Data Analytics Meetup – they’re a mix of bio/health/medical informaticians, machine learners, software developers, engineers, and data scientists, so it was a great opportunity to talk about OpenTrials to a slightly different crowd from usual. There were lots of interesting questions, some potential offers of help from those who supervise students, and the conversations continued long after the session ended. Thanks to Paul A for organising and Ben vZ for hosting!

Secondly, we took part in a Data Journalism Hack Day at King’s College London (KCL), in collaboration with Open Knowledge International and SoBigData. The day was organised for students of the KCL Data Journalism Course, and saw them working in groups on issues from international taxation, clinical trials, human rights violations, and natural resource extraction in the Global South. The students had already extracted data from various sources and their aim for the day was to work with data and domain experts to build a story around the data which they would subsequently write up.

The students working with the OpenTrials data had previously decided to focus on migraine trials and had already used the OpenTrials API to extract the relevant trials, importing them into Excel where they spent most of their time cleaning the data, analysing it, and visualising it.

It was an enjoyable day helping the students better understand how clinical trials work, what sort of issues may be worth considering for their story, and how the data can inform it. It was also interesting to see the OpenTrials data being used in a real, hands-on way, how powerful it can be to answer specific questions, along with challenges and limitations relating to missing data from some registries and non-normalised company names – issues we are aware of and want to address.

We’re looking forward to reading the finished clinical trials ‘exclusive’ by the students – meanwhile we don’t want to steal their journalistic thunder, so we won’t post any of the cool visualisations they’ve created yet, but once the story is live, we’ll link to it here.

Our talk covered some of the problems with the information architecture of evidence-based medicine, how OpenTrials aims to help fix them, a technical overview of the platform, how we import data, licensing issues, user examples, and a demo – here are the slides:

Along with some good questions and suggestions for OpenTrials, we also had a number of meetings, both planned and spontaneous, to discuss potential collaborations to improve OpenTrials functionality and integrate others’ data into our system.

A great example of this is the Risk of Bias data which the Cochrane Schizophrenia Group previously kindly gave to us (thanks a lot!). This is structured data produced by researchers, who have graded schizophrenia trials on issues such as blinding and selective reporting. Here are some examples of how we’ve integrated this data onto individual trial pages (hint: have a look at the ‘Methodological rigour’ section at the bottom of each page).

As always, if you have feedback about OpenTrials you can email us your thoughts, or file a GitHub issue if you’re familiar with it. We’ll keep you updated with any developments on the blog, but please make sure you subscribe to our newsletter and follow us on Twitter!

OpenTrialsFDA will be featured in the OHSU (Oregon Health & Science University) Open Science Data Jamboree & Hackathon. The event will focus on the power of open content and data to advance biomedical research and human health.  Dr. Erick Turner, Associate Professor at OHSU and Staff Psychiatrist at the Portland VA Medical Center, will introduce OpenTrialsFDA, which is a finalist for the highly competitive Open Science Prize (voting for the finalists is open until 6 January 2017 here).

OpenTrialsFDA aims to increase access to and the utility of FDA drug approval packages. These review packages often contain information on clinical trials that have never been published.  When these “inconvenient truths” are integrated into a systematic review or meta-analysis, it can sometimes dramatically shift a drug’s risk-benefit ratio. Although the FDA makes the reviews available via its web portal Drugs@FDA, the documents are notoriously difficult to search and aggregate. As a consequence, they are rarely used by clinicians and researchers, despite their immense value.  OpenTrialsFDA intends to change this by allowing researchers, clinicians, the public, developers, and third party platforms to access, search, and build upon this data.

Dr. Turner’s keynote talk will be followed by an overview of the NIH’s new clinical trial data sharing requirements and two concurrent hands-on sessions:

• The first will be a deep dive on searching the OpenTrialsFDA website, understanding the contents of an FDA drug approval package (including the statistical content), and how to use the data for different use cases, such as meta analysis and systematic reviews.  Dr. Turner will team teach the breakout with an OHSU biostatistician.
• The second will be a OpenTrialsFDA hackathon using the OpenTrials API.  Attendees will have the opportunity to use their research, development, clinical, and visualization expertise (to name a few) to explore, present, and build upon the OpenTrials data and the OpenTrialsFDA application.  Please note, no technical expertise is required, as there will be multiple ways of contributing.

The Data Jamboree is presented by Computational Biology and the OHSU Library.  All are welcome to attend, including OHSU faculty, students, and staff as well as Portland area science enthusiasts and developers!

• Date: (to be announced)
• Time 2:30 – 5:00 PM (hackathon will continue past 5:00)
• Location: Collaborative Life Sciences Building (CLSB), 3A003, OHSU, Portland, Oregon
• Food & Drink provided

Contact:  Robin Champieux, Scholarly Communication Librarian, [email protected]

General questions

Background

The OpenTrialsFDA team is one of the six finalists for the Open Science Prize: a global science competition to make both the outputs from science and the research process broadly accessible to the public. This is a collaboration between Dr. Erick Turner, Dr. Ben Goldacre and the OpenTrials team at Open Knowledge International. Competing for the $230,000 prize that will be awarded to the winner in February 2017, OpenTrialsFDA is working on a prototype that will make the information hidden in the user-unfriendly drug approval packages of the FDA (the US Food and Drug Administration) more easily accessible and searchable, and link these to documents and data related to clinical trials.

The work is closely related to the OpenTrials project, which aims to locate, match, and share all publicly accessible data and documents, on all trials conducted, on all medicines and other treatments, globally. OpenTrials is aggregating this information from a wide variety of existing sources and aims to provide a comprehensive picture of the data and documents on clinical trials conducted on medicines and other treatments around the world.

What is the scope of OpenTrialsFDA?

OpenTrialsFDA works on making clinical trial data from the FDA (the US Food and Drug Administration) more easily accessible and searchable. Until now, this information has been hidden in the user-unfriendly Drug Approval Packages that the FDA publishes via its dataportal Drugs@FDA. These documents are notoriously difficult to access, aggregate, and search, since the Drugs@FDA website itself is not intuitive to navigate, and the reviews are stored as individual PDFs which, except for relatively new drugs, cannot be searched.  OpenTrialsFDA will extract the relevant data from the FDA documents, link it to other clinical trial data and present it through a new user-friendly web interface. Any user will be able to type in a drug name, and see all the places where this drug is mentioned in an FDA document. Users will also be able to access, search and present this information through the application programming interfaces (APIs) the team will produce.

Who is behind OpenTrialsFDA?

OpenTrialsFDA is a collaboration between Dr. Erick Turner (a psychiatrist-researcher and transparency advocate), Dr. Ben Goldacre (Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford) and the team behind OpenTrials at Open Knowledge International.  

What is the Drugs@FDA portal?

Drugs@FDA is a searchable catalog of the US Food and Drug Administration (FDA) approved drug products, both prescription and over the counter, with links to documents relating to marketing approval. More information is available from their FAQ section.

What are FDA drug approval packages?

FDA Drug Approval Packages consist of FDA employees’ reviews of new drug applications (NDAs) submitted by pharmaceutical companies seeking approval to market their drugs in the US. These NDAs consist primarily of clinical study reports (CSRs). Some drugs have more than one approval package, because approval is granted not for a drug per se, but for each drug-indication combination.

How will OpenTrialsFDA improve access to the FDA data?

The team will scrape the FDA website and extract the relevant information from the PDFs through a process of OCR (optical character recognition). Through the new OpenTrialsFDA interface, users will be able to explore and discover the FDA data. In addition, the information will be integrated into the OpenTrials database, so that the FDA report can be linked to reports from other sources, such as ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed.

How does the FDA review process differ from the peer review process used for manuscripts submitted to journals?

The key is the before-versus-after-the-trial aspect. Before the sponsor can begin a trial in the US, it must submit the trial protocol to the FDA. By means of this “pre-trial” review of the protocol, the FDA learns (1) that that trial is to be conducted and (2) the nitty-gritty methodological details. That way, a few years later, when the sponsor submits its NDA, the reviewer compares the clinical study report (CSR) to the original protocol to determine (1) whether any studies have been omitted and (2) whether any outcomes have been switched. By contrast, in the world of peer-reviewed journal articles, there is no “pre-trial” review, allowing both (1) and (2) to take place.

Who authors FDA reviews?

The FDA review team is multidisciplinary, yielding different types of reviews. Medical reviews are usually authored by physicians who summarize and evaluates the data on drug efficacy and safety. Statistical reviews are carried out by statisticians who test whether they can replicate the sponsor’s results using the patient-level data submitted electronically. There are other review disciplines, e.g. chemistry, pharmacology-toxicology, whose reviews pertain primarily to the preclinical phase and early clinical (human) phases of drug development.

What is the Open Science Prize?

The Open Science Prize is a collaboration between the Wellcome Trust, the US National Institutes of Health (NIH) and the Howard Hughes Medical Institute. Its goal is to unleash the power of open content and data to advance biomedical research and its application for health benefit. The OpenTrialsFDA team is one of the six finalist teams that were selected in May 2016. All teams will showcase their prototypes at the BD2K Open Data Science Symposium on 1 December 2016, when public voting will begin. The public is asked to help select the most promising, innovative and impactful prototypes from among the six finalists – among which one will receive the grand prize of $230,000.  

Troubleshooting

In case you’re having difficulty finding the right information in the Drugs@FDA portal, here are some troubleshooting questions to help you:

Was the drug approved before or after 1997?
The FDA started posting reviews (Drug Approval Packages) in 1997. If the drug in question was approved in 1996 or earlier, you probably will not find it posted.

Are you sure that the use you’re interested in is not off-label, ie. that the FDA approved it for the indication you’re interested in?
Oftentimes drugs are used widely for a certain condition (indication) for which the FDA has not granted approval (off-label use). If you check the product label (e.g. at https://dailymed.nlm.nih.gov/dailymed/index.cfm) and look within the section on “Indications and Usage”, you will see a list of the approved indications.

Are you looking for indication #1 or one of the subsequent indications?
A given drug might be FDA-approved for multiple indications. The FDA is careful about posting the Drug Approval Package for the first indication (the one that allowed that drug to enter the market), but for subsequent indications (#2, #3, etc.), it’s more hit-or-miss.

If you are motivated, one way to get such reviews is to file a Freedom of Information request here; but be warned—the FDA might take months or longer to fulfill your request. If you do get such reviews, we would appreciate your sharing it with us so we can add to OpenTrialsFDA and make it readily available to others.

Did you use the generic name or brand name?
If you use the generic name, you will probably get lots of hits: one for when the molecule was first being introduced to the market plus hits for subsequent generic versions. The FDA does its review for efficacy and safety when the molecule is first introduced to the market. Years later, when the patent expires, generic equivalents get approved, but what the FDA cares about at this stage is bioequivalence (in terms of blood levels) to the original brand name version. Thus generics do not have to demonstrate efficacy and safety all over again, which is why these approval packages generally contain little or no such data.

Is the review an NDA or an ANDA?
Please see the question and answer immediately above. NDAs (new drug applications) correspond to brand name versions, while ANDAs (abbreviated new drug applications) correspond to generic equivalents. They are abbreviated because less is required of the sponsor (see above). Assuming you are interested in drug efficacy and/or safety (and not bioequivalence), you will want to focus on NDAs and ignore ANDAs.

Is this the original or a subsequent formulation of a drug?
This is similar to the issue brand name versions vs. generic versions (see above). The burden of proof regarding efficacy and safety is highest when a molecule is first entering the market. A given drug will often enter the market as an immediate release (IR) formulation, before which the FDA will do its most comprehensive review of drug efficacy and safety. Some years later, the sponsor may seek approval for a sustained-release (SR) formulation, but by that time, millions of patients have already been exposed to the IR formulation. Because it is relatively unlikely that the efficacy and/or safety profile will differ a great deal between the IR and the SR, the SR may have to do fewer clinical trials compared to its IR version.

Want to get involved? Have further questions?

Please email us at [email protected]. This email is sent to Emma Beer, the Project Manager for OpenTrials and Ben Goldacre, its Principal Investigator.  

For further project updates, check the OpenTrialsFDA page at https://fda.opentrials.net or follow us on Twitter: @opentrials

Now that the dust has settled, I wanted to give you a brief summary of what we got up to at the Hack Day and also give you some information on the API for those of you who want to play around with the data that powers OpenTrials.

Hack Day

We had a range of people attend our event in Berlin, from researchers, developers, artists, and industry consultants. Some were just interested in the concept of OpenTrials and its ability to improve medicine, but were new to coding, whereas others were seasoned developers used to hacking code and got their teeth straight into our API.

A range of ideas for projects, integrations, and improvements were discussed, including:

• Integrating OpenTrials information into WikiData
• Adding information on R&D costs to intervention pages
• Showing additional summary information in the search results (to avoiding clicking through to a trial’s page)
• Linking more academic publications to clinical trials listed in OpenTrials by searching the full text of journal articles for mentions of clinical trial IDs (we currently only look at the abstracts)
• Creating a two-way lookup table, showing what fields/variables can be expected from each clinical trial registry source
• Adding early FDA approval letters (not available from Drugs@FDA) to drug/intervention pages (subject to an ongoing, lengthy FOIA request)
• Displaying links between conditions, interventions, and trials (see diagram below)

Of particular mention is Matthias Koenig’s hack (see above) – more screenshots and code here.

The conversations of the day generally revolved around medicine, but covered everything from text and data mining, improving research quality, clinical trial transparency, through to open access and open science, We’d love to keep the conversations going, so if you’re inspired to work on something or have already started, please let us know and we’ll make sure to spread the word.

Whether you attended or are interested in working with the OpenTrials data, please join the conversation in our chatroom and view the code behind OpenTrials.

API information

If you’re a developer or data scientist who’s interested in playing with our underlying data, we have an API available (caveat emptor: early version, may change) – here are some documents to get you up to speed with the data:

• Introduction to the data
• Working with the OpenTrials API
• API documentation
• Project ideas

Update: If you use R, be sure to check out Darko Bergant’s great tutorial – Using OpenTrials API with R

NB. You can download a database dump of OpenTrials here.

We’re also happy to discuss potential collaborations, and ask that if you use our data for academic research, you cite our paper – thanks!

If you have any feedback, bug reports, or feature requests for OpenTrials please either email us or file an issue on our GitHub repo.

OpenTrials works like a search engine, with advanced search options for filtering results by criteria such as drug and disease area. All data and documents for each trial included are “threaded” together and presented alongside each other. At the World Health Summit, the team will be demonstrating how the OpenTrials interface works, including how to explore trials and filter results by criteria such as drug and disease area. They will also demonstrate the power of linking clinical trial information together, showing how it can be used to highlight important discrepancies in the data.

We want the information provided on OpenTrials to inform decision-making and lead to better medical services worldwide. We expect a range of potential uses for the platform:

• A public health researcher could find out more about the range of trials on a drug, searching by various criteria to match a specific population.
• A doctor interested in critical appraisal of research papers could see if sources of bias for specific trials have already been assessed by experts.
• A researcher could see if the same trial reports somewhat different methods or results in different places.
• A patient interested in participating in a trial for their condition could identify trials in their geographical area which are enrolling.

A crowdsourcing functionality allows users to contribute data and documents and to provide feedback on the accuracy of trial information.

OpenTrials currently extracts and displays data from ClinicalTrials.gov, EU CTR, HRA, WHO ICTRP, and PubMed, and risk of bias assessments from the Cochrane Schizophrenia group. After the beta launch, we plan to integrate systematic review data from Epistemonikos and other sources. There are seven additional sources of data that have been extracted, but can’t currently be displayed because of licensing issues – we are working with these sources of data to get permission to publish. We’ll keep updating the OpenTrials blog as they become available.

“There have been numerous positive statements about the need for greater transparency on information about clinical trials, over many years, but it has been almost impossible to track and audit exactly what is missing, or easily identify discrepancies in information about trials” explained Dr. Goldacre, the project’s Chief Investigator and a Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford. “This project aims to draw together everything that is known around each clinical trial. The end product will provide valuable information for patients, doctors, researchers, and policymakers—not just on individual trials, but also on how whole sectors, researchers, companies, and funders are performing. It has the potential to show who is failing to share information appropriately, who is doing well, and how standards can be improved.”

“OpenTrials is an important step towards ensuring researchers, journalists, and patient groups have access to the medical information they need,” said Pavel Richter, CEO of Open Knowledge International. “Through the OpenTrials platform, researchers can advance science more quickly, doctors can easily find the latest evidence to improve services, and patients can locate information about pressing public health issues. OpenTrials is a great example of the work we are doing at Open Knowledge International to equip civil society organisations with the tools and information they need to address social problems and improve people’s lives.”

The first phase of the Open Trials project is scheduled for completion in March 2017. For project updates, please follow @opentrials on twitter or get in touch with us at [email protected].  A Hack Day (a World Health Summit Satellite event) took place on 8 October in Berlin.  For more details, see https://opentrials.net/hackday

Further information on speakers and topics of the World Health Summit 2016:

www.worldhealthsummit.org/the-summit/program

www.worldhealthsummit.org/the-summit/speakers

The World Health Summit is open to media representatives:

www.worldhealthsummit.org/press-media/accreditation

Editor’s notes:

Ben Goldacre
Ben is a doctor, academic, writer, and broadcaster, and currently a Senior Clinical Research Fellow in the Centre for Evidence Based Medicine at the University of Oxford. His blog is at www.badscience.net and he is @bengoldacre on twitter. Read more here. His academic and policy work is in epidemiology and evidence based medicine, where he works on various problems including variation in care, better uses of routinely collected electronic health data, access to clinical trial data, efficient trial design, and retracted papers. In policy work, he co-authored this influential Cabinet Office paper, advocating for randomised trials in government, and setting out mechanisms to drive this forwards. He is the co-founder of the AllTrials campaign. He engages with policy makers. Alongside this he also works in public engagement, writing and broadcasting for a general audience on problems in evidence based medicine. His books have sold over 600,000 copies.

Open Knowledge International
https://okfn.org
https://opentrials.net
Open Knowledge International is a global non-profit organisation focussing on realising open data’s value to society by helping civil society groups access and use data to take action on social problems. Open Knowledge International addresses this in three steps: 1) we show the value of open data for the work of civil society organizations; 2) we provide organisations with the tools and skills to effectively use open data; and 3) we make government information systems responsive to civil society.

The Laura and John Arnold Foundation
www.arnoldfoundation.org
LJAF is a private foundation committed to producing substantial, widespread, and lasting reforms that will maximize opportunities and minimize injustice in our society. Its strategic investments are currently focused on criminal justice, education, public accountability, evidence-based policy, and research integrity. LJAF has offices in Houston, New York City and Washington D.C.

COS
http://centerforopenscience.org
COS is a non-profit technology company providing free, open source software and services to increase inclusivity and transparency of research. COS supports shifting incentives and practices to align more closely with scientific values.  COS develops the Open Science Framework as an infrastructure to enable a more open and transparent research workflow across all of the sciences.

World Health Summit
www.worldhealthsummit.org
Under the high patronage of German Chancellor Angela Merkel, French President François Hollande and European Commission President Jean-Claude Juncker, the WHS attracts about 1,800 participants from more than 80 countries. It is the premiere international platform for  exploring strategic developments and decisions in the area of healthcare.

The launch will take place at the ‘Fostering Open Science in Global Health’ workshop, with OpenTrials being represented by our Community Manager, Ben Meghreblian. The workshop will be a great opportunity to talk about the role of open data, open science, and generally how being open can bring improvements in medicine and beyond!

If you’ll be attending the conference or the workshop, we’d love to meet you – please do get in touch and let us know.

Hack Day

If that wasn’t enough, we also have a confirmed date and location for the OpenTrials Hack Day – it will take place on Saturday 8th October at the German office of Wikimedia in Berlin.

We’re inviting people from a range of backgrounds. So, if you’re a developer, data scientist, health technologist, open data advocate, or otherwise interested in health, medicine, and clinical trials, come along and learn more about the data that powers OpenTrials, how it’s structured, and how to use our API to search the OpenTrials database or build applications using the data.

On the day our technical lead and a domain expert will be on hand to explain the data and facilitate the day – we’re really looking forward to seeing what clever hacks and mini-projects you’ll create.

For those of you who have already asked, we’ll be releasing documentation on the OpenTrials API and database soon, but meanwhile if you’re interested in the event you’ll find more details on the OpenTrials Eventbrite page, or you can register quickly below.

It was great to speak to so many people who are interested in OpenTrials, both in terms of researchers who want to use the platform and those with a general enthusiasm for its impact on medicine.

Around 40 people attended our talk which explained why OpenTrials is an important infrastructure project for medicine, covered some of the technical aspects of the platform, details of what data we’ve imported so far, and lastly a quick demo.

If you’re feeling impatient, here are the slides from the talk, or scroll down for a summary.

Ben Goldacre and Vitor Baptista present OpenTrials at Evidence Live 2016 (photo by benmeg / CC BY)

What we’ve imported into the OpenTrials database so far

331,999 deduplicated trials, collected from three clinical trial registries: ClinicalTrials.gov 205,422 EU CTR 35,159 WHO ICTRP 298,688

• ~22,000 research summaries from the Health Research Authority
• ~510,000 publications from PubMed (~24,000 linked with trials)

Current functionality

• Basic search (by keyword)
• Searching for trials with publications
• Uploading missing data/documents for a particular trial
• Showing trials with discrepancies (e.g. target sample size)

What we’re importing next

• Systematic Reviews from:
  • Epistemonikos (26 sources)
• Clinical Study Report Synopses (e.g. YODA)
• Cochrane Schizophrenia data
• FDA Drug Approval Packages – OpenTrialsFDA won $80,000 to develop prototype to unlock ‘hidden’ data in non-searchable PDFs

Feedback and get involved

If you attended the talk and have any questions or feedback, please email us. And generally if you’re interested in contributing to OpenTrials, get in touch.

Want to get early access to the data and be a user tester? Sign up and we’ll be in touch soon.