Our Projects |
Projects, Templates, and Materials
EVALUATING REPRODUCIBILITY AND ROBUSTNESS OF REAL WORLD EVIDENCE FROM LARGE HEALTHCARE DATABASES
(ENCePP registration number: EUPAS19636)
Our objective is to produce empirically based recommendations on how to measure, document and achieve fully reproducible and robust findings from healthcare database studies.
We will start by identifying a random sample of 250 large healthcare database studies published in a leading clinical or epidemiology journal within the last 5 years. For this sample of 250 studies, we will evaluate transparency of reporting based on a catalogue of specific parameters identified by a joint task force between two leading research societies who focus on using databases to evaluate the use and value of drugs (International Society of Pharmacoepidemiology, International Society of Pharmacoeconomics and Outcomes Research). We will then attempt to replicate 150 studies from this sample using the same methods and data sources as reported in the original publication. After replicating these studies, we will evaluate the robustness of evidence in a subset (N = 50) of studies by applying a range of plausible alternative design and algorithm choices and varying assumptions regarding confounding and misclassification.
We will:
Recommendations on how to improve transparency and reporting will be developed collaboratively with a Scientific Advisory Board and closely aligned with professional society guidance and journal editor policies.
Our objective is to produce empirically based recommendations on how to measure, document and achieve fully reproducible and robust findings from healthcare database studies.
We will start by identifying a random sample of 250 large healthcare database studies published in a leading clinical or epidemiology journal within the last 5 years. For this sample of 250 studies, we will evaluate transparency of reporting based on a catalogue of specific parameters identified by a joint task force between two leading research societies who focus on using databases to evaluate the use and value of drugs (International Society of Pharmacoepidemiology, International Society of Pharmacoeconomics and Outcomes Research). We will then attempt to replicate 150 studies from this sample using the same methods and data sources as reported in the original publication. After replicating these studies, we will evaluate the robustness of evidence in a subset (N = 50) of studies by applying a range of plausible alternative design and algorithm choices and varying assumptions regarding confounding and misclassification.
We will:
- Describe how closely studies are replicated
- Describe areas where insufficient detail in reporting on scientific decisions are most common
- Identify operational parameters where insufficient reporting has the greatest impact on findings
- Explore predictors of insufficient reporting
Recommendations on how to improve transparency and reporting will be developed collaboratively with a Scientific Advisory Board and closely aligned with professional society guidance and journal editor policies.
GRAPHICAL DEPICTION OF STUDY DESIGN TO ASSESS THE EFFECTIVENESS OF MEDICAL PRODUCTS WITH HEALTHCARE DATABASES
STaRT-RWE: A STRUCTURED TEMPLATE FOR PLANNING AND REPORTING ON THE IMPLEMENTATION OF REAL-WORLD EVIDENCE STUDIES
Manuscript pre-print with template and examples
Real world evidence (RWE) generated from real world data (RWD) increasingly informs important decisions about the clinical effectiveness and safety of medical products and interventions. Unlike clinical trials, which can leverage the power of randomization, or non-randomized studies with prospective collection of data for a specific research purpose, most RWE studies make secondary use of longitudinal data collected as part of routine healthcare processes, including administrative claims and electronic health records. This involves numerous complex design and analytic choices.
The lack of detail and structure in RWE reporting often requires substantial reviewer time for assessment of studies based on secondary healthcare data. Unambiguous reporting on key design and study implementation parameters could not only streamline and increase efficiency of review for stakeholders, but also improve confidence in the ability to judge the quality of evidence. The level of specificity must be balanced against the burdens it imposes on those reporting and those reviewing study details. In alignment with International Council of Harmonization strategic goals, this public-private collaboration developed a structured template to support RWE study planning, implementation and reporting, based on a consensus document from professional societies.
The template specifies key study parameters clearly and concisely in tabular and visual formats, to fulfill several aims:
The template is focused on RWE on the effectiveness and safety of medical products and interventions and is compatible with multiple study designs, RWD sources, reporting guidelines, checklists and bias assessment tools. While the simplicity of a checklist is excellent for summarizing areas to report on, it leaves room for misinterpretation and ambiguity about important details of study implementation. We complement the checklist approach by developing a study implementation template where methods related items from existing checklists correspond to the main headings in structured tables where critical details are communicated. Template tables are accompanied by a detailed visual summary in the form of a design diagram. The template is intended to support research planning and preparation, then shared with the final study results to facilitate review and replication. A library of examples for different use cases were prepared to enhance usability.
Real world evidence (RWE) generated from real world data (RWD) increasingly informs important decisions about the clinical effectiveness and safety of medical products and interventions. Unlike clinical trials, which can leverage the power of randomization, or non-randomized studies with prospective collection of data for a specific research purpose, most RWE studies make secondary use of longitudinal data collected as part of routine healthcare processes, including administrative claims and electronic health records. This involves numerous complex design and analytic choices.
The lack of detail and structure in RWE reporting often requires substantial reviewer time for assessment of studies based on secondary healthcare data. Unambiguous reporting on key design and study implementation parameters could not only streamline and increase efficiency of review for stakeholders, but also improve confidence in the ability to judge the quality of evidence. The level of specificity must be balanced against the burdens it imposes on those reporting and those reviewing study details. In alignment with International Council of Harmonization strategic goals, this public-private collaboration developed a structured template to support RWE study planning, implementation and reporting, based on a consensus document from professional societies.
The template specifies key study parameters clearly and concisely in tabular and visual formats, to fulfill several aims:
- Serve as a guiding tool for designing and conducting reproducible RWE studies
- Set clear expectations for transparent communication of RWE methods
- Reduce misinterpretation of prose that lacks specificity
- Allow reviewers to orient quickly and find key information
- Facilitate reproducibility, validity assessment, and evidence synthesis
The template is focused on RWE on the effectiveness and safety of medical products and interventions and is compatible with multiple study designs, RWD sources, reporting guidelines, checklists and bias assessment tools. While the simplicity of a checklist is excellent for summarizing areas to report on, it leaves room for misinterpretation and ambiguity about important details of study implementation. We complement the checklist approach by developing a study implementation template where methods related items from existing checklists correspond to the main headings in structured tables where critical details are communicated. Template tables are accompanied by a detailed visual summary in the form of a design diagram. The template is intended to support research planning and preparation, then shared with the final study results to facilitate review and replication. A library of examples for different use cases were prepared to enhance usability.
FRAMEWORK FOR VISUALISING STUDY DESIGNS AND DATA OBSERVABILITY in ELECTRONIC HEALTH RECORD DATA