Real World Data RWD

Using RWD to verify RWE – What Could Go Wrong?

In Uncategorized by Michael Willis1 Comment

In recent years, the terms “real-world data” (RWD) and “real-world evidence” (RWE) have played an increasing role in health research and decisions. What do these terms mean – RWD and RWE, and, why now? According to the FDA1, “… real-world data are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.”

RWD can come from a number of sources, for example:

  • Electronic health records (EHRs)
  • Claims and billing activities
  • Product and disease registries
  • Patient-generated data including in home-use settings
  • Data gathered from other sources that can inform on health status, such as mobile devices

Real-world evidence is the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.

A recent publication by David Thompson, PhD, asked, “What could go wrong?”when using RWD to generate RWE for regulatory purposes? The focus of Dr. Thompson’s article is the on-going effort to “replicate” the results of randomized controlled trials [RCT] using RWD. I certainly agree with most (if not all) of Dr. Thompson’s comments.

However, I do have a few comments to add to this on-going discussion of RWD and RCT results. To start, these two sources of evidence have two different purposes. RWD are collected with (generally) no hypotheses in mind and are likely adequate for primary care and other purposes (see list above1).

RCT data are collected to demonstrate a possible causal connection between treatment and outcome, i.e., to address a specific hypothesis. That is to say, the RCT design works to maximize the internal validity of the trial. Simply put, internal validity addresses the degree to which the treatment or intervention (and not something else) produced the outcome; external validity has to do with generalizability of the results.

RCT results, by design, have low generalizability due to rigorous screening of participants, and a highly controlled intervention environment.

Because of these and other differences between RWD and RCT, each source is not better or worse than the other. We should be asking, better or worse for what purposes? Or, in the words of Dr. Thompson, “… impugning RWD sources and analytic methods for failing to align with RCT findings is inappropriate … “2 Results of multiple RCTs with similar treatments and outcomes do not always align.

In this, I do agree that the efforts to replicate RCT using RWD may not be fruitful.

One of the issues facing RWD, because they are not generally collected for some specific objective, is that collection does not follow a specified protocol. By this, I mean that something as simple as how blood pressure measurement is collected can vary from visit to visit (same participant), and from clinic to clinic. We would need to minimize variance due to measurement, and other sources of variance before RWD can become a reliable source of RWE.

Finally, scientific review of literature (meta-analysis) addresses the following: what do we know, what don’t we know, what else do we need to know, and with what level of confidence. Rather than try to replicate results of RCT with RWD, I propose that we use meta-analytic evidence to compare/contrast with those of RWE. By pooling results of multiple RCTs, we can bypass the need to replicate the point estimates of a single RCT with RWD.

These would be focused meta-analytic reviews, with the pooled studies having similar patient demographics, treatment/intervention, and outcomes to those of the RWD being analyzed.

The pooled research literature can provide context with which to interpret RWE. If meta-analytic evidence contradicts or support RWE we can then ask to what extent, and why.

Zung Vu Tran, PhD
Professor (ret.) of Biostatistics
Founder & Chief Science Officer
MedAware Systems, Inc.

References Cited:


2.   Thompson D. Replication of randomized, controlled trials using real-world data: what could go wrong? Value in Health 2

Please follow and like us: