The 19th century British statesman Benjamin Disraeli is famously known for the quote (possibly misattributed to him) that “there are three kinds of lies: lies, damned lies, and statistics.”
The wisdom of Disraeli can be seen when analyzing the recent hype over Galleri’s “Grail” test – a “liquid biopsy” that I have covered in a previous post. Grail tests a single blood sample for evidence of over 50 different cancers. The appeal is obvious. It would be wonderful to have a simple blood test every year or so as part of your checkup that can tell you if you have cancer or not and that can greatly increase the odds of early detection and cure.
Unfortunately, despite the hype surrounding the Grail test, this dream is not yet a reality.
First off, the hype: recently an advisory panel voted 6-4 to recommend the FDA approve Grail. This does not mean that Grail is suddenly available. It has in fact already been available for several years. Rather, if the FDA follows the panels suggestion, it would force various insurers to cover the test. At a cost of nearly a thousand dollars a pop, this would add up to a huge healthcare expense that will drive the cost of insurance premiums yet higher. While that’s a problem in and of itself, we of course cannot decide in isolation that the cost is too much. For if Grail truly reduced the risk of cancer deaths, it could well be worthwhile, even at huge expense. Unfortunately, the data Galleri itself has released to date is hardly impressive in this regard.
One study, conducted by Britain’s National Health Service, showed a slight trend toward a reduction in stage 4 cancers in those undergoing regular Grail screenings (suggesting maybe earlier cancer diagnoses), but it was not enough to meet statistical or clinical significance, and there was no obvious reduction in cancer mortality. This is not a terrible result – it suggests that with further improvements to the test, there may someday (not that many years off) be a real benefit. But as it stands now, the data is just not yet there that Grail testing leads to lives saved.
But that’s not my biggest concern about the test. Knowing patients as I do, I suspect that many will want to use the Grail test as a way to gain reassurance. I sympathize with this desire. Fear of cancer is commonplace, and a blood test that could tell patients with virtual certainty that they will NOT develop cancer in the next year or two would be a wonderful thing for peace of mind. But this is where Grail really falls down, and where we can learn something from the late Mr. Disraeli.
In medicine we have a statistic called “Negative Predictive Value,” which in plain English translates to: what are the odds that a negative result on a test means that I in fact do not have the disease. Or put another way – if I take this test and it comes back normal, how confident can I be that I really am in the clear?
Grail has a negative predictive value of 99.07%, which means that if I take the Grail test and get a normal result, there is a slightly better than 99% chance that I really do not have cancer. Conversely, there is only a bit less than 1% chance that I have received a false negative and in fact am going to be diagnosed with cancer in the coming year. At first glance, that sounds wonderful.
But consider this: in any given year, an average adult over age 50 in this country has only about a 1% chance of being diagnosed with cancer and about a 99% chance of not being diagnosed with cancer. (Of course this is an average, and these stats shift a bit with age and health status).
In other words , a negative result on a Grail test really does not add anything to your knowledge of whether or not you will develop cancer in the near future.
To put this another way, let’s imagine a new test that I will egotistically name “Fischer’s cancer test.” Fischer’s cancer test asks whether you have a nose on your face or not, and states that if you do, you will not develop cancer in the coming 12 months. Since (with vanishingly rare exceptions such as survivors of horrible accidents or Lord Voldemort) everybody has a nose on their face, 100% of the population will “pass” Fischer’s cancer test and get a reassuring result.
Think that sounds absurd? Well, Fischer’s cancer test has a 99% negative predictive value, i.e. 99% of the people who are told they needn’t worry about cancer because they have a nose on their face are in fact going to remain cancer free over the coming 12 months, whereas only 1% will have received a false negative. The misleading nature of this test is obvious, and you would never be taken in by this logic. And yet the Grail test is barely more effective than Fischer’s cancer test at ruling out cancer.
To look at the same problem from a slightly different angle, several studies have now shown that – of the people who go on to develop cancer over the ensuing 12 months after taking the Grail test – only about a third will have an abnormal test on Grail, while two-thirds will be given a negative result. This accounts for why the negative predictive value of the test is barely better than just looking at population averages (or looking at whether you have a nose on your face).
Of course, false negatives are not the only worry in medical testing. False positives are also a concern, and here the Grail test performs better. About 60% of positive Grail tests do in fact lead to a cancer diagnosis, meaning the test has roughly a 60% positive predictive value. As screening tests go, that’s actually not bad (by comparison, only about 5% of abnormal mammograms turn out to be cancer, so the positive predictive value of an abnormal mammogram is only about 5%). However, it remains unclear how many times a positive Grail test actually improves patient outcomes. Receiving a diagnosis of a cancer for which there is no cure, or a diagnosis of a cancer for which cure rates are virtually 100% even if they are not diagnosed early (for example skin or thyroid cancers) is not that useful.
When we think of screening tests in general, we really want tests that have a very good negative predictive value, far more than tests that have a good positive predictive value. Coming back to mammograms, they have a negative predictive value of about 99.8%. In other words, a woman who gets a normal mammogram can be reasonably confident that she does not have breast cancer, while the minority of women who get an abnormal mammogram can be quickly and inexpensively (for example with a repeat mammogram and/or ultrasound) sorted into those who do not have cancer and those who need a biopsy.
By contrast, a normal result on the Grail test provides no useful reassurance that you do not have cancer, and only a very marginal benefit, if any, in the event of an abnormal result.
For what it’s worth, I am not alone in the above opinions. Many doctors have expressed shock at the recommendation to approve Grail, and several oncologists who I respect have had nothing but negative things to say about this development. I will leave it to others to speculate why the FDA panel might have recommended Grail despite the fact that Galleri’s own data, published to put the best possible light on their product, shows that it is at best still very much a work in progress.
