Roundtable discussion: Does Avastin work on breast cancer? Should Medicare be paying for it?

(by Andrew Gelman)

Last week I posted a link to New York Times columnist Joe Nocera’s claim that Medicare is a corrupt system that is paying for a drug, Avastin, that doesn’t work for breast cancer patients.  I sent the pointer to several colleagues, along with the following note:

I have three goals here:

1. To get to the bottom of the Avastin story.

2. If Nocera is correct, to add the voice of the statistics profession to his push to have more rational rules for drug approval.

3. To promote the Statistics Forum, as part of a larger goal of engaging the statistics community in frequent public discussions.

In the old days, professional discussions were centered on journal articles. We still do this, but blogs and other online forums are becoming increasingly important. By contributing a couple paragraphs here (including links to further reading), you will be helping in all three goals above. Just send to me and I will post at the Forum (under your name).

(I myself am doing this in the spirit of service to the profession in my duties as editor of the Statistics Forum.)

I received several responses.  The quick summary is:

1.  Nocera claimed that Avastin doesn’t work for breast cancer.  Actually, though, the data seem to show that it does work.

2.  It’s not clear whether Medicare should be paying for the drug.  In any case, the drug reimbursement system is a mess.

Now here are the comments from the experts.

From biostatistician John Carlin:

I asked a colleague of mine about it – he sits on Australia’s Pharmaceutical Benefits Advisory Committee, which rather uniquely in the world (I believe) is charged with examining cost-effectiveness of new drugs in order to advise government as to whether they should be subsidised under the national Pharmaceutical Benefits Scheme (which pays the bulk of drug costs in this country). As I understand it, Avastin (bevacizumab) is still approved here as a treatment for certain indications but it is not approved for listing on the PBS, and without receiving a subsidy via that mechanism its cost is generally prohibitive.

See also this paper by John Ioannidis, which highlights issues about selective early evidence and multiplicity of comparisons for this drug:

From statistician Howard Wainer:

Data and evidence seems to have only a minor role in today’s political climate — witness the support of policies for cutting taxes to reduce the deficit. In healthcare note the reaction to the evidence-based decision to cut back massively on such diagnostic tests as PSAs and mammograms in which their huge number of false positives makes the probability o a true positive tiny and clinical results that confirm that early detection of breast cancer seems to have no effect on survival (with modern treatment) and the lack of a clear advantage for treatment over not for prostate cancer.

None-the-less the size of the diagnostic industry presses against rationality.

From a health-care economist:

From what I can understand, there are scientific, technical, and political forces at work here. The scientific issue is that Avastin is believed to work for some women, but not all.  No one knows which women will benefit, and so the FDA ordered Roche/Genentech to figure this out.  The technical issue is that by law, CMS has to cover any approved cancer drug for an off-label indication if the indication is included in one of several named drug compendia.  Because CMS does not like making these decisions, this generally suits CMS fine.  In this case, the National Comprehensive Cancer Network (NCCN), “a non-profit group of oncologists whose guidance is closely followed by leading treatment centers, has voted overwhelmingly in favor of maintaining its recommendation that Avastin should be used to treat breast cancer.”  The NCCN vote was 24-0, with one abstention.  That brings up the politics here.   The interesting dimension of politics is not the death panel charge — though that’s obviously there — but how organizations like the NCCN make their decisions.  There is an undercurrent of sleeziness that one picks up.  E.g., a bunch of members of the NCCN have ties to Roche; they make their money off this stuff; etc.  All of these play together to create outcomes like this.

From epidemiologist Sander Greenland:

I laud Andrew’s call to think about what might be done in the Avastin case in particular and the general basis for Medicare reimbursement. From the reported information the latter seems not to have much in the way of conflict-of-interest safeguards (unlike FDA panels). That problem, however, is not one of technical statistics or evidence evaluation – it’s a far more fundamental and touchy issue of protecting decision-making from the influences of vested interests.

From psychologist Jon Baron:

I have long advocated the use of cost-effectiveness analysis in health care.  (See, for example, the chapter on utility measurement in my textbook Thinking and Deciding.  The section on the Oregon Health Plan has been there since the 3d edition in 2000.)  Of course, the method has problems.  Measurement of utility is not rocket science.  (I compare it to the measurement of time in ancient Egypt.)  Nor is measurement of cost, for that matter.  But the issue is “compared to what?”  All other methods of allocating health resources seem more likely to waste resources that could be put to better use, saving lives and increasing quality of life, elsewhere.

A move in this direction in Obama’s health-care proposal was derided as “death panels”.  My understanding is that it is still there and could grow in the future.

I have not read the literature on Avastin.  But, from Nocera’s article and other things I’ve read about it, it seems to me that it would not pass a cost-effectiveness test with some reasonable limit on dollars per Quality-adjusted Life Year.  Whether it has a statistically significant benefit is beside the point.  The issue is, I think, our best estimate of its effectiveness given the data available.  If the FDA thinks it does no good at all, then it probably does not do very much good, and it is very expensive.  Some newer treatments for prostate cancer seem to fall in the same category, and it is possible that the same is true of the HPV vaccine (in the U.S., where Pap smears are routine).

Of course, any use of cost-effectiveness analysis needs to be careful about research, including “Phase 3” research on extensions of drugs to off-label uses and other issues not addressed before approval.

But the main problem is public opposition.  People have trouble with trade-offs and tend to think in terms of an opposition between “cover everything that might help” and “put limits on the amount of money that is spent and let someone else figure out how to allocate resources within those limits”.  The latter approach might work, but only if the “someone else” were willing to use cost-effectiveness and if the limits were sufficiently generous.

On the latter point, it seems to me that health care is very important and that its cost, relative to other ways of spending money, is not that high for what it does.  The idea that the total budget for health care should not increase seems wrong.  Health care is getting better, and thus it is reasonable for people to want more of it.  The idea of keeping the cost in line with inflation is like saying that, between 1945 and 1960, the amount of money spent on home entertainment increased too much and must be limited.

I saved the best for last.  Here’s statistician Don Berry, who is an expert on medical decision making in general and cancer treatment in particular:

The Avastin story is a very long one, with many turns. The FDA approval question for metastatic breast cancer, which is the issue here, has divided oncologists. There are rational arguments on both sides.

There is no question that Avastin “works” in the sense that it has an anti-tumor effect. Everyone agrees with that, well, with the possible exception of Joe Nocera. In particular, it clearly delays progression of metastatic breast cancer, which is the reason it was approved for treating that disease in the first place. The FDA reversed itself (for this disease but not for other cancers such as lung and colon that will remain on Avastin’s label) because Avastin has not been shown to statistically significantly prolong overall survival (OS). Some oncologists–actually, most oncologists–argue that progression-free survival (PFS) is clinically meaningful and should be a registration end point. The FDA’s position–and that of the Oncology Drug Advisory Committee (ODAC)–is that improved PFS is not usually enough to approve drugs without empirical evidence of improved OS to go along with it.

Genentech/Roche appealed to the FDA Commissioner after the FDA’s first decision (based on ODAC’s recommendation) to remove metastatic breast cancer from Avastin’s label. This kind of appeal is legal but is almost never used. Genentech mustered many arguments. Several were statistical. For example, they argued that none of the clinical trials in question were powered to show a benefit in OS. The following article addresses this question:

Broglio KR, Berry DA (2009). Detecting an overall survival benefit that is derived from progression-free survival. Journal of the National Cancer Institute 101:1642-1649.

This article demonstrates that it’s very difficult to power a study to show an OS benefit when survival post-progression (SPP=OS-PFS) is long, which it is in metastatic breast cancer, about 2 years in some of the Avastin trials. The article argues that even if an advantage in PFS translates perfectly into the same advantage in OS, the variability after progression so dilutes the OS effect that it’s likely to be lost.

The assumptions in the article about SPP are realistic. I know many example clinical trials in many types of cancer (and I know no counterexamples) where SPP is essentially the same in both treatment groups, even when the experimental drug showed better PFS than control. This is despite crossovers (to the drug when the control patient progresses) and potentially greater efforts by the clinicians in one treatment group to keep their patients alive. (The main reason that SPP is similar in the two treatment groups is that metastatic cancer is almost uniformly fatal and it’s hard to slow the disease after it’s set up housekeeping throughout the body.) It makes sense that a drug that was effective in delaying progression is not effective after progression because the drug is almost always stopped when the patient progresses, and the patient usually goes onto another drug.

For the full Avastin story check out The Cancer Letter and links provided therein.

5 Responses to “Roundtable discussion: Does Avastin work on breast cancer? Should Medicare be paying for it?”

  1. 1 ff November 28, 2011 at 6:40 pm

    Only 1 comment says anything about the actual evidence for the actual issue. The others all make theoretical comments about larger policy issues, not this one.

    Dr. Berry’s comment is fair, but slightly loaded. We have a medication with side effects (though fairly mild) and studies that were designed to show an improvement in survival that were unsuccessful.

    Dr. Berry’s view that PFS should be enough to get a drug approved is reasonable, but it is debatable and it is not the universal current policy standard. Medicine is filled with examples of medications that showed beneficial intermediate end-points with negative clinical end-points. Patients don’t care about PFS – it’s an imperfect marker for things they care about. FDA has a reasonable rule, they were clear about it for this drug when this study was designed, and the study was negative.

    • 2 statisticsforum December 1, 2011 at 5:50 pm


      Berry is not claiming that PFS is not the ultimate outcome of interest. The point is that we have data only on a sample, not the population of all people who might use the drug. Estimates from a sample have sampling variability. Berry is arguing that the the evidence is consistent with a benefit in overall survival as well, a benefit that is hard to detect given the small sample size.

      • 3 ri December 5, 2011 at 9:00 pm

        The FDA decision, as I understand it, is not solely based on whether the drug “works” (however that is quantified), it is rather based on whether the benefits of the drug outweigh the risks. For example, does an increase in median survival of X months (of drug A +B compared to drug B), outweighs the additional risks of A? As you can tell, this is a complex decision depending on the magnitude of X, the magnitude of the additional risks, any association between the risks and the benefits (i.e. are those who benefit and those have risks the same or are they different) and the variability of all of that.
        The panel of experts at the hearing answered questions on benefit/risk profile, not on benefit alone.

  2. 5 John Johnson December 6, 2011 at 4:22 am

    There’s a lot here to unpack. I agree to some extent with Berry’s comment that overall survival (OS) is a difficult endpoint to study in studies of short duration. It’s seductively simple, though, because it’s a “hard” endpoint (few people disagree on when a person dies), as opposed to PFS or anti-tumor activity, both of which involve some degree of subjectivity. (PFS can also be a pain to compute if you don’t do your data collection very carefully.) Add to this the issue that anti-tumor activity doesn’t always translate into long-term benefit because tumors can build up resistance much the same way that bacteria can build up antibiotic resistance. (And so patients switch drugs upon progressing, and complicate the OS endpoint.) Ultimately, the “softer” endpoints are used as arguments for conditional approval, with the promise that the evidence of improvements in OS is forthcoming.

    Is this the right way to go? Ultimately, the argument isn’t statistical any more, but rather brings in the ethical traps of cost vs. length of delay of mortality. If we can navigate this, we can come close to Jon Baron’s decision-making nirvana based on economics. Knowing the way we approach health care, I think this kind of decision making is far off, and the payers are left with having to react to incomplete and conflicting data.

    I’ll join my voice to those who want to move away from our addiction to the OS endpoint.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


The Statistics Forum, brought to you by the American Statistical Association and CHANCE magazine, provides everyone the opportunity to participate in discussions about probability and statistics and their role in important and interesting topics.

The views expressed here are those of the individual authors and not necessarily those of the ASA, its officers, or its staff. The Statistics Forum is edited by Andrew Gelman.

A Magazine for People Interested in the Analysis of Data

%d bloggers like this: