Why Products Fail in Phase III

Half of the small-molecule drugs that fail pivotal trials can't prove better efficacy than placebos. The most important predictor of failure: a drug with a novel mechanism of action. But for the most part, companies are pursuing a one-size-fits-all development strategy, using the same methods for developing drugs that modulate novel and precedented targets. Companies need to better differentiate their development strategies based on risk.

Half of the drugs that fail pivotal trials can't prove better efficacy than placebos. Companies need to better differentiate their development strategies based on risk.

By Tamara Elias, Maria Gordian, Navjot Singh, and Rodney Zemmel

The productivity challenges facing pharmaceutical companies' R&D groups are well known, as is one of the causes: high failure rates among products in clinical development. To figure out how much of these failures are within a drug company's control, we conducted an industry-wide analysis of the causes of attrition, focusing on Phase III. Failures and successes at this stage phase have the greatest potential impact on R&D expenditures since these projects cost much more than earlier stage compounds. They are also the most scrutinized by investors and as a result Phase III attrition has huge implications for company valuations. Failure at Phase III should be infrequent: the extensive testing drugs undergo in earlier phases should weed out most if not all problematic compounds.

But they don't. Despite the conventional wisdom that drugs have proven their safety and efficacy by the time they enter Phase III, a surprising number fail to reach the market. The failures share two key attributes: inadequately defined endpoints and new mechanisms of action. The objectivity with which endpoints were defined has a small but important effect on failure rates; and new mechanisms of action are much more likely to fail than proven mechanisms.

Pharmaceutical companies can convert these findings into at least three meaningful actions to reduce failures in Phase III trials. First, they should cut their losses earlier, by improving their decision making, especially in Phase II. Second, they should improve the strategic and operational execution based on compound risk: they should apply different development criteria and resources to compounds based on their risk profile. Finally, they should enhance the designs of their Phase II and Phase III trials.

Gathering the Data

To learn about the causes of attrition, we analyzed Phase III trial failures reported between 1990 and 2002. To ensure that we were working with consistent data, we focused on small molecules, excluding biologics, and restricted our analysis to the larger pharmaceutical companies. (See the sidebar "Behind the Analysis: Deriving the Attrition Rates" for details on our methodology.)

We defined "failure" as those cases in which a trial either ended early (except where it ended early because of strongly positive results) or did not produce the results that would ensure drug approval. Of the 656 Phase III compounds being pursued for multiple indications that we screened, 278 of these failed implying only 58% of these trials were successful.

Of the total pool of 278 failed trials, we analyzed the histories of a group of 212 compounds that fell into six therapeutic areas: CNS, infectious disease, cardiovascular, endocrinology, oncology, and respiratory. We found sufficient publicly available information to perform an analysis of 73 of the 212 compounds. Our analysis consisted of outside-in investigations – analyst reports, press releases, and other sources of public information – to understand the causes of failure. A panel of clinical experts reviewed our materials to ensure that we had interpreted the data correctly. Though this data set is limited, we believe that it is more robust than the internal information on trial failures available at any single pharmaceutical company.

Root Causes of Failure

To try to understand why these 73 drugs failed in Phase III, we analyzed three major areas of the studies: efficacy (compared with placebo), safety (also compared with placebo), and differentiation (against relevant comparators) (See Exhibit 1.) In a full 50% of cases, the drugs in question failed for lack of efficacy: the trials could not demonstrate that the drugs were more medically effective than the placebos. Another 30% of the failures came from safety concerns. In the final 20%, the new drugs could not be proven safer or more effective than the drugs already on the market.

The fact that fully half these compounds proved inefficacious in Phase III was astonishing – particularly since one of the key purposes of Phase II trials is to establish proof of efficacy in patients. Admittedly, Phase II trials can't eliminate all the uncertainty in a new compound, but they should remove most of it. And this lack of efficacy was not limited to one or two therapeutic areas: this was the case in all therapeutic areas although the absolute magnitude of this root cause of failure varied by therapeutic area. (See Exhibit 2.) We recognize that the "n" in each therapeutic area is quite small, but the trends emerge clearly even from a sample of this size.

New Mechanisms and Unclear End-Points

Efficacy failures can themselves be traced to a variety of causes. We explored two potential signals of success or failure: mechanistic novelty and endpoint definition.

For trial end-points, we used a qualitative definition. We considered end-points objective if researchers could measure them with diagnostic tests whose results could be easily reproduced, or with scales that were both professionally measured and widely used. We considered end-points to be less objective if they relied on less easily reproducible measurements, uncommonly used scales, or self-reporting by patients. As it turned out, trials with more objective end-points were generally more successful.

But an even more significant predictor of failure was novel mechanism. Even after the patient evaluation process in Phase II, drugs that used novel mechanisms of action failed more than twice as often in Phase III as those that used known mechanisms. And if drugs had both novel mechanisms and less objective endpoints, they failed 70% of the time (See Exhibits 3 and 4.) In contrast, drugs with validated mechanisms and objective endpoints failed just 25% of the time.

Make Tougher Decisions Sooner

Early trials should weed out ineffective compounds. This seems obvious. But the evidence suggests companies are not using Phase II trials to guide their judgment as rigorously as they should.

These observations were confirmed by clinicians and statisticians across the pharmaceutical industry, who consistently told us two things. First, they said that companies are not doing enough Phase II trials, such as dose ranging studies. Second, they explained that companies are not using their Phase II results to make the right decisions. More than once, we were told that companies had pushed through to Phase III compounds about which clinicians and statisticians had had reservations in Phase II.

The basic problem is wishful thinking. Sometimes project teams lose their objectivity about compounds, because they have worked on and championed them for years. Another reason is that senior managers bow to Wall Street: many feel pressure to deliver on their announcements, despite the uncertainties caused by the technical risks and subtleties of drug discovery. Some have announced pipeline goals with incentives of senior management tied to meeting targets around numbers of compounds in different stages.

Pharmaceutical companies that recognize these problems can make better choices. One option is to modify stage-gate processes so that higher-risk compounds, such as those with novel mechanisms of action, get additional scrutiny in Phase II. Companies can also encourage better decision making by changing their incentive systems. They might, for example, segment incentives to reward good decision-making on higher-risk compounds and encourage tough decisions. On the other hand, they might incentivize scientists working on lower-risk compounds based on faster development times, and ensuring commercial differentiation. For higher-risk compounds, they may also want to use independent review boards, composed of people outside the company.

Another novel approach could be the use of internal information markets. A company would use an IT system to enable access to enterprise perspectives on, for example, whether a specific compound was worth developing based on preliminary results. The approach typically includes an incentive for the broader organization to participate in a fun way, for example by creating a simulated stock market. This bottoms-up approach can help reveal interesting insights on whether the decision makers are in tune with the realities of the compound and are indeed making truly objective choices. Information markets can help companies make better decisions – and get better bottom-line results – by enhancing their forecasting, sharpening their reading of markets, and helping them to monitor their competitors more accurately.

Risk-Based Segmentation

But all of the ways to improve execution in late-stage testing require companies to segment their development candidates on the basis of risk and then differentiate their development tactics accordingly. For low-risk compounds, companies might want to increase the speed of clinical trials – even if this approach costs more. Finding a way to shorten the common six- to nine-month delay between the end of Phase II and the beginning of Phase III, for example, could help bring many successful drugs to market faster. In contrast, higher-risk compounds should undergo additional scrutiny, not only during Phase III but also during Phase II. Furthermore, new indications for these compounds should be added carefully for Phase III, given their higher attrition rates.

Another area to examine is team structure. Lower-risk compounds could benefit from groups that focus on making operations more efficient, speeding the drugs' passage to market. Higher-risk compounds might benefit from teams led by people with unusually good scientific and business judgment, who would be able to ensure that unpromising compounds are killed early. Given the fact that fewer of these high-risk drugs are likely to be successful, these team leaders may be able to manage several projects simultaneously, thus reducing costs. High-risk compounds will also benefit from a network of experts – both internal and external. Therapeutic area experts can provide important insights about disease mechanisms, and these insights can help inform the design and conduct of clinical trials, as well as the go/no-go decisions.

Risk based segmentation can also have implications for licensing and asset sourcing strategies. Often we find companies using standard attrition rates in Phase III to help them value and structure deals. They shouldn't: attrition rates vary too much. Deal terms should reflect these differences through appropriate adjustments in milestones and royalties. For example, compounds with known mechanisms of action could merit higher up-front payments, while novel ones could receive less up front and higher royalties down the line – even for compounds that are currently in Phase III. Pharmaceutical companies could also think about sharing the development costs for novel compounds with venture capitalists or other investors—granted they could structure a deal that provides the investor with enough upside to justify his risk.

Companies need to also think about how they can balance their late-stage portfolios more effectively, ensuring that their mix of high- and low-risk compounds matches their appetite for risk and their R&D strategy. They could strive for a better balance between large and small molecules, for example, since biologics—while not the focus of this analysis--fail less frequently in the clinic than chemical entities. Likewise, companies might want to pursue lower risk assets internally while building a network of alliances to share risk on chancier compounds, even in late stages of development.

Improving Clinical Trial Design

Another critical way to decrease Phase III attrition is to do smarter work in Phase II and in linking Phase II and III trials. One way to do this is to make sure that both Phase II and Phase III trials use a consistent set of end-points. Another is to make sure that Phase II trials include a more heterogeneous population, since more diverse populations will, in most cases, better anticipate the larger pools of patients to be studied in Phase III.

Pharmaceutical companies should make sure they settle on the right doses in Phase II trials, before advancing to Phase III. Again, that seems obvious but it can only be done if drug firms test a wider range of dosages in Phase II than most trials currently include. Companies should also consider pursuing adaptive trial design approaches to clinical trial design earlier. Such approaches can enable broader screening of data and refinement of designs based on continuous analysis of data. For example, companies can drop and add arms to Phase II trials depending on the efficacy outcome at different doses.

Companies should also be much more rigorous with compounds that looked only marginally efficacious in Phase II but that they have nevertheless decided to advance to Phase III. Study protocols should include multiple interim analyses; this way, the company can decide to kill unsuccessful projects earlier in Phase III than they do now. None of these suggestions will eliminate Phase III failures, but they should reduce the number and cost of failure. Moreover, as companies begin to explicitly and indeed publicly adopt them, they will as a consequence be responding to investor concerns. More and more, investors will prefer companies whose managements demonstrate that they understand pipeline attrition by segmenting their development-stage compounds by relative risk and applying the appropriate development strategies for each risk category. Big Pharma is no longer accorded premium valuations for merely meeting earnings expectations. Instead premiums come from the near term probability of revenue growth – primarily driven by their Phase III pipelines. Systematic risk-based segmentation of late stage compounds will provide investors tools to better assess these pipelines to justify appropriate share-price premiums. The one-size-fits-all development model might have worked when the vast majority of targets for new drug candidates were well understood. In the era of genomics, it doesn't.

The authors are members of McKinsey & Company's New York office. Tamara Elias is an engagement manager; Navjot Singh is an associate principal; and Maria Gordian and Rodney Zemmel are both partners. The authors would like to acknowledge the contributions of Naureen Alamin and Andy Qiu.

Behind the Analysis: Deriving the Attrition Rates

We used the Pharmaprojects database to compile a list of 1,515 small molecules for which Phase III results had been reported between 1990 and 2002. To this list we added another 112 compounds identified through the Evaluate database.

To ensure the consistency of our data, we focused on new chemical entities, excluding biologics, sponsored by Big Pharma. We then removed any compounds in ongoing trials, even if preliminary results had been reported, as well as studies of drugs already on the market.

From these compounds, we looked for those which had failed in Phase III trials. We used analyst reports and company press releases, as well as public databases like Medline to ascertain outcomes. We defined failed trials as those that ended early (unless the reason for early termination was strongly positive findings) and those that did not produce results sufficient to secure FDA approval for the studied compound. The trials that produced successful results were analyzed separately to allow us to ascertain the compounds' mechanisms of actions and to help calculate attrition rates.

Finally, we scrubbed through the data again to remove duplicate entries and trials that had erroneously been categorized as failures or that had actually reported results outside of our specified time period. This detailed evaluation also permitted us to identify the trials for which the root cause of failure could be determined. At the end, we had 73 total failed trials for which we could provide a reason for failure (See Exhibit 5.)

Failure to show efficacy vs. placebo was one of the most important reasons trials didn't work. But since very few oncology trials use placebo controls--instead testing new drugs against the best available treatment regimen--we had to use somewhat different definitions of the root causes of failure in oncology trials. If an oncology trial compared standard treatment against the combination of standard treatment plus a new drug (e.g., surgical resection vs. surgical resection plus vinorelbine/cisplatin for non-small-cell lung cancer), we considered the trial to be equivalent to a placebo-controlled study. If the new drug was ineffective, we defined it as having failed due to lack of efficacy vs. placebo.

In contrast, if the trial substituted the new drug for one of the components of a standard regimen (e.g., doxorubicin and docetaxel vs. doxorubicin and cyclophosphamide for breast cancer), we defined the trial as a comparator trial. In this case, we ascribed failure to lack of differentiation.

If a trial failed because of a single cause, that cause was assigned 1 point. If the trial failed because of two causes, each cause was assigned 0.5 points. (No trial failed because of more than two causes.) To calculate the percentage of failures, we summed the number of points assigned to each cause and then divided the total by the number of compounds. The results were rounded to whole numbers.

End-Point Objectivity

More objective end-points were defined as those that could be measured with easily reproducible diagnostic tests like ECGs for arrhythmia patients or changes in MRI lesion volume for stroke patients, or widely used scales based on interviews conducted by health professionals (e.g., the Hamilton depression scale). Less objective end-points relied on measurements that are not easily reproducible, scales that are not widely used, or patients' self-report (e.g., pain diaries).

The attrition rate was defined as the number of trial failures from 1990 to 2002 divided by the combined number of failures and successes. The number of trial successes was derived from our original database using the same procedure employed to identify trial failures.