25 Years of the Cost Effectiveness Acceptability Curve

In 1994 Ben van Hout introduced the concept of the cost effectiveness acceptability curve (CEAC). In the 25 years since, the CEAC has become a standard part of health economic evaluations and incorporated into health technology assessment guidelines. Professors Nancy Devlin and Andrew Briggs spoke at a reception at ISPOR Europe this year to celebrate these contributions. As 2019 comes to a close, OHE reflects on 25 years of the CEAC.

In 1994 Ben van Hout, along with Maiwenn Al, Gilad Gordon, and Frans Rutten, published an article introducing the concept of the cost effectiveness acceptability curve (CEAC). In the 25 years since, the article has received close to 1,000 citations; the CEAC itself has become a standard part of health economic evaluations, and incorporated into health technology assessment methods guidelines in the UK and internationally. ISPOR President and OHE Research Fellow Professor Nancy Devlin and Professor Andrew Briggs of the London School of Hygiene and Tropical Medicine spoke at a reception at the ISPOR Europe conference in November to celebrate Ben’s contribution. As 2019 comes to a close, OHE reflects on 25 years of the CEAC and summarises the presentations given by Professors Devlin and Briggs.

The CEAC was introduced as an approach to dealing – in a statistically meaningful way – with the uncertainty inherent in cost-effectiveness calculations. Uncertainty means decision makers may make wrong decisions – adopt a technology which turns out not to be good value for money, or not adopt one that is cost-effective. Ben Van Hout’s contribution was firstly to provide a mathematically robust means to estimate this uncertainty, and secondly to offer an intuitive graphical representation of it which could be interpreted by non-mathematicians.

The incremental cost effectiveness ratio (ICER) represents the additional cost of realising one extra unit of health effect, and is the key statistic needed to understand the cost-effectiveness of one healthcare treatment relative to another. Prior to the CEAC, estimates of uncertainty around an ICER relied on the calculation of confidence intervals around the joint distribution of the cost and effectiveness estimates. This presented two key mathematical issues.

Firstly, since the ICER is a ratio of two random variables, there are discontinuities around zero, which cause problems for estimating confidence limits. Secondly, the use of confidence intervals assumes that the ICER’s sampling distribution follows a normal distribution, which is problematic in the (frequent) situations when sampling may be skewed. As Professor Devlin explained at ISPOR, with the CEAC, van Hout introduced a novel approach drawing on Fieller’s theorem. This involves estimating the plane of cost and effect pairs which could occur with 95% probability – thereby avoiding discontinues around zero, and only assuming a normal distribution of the averages of the costs and effects (as follows from the Central Limit Theorem).

Whilst this approach presented a significant step forward in resolving the technical challenges in estimating the uncertainty around the ICER, the plane of cost and effect pairs is on its own difficult to interpret. As illustrated below, pairs typically appear in all four quadrants – with the possibility that one treatment is both more effective and cheaper than the other, and less effective and more expensive.

The crucial second step of van Hout’s contribution was to translate the plane into a single ‘acceptability curve’, which represents the proportion of the pairs that are acceptable for each given ICER. The CEAC thereby provides a single, easily interpretable statistic for policymakers: at a given cost for realising one extra unit of health effect, there is a certain probability that the treatment under evaluation is acceptable.

Professor Briggs, the foremost global authority on uncertainty in cost-effectiveness analysis, recounted at ISPOR how, in the academic year 1993/4 he had just embarked on a PhD on this topic. “Imagine my dismay when I learned of Ben’s article, published early in 1994 – how could I embark on a PhD in this area when the problems I had identified in my early reading had now been solved?”  Yet, he explained, “I had very good advisors, who recognized what I did not at the time:  that truly innovative papers often need assistance in showing their importance and relevance for the field.” Indeed, testament to Professor Briggs’s contribution to the popularisation of the use of the CEAC to support decision making was that “I remember a comment from Ben sometime after my PhD was completed: that I had become a greater advocate for the CEAC than he himself!”

Cost-effectiveness plane and cost-effectiveness acceptability curve comparing the intervention group to the control group. Reproduced from Li et al., 2018.

As Professor Devlin explained, the CEAC has become a mainstream part of health economic evaluations in the last 25 years. She noted that “the substantial impact of the paper is also clear from its inclusion in methods guidelines, including NICE’s methods for technology assessment and in the recommendations from the US 2nd Panel on cost effectiveness in Health and Medicine (JAMA, 2016) “. She also noted that some useful extensions to it have been introduced, “for example, to allow for the comparison of more than two alternatives, or graphically present the point at which one alternative becomes preferred to the other (the Cost Effectiveness Acceptability Frontier).”

Both Professors Briggs and Devlin noted that the value of the CEAC, in providing a widely understandable representation of uncertainty, has not only endured but is becoming ever more relevant as decision-makers face greater uncertainty about the efficacy of new healthcare treatments. As Professor Briggs summarized “The test of a good paper is its durability in terms of relevance.  I would argue that this was a paper before its time and, if anything, this paper is more relevant today than when it first appeared.  And while the methods for calculating the CEAC have thankfully evolved (no more grappling with double integration of the joint density of cost-effectiveness on the cost-effectiveness plane) – the usefulness of the concept has endured.  This is a paper that should rightfully be considered a classic in our field.”

Professor Devlin added that “this paper is one of many ground breaking pieces of work Ben has produced over the years in a career which has established him one of our most gifted and creative innovators in the field – and (on a personal note) someone who it is a real privilege to continue to work with.”

OHE joins Professors Devlin and Briggs in celebrating the contribution of the CEAC and its lead author, Ben van Hout, to health economics and decision making.

Further reading

• O’Brien, B.J. and Briggs, A.H., 2002. Analysis of uncertainty in health care cost-effectiveness studies: an introduction to statistical issues and methods. Statistical Methods in Medical Research, 11(6), pp.455–468. 1191/0962280202sm304ra.
• Briggs, A.H., Mooney, C.Z., Wonderling, D.E., 1999. Constructing confidence intervals for cost‐effectiveness ratios: an evaluation of parametric and non‐parametric techniques using Monte Carlo simulation. Statistics in Medicine, 18(23), pp.3245-3262. 1002/(SICI)1097-0258(19991215)18:233.0.CO;2-2.
• Fieller, E.C., 1954. Some Problems in Interval Estimation. Journal of the Royal Statistical Society: Series B (Methodological), 16(2), pp.175–185. 10.1111/j.2517-6161.1954.tb00159.x.
• Li, J., Parrott, S., Sweeting, M., Farmer, A., Ross, J., Dack, C., Pal, K., Yardley, L., Barnard, M., Hudda, M., Alkhaldi, G. and Murray, E., 2018. Cost-Effectiveness of Facilitated Access to a Self-Management Website, Compared to Usual Care, for Patients With Type 2 Diabetes (HeLP-Diabetes): Randomized Controlled Trial. Journal of Medical Internet Research, 20(6), p.e201. 2196/jmir.9256.
• National Institute for Health and Care Excellence, 2013. Guide to the methods of technology appraisal 2013: Process and Methods. Available online: https://www.nice.org.uk/process/pmg9/chapter/foreword
• Sanders, G.D., Neumann, P.J., Basu, A., Brock, D.W., Feeny, D., Krahn, M., Kuntz, K.M., Meltzer, D.O., Owens, D.K., Prosser, L.A., Salomon, J.A., Sculpher, M.J., Trikalinos, T.A., Russell, L.B., Siegel, J.E. and Ganiats, T.G., 2016. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA, 316(10), p.1093. 1001/jama.2016.12195.
• Fenwick, E., Claxton, K. and Sculpher, M., 2001. Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Economics, 10(8), pp.779–787. 1002/hec.635.