To Hell with the 3L! NICE’s Missed Opportunity to Upgrade Health Outcome Measurement

This blog post is the second in a series on the new National Institute for Health and Care Excellence (NICE) health technology evaluation manual. Each post provides a critical discussion on a particular topic, including the expected implications of the changes (or lack thereof) in the manual; what is still missing; and what further research is needed.

What is the purpose of a ‘methods review’ for a health technology assessment (HTA) agency? For NICE, to quote Sir Andrew Dillon, it is “part of the regular review and refresh of [NICE] methods to ensure that they are robust and up-to-date.” So you might be surprised when I tell you that NICE has chosen to stand by a 30-year old algorithm to value health improvements, despite a more robust (and more up-to-date) approach being available.

What’s the story?

I am writing, of course, about the EQ-5D. The EQ-5D is a tool for describing health states in a way that is relevant across disease areas and health care settings. Its demonstrated ability to detect changes in health resulting from many (perhaps most) health technologies has established it as a cornerstone for HTA in England and elsewhere. Crucially, researchers and technology developers use the EQ-5D to generate quality-adjusted life years (QALYs), NICE’s preferred value metric. And while alternative tools to the EQ-5D exist for this purpose, NICE has long favoured the EQ-5D.

The first version of the EQ-5D, now known as the EQ-5D-3L, was embraced by NICE and has been formally recommended since 2008. To generate QALYs from EQ-5D data, it is necessary to apply ‘weights’ to the responses. These are derived by surveying a representative general public sample and asking people about their preferences for different health states. The resulting weights are known as a ‘value set’. When NICE embraced the EQ-5D-3L, it embraced the available value set for the UK, based on a study conducted in 1995.

A lot has happened since 1995. NICE didn’t exist back then. The science of health measurement and preference elicitation has come on leaps and bounds. The EuroQol Group, which develops the EQ-5D, has spent millions of euros on research (some conducted by OHE). Most importantly, researchers developed an improved version of the EQ-5D, called the EQ-5D-5L, and a value set for England for the new instrument.

The EQ-5D-5L was first described in 2011, following a research programme led by OHE’s Mike Herdman. Then, in 2016, a team led by OHE published the EQ-5D-5L value set for England (subsequently published in Health Economics). These studies were part of a wider research programme that has comprehensively demonstrated the superiority of the EQ-5D-5L over the EQ-5D-3L. And, yet, even following the latest methods review, NICE is still recommending the EQ-5D-3L and its accompanying data on the preferences of the 1995 UK population.

NICE’s objection is not to the EQ-5D-5L instrument, but to the associated value set. This has created a very confusing situation. In the new manual, analysts are told that “NICE does not recommend using the EQ-5D-5L value set for England”. Then, in the same paragraph, we read that “[NICE] support sponsors of prospective clinical studies continuing to use the 5L version of the EQ-5D descriptive system to collect data on quality of life.” What a mess!

What went wrong?

In a recent editorial published in PharmacoEconomics – Open, I have tried to unpick what happened and why. Following the publication of the new EQ-5D-5L value set, NICE asked some academics to review it. Bear in mind that the publication had already undergone peer review with Health Economics, a leading academic journal. When NICE initiated this ‘quality assurance’ process, they did not clearly specify – or perhaps simply misspecified – its purpose.

If you ask an academic to identify some shortcomings in a piece of research, they will oblige. No study is flawless. There is no gold standard against which to compare the methods of a study of this kind.

The question that NICE should have asked, the only question of any relevance to their decision about the EQ-5D-5L value set, is whether it was superior to the current approach (i.e. using the 1995 study). In this respect, the problem facing NICE was very similar to the kind of decision problems they manage daily in health technology assessment. NICE did not specify the comparator, and so the comparator became perfection.

A missed opportunity

The methods review was an ideal opportunity for NICE to take stock of what has happened since 2018, when the ‘quality assurance’ report was first published, and to recognise it as a debacle. As new methods are developed, and as the public’s preferences evolve, use of the old EQ-5D-3L value set becomes increasingly problematic. NICE decisions may not reflect public preferences to the extent that we are now able to measure them.

In their avoidance of this issue, NICE continue to dig themselves deeper. Now we must wait for a new value set to be developed. In all likelihood, it will look very similar to the original, and the cost will be measurable in wasted time and potentially misallocated health care resources. And, if it doesn’t look similar, what then? NICE will face an even bigger challenge, as the new value set must surely face at least as much scrutiny as the original value set for England to justify the whole endeavour.

The weight of the evidence is strongly in favour of the EQ-5D-5L and its associated value set. That’s why NICE’s decision not to recommend the EQ-5D-5L value set for England in its new manual is a missed opportunity. Researchers and analysts are left wondering how long the current impasse will last. The solution is clear. To hell with the 3L!

Related Research

Sampson, C., Parkin, D. and Devlin, N., 2020. Drop Dead: Is Anchoring at ‘Dead’ a Theoretical Requirement in Health State Valuation?. OHE Research Paper.

Sampson, C., 2022. NICE and the EQ-5D-5L: Ten Years Trouble. PharmacoEconomics Open 6, 5–8. DOI

Devlin, N.J., Shah, K.K., Feng, Y., Mulhern, B. and van Hout, B., 2018. Valuing health-related quality of life: An EQ-5D-5L value set for England. Health Economics, 27(1), pp.7–22. DOI

Herdman, M., Gudex, C., Lloyd, A. et al., 2011. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res 20, 1727–1736. DOI

van Hout, B., Mulhern, B., Feng, Y., Shah, K. and Devlin, N., 2020. The EQ-5D-5L value set for England: response to the “Quality Assurance”. Value in Health, 23(5), pp.649-655. DOI

Related blogs

A series on the new National Institute for Health and Care Excellence (NICE) health technology evaluation manual

• • New NICE Manual for Health Technology Evaluations: A Critical Discussion on the Most Relevant Changes (or Lack Thereof)
  • Including Carer Quality of Life in Health Technology Evaluation: Are We There Yet?
  • The NICE Health Technology Evaluation Manual: A Fresh Perspective Needed?
  • NICE’s severity modifier: a step in the right direction, but still a long way to go