Skip to content
Globe Icon

But does it work? The challenge of evaluating complex integration projects: the experience from England

2014 Conference Presentation

Care integrationEvaluation EnglandUnited Kingdom

2 September 2015

But does it work? The challenge of evaluating complex integration projects: the experience from England

Ruth Thorlby, Nuffield Trust, United Kingdom

Abstract

In the past three years, the NHS in England has embarked on an ambitious programme to enable a faster pace and scale of integration between health and social care services, including supporting 14 ‘Pioneer’ innovation sites across England and creating a ‘Better Care Fund’ worth £3.8 billion from 2015/16 for regional initiatives in joint working between health and social care services. Many of the projects that are bidding for financial support from central government aim to streamline services across organisations for vulnerable groups, including people with long-term conditions, older people with multiple conditions, and people at the end of life. These integration projects are using a variety of techniques, including better identification of patients at risk of emergency hospital admissions, care-coordinators and multi-disciplinary teams to target and organise services around people in their homes and the community, enabling a wider range of services to be provided by both the state and voluntary sector, all underpinned by improved information systems that are shared between services.

The objective of these integration initiatives include better user experience and reducing costs to the health and social care system, particularly the pressure on emergency departments in hospitals. Emergency admissions in England have risen by 47 per cent between 1997/8 and 2012/30, compared to a 10% growth in the overall population. The evidence base underpinning these initiatives suggests that they will not necessarily meet all of their objectives, especially reducing pressure on hospitals.

This presentation will draw on the Nuffield Trust’s experience of evaluating complex integrated and community based care initiatives within the past five years. Over 30 projects have been evaluated, in collaboration with other organisations and many funded by the UK’s Department of Health. The projects have included projects using support workers to target older people at risk of hospital admission; ‘virtual wards’ which use multi-disciplinary teams to deliver home based care for at-risk patients; a large randomised controlled trial of tele-health and telemedicine; and an end-of–life home based nursing service. Many of these evaluations have investigated the impact of interventions on the use and cost of health services using primarily quantitative methods, including innovative ‘matched control’ techniques to track intervention patients compared to control group patients with similar characteristics and patterns of service use, to avoid the phenomenon of ‘regression to the mean’. The findings of these evaluations have shown that nearly all of them have failed to reduce hospital use (and in some cases have increased emergency admissions) with the sole exception of the end of life nursing care.

The presentation will explore the implications of these findings, including the challenges of evaluating projects which are implemented in different ways in different areas; how to account for contextual differences, such as organisational factors, leadership and differing timescales in implementation and enabling change in working practices. We will discuss the implications of our experience for designing intelligent evaluation that can both capture the complexity of these projects and be of benefit to those wanting to innovate in the future.

Audio