IAPT national and local performance data 

This page provides an overview of recent developments in the evaluation of the Improving Access to Psychological Therapy (IAPT) programme, and offers tools to help identify local Primary Care Trust (PCT) performance against national data. It also provides an opportunity to benchmark non-IAPT service performance data for recovery against the IAPT recovery benchmark. 

I would welcome comments on the content of this page, and the tools provided which are explained below. You can either call me, email me or use the website's contact form - you'll find all of those details here Contact me. I'm aiming to update this data and the tools when the next set of finalised performance data is published in and feedback will help to shape their development. 

Within this page you'll find the following sections:


IAPT performance – what’s the story so far?

There have been a number of programme evaluations of IAPT as it has unfolded. In addition to the evaluations of the early pilot sites in Doncaster and Newham, two further reports are of note:


  1. A comprehensive evaluation published in 2010 of the first wave of roll-out sites that started operation in 2008 1 
  2. A study of the first three years of IAPT operation published in November 2012

PCT’s have been required to submit data on a quarterly basis against a range of key performance indicators (KPI’s), for example the number of people in the local population who have depression and/or anxiety disorders, the proportion of people referred for psychological therapies who are entering treatment, and the proportion of people who have ‘recovered’. These data are publicly available and are published by the NHS Information Centre in the form of downloadable spreadsheets.

 

What follows is an exploration of quarterly IAPT performance for the six quarters between April 2011 – September 2012, based on submissions at individual PCT and Strategic Health Authority (SHA) levels.

  

While there are some encouraging overall signs within the data, including the fact that a good proportion of people can be shown to recover from their symptoms, the data raises some critical issues about the wide variations in performance across PCT’s with respect to rates of access to services, as well as rates of recovery.

 

The differences between the numbers of people being referred, and those with recovery data, is sometimes very stark. For an illustration see my letter Nine out of 10 people not helped by IAPT? published in  Therapy Today in February 2011 - it's one of the attachments below.

 

NOTE: The term recovery as used within IAPT has a very specific definition which may not be the same as your definition, if you have one. You'll find more details below in the section Benchmarking your own service recovery data against the IAPT data


References


1. North East Public Health Observatory (2010) Improving Access to Psychological Therapies: A review of the progress made by sites in the first rollout http://www.iapt.nhs.uk/silo/files/iapt-a-review-of-the-progress-made-by-sites-in-the-first-roll8208-out-year.pdf

2. Department of Health (2012)  IAPT three-year report: The first million patients.  http://www.iapt.nhs.uk/silo/files/iapt-3-year-report.pdf 



What’s the purpose of this analysis?

There are three basic aims to this analysis:

1.    To illustrate the variations in relative performance across PCT’s

2.    To show how to identify local PCT data from within the overall data

3.    To enable services that collect data from routine outcome measurement to contrast their own performance against national benchmarks

Relative performance across PCT’s

Some of the KPI data from the NHS Information Centre is in the form of raw numbers (e.g. the number of people entering treatment), and some in percentages (e.g. the proportion of people moving to recovery).

Data in number format isn’t so useful for comparing local and national data. The three benchmarks shown below are for data published using percentages, which are much easier to produce benchmarks from. From April 2011 PCT’s were required to report three indicators expressed as percentages:

1.    The proportion of referrals entering treatment

2.    Those entering treatment as a proportion of people in the local population who have depression and/or anxiety disorders

3.    The proportion of people completing treatment that are moving to recovery

The three indicators published as percentages are shown in the form of visual benchmarks, namely quartile ranges (essentially 25% slices) of the full range of performance against the relevant KPI.

By extracting data for local PCT’s it is possible to see where on the range an individual PCT lies.

Profiling local PCT data

In spreadsheet form the data is difficult to digest. To assist I’ve produced a Microsoft Excel template, which is a downloadable attachment below. Using this it’s possible to aggregate the data – for example from one PCT - from the numerous worksheets in the published KPI data, into one place to build an overall picture of performance. The overall national KPI performance data is shown in the template, and a column alongside this allows you to enter the relevant data for a local service, so that you can compare the two.

Since some of the KPI data is in number format only, which isn’t useful for comparing local and national data, I’ve also added some additional calculations in the template that will take the published numbers and turn them into a percentage (e.g. the number of people who are ‘moving to recovery’ as proportion of those completing treatment).

Please see the next section for more details. 

 


How to retrieve and benchmark data for your local service

1.    Download (rather than View) the summary spreadsheet template from the attachments at the bottom of this page called IAPT_summary_template_Feb2012.xlsx  Use the blue arrow on the right to download the sheet

2.    Download the IAPT data for the quarter you want to explore from the NHS Information Centre website here http://bit.ly/XmBxRC  

Note that within each section some data is final and some (earlier) data is provisional. This is because PCT’s are permitted an opportunity to amend provisional data at a later stage. It is therefore advisable to work with final data - the screenshots below illustrate. 

Select IAPT date range
Select the file you want to download. 

IAPT data file selections

3.    In the summary template, for each cell highlighted in orange, find the data for the relevant local KPI or indicator in the IAPT file you have downloaded as follows: 

4.    Go to the relevant worksheet tab

5.    Use Find (Ctrl+F ) in the worksheet document to find data for a local PCT. E.g. type ‘bristol’ in Find for data on Bristol PCT and it will find all entries for Bristol in that spreadsheet                                                                 

6.    Keep Find open for the next query 

7.    Enter data into the relevant cell in the Summary sheet

8.    NOTE: do not enter data into the green shaded cells – these contain functions which will automatically calculate the value for that cell based on data you have already entered 

9.    Go to the next worksheet and click Find next

10. Repeat till you have entered data for all the indicated cells

Once complete, you will be able to compare the performance of the local service against the national average. 

NOTE: 

Full definitions of terms used in the template spreadsheet can be found under a separate tab

All national data (both numbers and percentages) in the summary spreadsheet are drawn from the relevant sections of the Improving Access to Psychological Therapies, Key Performance Indicators (IAPT KPIs) - Q1 2012-13 final and Q2 2012-13 provisional: Q1 Final Tables [.xls]  spreadsheet, with the exception of data in the cells shaded green. These are my own calculations (to see how these were obtained see below under Additional Spreadsheet calculations)

The Summary sheet is protected to prevent accidental over-write of formulas within it. You can enter data, and if you wish to save it, simply save as a new file to your hard drive or network

KPI 5: The number of people who completed treatment in the quarter is not a direct subset of the number of people referred in the same quarter, as some may have been referred for treatment in a previous quarter

KPI 5: The number of people who have completed treatment (minimum 2 treatment contacts) during the reporting quarter

Disclaimer

While I have made every effort to ensure the accuracy of data in the summary worksheet I cannot be held responsible for any loss as a result of your interpretation or subsequent use of this data. It is intended not to provide or lead to definitive judgements about performance – it needs to be considered in the context of the local service and locality where it originated.

Similarly, I can take no responsibility for erroneous data in the original work, whether provided by the individual PCT or by the NHS Information Centre.



How to benchmark local data against the national IAPT ranges

Having identified the data for a local service you can now see where that data sits within the national range for the KPI's for which I’ve been able to construct a benchmark:

1.    The proportion of referrals that are entering treatment (discontinued from April 2012)

2.    The people who have entered (i.e. received) treatment as a proportion of people with anxiety or depression

3.    Recovery - the number of people not at caseness at their last session, as a proportion of people who were at caseness at their first session

The benchmarks are based the respective quarter for which there is final (rather than provisional) data. From April 2012 use of the first of these – the proportion of referrals entering treatment – has been discontinued. Thus, this benchmark is based on data for the quarter January - March 2012, and the others are based on data for the quarter April - June 2012. 

The three benchmarks below show the range of performance against each benchmark across all PCT’s that have valid data. That range is split into four equal parts or quartiles, hence each shaded block represents the range of 25 percent of services. The markers adjacent to each line that separates the quartiles give the range for that quartile.

For example, the range of recovery rates for services runs between 12% and 97% (Benchmark 3 below). The 25% of services with the lowest recovery rates achieved between 12% - 40% recovery. The 25% of services with the highest rates of recovery achieved between 51% - 97%, and so on.

For each of the three indicators for which it has been possible to construct benchmarks simply note the local PCT value for the relevant indicator and determine where on the benchmark range it lies and in which quartile.


Benchmark 1. The proportion of referrals that are entering treatment

Referrals entering treatment


Example: my local service shows 56% of clients referred entering treatment in Quarter 4 2011. That is below the national mean figure of 60%, and puts the service in the second quartile for services (49% - 60%) 


Benchmark 2. The number of people who have entered (i.e.received) treatment as a proportion of people with anxiety or depression 

People entering treatment as % of those with anxiety or depression

Benchmark 3. Recovery rate: Number of people not at caseness at their last session, as a proportion of people who were at caseness at their first session

IAPT recovery rates by PCT


Key points to note:
  1. There are just three benchmarks because these are the only KPI’s where a percentage figure is given. Other percentage figures in the Summary spreadsheet (shaded in green) are my own, however, to build benchmarks for these would require calculating a percentage figure for each PCT separately (around 450 calculations!)
  2. The IAPT programme target for patients moving to recovery is 50%, based on the proportion of all patients that complete treatment. The published headline recovery rate shown above excludes patients that were non-case at the start of therapy and for the quarter is 46%. By this measure the programme is approaching its target of 50% recovery. 

    However, if we contrast patients moving to recovery with those at other time points we find, for example, 

    As a proportion of all those completing treatment (the stated target for IAPT) the recovery figure is 39.3%. Further, of those entering treatment it’s 24.3%, and of those referred just 14.8%. In other words, less than fifteen in every hundred patients referred to IAPT services can be shown to have recovered. 



Benchmarking your own service recovery data against the IAPT data

If you have data about your own service performance you may be able to benchmark your own data against the national IAPT data – BUT FIRST, A WORD OF WARNING!

The term recovery has a specific meaning within IAPT which may not be the same as that used within other measures and measurement systems. Within the IAPT data recovery is defined as the:

“Number of people not at caseness at their last session, as a proportion of people who were at caseness at their first session”

This simply means that clients moved from a score which indicated a clinical level of symptoms, to one more representative of the normal or non-clinical range. In the case of GAD-7 the cut-off is 8, so recovery involves moving from a score of 8 or more to a score of 7 or less (scoring range is 0 – 21). For PHQ-9 the cut-off is 10, and recovery involves moving from a score of 10 or above to a score of 9 or less (scoring range is 0 – 27).

In other words, recovery in IAPT terms may involve just a one point reduction in the client’s score.

Elsewhere, the term recovery is defined differently. A move from a clinical level to a non-clinical level of symptoms is also known as clinical improvement. But as it may involve only a single point change, there is also a question of whether that change is reliable e.g. whether it might have equally occurred by chance. The threshold for reliable change for GAD-7 is 4 points and for PHQ-9, 6 points. From the PCT data it is not possible to know for what proportion of the clients who showed recovery, that improvement was also reliable. 

You can find definitions of these terms elsewhere on this site in the glossary under Getting Better Outcomes

Recovery in other systems, for example the CORE system, involves achieving both clinical and reliable change, so it’s a rather higher bar.

In other words, to compare recovery in IAPT terms with the CORE system results, you need to look at the proportion of clients that achieve clinical change only as reported by the CORE system or software, not at the system's clinical and reliable change which is a more stringent criterion.


How has IAPT developed over time?

For each of the two current benchmark indicators  i.e. people who have entered treatment as a proportion of people with anxiety or depression, and the proportion of people recovered, I have plotted the quartile ranges over time for each on the two graphs immediately below. This allows us to see trends in the data. 

The first graph shows the quartile ranges over the six quarters between April 2011 – September 2012. The data for each quarter shows the highest and lowest proportions, by PCT, of people entering treatment as a proportion of those in the local population with anxiety or depression, as well as markers for the 25th, 50th and 75th percentiles.

Patients entering treatment by Quarter

The data over the six quarters shows wide variations in the proportions of people entering therapy by PCT area. The average range over all six quarters is between 0.18% and 6.02%. This suggests that patients in the area with the highest proportion have a thirty three times greater chance of entering therapy than those in the lowest area.  

Slightly less stark, but still significant ranges exist for the proportions of people moving to recovery (below). Beneath the headline recovery rates for each quarter there exist wide variations between PCT’s. Excluding recovery figures of 100% and over, the average range over the five quarters with finalised data shows a minimum rate of 11.2% and a maximum of 81.8%.  If accurate, then patients in the PCT area with the highest rate appear to have a seven times greater chance of recovery than those in the area with the lowest rate.  


IAPT recovery rate trends by PCT area


Note:  the recovery rates for PCT’s are based on the proportion of patients completing in a quarter who were above case level at start of treatment and are below at completion. By these criteria a recovery rate of above 100% would appear impossible. The recovery figures shown in the graphs above therefore exclude four instances of PCT data that show recovery rates of 100% (twice), 134% and 157% that are assumed to be erroneous. They do, however, include counts of 0% recovery in the first two quarters, which may or may not be accurate.

The accuracy of other recovery data cannot be confirmed, but the extremes reported in the earlier quarters do not appear to be evident in the data for the fifth quarter, which may reflect improving data quality. 



Additional Spreadsheet calculations

All data (both numbers and percentages) in the Summary spreadsheet is drawn from the relevant sections of the IAPT_KPIs_Q1_v2 document, with the exception of data in the cells shaded green. These percentages are calculated as follows:

% of people referred as a proportion of the people with depression or anxiety disorders: KPI 3a: The number of people who have been referred for psychological therapies, divided by (/) KPI 1: The number of people who have depression and/or anxiety disorders

% of people waiting more than 28 days as a proportion of those referred:      KPI 3b: The number of active referrals who have waited more than 28 days from referral to first treatment/first therapeutic session (at the end of the reporting quarter) / KPI 3a: The number of people who have been referred for psychological therapies

% of referrals that are entering treatment:  KP4:  The number of people who have entered (i.e. received) psychological therapies during quarter KPI 3a: The number of people who have been referred for psychological therapies

% people completing treatment that were not case at commencement: KPI 6b The number of people who have completed treatment not at clinical caseness at treatment commencement / KPI 5: The number of people who have completed treatment during the reporting quarter. 

% recovered as proportion of those referred: KPI 6a: The number of people who are moving to recovery KPI 3a: The number of people who have been referred 

% recovered as proportion of all those entering treatment: KPI 6a: The number of people who are moving to recovery KPI 4: The number of people who have been entered psychological therapy

% recovered as proportion of all those completing treatment: KPI 6a: The number of people who are moving to recovery KPI 5: The number of people who have  completed therapy 




Feedback please!

I would welcome comments on the content of this page, and the tools provided. You can either call me, email me or use the website's contact form - you'll find all of those details here Contact me 

Thank you!



Ĉ
Barry McInnes,
20 Feb 2013, 08:48