july 2018 gp patient survey y12 questionnaire... · 2018-08-01 · the nhs england insight &...

17-043177-01 | Version 1 | Public | This work was carried out in accordance with the requirements of the international quality standard for Market Research, ISO 20252, and with the Ipsos MORI Terms and Conditions which can be found at http://www.ipsos-mori.com/terms. © NHS England 2018

July 2018

GP Patient Survey Questionnaire redevelopment

NHS England and Ipsos MORI

Ipsos MORI | GP Patient Survey: Questionnaire redevelopment


17-043177-01 | Version 1 | Public | This work was carried out in accordance with the requirements of the international quality standard for Market Research, ISO 20252:, and with the Ipsos MORI Terms and Conditions which can be found at http://www.ipsos-mori.com/terms. © NHS England 2018



Contents Acknowledgements ........................................................................................................................ 4

Glossary ............................................................................................................................................ 5

1 Background ................................................................................................................................. 1

1.1 What is the GPPS? .............................................................................................................................................. 1

1.2 Why did the questionnaire need to change? ................................................................................................. 1

1.3 A framework for questionnaire redevelopment ........................................................................................... 2

2 Governance ................................................................................................................................. 5

2.1 GPPS core project group ................................................................................................................................... 5

2.2 GPPS steering group ......................................................................................................................................... 5

2.3 Questionnaire development advisory group ................................................................................................. 6

3 Analysis of existing GPPS data ................................................................................................. 8

3.1 Introduction and objectives ............................................................................................................................. 8

3.2 Key findings and recommendations from analysis of existing GPPS data ............................................... 9

4 Stakeholder engagement ........................................................................................................ 13

4.1 Introduction and objectives ........................................................................................................................... 13

4.2 Stakeholder engagement about specific topics .......................................................................................... 13

4.3 Stakeholder workshop .................................................................................................................................... 18

4.4 Stakeholder engagement about the EQ-5D ................................................................................................. 26

4.5 Key findings and recommendations from stakeholder engagement ...................................................... 28

5 Engagement and research with patients ............................................................................... 31


5.2 Website feedback............................................................................................................................................. 31

5.3 Qualitative research ......................................................................................................................................... 36

5.4 Service user testing .......................................................................................................................................... 41

6 Cognitive testing ...................................................................................................................... 46


6.2 Cognitive testing .............................................................................................................................................. 47

6.3 Key findings and recommendations from cognitive testing .................................................................... 47

7 Next steps .................................................................................................................................. 53



List of Tables

Table 1.1: The GPPS questionnaire redevelopment timetable 2017 ........................................ 3

Table 3.1: Measures used for the survey response analysis ...................................................... 8

Table 4.1: Summary of stakeholder engagement about specific topics ................................ 14

Table 4.2: Current EQ5D national indicator trends ................................................................... 27

Table 4.3: Summary of EQ5D national indicator changes over time ...................................... 27



Acknowledgements

This report was written by the NHS England Insight & Feedback team and the Ipsos MORI GP Patient

Survey team, but was made possible by the support and advice of many individuals and

organisations; we would like to thank everyone who contributed to this process.

The NHS England Insight & Feedback team can be contacted at: [email protected].

The Ipsos MORI GP Patient Survey team can be contacted at: [email protected];

[email protected]; [email protected].

mailto:[email protected]






Glossary

Clinical Commissioning Group (CCG)

Clinical Commissioning Groups (CCGs) are clinically-led statutory NHS bodies responsible for the

planning and commissioning of healthcare services for their local area; there are currently 195 CCGs

in England. CCGs were created following the Health and Social Care Act in 2012, and replaced

Primary Care Trusts on 1 April 2013. All GP practices belong to a CCG, and many GP services are co-

commissioned with CCGs.

CCG Improvement and Assessment Framework (IAF)

NHS England has a statutory duty (under the Health and Social Care Act (2012)) to conduct an annual

assessment of every Clinical Commissioning Group (CCG). This is managed by the Improvement and

Assessment Framework (IAF), first introduced in April 2016, which is designed to drive improvement

by enabling local health systems and communities to assess their own progress from ratings

published online.

Commissioners

Commissioners are those involved in the planning and purchase of NHS and publicly funded adult

social care services. NHS England is responsible for commissioning high quality primary care services

for the population of England. Its commissioning policy is to move towards more place based,

clinically led commissioning, and sharing or delegating commissioning of primary medical care

services to Clinical Commissioning Groups (CCGs). In April 2018, 178 CCGs were fully delegated, 10

had joint arrangements and 7 had greater involvement (further information is available here).

Department of Health and Social Care (DHSC)

The Department of Health and Social Care (DHSC) is a ministerial department, responsible for

leading, shaping and funding health and social care in England.

EQ-5D

EQ-5D is a standardized instrument developed by the EuroQol Group, often used as an outcome

measure of health-related quality of life. The EQ-5D descriptive system was included in GPPS from

2011 to 2017, and comprises five dimensions: mobility, self-care, usual activities, pain/discomfort and

anxiety/depression.

https://www.england.nhs.uk/ccgs/

https://www.england.nhs.uk/commissioning/ccg-assess/

https://www.england.nhs.uk/commissioning/ccg-assess/

https://www.england.nhs.uk/commissioning/pc-co-comms/

https://www.gov.uk/government/organisations/department-of-health-and-social-care

https://euroqol.org/



Extended access

Extended access to general practice will mean that patients can access pre-bookable appointments

with GPs, nurses or other healthcare professionals on weekday evenings, Saturdays and Sundays. This

will include peak times of demand including bank holidays and across the Easter, Christmas and New

Year periods. Extended access to general practice is available in some areas of the country now and

will be available to the whole population from 1 October 2018.

GP Forward View

The General Practice Forward View (GP Forward View) was published in April 2016 by NHS England.

Developed with the Royal College of General Practitioners (RCGP) and Health Education England

(HEE), it sets out plans to improve patient care and access, investing in new ways of providing primary

care.

GP practice/surgery

The term GP surgery refers to a single site. It is increasingly common for a GP practice to consist of

several sites (i.e. several GP surgeries) at which patients can access appointments. A GP practice

usually consists of a team of healthcare professionals that includes – GPs, nurses, healthcare

assistants, practice managers and receptionists. Practices work closely with other healthcare

professionals such as health visitors, midwives, mental health services and social care services, in the

delivery of patient care.

General practice pharmacists

General practice or clinical pharmacists work within general practice consulting with and treating

patients directly. For example, they might provide extra help to manage long-term conditions, advice

for those on multiple medicines and better access to health checks, helping GPs manage the

demands on their time.

GP access hub

GP access hubs offer patients improved access to general practice services. The services offered in GP

access hubs may differ depending on the location however all will offer pre-bookable evening and

weekend appointments for patients registered with practices in the local area. Patients may be offered

an appointment at their nearest GP access hub if their own practice is unable to offer them an

appointment at a time that is convenient for them, including on weekday evenings, Saturdays and

Sundays. GP access hubs are not currently available across the whole of England, with different

arrangements in place locally.

https://www.england.nhs.uk/wp-content/uploads/2016/04/gpfv.pdf

https://www.england.nhs.uk/gp/gpfv/about/

https://www.england.nhs.uk/gp/gpfv/workforce/building-the-general-practice-workforce/cp-gp/



Multi-coding

All questions in any survey are designed for a participant to choose either one answer from those

presented, or more than one answer (multi-coding). For most data collection methods it is possible to

control this, for example, those administered face-to-face, via telephone or online, where these rules

can be pre-programmed for each question. However, in any paper-based self-completion survey it is

possible for participants to select multiple responses in instances where they are expected (or

instructed) to give only one.

NHS Outcomes Framework

The NHS Outcomes Framework is a set of indicators developed by NHS Digital and the Department

of Health and Social Care (DHSC), designed to provide national level accountability for the outcomes

the NHS delivers. The indicators are grouped around five domains, focused on improving health and

reducing health inequality:

▪ Domain 1 - Preventing people from dying prematurely

▪ Domain 2 - Enhancing quality of life for people with long-term conditions

▪ Domain 3 - Helping people to recover from episodes of ill health or following injury

▪ Domain 4 - Ensuring that people have a positive experience of care

▪ Domain 5 - Treating and caring for people in a safe environment and protecting them from

avoidable harm

Official statistics

Official statistics are statistics published by a government department, agency or public body,

governed by a Code of Practice for Official Statistics.

The data produced by GPPS meets the main criteria set for an Official Statistic, including being used

publicly in support of major decisions on policy and resource allocation, being in the public interest,

being made available on a regular basis (annually), having a robust methodology, being nationally

representative, and being available in the public domain (published openly on the GPPS website).

https://beta.digital.nhs.uk/publications/ci-hub/nhs-outcomes-framework

https://www.statisticsauthority.gov.uk/wp-content/uploads/2015/12/images-codeofpracticeforofficialstatisticsjanuary2009_tcm97-25306.pdf



Patient Participation Group (PPG)

A Patient Participation Group (PPG) is a group made up of volunteer patients, the practice manager

and one or more of the practice GPs who meet on a regular basis to discuss the services offered and

improvements that would benefit patients and the practice. The first groups began in 1972, and in

2016 became a contractual requirement for all practices. PPGs are represented at the national level

by the National Association for Patient Participation (NAPP).

Prevalence

Prevalence is the proportion of a particular population found to be affected by a medical condition at

a given time. More information on how prevalence is estimated and reported can be found here.

Think tank

“Think tank” is a collective term to describe public policy research institutes that aim to play a key role

in making and influencing global, regional and national policy, by engaging in research and advocacy

in their specific area of expertise.

http://www.napp.org.uk/index.html

https://www.nimh.nih.gov/health/statistics/what-is-prevalence.shtml



1 Background

Ipsos MORI | GP Patient Survey: Questionnaire redevelopment 1


This report details the development of the GP Patient Survey (GPPS) questionnaire ahead of Year 12

of the survey (fieldwork and publication in 2018). It summarises the work that was conducted in

several stages, engaging a range of stakeholders.

1.1 What is the GPPS?

The GPPS is a large-scale England-wide survey funded by NHS England and currently administered

by Ipsos MORI. It provides practice-level data about patients’ experiences of their GP practices, which

is comparable across organisations (for practices and CCGs, for example) and over time. The findings

are published online (at www.gp-patient.co.uk) and can be used by GP practices and other

organisations to inform decisions about local health services. The survey produces Official Statistics

that feed into domains of the NHS Outcomes Framework, CCG Improvement and Assessment

Framework and is used by the Care Quality Commission in their Insight Model. More generally, the

data are used to measure how well GP services are serving patients. In this way, the survey provides

invaluable insight to assist with improvement and accountability, and also gives patients information

to help them make choices about which GP service they use.

The GPPS was first published in June 2007, and in that time the questionnaire has developed from

two to eight pages, with the last major set of changes to individual questions being implemented in

2011 (Year 6), following guidance from the Department of Health (DH) who were then responsible for

the survey.1

1.2 Why did the questionnaire need to change?

Primary care services in England are currently undergoing significant change, as set out in the GP

Forward View, which outlines NHS England’s vision for primary care. These changes include providing

extended access to appointments including in the evening and at weekends, as well as introducing

new ways for people to contact their GP surgery and make appointments. New services are also

rolling out across practices, such as access to general practice pharmacists, mental health

professionals and other practitioners.

1 These changes were wide ranging to ensure that the questionnaire met certain commitments within the NHS Outcomes

Framework, covering the full patient journey (from making appointments to the consultation). This process was also

designed to ensure the questionnaire included a measure of health-related quality of life, especially for those with long-

term conditions, as well as focusing on services outside general practice such as NHS Dentistry and out-of-hours

provision.

1 Background

http://www.gp-patient.co.uk/





These changes will affect how patients experience local health services, and in order to ensure that

the GPPS remains relevant in this context (and continues to ask about the right things, in the right

way), it was essential to review the questions on access and experience. However, the reviewing

process also identified an opportunity to make other improvements to the questionnaire, including

the sections which address “Managing your health,” “Your state of health today,” and “Planning your

care,” to provide a better understanding of how local care services are supporting patients to live well,

particularly patients with long-term care needs.

It should be noted that throughout the development process the need for change was balanced with

recognition of the importance of retaining questions that continue to be widely used, with the caveat

that the wider set of changes may impact data trends going forward. Guidance on how these

changes impact on trend data is available on the GPPS website.

1.3 A framework for questionnaire redevelopment

From the outset, the team at NHS England developed a framework for the questionnaire

redevelopment process which included a number of elements:

• Analysis of 2016 GPPS data: an initial stage looking at how well the existing questions “work,”

including: identification of questions with very high or low percentages responding (ceiling or

floor effects); instances where non-response is high, or a high proportion respond “don’t

know/can’t say/doesn’t apply” or “neither;” and looking at correlations between questions.

This stage also involved a review of cognitive testing in previous waves to provide insight into

potential issues with existing questions as well as changes that had already been tested;

• Establishing a governance framework: while the process was managed by the NHS England

Insight & Feedback team with support from the Medical and Nursing Analytical Unit, it

required additional governance to provide direction and leadership throughout. This involved

establishing an advisory group of senior policy leads within NHS England to ensure the newly

designed questionnaire effectively met users’ needs. This group worked alongside the already

established GPPS steering group to advise on priority data requirements and discuss and

approve proposals;

• Stakeholder engagement: wider engagement with over 200 stakeholders including clinical and

policy specialists from across the NHS, as well as those from academia, the voluntary and

community sector and think tanks. This involved individual interviews and meetings as well as

a larger stakeholder workshop;

https://www.gp-patient.co.uk/surveysandreports2018



• Engagement and research with patients: a broad programme of engagement including an

online feedback exercise hosted on the GPPS website that invited wider views from the public,

qualitative research with patients focusing on experience of extended hours access, and

further service user testing of specific questions and concepts; and

• Cognitive testing: the final stage in the process involved cognitive testing of the revised draft

questionnaire, involving 40 interviews over three rounds, to check detailed understanding

among patients.

This report explores the work done at each of these stages in more detail. The redesigned

questionnaire is available on the survey website. The appendices are provided under separate cover

in the same location.

Table 1.1: The GPPS questionnaire redevelopment timetable 2017

Action Timing

Insight & Feedback team planning/scoping January - February

Analysis of existing GPPS data April

Stakeholder engagement

Stakeholder engagement (access & experience, LTCs, self-

care)

May - July

Workshop July

Engagement and research with patients

Online feedback exercise June

Qualitative research with general public June-July

Service user testing July

Finalise questionnaire and testing

Collate outcomes from stakeholder engagement to feed into

questionnaire design

July

Cognitive testing of questionnaire August - October

Note that the survey advisory group and GPPS steering group provided input at key points

throughout this process.

http://www.gp-patient.co.uk/surveysandreports2018



2 Governance



As well as the external scrutiny provided by the stakeholder and wider public engagement described

in this report, it was important to establish a clear governance process to provide direction and

leadership for the proposed changes, including a number of advisory groups. The key elements of

this governance are described below.

2.1 GPPS core project group

The NHS England Insight & Feedback team managed the survey redevelopment, working closely with

Ipsos MORI and NHS England’s Medical and Nursing Analytical Unit and Improving Access to General

Practice teams.

Governance of the survey redevelopment was managed on a day-to-day basis by the Senior Insight

Account Manager in the Insight & Feedback team and by the Research Manager at Ipsos MORI. A

detailed timetable, including opportunities for stakeholder input, was agreed and a risk register was

developed to ensure any risks to the work were proactively managed.

2.2 GPPS steering group

The survey has a permanent steering group, which consists of representatives from NHS England

policy teams, the Department of Health and Social Care, the Care Quality Commission, the British

Medical Association (including representation from the General Practitioners’ Committee and a Local

Medical Committee), the Royal College of General Practitioners, the National Association of Patient

Participation, lay membership of local commissioning groups (CCGs) and academic researchers who

use GPPS data. The group meets between two and four times a year, at key points in the survey

lifecycle.

The steering group is designed to provide a level of independent scrutiny to the overall direction of

the GPPS, including the survey design and publication. The group provided significant input into the

questionnaire design process and subsequent content. Two steering group meetings focused on the

questionnaire, with part of the July and all of the October 2017 meetings dedicated to these

discussions. This allowed the group to challenge decisions and provide valuable guidance, from

across a number of sectors, greatly benefitting the decisions made.

2 Governance



2.3 Questionnaire development advisory group

An advisory group was convened specifically to oversee the questionnaire redevelopment process,

and included representatives from NHS England with expertise in primary care. The GPPS policy

sponsor, the Director of Improving Access to General Practice, was part of this group and was

ultimately responsible for agreeing changes to the questionnaire on behalf of NHS England.

The group as a whole met twice: initially in June, at the beginning of the process, to collect views on

policy areas that needed to be included in the revised questionnaire, then again during August to

update and seek views on the direction of the redesign, reviewing much of the feedback already

collected and potential changes as a result. The group were sent several versions of the questionnaire

throughout for comment. Their expertise was also used to address any specific queries the Insight &

Feedback team had about the content.



3 Analysis of existing GPPS data



3.1 Introduction and objectives

In April 2017, NHS England undertook analysis of GPPS data published in July 2016. This work

scrutinised responses on a question-by-question basis to identify potential discrepancies in how

participants answer the survey questions as this may indicate issues with comprehension. The analysis,

in particular, highlighted:

• questions with a significant proportion of non-responders;

• questions with evidence of floor or ceiling effects; and

• areas where participants answer in error.

This stage also involved a review of cognitive testing in previous versions of the GPPS questionnaire

to provide insight into potential issues with existing questions as well as changes that had already

been tested.

Table 3.1 below provides more detail on the analysis and measures used.

Table 3.1: Measures used for the survey response analysis

Issues looked for Measure

Ceiling/ floor effects Questions where a high proportion of participants are

clustered at one particular end of the answer scale - more

than 90% or less than 10% of participants answering

positively/negatively (e.g. very + fairly good).

High proportion of non-

response

Questions not answered by 5% or more of participants (or

more than 5% of those expected to answer if a filter question).

High proportion select “can't

say”, “don't know”, “doesn't

apply”

Questions where 5% or more of responses were attributed to

“can’t say”, “don’t know” or “doesn’t apply”.

High proportion select “neither” Where 10% or more of responses were attributed to “neither”.

Correlation between questions Pearson's correlation with other questions greater than or

equal to 0.8 or less than or equal to -0.8 (inverse correlation).

Correlation testing carried out using practice level scores,

where a question has multiple variables (i.e. multicode

questions are split into separate variables in SPSS) correlation

within the question was removed from the analysis.

3 Analysis of existing GPPS data



The analysis offered an evidence base of response patterns that could be used to supplement

discussions across the programme of questionnaire redevelopment work. It was circulated to key

stakeholders and was used in initial discussions with the advisory and steering groups, as well as being

referenced at key stages throughout the process.

Note that while all questions were included in the analysis, for some questions particular response

options or combinations were the focus (for example, for Q4 (Helpfulness of receptionists at GP

surgery) the combination of very/fairly helpful was the focus). In all instances, the analysis considered

the proportion not answering the question, those selecting “don’t know,” “can’t say” or “doesn’t

apply,” and those selecting “neither,” where relevant.

3.2 Key findings and recommendations from analysis of existing GPPS data

The key findings from this analysis that supported the questionnaire redevelopment are detailed

below.

3.2.1 Overall survey structure

In common with other paper-based self-completion surveys the analysis showed that the number of

participants who answer questions in error (based on the expected routing) was relatively high for

some questions in the 2015-2016 questionnaire. This suggests a need to re-consider pathways for

participants to navigate the questionnaire, for example, via routing and other signposting.

Related to this, the review also suggested that instructions about multi-coding can be inconsistent,

which appears to cause some confusion. For example, in some instances multi-coding is explicitly

permitted or disallowed, while in others it is permitted but not signposted. The inconsistency could be

due to how the questions were developed and the findings of previous cognitive testing. The data

shows that as a result of these inconsistencies participants can take different approaches to answering

the same question, suggesting a need to take a consistent approach to signposting on all multi-code

questions.

3.2.2 Questions on access

One in seven (16%) participants of the 2016 survey indicated that they had not seen or spoken to a

GP in the previous 12 months, rising to 32% for a nurse. These questions were positioned at the start

of the questionnaire, but were not relevant to all participants (the survey sample is based on those

registered with a GP practice, and not only those who have had a recent appointment).



In terms of booking appointments, the very low proportion (0.1%) saying they normally book by fax

machine (Q5) was expected, but the consistency in this level of response over time also suggested

that this option could be removed.

The data analysis found a number of issues in the section on making an appointment (Q10 to Q17).

High proportions of participants were answering in error, for example, 23% of all those returning a

questionnaire answered Q17 when they should not have done (what people did if they were not able

to get an appointment or the appointment they were offered was inconvenient). These discrepancies,

in the way that participants were answering specific questions, also suggested a possible lack of

salience of the question or issues with clarity of question routing.

3.2.3 Questions on experience

Questions on overall experience and recommending a GP surgery (Q28 and Q29) on the 2016

questionnaire were found to be highly correlated, suggesting an opportunity to streamline the

content. Further consultation was recommended to understand which measure was more widely

used, along with cognitive testing to check comprehension.

There was also evidence of correlation between questions measuring the quality of communications

with the GP and nurse, in particular the questions covering last GP and last nurse appointment, which

suggested the opportunity to rationalise the question sets by removing some of these (Q21a-e and

Q23a-e).

3.2.4 Long-term conditions, self-care and health-related quality of life

The analysis showed that some participants incorrectly answer the question on prevalence of long-

term condition (Q30), with around 10% of all those returning a questionnaire not recognising that

they have a condition when asked as part of a general question but then going on to answer a

follow-up about the specific type of condition they have (Q31). In addition, there is evidence of non-

response at Q31, with 8.1% of those returning a questionnaire not answering the question when they

should have done. This lack of comprehension and salience may be explained by the absence of a

standardised definition for a long-term condition (Q30) as well as by the length and complexity of the

list of conditions (Q31). These questions were a specific focus for the redevelopment, with

stakeholders also supporting the need to review them (see chapter 4).



Analysis showed both ceiling and floor effects when asking about confidence in self-management of

long-term conditions (Q33). Across all of the questions on care planning (Q36-Q39), data suggested

a potential lack of comprehension, with a high level of respondents answering in error. For the

question on use of their care plan to manage health day-to-day (Q38), 13.6% of respondents

answered this question in error, compared with 2.9% who answered the question following the

routing and gave a valid response.



4 Stakeholder engagement




To ensure the redevelopment of the GPPS questionnaire reflects the changing primary care landscape

and meets the needs of data users, an extensive engagement exercise took place, with views collected

from approximately 250 stakeholders from the NHS, Department of Health and Social Care, arms-

length bodies, academia and the voluntary and community sector.

Teams across NHS England were represented, including the Medical, Nursing, Operations and

Information, and Strategy and Innovation Directorates. External organisations represented included:

Department of Health and Social Care, Care Quality Commission, NHS Digital, Clinical Commissioning

Groups (CCGs), GP partnerships; British Medical Association, Royal College of GP’s, King’s Fund,

Nuffield Trust, Health Foundation; Patient Participation Groups (PPGs), Healthwatch England, C4CC;

National Voices, Stroke Association, Alzheimer’s Society, Mind; the Universities of Cambridge, Oxford,

Exeter, Dundee, Leeds, Bristol and Manchester.

This chapter focuses on the engagement which took place in three strands:

▪ with stakeholders about specific topics;

▪ a stakeholder workshop; and

▪ consultation on EQ5D (Q34).

This engagement piece informed all stages of the development work. Suggestions from stakeholders

were fielded, discussed and included where appropriate, and changes were presented to and further

discussed with these individuals and groups to ensure potential risks were identified and could

therefore be minimised.

4.2 Stakeholder engagement about specific topics

The overall process of stakeholder engagement focused in particular on a number of areas within the

questionnaire highlighted for potential change: long-term conditions (LTCs), self-care and care

planning, mental health conditions, web services, nursing, carers and gender identity. The table below

summarises the key elements of this process along with their objectives; each is then explored in

more detail in the following sections. This concludes with a summary of the decision-making

processes involved in taking suggestions for change forward into the next phase of the

redevelopment.

4 Stakeholder engagement



Table 4.1: Summary of stakeholder engagement about specific topics

Stakeholder engagement Purpose

Engagement with

stakeholders and policy

leads in relation to LTCs

and the GP Forward View

To identify issues with proposed questions, policy

preferences, and criteria for shortening the long-term

condition item list. This engagement involved three

elements: initial engagement in 2016; discussions with the

Care Quality Commission (CQC) on LTCs: and further

engagement with stakeholders and policy leads.

A self-care/care-planning

working group

Comprising policymakers, experts and stakeholders

(including patients) to ensure this area of questions

remained relevant.

Engagement with owners

of national performance

indicators that use GPPS

data

Both analysts and policymakers from NHS England,

Department of Health and Social Care, the Care Quality

Commission and NHS Digital. To understand which data

are used for indicators and where alternatives may exist.

This included specific engagement with users of the EQ5D

question (Q34) about its possible removal (discussed in

more detail in section 4.4 below).

Additional stakeholder

engagement

With policymakers, experts and patients concerning mental

health policy, carers’ policy, GP web services and GP

nursing.

4.2.1 Engagement with stakeholders and policy leads in relation to LTCs and the GP Forward View

A programme of engagement with stakeholders and policy leads, relating to LTCs and the GP

Forward View, was undertaken. This broadly comprised three stages, as outlined below.

Initial engagement with stakeholders and policy leads in 2016

The Insight & Feedback team initially reviewed the content of the GPPS during the summer of 2016

with a view to improve the long-term conditions question and to provide data to support the

ambitions of the GP Forward View. The GPPS steering group had suggested that these questions

could be improved, both in terms of the order in which they were asked and the wording of the

response options. The group also questioned the extent to which the right list of conditions was

included.



As a consequence, the Insight & Feedback team conducted initial stakeholder engagement with

colleagues focusing on long-term conditions (LTCs) including: the conditions captured by the

questionnaire, how patients are supported to manage their health (including care planning), along

with a measure of health-related quality of life. These issues were also discussed at an LTC workshop

that focused on multi-morbidity. From this engagement, stakeholders identified that the list of LTCs

used as part of the survey could be improved.

This LTC review coincided with the early stages of implementation of the GP Forward View. The

ambition was to revise the questionnaire to track progress against the aims of the GP Forward View.

However, there was a shared view between the NHS England Improving Access to General Practice

and Insight & Feedback teams that there was insufficient time to adequately develop and test

questionnaire changes ahead of fieldwork in 2016-17. It was therefore decided that it would be more

appropriate to postpone changes to the main GP practice sections of the questionnaire (Q1-Q28)

until 2017-18.

Following this decision to retain the opening sections of the GPPS questionnaire for a further year, it

was agreed that the time series on having and living with LTCs should likewise be maintained, and it

was agreed that no changes would be made at this point.

Engagement with the Care Quality Commission

During 2017 and before the redevelopment of the GPPS commenced, the Care Quality Commission

(CQC) redeveloped the LTC questions used in the 2017 Inpatient Survey. The Inpatient Survey is part

of the National Patient Survey Programme of NHS surveys, which covers a range of topics including

adult inpatient services, children and young people’s inpatient and day-case services, community

mental health services and maternity services. The aspiration for the LTC questions is that they should

be aligned as much as possible across both these and other health surveys led by NHS England. To

this end, the CQC asked for feedback on two proposed versions of the LTC questions to understand

which would produce the most valuable data for NHS England:

• Option A, focused primarily on how the patient is impacted by their long-term condition(s);

and

• Option B, focused on a number of specific conditions.



The Insight & Feedback team collected views from NHS England stakeholders to inform the decision-

making process. The exercise was also deemed valuable ahead of the planned changes for the GPPS,

as it provided important insight into NHS England’s needs. Based on feedback the decision was made

by CQC to develop option B, focusing on specific conditions with some focus on impact (see

http://nhssurveys.org/survey/1997), as this was felt to be of most benefit to stakeholders.

2018 survey engagement with stakeholders and policy leads

Before continuing with further stakeholder engagement covering LTCs, self-care and care planning,

the views already gathered were evaluated and the existing data analysis and review was completed

(see sections 4.2.1 and 4.2.2 and chapter 3). This supported more detailed conversations with

stakeholders around proposed revised and new questions, policy preferences and criteria for

shortening the LTC item list.

In terms of LTCs, in order to establish which conditions are most relevant to delivery of primary care,

data about the prevalence of LTCs within the population was reviewed. This was complemented by

analysis of GPPS survey data, looking at cases where participants indicated they did not have a LTC in

response to a general question but then selected a condition from a detailed list in the following

question (Q30 and Q31).

In addition, to evaluate the utility of existing questions a review of the indicators that use LTC data

from the GPPS was carried out to better understand its applications, and to provide context ahead of

making any changes.

Together, this information was used to propose a set of questions that could be considered for use in

the GPPS. These continued to cover measurement of LTCs as well as new questions focusing on frailty

and polypharmacy. To develop these questions, feedback was requested from stakeholders with

expertise in LTCs, self-care and self-management (with around 20 responding). In combination, this

feedback demonstrated that while stakeholders recommended substantial change, there was no clear

consensus: while some felt the LTC questions should be refined to more accurately measure

prevalence of LTCs, others focused on removing EQ5D so that additional questions covering other

priority areas could be included.

To inform opinion further, a paper was presented to the Long-Term Conditions team within NHS

England. The paper detailed the proposed changes to the questions and outlined the implications.

As a result of feedback from the Long-Term Conditions team the questions were further refined with

a sub-set of stakeholders considered to be subject-matter experts to ensure there was a clear use for

data on polypharmacy and frailty.

http://nhssurveys.org/survey/1997



4.2.2 Self-care and care-planning advisory group

Initial stakeholder engagement on the topic of LTCs indicated that questions on self-care, self-

management and care planning needed revising. A separate self-care advisory group was formed to

review these questions, consisting of subject-matter experts within NHS England, the Department of

Health and Social Care, the Picker Institute, National Voices, the University of Oxford, along with a

patient representative. The group met once face-to-face in June to discuss potential content, and

corresponded further virtually. The Insight & Feedback team also sought input from the NHS England

Self-care and Personalisation and Choice teams throughout the process. Overall feedback highlighted

that the care planning questions in particular were too narrow, only focusing on written care plans

and excluding any substantive care planning conversations that may have occurred between patients

and healthcare professionals. With regard to self-care and self-management, feedback focused on

ensuring that the survey is able to appropriately measure confidence around managing any issues or

challenges caused by a LTC and that it continues to measure the support offered to manage

conditions. It was felt that the previous questions did not discriminate well between different levels of

capability and confidence.

4.2.3 Engagement with indicator owners

As well as the wider stakeholder engagement, the Insight & Feedback team corresponded directly

with stakeholders who use GPPS data for indicators such as those in the NHS Outcomes Framework,

the CCG Improvement and Assessment Framework and on myNHS, amongst others. These indicator

owners were sent correspondence early in the process and invited to a teleconference for data users,

informing them of the work and potential implications. In particular, detailed work was undertaken

with indicator owners who use EQ5D (Q34), due to the fact that other senior stakeholders were

suggesting this metric should be removed in order to make room for other questions. More

information on the work around EQ5D is provided below (see section 4.4).

4.2.4 Additional stakeholder engagement

As well as the key engagement exercises explained above, several separate exercises ran alongside

the work on LTC and self-care/care planning, including exploration of questions about mental health

conditions, web services, nursing, carers and gender identity. For each area the Insight & Feedback

team engaged teams within NHS England and stakeholders outside of the organisation, where

relevant, to collate suggestions for content and consider the appropriateness of each among

policymakers, experts and patients.



4.2.5 Decision-making processes as a result of stakeholder engagement

The stakeholder engagement exercise was large and complex, with decisions being made around

question or topic inclusion based on there being a strong rationale, such as a question measuring a

policy area of interest to NHS England or the health service in general. The following criteria were

used to assess the suitability of questions for inclusion:

▪ a key area of NHS England policy or stakeholder interest;

▪ a topic area of sufficient relevance to patients’ experience (for example, a question focusing on

something very specific and niche would unfortunately not receive as many survey responses or

provide as much value as a question that the majority can answer);

▪ a concept that is not too complex and that can be effectively measured in a paper-based

questionnaire and;

▪ the appropriateness of such a question being included in a general practice questionnaire, for

example, if it is already collected elsewhere in a similar form or by other means.

Each suggestion was treated individually, keeping the above principles in mind when making

decisions. It was also the case that many of the questions and concepts were further scrutinised

during cognitive testing (see chapter 6).

4.3 Stakeholder workshop

4.3.1 Introduction

NHS England used the large volume of information collected in the early stages of the questionnaire

design process, from policy teams, survey users and other stakeholders, as well as from patients, to

produce a broad outline of proposed changes to the questionnaire.

In order to progress these suggestions further, NHS England and Ipsos MORI designed and ran a

questionnaire development workshop. The main focus in the workshop was on access and making an

appointment, but it also covered reactions to the proposed changes in general. This took place on

13th July 2017, and drew on the expertise of 36 stakeholders. Discussions focused on developing

proposed changes as well as deliberating the value of new measures against the potential loss in data

from questions deemed fit for removal.



4.3.2 Aim

The workshop aimed to draw on the expertise of a wide range of stakeholders in order to design the

redeveloped GPPS questionnaire in a way that future-proofed the content in light of current and

planned changes to service provision.

Given the space restrictions of the questionnaire (limited to eight pages) the workshop was framed as

a means of prioritising new areas of measurement and exploring how they could be integrated in a

meaningful way, while balancing this with retaining existing questions where relevant. As such there

was recognition that not all requests could be accommodated, and that the workshop offered an

opportunity to provide feedback to ensure the space was used in a way that maximised utility across

policy areas.

4.3.3 Workshop structure

Given the scope of change it was important to consult with specialists across a range of areas of

expertise. Thirty-six representatives from the following types of organisation attended:

• Representatives of GP Practices and Patient Participation Groups (PPGs):

• CCG representatives;

• Think-tanks;

• Academia;

• Regulators and Government Departments;

• Commissioning organisations; and

• Medical membership bodies.

Attendees were invited based upon their prior involvement in the survey or the survey redevelopment

process specifically, as well as those with particular expertise in primary care.

Whilst it was important to gain views on the wider questionnaire work, the focus of the workshop was

on the sections of the questionnaire that explore access to GP practice services including booking

appointments (see Appendices, provided under separate cover).

The workshop was divided into three key sessions (a detailed agenda can be found in the

Appendices).



1. Reaction to proposed changes to the GPPS questionnaire

Participants were given the opportunity to consider the proposed changes as a whole and highlight

areas where it was felt the questionnaire did not sufficiently reflect changes in provision over the next

five years, as well as any concerns in relation to the content proposed for removal.

2. Access and making appointments – areas of focus

This session aimed to develop a shared understanding of the different aspects related to contacting a

GP practice and making an appointment, with five areas of focus:

• What are the different ways and types of contact patients will have with their GP practice?

• Why do patients need to be seen?

• Who do patients see or speak to?

• When are patients “seen”?

• Where are patients “seen”?

3. Access and making appointments – journey mapping

Patient journey maps produced as an output of the qualitative research (see section 5.3) were used to

stimulate discussions around people’s complex interactions with primary care and how they could be

captured effectively through the survey. The maps focused on five key areas; a common theme across

four areas was the manner in which “extended access” (access to routine appointments at evenings

and weekends) was used:

• Extended access for convenience;

• Extended access for a same day appointment;

• Extended access at another surgery;

• Extended access at a location other than a GP surgery (for example an Extended Access Hub);

and

• Online service use.



Attendees of the workshop were divided into five breakout groups to ensure all topics could be fully

explored over the course of the workshop. Further, to ensure a range of perspectives were presented

on each of the topics a mix of attendees were assigned to each group. Each group was facilitated by

a member of the Ipsos MORI GPPS team, and supported by a member of the NHS England GPPS

team, where possible. Notes were transcribed in each group in real-time by experienced note-takers.

In addition to the three key breakout sessions, two plenary sessions were held to enable discussion

and reflection among the group as a whole.

4.3.4 Key findings

Stakeholder feedback was recorded at the question-level and a log was developed to track the

outcome for the questionnaire and associated rationale. All feedback was considered but the

discussion below focuses on the findings that were drawn upon to develop a first draft of the 2018

GPPS questionnaire.

• Ensuring appropriate terminology around general practice is used for each policy area

Historically the questionnaire has focused on measurements at the level of “GP surgery,”

however this concept is somewhat outdated as it is increasingly common for a GP practice to

consist of several sites (i.e. several GP surgeries) and patients can in theory be given an

appointment at any of these. Taking this a step further, new forms of service provision are no

longer centralised around GP practices such that patients can now attend a general practice

appointment at a range of locations such as an extended access hub in the same building as

their surgery, a hub not in the same building as their surgery, another surgery that is part of

their GP practice, or they can have a home visit. As such, workshop participants felt that the

framing of questions should be reviewed to ensure that access is measured appropriately for

appointments at a GP practice and appointments in the context of general practice services as

a whole.

• Reflecting the changing definition of appointments

The definition of an appointment is becoming more diverse, with an increasing use of online

methods, such as email, being used to communicate with healthcare professionals. Patients

may for example be offered an appointment to speak to someone online, or be visited in their

home. In the context of the questionnaire, the types of appointment measured should be

updated to reflect this diversification.



• Importance of understanding patient expectations

A finding that came through strongly in the qualitative research (see section 5.3), which fed

into the workshop, was that the urgency of securing an appointment was often much greater if

the appointment was for a child or if it was an unplanned appointment for someone with a

long-term condition. From discussions in the workshop it was felt that by including a measure

of who the appointment is for, a better understanding of patient expectations of the

appointment process would be obtained (e.g. whether the appointment was being made for a

child under the age of 16).

• Incorporating appropriateness of treatment

The 2017 (Year 11) questionnaire focuses on the type of appointment the patient wanted,

when the patient wanted to be seen, and whether these elements marry up with what actually

happened. In new forms of service provision the emphasis can be on healthcare professionals

directing patients to the right place, and giving them the right information in order for an

appropriate decision to be made about their needs. As such it was felt that contextual

elements about the appointment-making process needed to be added to understand whether

the options presented to a patient were appropriate for health needs. Therefore additional

questions were added around who the appointment was for, the level of concern associated

with their health, the level of choice offered to the patient and reasons for not taking an

appointment.

• Focusing on outcomes

There was a consensus at the workshop that who a patient saw becomes less important,

particularly in light of the fact that patients often do not know who they saw or spoke to (more

so the case for phone consultations); instead the focus should be on whether the patient was

satisfied with the process of making an appointment and if the outcome met their needs.

• Measuring pre-contact management

Pre-contact management is the management of a health complaint prior to making contact

with general practice for the purpose of making an appointment. This topic area has so far not

been measured in the questionnaire. To understand whether other services are only being

drawn upon when necessary pre-contact management acts as a useful proxy of whether

patients are seeking to address health complaints through other means and therefore only

referring to general practice when appropriate. For example, if a condition of is limited

concern, are patients engaging with other NHS platforms such as NHS 111 or pharmacists as a

point of first contact with primary care?



• Measuring quality of care from healthcare professionals as a whole

In the 2017 (Year 11) questionnaire specific sections were dedicated to measuring last contact

with a GP and with a nurse. Often this data is of limited use at a practice level, as it is not

possible to link responses to individual GPs or nurses. It was therefore suggested that

capturing the last healthcare professional a patient saw and their experience on that occasion

would be sufficient. It would also allow for a general understanding of performance of

different types of healthcare professional, for example, at CCG or national level. Furthermore,

due to the integration of pharmacists and mental health professionals into general practice,

workshop participants suggested that the list of healthcare professionals should be expanded

to provide detailed data on a wider range of clinicians.

• Updating competencies

Some of the existing competencies used to measure quality of care are highly correlated. It

was recommended that the measure “explaining tests and treatments” was removed as these

elements are not applicable to every appointment.2 For example, if a patient presents with a

condition unsuitable for tests and treatments, such as a common cold, they will not require

tests or treatment. It was also suggested that the measure, “involved you in decisions about

your care,” should be replaced with “involved as much as you wanted to be” to align with CQC

surveys. This measure acknowledges that patients may wish to be involved in their care to

varying degrees.

• Better understanding of the type of appointments offered to patients, and uptake of services

Due to the diversification of appointment types on offer in terms of place, type or healthcare

professional, it was felt that the questionnaire should do more to capture whether patients are

being offered a choice of appointment. In circumstances when they do not take an

appointment it was felt the questionnaire should capture why this is. Workshop participants

agreed that this will aid understanding of how services may need to change in specific areas to

better align with the needs of the population.

2 This question was later replaced with a measure on whether or not a patient felt their needs were met.



• Shifting questions on opening hours to examine availability of appointments as a whole

The 2017 (Year 11) questionnaire measures satisfaction of opening hours at GP surgery level,

and convenience of different opening times. This assumes that general practice appointments

are only available at the GP surgery level, however in reality when GP surgeries are closed

alternative routes of accessing general practice appointments are becoming more widely

available. A more accurate measurement would therefore be awareness of, and satisfaction

with, the times that patients believe they can obtain a general practice appointment.

• Including GP practice websites as part of the measures of access

With the broadening offer of online services, GP practice websites are an increasingly useful

resource for patients to access services, yet there is no measure of whether patients feel they

are effective. Therefore, the suggestion was made about measuring on site accessibility.

• Removing the subjective question on waiting times within the surgery

The 2017 (Year 11) questionnaire contains two measures on waiting times, one captures the

time patients have to normally wait to be seen after their appointment time and the other asks

how they feel about this wait. The consensus was that the former should be retained as it is

less subjective, and is therefore a more robust measure of improvement in waiting times.

• Removing the question on recommending a GP surgery

There is a measure of overall experience of a GP surgery and another relating to

recommendation of the GP surgery in the 2017 (Year 11) questionnaire. As these two

questions are highly correlated it was agreed that one was sufficient and that the overall

experience question was deemed to be of the most value to the GPPS, not least because of

the larger number of indicators that use this question. Also, the recommendation question is

currently captured by the Friends and Family Test (FFT).

• Needing a clearer definition of long-term conditions

The definition of long-standing health conditions in the 2017 (Year 11) questionnaire is open

to interpretation; no description is provided, and as such patients may not be reporting

conditions consistently. To ensure more consistent self-reporting it was felt that a clearer

definition should be supplied (for example, lasting or expected to last for 12 months or more)

and that the terminology should be updated to clearly encompass physical and mental health

conditions, disabilities or illnesses, rather than only mentioning “health conditions.”

https://www.england.nhs.uk/fft/



• Including measurements of frailty and loneliness

Frailty and loneliness are important issues to measure given that from October 1 2017 there

will be items within the GP contract around assessing the level of frailty of patients. This idea

was raised in the stakeholder engagement, and its importance was reiterated by workshop

attendees (some of whom were also involved in the stakeholder engagement piece). It was

therefore suggested that measurement of these issues should be added to the questionnaire.

• Including measurements of mental health

Mental health is a priority area in the NHS. The 2017 (Year 11) questionnaire seeks to identify

long-standing health conditions and medical conditions and in doing so it assumes patients

will identify mental health conditions under these terms. People with mental health conditions

may not be diagnosed and therefore may not self-identify in these questions. This was raised

as an issue to address in the stakeholder engagement exercise, and was subsequently

supported in the workshop, where participants suggested including a question on mental

wellbeing generally, including, for example, recognition and understanding of mental health

needs more broadly rather than a specific question on mental health conditions.

• Shifting focus from a care plan to the process of care planning

The current questionnaire focuses on the care plan itself, however it was felt to be more

pertinent to measure the process of care planning and whether it was inclusive, centring

around the needs of the patient, and whether as a result they feel able to manage their

condition. Workshop participants felt this should provide useful data around whether the care

planning process is working effectively.

• Refining measures relating to continuity of care

Prior to review there was a single item related to continuity of care (the question on how often

patients see or speak to the GP they prefer). Evidence suggests that as care is re-shaped to

enable easier and quicker access to services, continuity in primary care may diminish. The

questionnaire currently measures whether a patient has a preferred GP and how often they

speak to them, but evidence suggests that patients do not always wish to see their preferred

GP. In certain situations patients may choose to prioritise the convenience of an appointment,

or would prefer to be seen sooner than when their preferred GP is available. Therefore it was

concluded that the nuances of when a patient would like to see a preferred GP and whether

this requirement is met are more important than the current measures.



• Use of EQ-5D

Given the importance of EQ-5D to measurements of health economics, the value of including

it in the GPPS questionnaire was debated at the workshop. This is discussed in more detail in

the section about EQ-5D below.

4.4 Stakeholder engagement about the EQ-5D

4.4.1 Introduction

EQ-5D is a standardized instrument developed by the EuroQol Group as a measure of Health-Related

Quality of Life (HRQL), consisting of a descriptive system and the EQ Visual Analogue Scale (VAS). For

a number of years the GPPS questionnaire has included the EQ5D descriptive set of questions about

the state of a patient’s health (Q34), but not the VAS. The descriptive question set comprises five

dimensions: mobility, self-care, usual activities, pain/discomfort and anxiety/depression. As part of the

survey redevelopment a wide-ranging stakeholder engagement exercise investigated the utility of this

set of measurements in the context of the GPPS, including engagement with affected indicator

owners and current users of EQ5D.

4.4.2 Key findings

As part of the redevelopment of the GPPS questionnaire, we asked stakeholders their view on the

EQ5D.

Data from the EQ5D in GPPS has been used in key indicators by NHS England and Department of

Health and Social Care, and is also desired for use in economic analysis by the Department. The

question has been in the survey since 2007 and so has provided consistent year on year data.

Many of the stakeholders consulted suggested that aside from its employment in national indicators,

the EQ5D question set in the context of the GPPS is of limited use, particularly for those who are not

experts in using EQ5D data. Part of the lack of perceived utility was thought to derive from the fact

that the GPPS is a patient experience survey. As a metric of HRQL, EQ5D provides its greatest value

when employed as a longitudinal measurement to understand the health gain of an intervention (e.g.

by being included in a patient reported outcome measure (PROM) before- and after-survey). On a

cross-sectional study like the GPPS, EQ5D can only provide a snap-shot measurement of HQRL. As

such, in this context EQ5D does not provide outcome data that reflects the quality of primary care.

There are many potential factors influencing HQRL, encompassing environmental factors and

individual behaviours, such that the GPPS EQ5D results cannot be legitimately ascribed to local health

and care organisations. For this reason, the argument for collecting the data in this context at such a

large scale appears weak.

https://euroqol.org/eq-5d-instruments/

https://euroqol.org/euroqol/



In addition, the EQ5D data collected via the GPPS does not vary greatly from year to year (see tables

4.2 and 4.3), therefore it could be argued that it need not be collected on an annual basis, and hence

should not be permanently included in the GPPS. A further benefit of removing EQ5D, recognised by

many stakeholders, would be to allow space for other, more immediately useful or relevant questions,

particularly since, due to licensing requirements, EQ5D takes up almost a full page of the

questionnaire.

Table 4.2: Current EQ5D national indicator trends

CCG OIS/NHSOF

indicator 2011/12 2012/13 2013/14 2014/15 2015/16 2017

HQRL of people

with LTCs 0.743 0.744 0.743 0.743 0.741 0.737

HQRL of people

with 3+ LTCs 0.440 0.473 0.474 0.470 0.463 0.462

HQRL for people

with a long-term

mental health

condition*

0.527 0.529 0.528

HRQL of carers 0.815 0.807 0.804 0.804 0.800 0.797

Average HQRL for

all GPPS participants 0.829 0.821 0.820 0.821 0.820 0.816

Table 4.3: Summary of EQ5D national indicator changes over time

CCG OIS/NHSOF indicator

Change

since

2011

Change

in last

year

HQRL of people with LTCs -0.006 -0.004

HQRL of people with 3+ LTCs 0.022 -0.001

HQRL for people with a long-term mental health

condition 0.001* -0.001*

HRQL of carers -0.018 -0.003

Average HQRL for all GPPS participants -0.013 -0.004

* Only collected since 2012 and not yet published for 2017



4.4.3 Outcome

On balance, weighing the above arguments, the Insight & Feedback team recommended that EQ5D

should be removed and NHS England’s Chief Executive agreed with this recommendation.

This recommendation was on the basis that the loss of EQ5D would be offset by the utility of new

data about areas of specific current interest. Furthermore, it was agreed that a national measure of

EQ5D would instead be collected via NHS Digital’s Health Survey for England (HSE). The extensive

content of HSE fits more closely with EQ5D, meaning that findings can be analysed alongside a

greater range of drivers and indicators of health and wellbeing. HSE is a well-respected survey with a

scientific methodology and a high response rate, and also collects information about long-term

conditions and caring responsibilities, such that it could potentially be used to supply all of the

existing indicators on the NHS Outcomes Framework that use EQ5D data from the GPPS. However, it

was noted that a new set of trend data would have to be calculated using previous editions of HSE,

and that analysis of EQ5D would be restricted to the national level (HSE has an adult sample size of

8,000).

4.5 Key findings and recommendations from stakeholder engagement

Across all three strands of stakeholder engagement (engagement on specific topics, workshop, EQ5D)

it was clear that were a number of topic areas seen as more desirable from a policy perspective and

that have a clear immediate use, which included the following:

▪ Use of the GP practice’s website (to measure how useful and accessible this is);

▪ Who the most recent appointment booking was for (e.g. whether it was for a child);

▪ Level of concern about the need for the appointment (to measure felt need as opposed to

clinical need);

▪ Pre-contact management (such as searching for information online before contacting the GP

practice);

▪ Whether a choice of appointment was offered (and whether this was satisfactory);

▪ What kind of healthcare professional they last saw (including pharmacists and mental health

professionals);

▪ Whether any mental health needs have been recognised and addressed;



▪ Whether the patient’s needs were met overall;

▪ Frailty (problems with mobility or frequent falls);

▪ Loneliness and isolation; and

▪ Polypharmacy (regularly taking 5+ medicines).



5 Engagement and research with

patients




Feedback from patients was imperative to ensure that the redesigned GPPS questionnaire is as

reflective as possible of the views of those who complete it. This engagement and research with

patients consisted of three work streams that fed into wider decisions about content:

• an online questionnaire to collect views from data users of the GPPS and other stakeholders,

as well as members of the public;

• qualitative research with patients to better understand their service use; and

• specific condition-based service user testing when refining the questionnaire.

This work took place between June and August 2017, with views collected from approximately 150

people in total.

5.2 Website feedback

5.2.1 Background

To gather feedback from the general public, a short online questionnaire was advertised and hosted

on the GP Patient Survey website (www.gp-patient.co.uk), accessible via the banner on the home

page. The questionnaire was available between 16 June and 14 July 2017.

Information about how to take part via the online feedback form was shared with a range of

stakeholders via NHS corporate bulletins and also via the Department of Health and Social Care’s

Voluntary and Charity Sector Update. It was communicated to NHS England’s regional networks and,

in turn, they cascaded the information to their local stakeholders. The opportunity to provide

feedback was also publicised on Twitter.

5.2.2 Aim

The aim of the website feedback exercise was to ask the public and any additional stakeholders not

currently involved in the redevelopment of the GPPS to provide their input into the questionnaire

redesign by responding to four open-ended questions:

• Are there any sections or questions of the GPPS questionnaire you wish to remain unchanged?

• Are there any sections or questions of the GPPS questionnaire that you think need to be

changed or should be improved?

5 Engagement and research with patients




• What, if anything, should be added to the GPPS questionnaire?

• Do you have any other comments about potential changes to the GPPS questionnaire?

In total, 78 responses were received. Seventy participants provided information about their role,

including a mix of members of the public (21), healthcare professionals (20), healthcare managers

(19), service commissioners (eight) and students/researchers (two).

The feedback was analysed and used to inform discussions on survey redevelopment. This section

summarises key findings. A fuller version is available in the Appendices available on the survey

website.

5.2.3 Key findings

Overall feedback

A small number thought that the survey should not change, either because they felt the current

questions were adequate, or because they wanted to retain trend data. A few people, all working in

the health sector, felt that the survey should be discontinued in favour of using public money for

direct patient care instead. Some participants did not feel there were any specific questions or areas

they would amend. However, the vast majority said they valued the GPPS and identified specific

questions or areas for improvement.

Accessing your GP services

The majority of those giving feedback placed high value on this section as an important tool for

assessing GP practice performance but they highlighted a need to revise the contact options to better

reflect current methods, for example, including telemedicine and web consultations and removing by

fax.

It was acknowledged that GP practice websites are a key source of information for patients, and

feedback suggested including a question on how easy they are to use in order to identify areas for

improvement. It was also suggested that there should be distinct measurement of the online services

on offer, including their functionality.

Making an appointment

Those providing feedback highlighted that the current questionnaire asks when patients wanted to be

seen but felt that perhaps a more appropriate measure for assessing the quality of their pathway into

care is to ask when they needed to be seen.





Another key topic was the shift in general practice to the availability of appointments with a wider

range of staff (rather than just a GP or nurse) and the need for the questions to reflect this, perhaps

by adopting the more generic term “health professional.” One suggestion was to collect information

on whether a patient is directed towards other available services that might be more appropriate,

either by the receptionist or through telephone triage, in contrast to people approaching other

services as a result of their own initiative.

Waiting times

Those providing feedback pointed out that waiting times may be longer than patients want or expect

when GPs try to help patients with more than one issue or because of those with complex issues who

need longer appointments. As a result it was suggested it would be useful to understand how patients

use appointments, for example, by including questions on whether the patient had more than one

issue they wanted to discuss, whether appointment times should be longer, and whether patients felt

they were given enough time during the appointment.

Last GP appointment/Last nurse appointment

Some participants suggested combining the questions about last GP and nurse appointment to

create a more generic “healthcare professional” question. This would fit changes to the model of

general practice, where patients will see a range of different professionals. This change to the

questionnaire would allow the quality of the consultation to be captured more accurately.

Accessibility of information provided during the appointment was felt to be important, including

whether the patient was able to understand the diagnosis, treatment or advice given. In particular,

participants mentioned barriers such as jargon or understanding of English.

Another suggestion was to establish satisfaction with prescribing as well as whether non-

pharmaceutical options were offered.

Some providing feedback mentioned shared decision making as a relevant topic for inclusion. For

example, in terms of involvement in treatment and decisions made, suggestions included whether all

treatment options were discussed, including risks and benefits, whether the healthcare professional

took time to understand the patient’s values and whether patient decision aids were used to support

discussions.



Opening hours

Opening hours are considered a complex area for GP practices due to geographic variation in

arrangements and varied awareness of the options available as well as the distinction between

extended access and late opening hours. Consequently, new questions to measure patient

understanding were proposed, including awareness of opening hours and whether published

opening hours are adhered to.

Overall experience

The 2017 (Year 11) questionnaire has two questions relating to overall experience; a good/poor rating

of overall experience and recommendation of the GP surgery to someone who has just moved to the

area (Q28 and Q29 respectively). Overall experience was identified as a key indicator for overall

quality of the practice by some, and they emphasised the significance of this question. Some noted

that the question relating to recommendation is similar to the Friends and Family Test (FFT) and

queried the value of also using this question in an annual survey.

Managing your health

Some participants identified issues with the question measuring the existence of long-term conditions

(Q30); the terminology “long-standing” was felt to be potentially misleading, for example, possibly

capturing a number of more minor conditions such as hay fever.

In addition, some amendments were proposed to the list of long-term health conditions. For

example, one suggestion was to include HIV as it was felt this would help reduce the stigma of HIV

and raise awareness of the need for primary care services to be more knowledgeable about the virus.

There was specific support for retaining “Arthritis or long-term joint problem” and “Long-term back

problem” within the list.

Another suggestion for this section was to consider a specific question on sensory impairment and

communication needs, focusing on the use of sign language, interpreters and translators for other

languages.

Your state of health today

The 2017 (Year 11) questionnaire contains a question-set known by the term EQ5D (see section 4.4).

Some participants suggested that this section should be removed entirely, explaining that it is biased

towards physical over mental health, and feeling it could be replaced with a better measure of

subjective wellbeing. It should be noted that some participants mentioned their own tools for

measuring subjective wellbeing in place of EQ5D, and therefore have an interest in the GPPS

adopting these.

https://www.england.nhs.uk/fft/



Conversely other feedback highlighted the value of EQ5D for research, for example, into

understanding the impact of musculoskeletal conditions.

Planning your care

It was felt by some that this section should be removed for a variety of reasons. For example, care

plans are often completed by community services but responses would be attributed to a GP practice

and if negative could unfairly reflect on that practice. Further, it was felt that care planning is of limited

relevance to the majority of participants as it only applies to those with a long-term condition.

Where it was suggested the care planning questions should be retained, it was proposed they should

be re-focused to move away from care plans specifically and instead ask about the care planning

process; suggestions included whether patients had a planned discussion about how to manage their

health, whether they had enough information to prepare for this and whether the plan was

developed with their GP.

Out of hours

Those providing feedback commented that the questions within this section do not reflect the

complexity of out-of-hours services.

NHS dentistry

There were mixed views on the inclusion of dentistry questions in the GPPS. Some felt this section

should be removed or addressed in a separate questionnaire. Others thought there should be

additional questions in the dentistry section to act as parallels to the GP questions. Further

suggestions included asking when the patient last visited a dentist and whether it was an NHS

appointment. It was also suggested there should be a question to establish whether people know that

NHS dentistry is available.

Some questions about you

Some feedback highlighted the need for a wider variety of gender options.

Feedback was also received in relation to content on carers. The questionnaire contains a question on

unpaid caring responsibilities, however unpaid carer status is not offered as an option within the

employment question.



Another suggestion focused on the support received by those with caring responsibilities, including

support from local services or organisations in order to take care of their own health, help with

looking after the person they care for and the need for additional support. The aim would be to

identify whether carers require more support and whether these responsibilities impact on their own

health.

Feedback on the smoking habits question suggested the inclusion of an additional option of “e-

cigarette user”.

5.3 Qualitative research

5.3.1 Background

While changes are underway in the delivery of GP services, it is clear from previous research that

patients are often not aware what provision is available.3 This is partly a result of the complex

landscape, with multiple models and modes of access being offered around the country, each at

different stages of development. Although patients do not need to understand which model is

offered in their area, for a newly developed GPPS questionnaire to be relevant, it must effectively

capture their views and experience of extended GP access and it must also do this in a way that

reflects the language and terminology they would use. As a result, NHS England identified a need to

conduct qualitative research among patients in order to understand their different levels of awareness

and understanding of these services and how this may impact on the redesigned questionnaire.

This section outlines the research objectives and approach to this qualitative research, which was

conducted in June 2017. A key output from this element was a series of journey maps, outlining

patients’ experiences of a range of scenarios of accessing GP services, which fed directly into the

stakeholder engagement workshop (section 4.3).

5.3.2 Objectives

The qualitative research with patients was designed to understand three key objectives:

▪ What primary care provision looks like from a patients’ perspective;

▪ What factors influence patients’ overall experience of access; and

▪ How these experiences might impact the redesigned GPPS questionnaire.

3 A finding from the redevelopment of the out-of-hours questions for the GPPS in Year 10 (2015) as well as in the evaluation of Wave 1 of the GP Access

Fund.



Each of these objectives was further broken down into a more detailed set of research questions,

designed to ensure that the concepts presented in the questionnaire relating to service access and

experience are relevant to patients regardless of their local model of care as well as its stage of

development. The following sections outline the research approach, followed by a summary of the

key findings.

5.3.3 Research approach

The approach involved in-depth interviews with patients in a range of locations, designed to broadly

reflect the varying levels of maturity of extended access provision. NHS England identified three broad

types of extended access area and a number of suitable locations within each to ensure a good

regional spread as well as urban/rural coverage:

▪ Full provision: areas with well-established communications in place around the availability of

extended access models (24% of all GP practices in March 20174): Coventry, Slough and

Wakefield. There was a particular interest in patients who might have experienced extended

access to help understand whether it is recognised as a concept in areas where it is offered (to

establish questions about it would be understood);

▪ Partial provision: areas where extended access is developing (64% of all practices): Lewisham;

and

▪ No extended access or provision (13% of all practices): Bath.

Research was also considered in two further “partial provision” areas (North Derbyshire and West

Lancashire) but further investigation found they did not fully meet the criteria required.

NHS England provided some information about each location and the provision available in order to

help target recruitment in specific areas where extended access was available. This information was

provided to recruiters, who used a detailed recruitment screening questionnaire in order to ensure

that patients experiencing a range of types of access were included. Given the nature of the research

it was important that within each location patients had a range of levels of knowledge and experience

of extended access appointments, covering evening and weekend appointments both at their own

surgery and another location. Beyond this, it was also important that a range of other demographic

factors were considered which may impact on interactions with GP services (including gender, age,

parental status, long-term conditions, work status and ethnicity).

4 https://www.england.nhs.uk/statistics/statistical-work-areas/extended-access-general-practice/

https://www.england.nhs.uk/statistics/statistical-work-areas/extended-access-general-practice/



NHS England also wrote to the CCGs involved asking them to raise awareness of the research with

local GP providers, Commissioners and NHS England teams to enable them to confirm its legitimacy

with patients if queried.

Reflecting the complexities of extended access provision, during fieldwork it was found that patients

often confused experience of extended hours appointments at other surgeries with attending a walk-

in centre. This led to a pause in interviewing while NHS England and Ipsos MORI worked to identify

locations, such as hubs, that would secure a high incidence of patients attending appointments

elsewhere. However, a pragmatic decision was made to stop interviewing, based on the balance of

when the information was needed for the questionnaire development process, and when further

interviews could likely take place.

In total, 21 interviews were completed in five locations. Details of these, along with other participant

characteristics, are provided in the Appendices (published under separate cover).

5.3.4 Research materials, fieldwork and analysis

A detailed discussion guide was developed to reflect the research objectives, designed by Ipsos MORI

and NHS England. The full guide is contained in the Appendices to this report.

Fieldwork was conducted by experienced in-depth interviewers between 26 June and 4 July 2017, and

interviews were recorded, with participants’ permission. Patients were offered a cash incentive of £50

to thank them for their time and cover any costs incurred in participating.

Regular team analysis sessions were held throughout the fieldwork period, a crucial component of

any qualitative methodology.

5.3.5 Key findings

The following section outlines the key findings, focusing on the first two research objectives.

Throughout, the third objective was held in mind, reflecting on whether patient experiences are

changing and how this may influence the redesigned questionnaire. For example, this helped the

research team to identify which elements of the existing questionnaire continue to resonate and

which need to change to fully capture patient experiences, as well as to examine the language used.

These findings were used to inform the subsequent development of the questionnaire, to ensure the

concepts and terminology used reflect patients’ understanding across areas with varying levels of

development of extended access provision.


https://www.gp-patient.co.uk/surveysandreports2018



Objective 1: What does primary care provision look like from a patient’s perspective

In order to understand how patients experience primary care provision, the research explored

different journeys that patients take and how well they understand the services available to them,

including the concept of a “GP practice,” particularly focusing on awareness of how this may have

changed in areas with extended access.

As models of extended access develop it is likely that the concept of an individual’s own “GP” or “GP

practice” will become less distinct. However, at this time patients across all types of area still strongly

related to the idea of their “own” GP practice: this was understood as the site at which they were

registered (even those who are at practices across multiple sites).

Patients had mixed views about the concept of “my GP.” Some identified this as the GP they were

officially registered with but indicated that they were happy to see anyone. Others thought of

someone they preferred to see, often related to continuity of care or personal preference.

The majority continued to book appointments by telephone and by contacting their “own” GP

practice, particularly if they felt they need to be seen urgently or were unsure who they needed to

see. However, for less urgent appointments a variety of methods were used, including face-to-face,

telephone, through the practice website, as well as via an app. In two areas, patients could book using

an automated telephone service. Uptake of online booking systems was varied. Among those not

using it, there are a variety of reasons for this, for example, finding it difficult to do or simply not

getting around to it, including among those who are more “tech savvy”.

Many also referred to the triage process by receptionists during appointment making: not all are

comfortable with this but they did understand the reasons behind it and were happy to be guided to

the most relevant healthcare professional or type of appointment (such as telephone or face-to-face).

For the most part patients spontaneously knew the days their GP practice is open, but not necessarily

the hours, particularly if this varies across the week. Knowledge of extended hours appeared to be

largely related to previous use of these services. Patients were often only aware of extended hours as

a result of being offered an appointment at this time, with those in full-time work being more likely to

know. Some suggested more needs to be done to promote awareness of extended hours. However,

there was also some evidence that these appointments are not always offered freely, but are used on

a priority basis relating to the urgency of the issue.

Patients were unclear about who is responsible for delivering extended access provision, a confusion

that partly stems from the fact that in some cases this is being offered on the same site as their own

practice, even if the GPs available are different.



Awareness of the availability of other health services within each surgery was limited, for example,

pharmacy access and physiotherapy appointments.

There is some blurring between patients’ understanding of extended access models and out-of-hours

care. For example, some felt they had attended a scheduled appointment with a specific GP at

another surgery, but on probing had actually attended a walk-in centre. However, patients were able

to distinguish extended access from urgent care.

Objective 2: What factors influence patients’ overall experiences of access?

The second objective focused on the factors influencing patients’ overall experience of access to

ensure that these could be effectively measured in the redeveloped questionnaire. Key questions

posed were around what choices patients make when accessing GP services and how they assess

whether their needs were met, but also looking at how expectations may vary according to the

particular circumstances (for example, reason for contact, timing, urgency).

Patient expectations appeared largely driven by their own assessment of the level of urgency of their

particular health issue, or the health of the person they are making the appointment for. In situations

deemed urgent, the most important factors are ease of making an appointment and seeing an

appropriate healthcare professional as soon as possible in order to find a solution. In such

circumstances, the exact timing of the appointment is less important than how quickly they are seen,

and most described being willing to sit and wait. The individual they see is also less important than

feeling the issue is resolved quickly. For non-urgent issues, expectations tend to be higher, with

patients wanting a convenient appointment that fits around their lifestyle and commitments: they

want a set appointment with limited waiting time, consistency in healthcare professional, particularly

for ongoing conditions, and to be given sufficient time during their appointment. Patients are also

focused on resolution, with satisfaction being driven by a positive outcome or ongoing support, such

as a prescription or referral.

While expectations also vary between what patients consider to be “in-hours” and “out of hours” care,

there are commonalities between them, with patients expecting the same level of care and a clear

course of action that is fully explained by the end of their appointment.

Throughout these discussions, it is worth noting that patients were mindful of the broader context of

demands on the NHS, and seemed generally accepting of current opening hours and level of

provision in their area.



5.4 Service user testing

5.4.1 Introduction

Prior to the 2018 redevelopment of the GP Patient Survey, four questions were asked about long-

term conditions. These sought to ascertain:

▪ whether the participant has a long-term condition;

▪ which condition(s) the participant has, selected from a list of 17 answer codes (14 relate to

specific conditions or condition groups and one encompasses “Another long-term condition”)5;

▪ whether the participant feels supported by local services to manage their condition; and

▪ how confident the participant feels in manging their own health.

As part of the survey redevelopment, these questions were considered within the long-term

conditions review (see chapter 4).

An important stage in the development of the questions was testing newly proposed questions with

service users and networks of patients living with specific long-term conditions.

5.4.2 Aims

As part of primary engagement with stakeholders across NHS England and the wider health system,

including the voluntary and community sector, five revised questions on long-term conditions were

proposed. As before, these sought to measure prevalence and type of condition, as well as support to

manage conditions and confidence to self-manage. A new question was also put forward which

sought to measure the impact of the reported condition(s) on the participant’s ability to carry out

day-to-day activities.

In addition to these amendments, it was also proposed that an additional six questions on care

planning, previously asked of all survey participants, would only be asked of those who self-reported

having a long-term condition.

The questions were tested with groups of patients who had one or more long-term conditions in

order to ensure that they were relevant and understandable ahead of the more detailed scrutiny

around the wording that would occur during cognitive testing. The primary purposes of the service

user testing were to:

5 The other two items in the list allow the participant to say they do not have a condition or that they prefer not to say.



▪ ascertain whether the service user would relate with the chosen definition of “long-term

condition”;

▪ understand whether the service user would actively select their condition or condition group

from the revised condition list;

▪ ascertain whether proposed changes to question wording and answer scales would be met with

participant error;

▪ ensure that the content of the question set (including the additional proposed questions) was of

importance to the service user; and

▪ understand whether any questions should be removed or could be added.

5.4.3 Research approach

Five key groups of users were identified across a number of different long-term conditions, and the

proposed question set (totalling 11 questions, along with an accompanying set of instructions) were

sent to network managers and engagement leads within NHS England and across the voluntary and

community sector. The service user groups engaged with were:

▪ Mental health;

▪ Alzheimer’s and dementia;

▪ Learning disability;

▪ Stroke; and

▪ Cardiovascular.

Service users were asked to answer the questions as if they were filling out the survey under normal

circumstances. They were also provided with an instruction sheet that asked them to respond to a

series of questions, along with prompts to encourage them to think about key aspects of design such

as question wording or length. They were asked to return feedback within five working days.



5.4.4 Key findings

Responses were received from a range of service users who spanned all of the condition groups.

Feedback was rich and varied and, as expected, was often highly personalised to the individual

experience. In particular, this was reflected in the array of suggested additional questions including:

▪ Has anyone got power of attorney over you?

▪ If you have been referred to another service for your condition, did you feel provided with

enough information and support from your GP practice to access that service?

▪ Do you think the reception staff have an awareness of your condition?

▪ Does someone provide extra help to you now for day-to-day tasks?

Some of the suggested questions had the potential to collect data on areas of policy yet unmeasured

by the survey (such as onward pharmacy referrals), whilst others were in essence revealing of the

user’s experience of living with their condition (such as integrated care for community stroke services).

Ultimately, these questions were considered to be befitting of smaller, local surveys than the GPPS,

given that they would likely be relevant to only a small proportion of patients.

However, there was some clear consensus around specific issues in relation to the existing question

set:

▪ Many service users, particularly those identifying as having a cardiovascular condition, failed to

identify themselves as having a long-term condition under the provided definition (a physical or

mental health condition, disability or illness, lasting for or expected to last for 12 months or

more). However, these users were able to select their condition from the list provided in the

following question. Feedback indicated that those with low-impact and stable conditions that

are managed or controlled on a day-to-day basis (such as hypertension) were not considered to

be “conditions” or “illnesses” by those experiencing them. It was resolved to explore this further

in cognitive testing.

▪ Feedback included a number of suggested additions to the list of conditions, including “multiple

sclerosis” and “heart failure.” While ideally many more conditions would be included, this list was

developed through detailed conversations with stakeholders and based on the prevalence of

conditions within the general population.



▪ Issues of comprehension were raised with the questions on care planning and support, in

particular with the terms “managing” (in relation to managing a condition) and “plan.” Feedback

from service users indicated that these terms were considered to be jargon and that further

clarification was needed. Text was drafted to accompany the questions in which these terms

appeared, and this was taken forward to cognitive testing.

▪ Responses also served to highlight the need to account for variations in how a condition affects

one’s life. This feedback was applicable to many of the questions asked, and was deemed to be

especially relevant to service users living with mental health conditions, the symptoms of which

may be less consistently felt.

▪ In particular, feedback from service users living with a long-term mental health condition called

for parity in the list of conditions, which were felt to be heavily weighted towards physical health

conditions.

In the main part, however, there were contradictions in the feedback: while some were satisfied, other

service users asked for a complete overhaul in question wording. Contradictions were especially

evident in feedback from patients with a mental health condition. In particular, these participants

expressed concern about the terminology, “mental health condition,” to capture a range of conditions

with very different symptoms, effects and degrees of severity. This highlighted that more work still

needs to be done to understand how best to concisely collect information about mental health

conditions through a questionnaire.

In general, however, the majority of service users who fed back felt that the questions were worded

appropriately and sensitively, and that the right questions were asked and in the right order. Changes

made were taken forward to cognitive testing. Additional work into soliciting feedback from service

users who self-identify as having a mental health condition is ongoing.



6 Cognitive testing




Following the public engagement and stakeholder workshop, a new questionnaire was drafted in co-

ordination with NHS England. This aimed to take into account the changing service provision across

the country, to ensure that the questionnaire could continue to accurately measure patients’

experiences. However, the language and concepts used needed to be verified via a programme of

cognitive testing.

Cognitive interviews are a type of in-depth interview that serve to test both how the wording of

individual questions is understood and how sets of questions work to influence each other in the

context of a wider questionnaire. Specifically, cognitive interviews aim to determine how individuals

perceive each question alone and as part of a survey instrument. The cognitive interviews are

administered face-to-face with the interviewer observing the participant while completing the

questionnaire. This can be done in a variety of ways – in this case, participants were also encouraged

to “think aloud” as they completed the questionnaire, to allow for further understanding of

participants’ thought processes when answering the questions. The interviewer then asks about

specific aspects of the questionnaire to understand how participants determine their answer to each

question and to ensure that questions are understood as intended. Participants’ feedback is then used

to clarify question meaning where necessary and improve the survey as a whole.

There were two key objectives to this research:

▪ To ensure that the questions were able to measure people’s experiences of general practice

services;

▪ To ensure the changes did not impact the capacity of the public to understand the questions.

This chapter outlines the research approach used and outlines the key findings from each round of

testing. The feedback from these cognitive interviews fed directly into the redesign of the

questionnaire for 2018 (Year 12) of the GPPS.

6 Cognitive testing



6.2 Cognitive testing

6.2.1 Research design

The approach involved three rounds of cognitive interviewing with patients. Given the nature of the

research it was important that a range of demographic factors were considered which may impact on

interactions with GP services (including gender, age, parental status, long-term conditions, work status

and ethnicity), as well as specific factors relevant to certain questions being tested (such as whether

they had booked appointments for themselves or someone else, or attended general practice

services at a location other than their GP practice).

The specific quotas for each round of testing were agreed with NHS England. Recruitment was

conducted face-to-face with a detailed recruitment screening questionnaire used to identify

individuals who met the quota criteria.

In total 40 interviews were completed. Details of the participant characteristics are provided in a

separate cognitive testing report.

6.2.2 Research materials, fieldwork and analysis

A detailed discussion guide was developed to reflect the research objectives, designed by Ipsos MORI

and NHS England. This was adapted for each round, to reflect the changing questionnaire and

specific priorities. The full guide is included in the cognitive testing report.

Fieldwork was conducted by experienced in-depth interviewers, between 30th August and 12th

October 2017, and interviews were recorded, with participants’ permission. Participants were offered

a cash incentive of £40 to thank them for their time and cover any costs incurred in participating.

Regular team analysis sessions were held throughout the fieldwork period, a crucial component of

any qualitative methodology.

6.3 Key findings and recommendations from cognitive testing

Overall, in all three rounds of testing, the questionnaire was found to be clear, straightforward and

easy to complete. This likely reflected the previous elements of the questionnaire development work

conducted, including the qualitative work with patients, to ensure that the participants’ journeys could

be captured in full, in the language that they use.



The questionnaire was iteratively improved for each round of testing and the issues to resolve

became more specific as general clarifications were implemented. This report only pulls out the key

findings from each round, and does not address specific wording within questions. A full break-down

of changes to the questionnaire is included in a separate cognitive testing report.

6.3.1 Round 1

During Round 1, a particular focus for the cognitive testing was looking at participants who had not

been able to get an appointment when they tried to, and those with caring responsibilities. Therefore,

additional quotas were included specifically for these groups at this stage.

Participants generally found the questionnaire straightforward to respond to. However, some key

areas were flagged for review. These included:

1) Questionnaire routing: Participants struggled with the routing in several places, mostly

completing questions that they should have been routed away from. As a result, where routing

was particularly important, the questionnaire was reformatted to ensure these questions were

on the same page and arrows could be used to highlight which questions participants should

answer.

2) Multi-coding: There was some confusion about multi-coding on certain questions. As a result,

questions and instructions were reviewed, to ensure it was clear where participants could

select more than one answer.

3) Length of lists: Some of the lists were found to be too long, making it difficult for participants

to find the answers that were appropriate to them. These lists were then reviewed and

condensed where possible, to allow for easier completion.

4) Use of online services: Questions about use of online services, particularly what constitutes a

practice website or an NHS website, were unclear to some participants. There was also some

confusion over whether ease of use of online services would include being registered to use

these services. These questions were then revised, in co-ordination with NHS England’s online

team, to ensure they more closely reflected patient experiences.

5) Booking an appointment for someone else: Where participants could answer questions about

booking an appointment for someone else, some of the answer options needed updating to

reflect that experience.



6) Questions about “Your Health”: The questions about mental health needs, unplanned hospital

admissions and frailty were reviewed in response to feedback collected. These questions

continued to be developed across all three rounds, with input from relevant teams at NHS

England.

6.3.2 Round 2

Following the changes made in response to the feedback collected in Round 1, a second round of

cognitive testing was undertaken. For this round, there was a particular focus on participants who had

not been able to get an appointment when they tried to, those with caring responsibilities, and those

who had booked an appointment for someone else the last time they tried to get an appointment.

Again, participants were generally positive about the questionnaire and their ability to complete it.

However, some key areas raised for review were:

1) Thinking of all general practice locations: There was concern that instructions might not be

clear enough to encourage participants to think about all places where they could make a

general practice appointment, particularly those based at practices with more than one site.

This was therefore raised as a priority for the third round of cognitive testing.

2) Times general practice appointments were available: This question was focussed on measuring

the times when general practice appointments were available. The wording therefore needed

to be revised, as earlier iterations meant participants focussed on either the times their

particular practice was open, or on times when they needed to call in order to make an

appointment, not when the appointment itself would take place.

3) Specific experiences of making an appointment: The questionnaire did not capture the

experiences of those who were making more than one appointment at a time or felt they had

not been offered an appointment at all. As a result, some additional instruction text was added

to encourage participants to focus on one appointment, and answer options were reviewed to

allow for instances where participants felt they had not been offered an appointment.

4) Booking an appointment for someone else: Some participants struggled with the answer

options regarding who they had booked an appointment for, particularly with the definitions

of “caring responsibility,” and what this would cover. As a result, these were clarified to allow

for clearer responses.



6.3.3 Round 3

Following the changes made in response to the feedback collected in Round 2, a third round of

cognitive testing was undertaken. For this round, there was a particular focus on participants who had

booked an appointment for someone else last time they had booked an appointment, and

participants who had attended general practice appointments at locations other than their GP

practice.

Once more, participants found the questionnaire clear and easy to complete. However, in the final

round of testing, the following areas were raised as areas to consider in the final drafting:

1) Thinking about multiple general practice locations: Following the concerns raised in round 2,

participants were particularly probed on where they were thinking about all places where they

were able to get a general practice appointment. Although participants generally focussed on

the practice they were registered at, this was mostly due to limits or a lack of awareness on

where they were able to get appointments. As changes to the structure of general practice

continue, this will need to be monitored to ensure these experiences are captured.

2) Multi-coding of single-code questions: There were also some issues where participants multi-

coded questions that had been set-up as single-code questions. These were discussed with

NHS England, so that, where possible, the answer options emphasised that only one answer

option should be selected. In these cases, it was also discussed what to do with the data

should someone multi-code the answer in the final survey.

3) Choice of appointment: When thinking about the choices that were offered when booking,

participants tended to focus on the choice that was most important to them, rather than all

the choices this represented. As a result, the options were narrowed to encourage participants

to code all areas that were important to them.

4) When the appointment took place: The question asking about the length of time between

when the appointment was originally made and when it took place was revised several times

for clarity. Although participants were eventually able to answer the question in earlier

iterations, some had to read it several times to ensure they understood it.

5) Mental health: Participants who had previously been diagnosed with a mental health condition

found it difficult to answer the question which asked whether their mental health needs were

recognised, because they felt this was not always relevant to every appointment. As a result,

an additional answer option was included.



6.3.4 All three rounds

In all three rounds of cognitive testing, there were specific long-term conditions (such as high blood

pressure) which participants did not consider a to be long-term condition. This meant they would

initially say they did not have a long-term condition, but when presented with a list would select one.

This was mostly related to the impact they felt their condition had on their life. This had also been the

case in the previous version of the questionnaire, which had resulted in the overall long-term

conditions variable being back-coded from the specific list.

Participants were also particularly questioned on the use of “GP practice” and “general practice”

across all three rounds. Participants generally understood both to refer to their own surgery, where

they were registered. However, where participants had experience of attending general practice

appointments at more than one site, they were thinking more widely about the places they could

attend appointments. Combinations of both “GP practice” and “general practice” in a single question,

however, did cause confusion for participants. As a consequence, the term “GP practice” was used to

frame questions about a patient’s specific GP practice; where a GP practice consists of more than one

site, all sites were to be considered. In contrast, the term “general practice” was used where

participants were required to think about all places where they could make a general practice

appointment (not just their own GP practice). Participants generally focussed on the practice they

were registered at, irrespective of the term that was used. This was mostly due to limits or a lack of

awareness on where they were able to get an appointment. However, where participants had

experience of attending general practice appointments at more than one site, they were able to think

more widely about the different places at which they could attend appointments.



7 Next steps



The questionnaire was signed off on 10 November 2017, before being graphically designed for

printing. This final questionnaire took into account all the development from:

▪ analysis of existing GPPS data,

▪ engagement with stakeholders – through the steering group, questionnaire development

advisory group, workshop and on specific issues,

▪ engagement with patients – through the website, qualitative research and service user testing,

and

▪ cognitive testing.

Fieldwork for Year 12 of the GP Patient Survey, using this updated questionnaire, ran from 2 January –

9 April 2018.

Results are due to be published on 9 August 2018 and will be made available via the GPPS website.

7 Next steps



Ipsos MORI | GP Patient Survey: Questionnaire redevelopment A1


For more information

3 Thomas More Square

London

E1W 1YW

t: +44 (0)20 3059 5000

www.ipsos-mori.com

http://twitter.com/IpsosMORI

About Ipsos MORI’s Social Research Institute

The Social Research Institute works closely with national governments, local public services and the not-for-profit sector.

Its c.200 research staff focus on public service and policy issues. Each has expertise in a particular part of the public sector,

ensuring we have a detailed understanding of specific sectors and policy challenges. This, combined with our methods

and communications expertise, helps ensure that our research makes a difference for decision makers and communities.

july 2018 gp patient survey y12 questionnaire... · 2018-08-01 · the nhs england insight &...

Documents