he course and module evaluation conference - paul bennet
TRANSCRIPT
Planning for enhancement using programme surveys
Dr Paul Bennett
Informed by work at theHigher Education Academy
∂
Outline
1. Purpose and challenges of programme surveys• Purposes and types of programme survey • Challenges of student surveys• Challenges of‘programme-ness’
1. Driving enhancements to education• Survey results as quality indicators• Engaging academic staff
are the main strengths and limitations of using the NSS for informing changes to teaching?
∂
Purposes of programme surveys
NSS Review• Providing information to prospective students• Providing information to HE providers
(for quality assurance and enhancement)• Ensuring public accountability
(especially informing government and funders)• Informing current students strengths and
limitations of using the NSS for informing changes to teaching?
∂
Purposes of programme surveys
QAA UK Quality Code (B5 – Student Engagement)• Questionnaires as one of a wide range of
mechanisms by which students can be involved in quality systems
• Survey results themselves part of the information base for student-staff discussions
∂
Types of student surveys
• Satisfaction• Experience • Climate (more common at institution level)
• Engagement• Diagnostic (more common at module level)
Questions may attempt to extract: opinions, attitudes, factual information, perceptions, motivations, expectations, behaviours, development, understanding…
∂
Love? Hate?
Efficient OverusedDemocratic ConsumeristDigestible ShallowComparable MisleadingReliable InvalidMotivating Demotivating
SURVEYSSTUDENT EXTRACT
Challenges of student surveys
∂
Challenges of ‘programme-ness’
• Defining the correct population• Long time period with unknown reference points• Multiple experiences, modules, lecturers• End of programme – tend to be summative• Game playing / desirability bias• Generic questions• Masks variability of experience between students
∂
Discussion
1. Which programme surveys do you use and for what purposes?
2. What are the main challenges in meeting those purposes?
3. How can we best deal with ‘programme-ness’?
∂
Survey results as quality indicators
• As with any research method, surveys provide a partial representation of …
• Do programme surveys purport to indicate ‘quality’?
Graham Gibbs (2010) Dimensions of Quality• Measures of quality should relate to educational gain• Measures of effective practice (student engagement,
intellectual challenge, deep learning) are good predictors of educational gain
• Such measures are found in the National Survey of Student Engagement (NSSE) but not in the UK’s NSS
∂
Survey results as quality indicators
Implications
•Survey results should be triangulated with other sources of information
•Don’t just mirror the NSS in internal programme surveys or module evaluations
•Engagement surveys being trialled in UK context (HEA ‘UKES’ pilot promising, new NSS pilot in progress)
•BUT the NSS will remain a priority in University planning and quality enhancement – engaging staff vital
∂
Engaging academic staff
• Staff sceptical of survey metrics:
- Reductionist - Data problems, aggregation, programme-ness…- Response rates - Question focus and language- Top down approach with a ‘deficit’ focus – ‘a stick to beat us with’
• Why are the results as they are?
∂
Engaging academic staff
• Opportunities to unpack and openly discuss results
• Focus on positive areas and sharing best practice
• Triangulation with other information and expertise
• Further research and investigation to ‘drill down’
• Staff – student engagement
∂
Progress in use of the NSS for enhancement
- ‘Traffic-light’ performance monitoring
- Use in staff meetings and action-plans
- ‘You said, we did’
- Further research and investigation
- Staff-student-manager dialogue (formal/informal)
- Staff-student events
- Students as partners (e.g. in action planning)
∂
Discussion
1. How do survey results inform planning for enhancement at your institution?
2. What are the challenges in ‘drilling down’ into results to find out what the issues are?
3. How best can academic staff be engaged in discussion of survey results?
∂
• Programme surveys are useful research method giving a partial representation of experience
• Recognising strengths and limitations is key to planning effective enhancement
• Needs to be triangulated with other information – e.g. module evaluation, qualitative feedback
• Importance of staff and student engagement
Summary
‘Making it Count’
- using the NSS for enhancement
www.heacademy.ac.uk/nss
16
Planning for enhancement using programme surveys
Dr Paul Bennett
Informed by work at theHigher Education Academy