engaging the student voice to enhance student services dr hamish coates [email protected] november...
TRANSCRIPT
Focus of the presentation
• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?
Quality data for management
• The quality of ‘education’ data underpins the cogency of many strategic and operational decisions
• Senior managers require timely access to informative and robust data
• But often the wrong data is collected• But much data management within Australian
higher education institutions is ad hoc We need to think strategically about what data
is required for institutional management We need efficient systems for analysing and
interpreting large amounts of complex data
Identifying existing data
• Does the institution have central data management protocols?
• What data exists centrally, and within faculties, departments, units, subjects…?
• What is the nature and limitations of existing data?
• What is the strategic or operational relevance of the data?
• How have the data and collections been developed?
Reviewing data needs
• What is required to bring disparate data together – initially, and in routine ways?
• How is this data best analysed and reported, and who should do this?
• What are the relevant points of reference – norms or benchmarks/criteria?
• What ‘gaps’ exist in the data collection?• What redundancies exist, and what are the
legacy issues?
Focus of the presentation
• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?
A summary of existing data…Inputs Processes Outcomes
Institutions Capital resourcesInstitutional
reputation
Institutional contextsLearning systems
GrowthReputation
Staff Teaching qualifications
Teaching experienceTeaching staff
numbersExpenditure per
studentStaff/student ratios
Teaching processes Teaching experienceTeaching resources
Students Entry scoresStudent diversityDemand/selectivityStudy access/equity
Engagement processes
Retention and attrition
Course completionsGraduate and generic Engagement
outcomesLearning outcomesGraduate
destinations
Internal ‘student support’ data
0 10 20 30 40 50 60 70 80 90
Internal satisfaction surveys
Equity success indicators
Usage statistics for student services
Internal reviews
Programs offered in flexible mode
CEQ / PREQ scales - support services
E-learning Help Desk response times
Size of library collection
Expenditure on services
Range of student services
Online student support / enrolment
Per cent of institutions (2004)
External ‘student support’ data
CEQ Learning Resources Scale (8/38 institutions)• The library resources were appropriate for my needs• The study materials were clear and concise• It was made clear what resources were available to help me learn• Course materials were relevant and up to date• Where it was used, the information technology in teaching and learning
was effective
CEQ Student Support Scale (18/38 institutions)• I was able to access information technology resources when I needed
them• Relevant learning resources were accessible when I needed them• Health, welfare and counselling services met my requirements• The library services were readily accessible• I was satisfied with the course and careers advice provided
Data use by institutions
0 10 20 30 40 50 60 70 80 90
Strategic planning and management
Reporting eg annual report, DEST profiles, AUQA
Support budget process
Course/curriculum reviews/development
Quality enhancement
Academic Organisational Unit reviews or evaluation
Benchmarking
Marketing/Promotion
Staff performance management
Distribute teaching funds
Reward/incentive strategies
Per cent
Focus of the presentation
• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?
Exploring ‘student engagement’…• Why take the ‘engagement approach’?• How can we factor ‘insights’ on student
engagement into quality management?• Why isn’t student engagement information
implicit in quality educational management?• Who defines how students should interact with
universities?• Who benefits from evidence-based
understandings of engagement?• What value could engagement insights add to
higher education?
Inputs Processes Outcomes
Institutions Capital resourcesInstitutional reputation
Institutional contextsLearning systems
GrowthReputation
Staff Teaching qualificationsTeaching experienceTeaching staff numbersExpenditure per studentStaff/student ratios
Teaching processes Teaching experienceTeaching resources
Students Entry scoresStudent diversityDemand/selectivityStudy access/equity
Engagement processesRetention and attrition
Course completionsGraduate and generic Engagement outcomesLearning outcomesGraduate destinations
Educational indicators
The idea of ‘student engagement’• Individuals learn and develop through behavioral,
cognitive and affective involvement with key educational and social practices
• People learn when staff and institutions provide supports likely to encourage involvement
• A specific idea which encompasses a broad range of relevant academic and non-academic phenomena
• Measures of engagement provide an index of whether students are engaging with university in ways likely to stimulate development and generate high quality learning
• Student engagement is not ‘satisfaction’, not ‘retention’, not just/primarily the evaluation of pedagogy, not an attitudinal/opinion survey, and not ‘student experience’
Engagement as a development
• Developed from interpretations of individual learning and development as constructive participation in university communities
– Astin’s ‘involvement principle’– Pace’s ‘quality of effort’– Tinto’s work on integration and departure
• Based on the identification of activities and conditions linked empirically with effective learning
– Chickering & Gamson’s ‘seven principles’– identification of process factors by Pace, Pascarella
& Terenzini, Astin, and Ewell
Engagement as a reaction
• Engagement offers a valid and consistent measure of the quality of education
• Limitations of alternative means of evaluating the quality of university education
– institutional resources and reputations– measures of teaching quality and teacher
qualifications– student input, progression and outcomes
The significance of engagement• Reflects students’ intrinsic involvement with
education• Offers a generalisable indirect measure of
educational outcomes• A direct measure of involvement in key processes• Focuses quality considerations on student learning
and development• Reflects the wide range of educationally meaningful
interactions that students have with their universities• A coincident measure of student activity that can be
used to evaluate and manage the quality of higher education
Typical qualities of student engagement
• Constructive teaching• Supportive learning environment• Teacher approachability• Student and staff interaction• Academic challenge• Active learning• Complementary activities• Collaborative work• Beyond class collaboration• Online forms of engagement
Styles of engagement
Possible measurement approaches
• One-to-one interaction
• Direct naturalistic observation
• Administrative data
• Time or activity diaries
• Assessment
• Interviews and focus groups
• Questionnaires and surveys
Student experience instruments
• College Student Experiences Questionnaire (CSEQ)• College Student Expectations Questionnaire
(CSXQ)• College Student Survey (CSS)• Freshman Survey (FS)• Your First College Year (YFCY)• National Student Survey (NSS)• First Year Experience Questionnaire (FYEQ)• Course Experience Questionnaire (CEQ)• Postgraduate Research Experience Questionnaire
(PREQ)
Student engagement instruments
• College Student Report (CSR)• Student Engagement Questionnaire (SEQ)• (CEQuery)
Levels of engagement
0.00 1.00 2.00 3.00
Constructive teaching
Supportive learning environment
Teacher approachability
Academic challenge
Active learning
Student/staff interaction
Complementary activities
Collaborative work
Beyond-class collaboration
Never Rarely Sometimes Often0.00 1.00 2.00 3.00
Constructive teaching
Supportive learning environment
Teacher approachability
Academic challenge
Active learning
Student/staff interaction
Complementary activities
Collaborative work
Beyond-class collaboration
Never Rarely Sometimes Often
0
1
2
3
OEOA
LOA
R OT OC OSI
OCS CT
SLE TA AC AL SSI
CA CW BCC
English not first language
English first language
often
some-times
rarely
never
Using ‘engagement’ data
• Guidance for prospective students on expectations, experiences and supports at an institution
• Data on resource use, management and educational value
• Information for internal and external quality enhancement activities
• Insight for graduate employers on individuals’ involvement with their work
• Feedback for teaching academics to help them manage and improve pedagogy and learning
• Advice for current students on course quality and learning practices, retention and involvement
Focus of the presentation
• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?
CEQuery context• Graduate Careers Australia conducts an annual survey of
graduates using the Graduate Destinations Survey (GDS) and the Course Experience Survey (CEQ) – these are sent out together
• These surveys capture a range of demographic variables linked to responses on graduate outcomes (GDS) and satisfaction (CEQ)
• The CEQ also has sections for written comments under the headings “Best aspects” and “Needs improvement” – these are analysed by CEQuery
• This enables the student’s qualitative comments to be linked with demographic and field of education variables, satisfaction scales and graduate outcomes
The opportunity and need
• Need not just to gain but retain students
• Rapid developments in I.T.
• 750,000 CEQ comments - unanalysed
• Listen to the ‘voice’ of students and not just rely on quantitative measures
What is CEQuery?• IT-enabled analysis of large amounts of
qualitative data on HE learning & teaching• Wide range of custom analyses against any of
the CEQ student demographic categories or items
• Results in tabular, graphical or digital modes• Allows user to test the veracity of the coding by
showing the comments that made up the count• Five domains and 26 subdomains
CEQuery coding domains
• Assessment
• Course design
• Outcomes
• Staff
• Support
CEQuery subdomains
• Assessment– Expectations– Feedback– Marking– Relevance– Standards
• Course design– Flexibility– Learning methods– Practice-theory links– Relevance– Structure
CEQuery subdomains
• Outcomes– Further learning– Intellectual– Interpersonal– Personal– Knowledge/skills– Work application
• Staff– Accessibility– Practical experience– Quality and attitude– Teaching skills
CEQuery subdomains
• Support– Infrastructure– Learning resources– Library– Social affinity– Student administration
– Student Services– Course administration
Hit Rates: Domain Level
Staff, Teaching Skills, Needs Improvement
• Some lecturers provided poor lectures and in some subjects the questions in the final exam were nothing like the ones we had been taught to answer.
• Affordable technology needs to be better -utilized as teaching aids by staff who know that teaching is more than standing up the front droning on
• Lecturers need to be brought in line now traditional teaching methods are becoming redundant to tutorials. They need to learn how to actively involve us, get us working creatively, experimenting – less chalk and talk.
• The teaching staff should give more constructive feedback and make learning more interactive during classes.
• The methods employed by lecturers are boring they need to participate more with audience to encourage participation in discussions and enhance interest.
Validity of the results
• CEQuery comments hit rate: 80%– Of the remaining 20% not hit in CEQuery, half are
not meaningful (cannot be coded)
• Coding of comments to sub-domains by CEQuery checked by multiple raters– Accuracy approx 90%
• CEQuery dictionary of words and phrases has been refined and updated, and custom dictionaries have been produced
A review of CEQuery
• Geoff Scott (2005). Accessing the Student Voice. DEST: Canberra.
• Analysed comments made by 95,000 graduates between 2001-4 at 14 universities
• Key insights into CEQuery use, correlates of response, key aspects of engagement, ‘best aspects’ and ‘improvement areas’
• Findings validated through a series of national workshops
Subdomain BA count NI count BA/NI (QUALITY)
Outcomes – personal (OP) 1668 88 19.0
Outcomes – further learning (OF) 347 24 14.5
Outcomes – intellectual (OI) 7197 541 13.3
Outcomes – unspecified (OU) 1126 173 6.5
Support – unspecified (SU) 277 54 5.1
Outcomes – interpersonal (OIP) 2455 595 4.1
Outcomes – knowledge/skills (OK) 8037 2746 2.9
Staff – accessibility (SA) 12748 10611 1.2
Course - Practical–theory links (CP) 9157 7658 1.2
Outcomes – work application (OW) 4715 5248 0.9
Assessment - unspecified 179 251 0.7
Support – learning resources (SR) 3970 6006 0.7
Course design – relevance (CR) 6335 9658 0.7
Staff – teaching skills 5548 9969 0.6
Support – infrastructure/environment (SI) 3423 6353 0.5
Support – library (SL) 1018 1933 0.5
Support – student services 784 1808 0.4
Assessment – standards (AS) 1873 5449 0.3
Support – student administration (SAd) 1078 4095 0.3
Course design – structure (CS) 3579 15668 0.2
Assessment – marking (AM) 386 2045 0.2
Assessment – expectations (AE) 308 1794 0.2
Assessment – feedback (AF) 316 2792 0.1
Perceptions of quality by subdomian
Subdomain BA count NI count TL count (IMPORTANCE)
Course – methods (CM) 22231 18338 40569
Staff – quality (SQ) 17417 13512 30929
Staff – accessibility (SA) 12748 10611 23359
Course design– flexibility (CF) 12754 10579 23333
Course design – structure (CS) 3579 15668 19247
Course - Practical–theory links (CP) 9157 7658 16815
Course design – relevance (CR) 6335 9658 15993
Staff – teaching skills 5548 9969 15517
Support – social affinity (SSA) 7249 3683 10932
Outcomes – knowledge/skills (OK) 8037 2746 10783
Support – student administration (SAd) 1078 4095 5173
Assessment – relevance (AR) 2537 1400 3937
Assessment – feedback (AF) 316 2792 3108
Outcomes – interpersonal (OIP) 2455 595 3050
Support – library (SL) 1018 1933 2951
Support – student services 784 1808 2592
Outcomes – personal (OP) 1668 88 1756
Course design – unspecified (CU) 1079 574 1653
Outcomes – unspecified (OU) 1126 173 1299
Staff – unspecified (SU) 347 334 681
Assessment - unspecified 179 251 430
Outcomes – further learning (OF) 347 24 371
Support – unspecified (SU) 277 54 331
Importance perceptions by subdomian
Clusters of the best learning methods
Cluster Hits
Face to face 11693
Simulations and labs 1357
Independent study and negotiated learning 3572
Practice-oriented and ‘real world’ 9329
ICT Supported 836
Clustered best learning activities in Business
Face to Face Lecture, group project, tutorial, class-
work exercise, discussion and sharing ideas, seminar and individual presentation, workshop, debate, 1:1 mentor, conference, symposium, forum, exhibition, peer assistance, group dynamics exercises, critique of student production, buzz group
Independent Study Completing assignments, thesis, self
teaching materials, self-directed study, project report writing, learning contract
Simulations & Labs Mock trial, role play, hypothetical,
games, experiments, labs, simulation
Practice oriented/real world Clinical placement, practicum,
practical experience, PLT, Coop Ed, work experience, hands on practice, field study/site visit, practical work,camps, real world problems, guest practitioners, professional mentor, design studio, artistic production, study abroad, case study
ICT-enabled learning Online search, web-based learning,
email, phone, teleconference, CD, DVD, tele-tutorial, radio, TV, digital image access, computer assisted assessment, simulations, computer generated demonstrations of processes, interactive programs
Most common best aspects by field of education
Science & Built Environment Team/group project, assignments, field
study/site visit, hands on practice, lecture, class exercises, laboratory work, practical work, practical experience
HealthClinical placement, practical experience, lecture, hands on practice, assignments, tutorial, class exercises, group project work, labs, practicum, work experience
EducationPracticum, practical experiences, assignments, hands on practice, lecture, tutorial, class discussion, class exercises, team/group project work
Management & Commerce Team/group project, assignments,
lecture, class exercises,, seminar – individual presentation, tutorial, discussion, case study, real world problems to solve, work experience
Society, Culture & Creative ArtsAssignments, class exercises, lecture, tutorial, group project, class discussion, hands on practice, practical experience, seminar – individual presentation, practical work
CEQ scales and CEQuery domains• All five domains significantly associated with all CEQ
scales• Best aspects comments predict higher CEQ scale
scores and needs improvement comments lower CEQ scale scores
• Strongest correlation with the Good Teaching Scale, Overall Satisfaction Item (OSI) and Clear Goals and Standards Scale
• For example: given OSI score, odds of positive rather than a negative comment given OSI score: Assessment 1.22, Course Design 1.34, Outcomes 1.40, Staff 1.21, Support 1.21 (eg: positive ‘support’ comment 21% more likely for each step up 5-point CEQ scale)
Assess-ment
Coursedesign Outcomes Staff Support
University X X X X X
Field of education X X X X X
Qualification X X X X
Fees X X X
Gender
Age X X X X
Attendance type X X
Attendance mode X X X X
Year data X X X
Australian residence X X X X
ATSI
Conditions that enhance engagement• A sound, responsive, flexible, relevant, clear and mutually
reinforcing course design—a design that uses an appropriate variety of interactive, practice-oriented and problem-based learning methods
• Capable, committed, accessible and responsive staff being in place to deliver and improve the design during implementation
• Efficient and responsive administrative, IT, library and student support systems actively working together to support its operation
• Relevant, consistent and integrated assessment of a university standard that the course’s design, learning methods and resources specifically enable students to complete
Key findings• It is the total experience that retains students• Graduates see certain experiences are more ‘important’
than others, and identify high quality areas as well as those in need of improvement
• Areas with highest hits identify key QA checkpoints for course design, accreditation and review
• Variation between field of education and institution suggests opportunities for benchmarking
• Should address the areas with low odds of a best aspect comment (assessment, course structure, support)
• Follow up specific issues, such as the low level mention of ICT as a best aspect
Focus of the presentation
• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?
Using findings to enhance practice
• Student engagement data can be used to enhance the quality of education and the university experience
• Strategic use of ICT/LMS/portals in ways that supplment and enhance the student experience
• We need to develop measures of engagement, and make better use of the CEQuery tool
• We need to develop ways of factoring evidence-based perspectives on student engagement into practice
Use CEQuery• Available to all universities• Analyse feedback data from many survey
instruments• Can be tailored for local contexts• Need to enter open-text response• Need to position within strategic reporting and
planning cycles• Identify good practice, internal benchmarking,
priorities for improvement
Review student engagement• Ensure feedback instruments are harnessing
data on what really matters for high quality learning and support – we’re not measuring key aspects of higher education
• Ensure analysis and reporting approaches are appropriate
• Weave ‘student engagement’ ideas into operational and strategic plans, and practices
Help students learn to engage • Produce resources which help students
understand key aspects of being a university student
• Develop targeted programs and spaces to help students learn about university
• Develop practices which encourage engagement
• Provide students with feedback on their views
Manage student engagement• Shape support systems in response to known
individual and cohort differences, perceptions and reports
• Shape adaptive spaces, portals, services, cultures, activities, programs, approaches, people…
• Study student readiness, identify at risk students, studying student transitions through study
• Develop seamless links between academic and non-academic aspects of university
Benchmarking
• Make comparisons within and between institutions
• Develop meaningful points of reference and for improvement
• Develop ‘learning communities’ to generate and disseminate key ideas
At your institution…
• How could student engagement data be better used at your institution?
• What changes in data collection, analysis and reporting may be required?
• What programs or practices could be altered, and how?