outcomes of social work education (oswe): findings from the projects
DESCRIPTION
Outcomes of Social Work Education (OSWE): Findings from the Projects. Hilary Burgess, John Carpenter, Joanna Fox, Roxana Anghel, Sharon Vitali, Juliet Koprowska, Kish Bhatti-Sinclair, Anne Quinney, Imogen Taylor, Clare Ockwell, Suzy Braye, Michelle Lefevre. Outline of Session. - PowerPoint PPT PresentationTRANSCRIPT
1
Outcomes of Social Work Education (OSWE):
Findings from the ProjectsHilary Burgess, John Carpenter, Joanna Fox, Roxana Anghel, Sharon Vitali, Juliet Koprowska, Kish Bhatti-Sinclair, Anne Quinney, Imogen Taylor, Clare Ockwell, Suzy Braye, Michelle Lefevre
2
Outline of Session
1. Introduction to OSWE project2. Presentation of findings from six partners in
England with questions for clarification.3. Discussion of the findings in general and their
implications.4. Key issues in doing outcomes-focussed
research on Social Work Education.5. Open discussion.
3
Aims of OSWE (3 year project)
To test the feasibility of outcome measures and research designs in Social Work Education
To generate high quality evidence about the effectiveness of methods of SWE
To build capacity and capability amongst academics, including service users
To use opportunities to compare and contrast practice between programmes
4
A Collaborative Capacity & Capability building model
Peer learning through Action Learning Set Support, advice and mentoring, F2F, e-mail and by
phone (e.g. data analysis).
See Burgess, Hilary and Carpenter, John (2008) 'Building Capacity and Capability for Evaluating the Outcomes of Social Work Education (the OSWE Project): Creating a Culture Change', Social Work Education.
5
Measuring Outcomes
Measuring changes over time Before-after designs Using and adapting measures
6
Primary levels of Outcomes (after Carpenter, 2005)
Levels of Outcome 1.Learners’ reactions2.Modifications in attitudes and
perceptions Attitudes Motivational
3.Acquisition of knowledge and skills
Procedural Strategic knowledge Initial skills Compilation skills
4. Changes in Behaviour5. Benefits to users and carers
Focus
Attitudes to race and racism (Soton)Attitudes interprofessional practice
(Sussex)
Understanding partnership with SU&C (Anglia Ruskin)
Communication skills with children (Sussex)
Interviewing and communication skills (York)
Use of research skills (Bournemouth)Acquisition of Soc. Wk. competences
(Oxford Brookes)
7
The outcomes of teaching and learning about ‘race’ and racism
Kish Bhatti-SinclairDivision of Social Work StudiesSchool of Social SciencesUniversity of Southampton
8
Aims and Methods
Students’ understanding and experience of ‘race’ and racism; and
How ‘race’ and racism is addressed in the degree curriculum
Questionnaires to BSc social science students in week 2 of Semester 1 (153 students) and in Week 8 (71 students) of Semester 2.
34 respondents completed both sets. Social work students = 41.2%.
9
Main sources of learningabout race and racism (frequency)
TYPES OF LEARNING Seminars 17 Course units 16 Personal study 9 Shadowing soc wkr 4 Peer Group 3 Personal tutors 2
SPECIFIC COURSE UNITS 2 out of total of 8 (3 SW)
Social problems and social policy - 6
Sociology of everyday life - 2
10
Results
A small increase in self-rated knowledge of racism – 10%.
A increase in respondents’ confidence in challenging racism (but not statistically significant.)
Overall results suggest that formal teaching about ‘race’ and racism did influence knowledge although the impact was small.
11
Limitations
Sampling – not random. Results not generalisable – must be
interpreted with caution. Knowledge of racism based on self rating - not
possible to obtain an objective measure of knowledge.
12
Outcomes of “Using Research for Practice” using blended learning.
Bournemouth UniversityAnne Quinney
BA Social Work Year 2 students.
13
Methodology
Pre-Post testsMeasure: Research Self Efficacy (RSE) scale
(Holden et al., 1999). Subscales on research knowledge and skills. PLUS 5 items “using computer and information technology”.
Analysis: Paired t-tests for difference in mean scores.
14
Findings: Research Knowledge and Skills
“How confident are you that you can successfully analyse basic quantitative and qualitative data?”
Scale 0-10 0
1
2
3
4
5
6
Question9
T1
T2
15
Findings: using computer and information technology
“How confident are you that you can successfully access research findings from research bodies, social work organisations, government departments etc (e.g. JRF, SCIE) using the internet?”
0
1
2
3
4
5
6
7
8
Question14
T1
T2
16
Next Steps and Uses
1. Do Research Self Efficacy scores and assignment marks correlate?
2. Compare with another programme
Uses:– Adjust the curriculum for (1) present students who have to do a
dissertation in year 3 and (2) next year’s course. – Student self-assessment: “what I need to learn.”
– The RSE scale has been adopted by ESRC Researcher Development Initiative.
17
Outcomes of teaching about Partnership & Interprofessional Practice
Imogen Taylor, Clare Ockwell, Suzy Braye
University of Sussex
Note: Class-room based module does not include students from other professions.
18
Methodology
Outcomes of learning and teaching on BA & MA students’ attitudes – Stage 1 pre module teaching– Stage 2 post module teaching – [Stage 3 end of course]
Validated scales from University of West of England IPE programme
– Communication and teamwork (CT)– Interprofessional learning (ILL)– Interprofessional interaction (II)– Interprofessional relationships (IR)
19
Mean ratings at start/end of module (BA)
BA - Comparison of scores for each scale at Time 1 and Time 2
0
5
10
15
20
25
30
35
40
CT1 CT2 IL 1 IL 2 II 1 II 2 IR 1 IR 2
Scales and time point
Scor
e
20
Differences in attitudes
by years of pre-course experience
BA ratings for Communication and Teamwork Scale and number of years of experience
0
1
2
3
4
5
6
7
8
9
10
pos neutral neg pos neutral
T1 CTS rating T2 CTS rating
No. o
f res
pond
ents
MA ratings for Comunication and Teamwork Scale and years of experience
0
1
2
3
4
5
6
7
T1 CTS rating T2 CTS rating
No. o
f res
pond
ents
0-1 yr
2-5 yrs
5 yrs ormore
21
Next steps
Complete analysis of BA/MA T1 and T2 Collect and analyse end programme BA/MA
data (T3) Compare with UWE IPE findings Introduce new comparator programme (any
offers?)
22
Outcomes of Teaching and Learning communication skills
Juliet Koprowska
University of York
23
Research question & methodology
Are communication skills improved through learning within the university?
Methodology: multiple-measure case study design.
Sample: year 1 BSc Social Work
24
Research methods
Self-efficacy scale Recorded interview with service user-actor Self-evaluation of interview Service user-actor evaluation Assessment of video-recording Feedback from service users on placement
T1 = prior to teaching, T2 = 3 months later, after teaching, T3 = 9 months later, after first placement
25
Student A
0
2
4
6
8
10
1 2 3
Time
Self-efficacy Self-evaluationService user-actor evaluation Assessment
26
Student B
0
2
4
6
8
10
12
1 2 3
Time
Self-efficacy Self-evaluation
Service ucer-actor evaluation Assessment
27
Implications
Students may lose confidence as they acquire knowledge and understanding
Potential for formative assessment Effect of placement needs exploration Qualitative discussion with students and actors would
deepen understanding Should tools become regular measures to increase
participation in research, with retrospective permission?
28
Communication Skills with Children & Young People
Michelle LeFevreUniversity of Sussex
How does a Programme contribute to students’ development of confidence and competence in communicating with children and young people?
29
What and How are these learned?
‘Skilled/effective communication’ is not just techniques and micro skills. Also underpinning knowledge, values and ethical commitments, personal qualities and emotional capabilities.
Learned throughout the whole programme not just in focused skills teaching – and also through other personal and professional experiences.
30
MethodologyProspective Evaluation – 4 stages
1. Questionnaire – Students’ personal characteristics: Have particular kinds of student
learned most/least?– Measuring self-confidence in communication with children at
different stages. – Which aspects of the programme facilitated your confidence and
skills?
2. Case Vignette tool– Students demonstrate knowledge of planning, implementing,
reviewing and reflecting on communication with children.– More objective measure (though analysis is subjective?)
31
Key Findings
1. Students’ confidence has increased.2. Direct practice in placement the most important,
followed by Child Development 3. All aspects of focused skills teaching were
perceived as helpful to learning4. Knowledge of aspects of communication skills
increased (often significantly) - but not in every domain.
5. Still to analyse relation of characteristics to learning and mapping individual students through programme.
32
Learning about working in partnership with service users
Roxana Anghel & Joanna Fox
Anglia Ruskin University
Exploring the use of Concept Mapping
33
“Schematic representation of an individual’s understanding of a knowledge domain in the form of concepts meaningfully linked in propositions.”
Design – CM and questionnaire at T1 (induction week BA); T2 (end year
1); T3 (end year 2)– CM task, unstructured - ‘Working in Partnership with Service
Users’– Questionnaire – most impacting learning opportunities.
Analysis of CM – Generate scores based on validated concept-links– Raters included service user and researcher.
Method – Concept Mapping (CM)
34
Concept Map at start of programme
35
Concept map at end of Year 1
36
Concept Map at end Year 2
37
Findings – “Working in Partnership”
0
20
40
60
80
100
120
140
160
T1 T2 T3
1
2
34
5
6
Map
Sco
res
38
Sources of Learning
At T1: Ethics and Values; observational practice; Poverty, Social Exclusion and Social Work, and academic reading.
At T2: Social Work with Children and Families, Social Work with Adults, practice placement; Principles and Skills of Social Work; academic reading; and discussing with colleagues
39
CM used on two other modules
Relationship of Map Scores to Assignment Marks
30
35
40
45
50
55
60
65
70
75
30 40 50 60 70 80 90 100
Marks
Map
Sco
res
Relationship of Map Scores to Assignment Marks
40
45
50
55
60
65
70
75
40 50 60 70 80 90 100 110
Marks
Map
Sco
res
• Module B – Map scores and marks did not correlate
020406080
100120140160180
1 2 3 4 5 6 7 8 9 10 11
T1
T2
• Module A – Map scores correlated with marks
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13
T1
T2
• Module B (Year 1 MA, 11 pairs) – 72% T2 maps increased content and quality
• Module A (Year 3 BA, 13 pairs) – 46% T2 maps increased content and quality
Students
Ma
p S
co
res
Students
Ma
p S
co
res
40
Discussion
– Small numbers, so the value of CM as summative method in assessing outcomes needs to be explored further
– Success depends on the clarity of instructions and purpose, familiarity and time available.
– Uses: self-assessment, visual learning aid, formative assessment (identify gaps), as well as assessing outcomes.
– Being firmly based on “meaningful” learning CM can be applied to
many area of social work.
41
Measuring the Acquisition of Competency
Sharon Vitali
Oxford Brookes University
42
Methodology
Cohort Sample BSc Social Work– (n=38 at beginning n=34 at present = 89%)
Longitudinal - Baseline to Graduation Repeated Measures Fixed Outcomes (National
Occupational Standards Units) RICET Multiple Methods
43
Year One - Semester One
Baseline
December
RICET
RICETPERSONAL LEARNING PLAN
44
Consider your current state of competence (knowledge/skills) and indicate a score (from the guide below) for each of the learning outcomes listed.
0 = Can not produce any evidence of competence.
1 = Understands the learning outcome, but can produce only limited or no evidence of appropriate attempts to put it into practice. Much more knowledge/practice needed.
2 = Understands, and can offer evidence of tentative attempts to integrate into current knowledge/skill base.
3 = Demonstrates competence with some regularity.
4 = Advanced understanding and demonstrating adequate level of integration of knowledge, skills, and appropriate application.
5 = Clearly understands and demonstrates consistent and appropriate application of knowledge and skills in practice.
45
Acquisition of NOS Competencies Year One
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Unit 1
Unit 2
Unit 3
Unit 4
Unit 5
Unit 6
Unit 7
Unit 8
Unit 9
Unit 10
Unit 11
Unit 12
Unit 13
Unit 14
Unit 15
Unit 16
Unit 17
Unit 18
Unit 19
Unit 20
Unit 21
Units of Competency
Sco
re
Student Baseline Student End Sem1Tutor End Sem1 Student End Sem2
Academic Average = 61.4%
Video Scores SU=1.8 ER=2.2
46
Year Three - Semester Two
May (70 DAY PLO)
RICETPRACTICE & PORTFOLIO
TUTOR EVALUATION
PRACTICE & PORTFOLIOP/T EVALUATION
CRITICAL CAREERREVIEWTUTOR
EVALUATION RICET
47
Acquisition of NOS Competency Unit 4: Respond in a Crisis Situation
2
0333
43
43
4
1
5
1
Level of Competency (0-5)
PA End Place2
ST End Place2
PA Mid Place2
ST Mid Place 2
PA End Place1
ST End Place1
PA Mid Place1
ST Mid Place1
ST End Yr1 S2
Tutor End Yr1S1
ST End Yr1 S1
ST Baseline
48
Types of DataY1-S1
RICET
numeric
LEARNING PLAN
p/f
RICET
numeric
Grades
numeric
Y1-S2
INTERVIEW
numeric
RICET
numeric
PORTFOLIO
p/f
INTERVIEW
numericGrades
numeric
Y2-S1
RICET
numeric
RICET & PORTFOLIO
Numeric & p/f
PORTFOLIO
p/fGrades
numeric
Y2-S2
RICET
numeric
RICET & PORTFOLIO
Numeric & p/f
PORTFOLIO
numericGrades
numeric
Y3-S1
RICET
numeric
RICET & PORTFOLIO
Numeric & p/f
PORTFOLIO
p/f
CASE VIVA
numericGrades
numeric
Y3-S2
RICET
numeric
RICET & PORTFOLIO
Numeric & p/f
PORTFOLIO
numeric
CRITICAL CAREER REVIEW
numeric
Grades
numericNOW
end
49
Overview of the findings from projects increases in:
1. Attitudes to race and racism - but not to self-confidence in challenging racism.
2. Self-efficacy following learning (e.g. research skills) Do these correlate with marks?
3. Conceptual understanding (C Maps, communication vignettes).
4. Ratings of Competence (RICET)
50
But:
1. No measured increase in Communication skills
2. No overall increases in attitudes to partnership working
3. Measures do not necessarily correlate with marks.
And we need to strengthen research designs, especially comparative studies.
51
Some Questions for Discussion
1. What methods of learning and teaching should we compare? Are you interested?
2. What is their potential for use in formative evaluation and self-evaluation by students?
3. Should we use these measures routinely (like Oxford Brooks) to assess student outcomes?
4. Are you interested in replicating and developing these methods?
52
Key issues
1. Formulating an appropriate, specific research question2. Selecting/adapting appropriate methods/measures for
measuring change3. Negotiating student engagement, balancing ethical
concern to allow opt-out with maximising engagement to promote learning
4. Establishing involvement of service users/carers5. Getting support from colleagues6. Freeing space/time/resources for the research
53
Key issues (cont)
7. Balancing roles of teacher and researcher (sometimes complementary, sometimes conflicting)
8. Accessing expertise/learning to analyse and interpret data
9. Finding comparison sites
10. Feedback to programme/colleagues about knowledge/skills gained and implications for rest of programme
54
Thank you
To our sponsors: SCIE, SWAP and IRISS
Contacts: [email protected]@bristol.ac.uk
Resource:Evaluating Outcomes in Social Work Educationhttp://www.iriss.ac.uk/node/88