James Young, MSW, MPA Mary Secret, Ph.D.

Angie Mann-Williams, LCSW

Virginia Commonwealth University School of Social Work

Brief review of the evaluation literature Methodology/ Model Plan

Evaluation Design Student Learning Outcomes

Questionnaires (SLOQ) Resources

Results Implications

http://hbseevaluationapm2010.blogspot.com/

Over 4.6 million students took at least 1 online course in 2008. More than 1 in 4 higher ed. students now

take at least one course online. (Allen & Seaman, 2010)

From Meta-Analysis of Online Learning: On average, students in online learning

conditions performed better than those receiving face-to-face instruction. (Bernard et al., 2004; U. S. Department of Education, Office of Planning, Evaluation, and Policy Development, 2009).

“Course and program outcomes achieved in social work distance education courses and programs are comparable to those achieved in traditional face-to-face programs”. (Macy, Rooney, Hollister, Freddolino, 2001).

No differences on educational outcomes or satisfaction. (Wilke & Vinton, 2006).

No significant differences between Online or Face-to-Face groups in Practice, Policy, and HBSE courses. (Seabury, 2005; Siebert, Siebert, & Spaulding-Givens, 2006; Woehle, Quinn, 2009; York, 2008).

Student socialization to the profession? Online course design and evaluation?

Content Pedagogy

Follow up with Alumni? Evaluating student’s technical expertise?

Readiness for online learning and Success Resources

Technology Time Challenge with the rapidly changing nature of

technology & trying to evaluate online social work courses.

Explore differences and similarities between online/hybrid courses and face-to-face courses in the following areas:

Achievement of student learning outcomes as informed by selected EPAS Competencies

Inputs / Resources Instructor time Type of Technology resources Type of Class room activities and assignments

Student Perspectives

IRB approved Post-test only, quasi-experimental

research design Instructors in three sections of HBSE

course agreed to participate in study One online course taught by experienced

faculty (19 students) served as experimental group

One face-to-face course taught by doctoral student who served as teaching assistant for the online course (20 students) served as comparison group

Another face-to-face course taught by an experienced adjunct (19 students) served as second comparison group

All three courses followed the same basic syllabus, used the same textbook (Hutchison), used the same standardized power

point slides that accompanied the textbook

had the same assignments

Online, 3 in-class meetings (begin, middle, and end of semester)

primarily weekly graded collaborative learning groups on Blackboard

Face to Face, weekly in-class meetings

primarily lecture, class discussion, and small group work in classroom

The online course and one of the face-to-face courses used the same collaborative learning group model, including the use of the same weekly group discussion questions

The online course instructor served as mentor for the face-to-face courses taught by an adjunct faculty and a doctoral student

The doctoral student served as teaching assistant for the online course

Goal Instrument Administration Learning Outcomes

Learning Outcomes Instrument (Researcher Developed)

Last Class Period

Input/Resources

Record of Faculty Resources Form (Researcher Developed)

Completed by faculty on a weekly basis and turned into GRA

Student Perspectives

University End of Semester Course Evaluations

Online through the University system

Student Profile

0

10

20

30

40

50

60

70

80

90

100

Female Age under 30 Full-time Employed HS Prior Online Online post sec Psych UG SW UG

* Stat Sig Diff

Pe

rce

nt

*

*from data collected on the Learning Outcomes Instrument

More part-time students self-selected into online class

online face to face

Count 10 32

Expected Count 13.8 28.2

Full-time

% within Status Full or Part-

time

23.8% 76.2%

Count 9 7

Expected Count 5.2 10.8

Part-time

% within Status Full or Part-

time

56.3% 43.8%

Count 19 39

Expected Count 19.0 39.0

Total

% within Status Full or Part-

time

32.8% 67.2%

Four vignettes (case studies), each focusing on a different life stage Case Study 1: Pregnancy, Early

Childhood, Family Life Stage Case Study 2: Late Adulthood, Infancy

and Toddlerhood Case Study 3: Adolescence Case Study 4: Middle Childhood AND

Young Adulthood

10 – 12 questions per case study, developed to reflect the following EPAS teaching competency and practice behaviors:

Educational Policy 2.1.7—Apply knowledge of human behavior and the social environment. P.B 2.1.7 a Utilize Conceptual Frameworks To Guide

The Processes Of Assessment, Intervention, And Evaluation.

P.B 2.1.7b Critique And Apply Knowledge To Understand Person And Environment

Educational Policy 2.1.4 – Engage Diversity And Difference In Practice

Each question identified for focus on different area relative to EPAS:

C -- utilization of conceptual frameworkE -- critique and application of evidence-

based knowledgeD-- engagement to diversity and difference

(c) Rosa, Emma and Marie’s stories unfold as they progress through life. The term used to understand this relationship between time and human behavior and examines how chronological age, relationships, common life transitions, and social change shape people’s lives from birth to death.

Systems theory Life course perspective Cohort Study Developmental psychology none of the above is an appropriate term

(D) A social worker should be careful about applying Piaget’s theory of development when working with Maria because:

• Maria is at the autonomy vs shame level of psychosocial development

• Maria’s mother, Aida, was a teen parent who smoked

• The theory does not account for variation in culture, family, and significant relationships on cognitive development

• Maria has no contact with her father and he might want to remove her from the Surez family after her mother Aida was lost in the 9/11 tragedy

• All of the above are equally important factors to consider when applying Piaget’s theory

(e) One of the best evidence-based strategies to reduce the risk factors and increase the protective factors for Tiffany’s 3 and 5 year old sons at this point in their lives is:

– Attendance at bi-weekly church services– Enrollment in a high quality early

childhood education program– Parenting classes for Tiffany and Ruby– Have Tiffany marry the father of her

children – All of the above would be equally

effective strategies for the two boys

The instrument was administered during the last class session of the semester

The instructors participating were provided a script to read to students participating in the evaluation

The instructors collected the instruments and placed them in sealed envelopes.

The envelopes were given to a graduate research assistant ensuring that the students information remained confidential

Questions recoded so that correct answer = 1, incorrect answer = 0(multiple response questions earned multiple points)

Correct answers summed by EPAS focus Higher scores = greater knowledge Possible points by EPAS focus area

▪ Conceptual framework = 35▪ Evidence-based knowledge = 26 ▪ Diversity = 5

TOTAL = 66

Mann Whitney non parametric for independent groups to determine differences between the two face to face classes. No differences; two face to face classes

combined for further analysis Mann Whitney non parametric to

determine differences between the online class and the face to face class(es)

Students in online class scored higher in conceptual knowledge

(p = .003) and in overall knowledge (p = .006)

No statistically significant difference in evidence-based knowledge (p =.095 ) or diversity knowledge ( = .202)

Learning Type of Class N Mean

Online 19 25.15 Conceptual

Face to face 39 22.05

Online 19 2.68 Diversity

Face to face 39 2.30

Online 19 19.78 Evidence

Face to face 39 18.56

Online 19 47.63 Total Learning

Face to face 39 42.92

Differences in learning outcomes related to:

Part-time/full time status NO Undergraduate Major NO Prior Online NO Student Age NO

Questions about: Professor behaviors and attitudes Student general learning experience in

the course Student knowledge and skill acquisition

related to course objectives

Differences in learning outcomes related to:

Part-time/full time status NO Undergraduate Major NO Prior Online NO Student Age NO

Questions about:Professor behaviors and attitudesStudent general learning experience

in the courseStudent knowledge and skill

acquisition related to course objectives

PROFESSOR BEHAVIORS AND ATTITUDES MEAN

Total 3.73

Face to Face 3.744

Online 3.722

Items included in Professor Behaviors (4 = Strongly agree; 3 =Agree; 2= Disagree; 1=Strongly disagree) The instructor ….. • created an environment of trust and fairness.• encouraged discussion, participation, and questions.• treated students with courtesy and respect• demonstrated sensitivity to my needs as a student• clearly defined the requirements for my work in the course• presented course material in an organized and informative manner• provided helpful feedback on papers, tests, presentations, or other assignments• addressed social and economic justice as related to the course• addressed diversity and oppressed populations as related to the course

STUDENT LEARNING EXPERIENCES IN THE COURSE

MEAN

Total 3.65

Face to Face 3.62

Online 3.73

As a student in this course I ….. (4 = Strongly agree; 3 =Agree; 2= Disagree; 1=Strongly disagree) • engaged in critical thinking and analysis • was motivated to learn about the course subject• developed greater self-awareness• developed significant skills as a result of the course• increased my knowledge of the subject matter of the course• increased my knowledge of social and economic justice as related to the course• increased my knowledge of diversity and oppressed populations as related to the course• Prior to taking this course I was interested in the course content

STUDENT KNOWLEDGE AND SKILL ACQUISITION RELATED TO COURSE OBJECTIVES

MEAN

Total 4.36

Face to Face 4.2920

Online 4.5147

To what degree did this course contribute to your ability to…(5=A great deal; 4=A lot; 3=Somewhat; 2=A little; 1=Not at all)•Practice personal reflection and self-correction to assure continual professional development•Recognize and manage personal values in a way that allows professional values to guide practice•Make ethical decisions by applying standards of the National Association of Social Workers Code of Ethics and, as applicable, of the International Federation of Social Workers/International Association of Schools of Social Work Ethics in Social Work, Statement of Principles•Tolerate ambiguity in resolving ethical conflicts•Apply strategies of ethical reasoning to arrive at principled decisions•Distinguish, appraise, and integrate multiple sources of knowledge, including research-based knowledge, and practice wisdom•Recognize the extent to which a culture’s structures and values may oppress, marginalize, alienate, or create or enhance privilege and power•Understand the forms and mechanisms of oppression and discrimination•Use research evidence to inform practice•Utilize conceptual frameworks to guide the processes of assessment, intervention, and evaluation•Critique and apply knowledge to understand person and environment•Substantively and affectively prepare for action with individuals, families, groups, organizations, and communities•Use empathy and other interpersonal skills•Gain sufficient self-awareness to eliminate the influence of personal biases and values in working with diverse groups•Recognize and communicate their understanding of the importance of difference in shaping life experiences

Student timeStudent Time

0

10

20

30

40

50

60

70

80

90

100

Workloadappropriate for

course

1-2 hrs prep 3-5 hrs prep 5hrs+ prep 2-4 hrs class w ork more than 4 hrs

Pere

nt Online

Face to Face

More part-time students self-select into Online Course

Greater Conceptual and Overall Learning Outcomes in Online Course

More expenditure of faculty time in Online Course Greatest difference in grading Blackboard CLG’s

Similar use of technology in both courses Student perception of learning slightly higher in

online course Student perception of instructor slightly higher in

face to face course Student perception of workload demands greater

in online course

Evaluations of overall social work programs as opposed to courses

What happens to fieldSocial work educators inclination,

knowledge, skills regarding online education

Allen, I.E., & Seaman, J. (2010). Learning on demand: Online education in the United States, 2009. Babson Survey Research Group.

Bernard, R.M., et al. (2004). How Does Distance Education Compare With Classroom Instruction? A Meta-Analysis of the Empirical Literature? Review of Educational Research, 74(3), 379-439.

Coe, J., & Elliott, D. (1999). An evaluation of teaching direct practice courses in a distance education program for rural settings. Journal of Social Work Education, 35(3), 353-365.

Coe Regan, J.R., Freddolino, P., Schoech, D., & Watkins, J. (2007). Utilizing evidence-based research in developing best practice guidelines for distance education in social work. Paper presentation at the Council on Social Work Education Annual Program Meeting (CSWE-APM) in San Francisco, CA on October 30, 2007.

Coe Regan, J.R., & Youn, E.J. (2008). Past, present, and future trends in teaching clinical skills through web-based learning environments. Journal of Social Work Education, 44(2), 95-115.

Faux, T., & Black-Hughes, C. (2000). A comparison of using the internet versus lectures to teach social work history. Research on Social Work Practice, 10(4), 454-466.

Hutchison, E. (2008). Dimensions of human behavior: The changing life course (3rd ed.). Thousand Oaks, CA: Pine Forge Press.

Macy, J.A., Rooney, R.H., Hollister, C.D., & Freddolino, P.P. (2001). Evaluation of distance education programs in social work. Journal of Technology in Human Services, 18(3/4), 63-84.

Newhouse, J.J. (2001). Successful distance learning graduate education in human services. Journal of Technology in Human Services, 18(1/2), 33-50.

Packard, T. & Austin, M. (2009). Using a Comprehensive Case-Based Examination To Evaluate and Integrate Student Learning in Social Work Administration', Journal of Teaching in Social Work, 29: 2, 204 - 215.

Seabury, B.A. (2005). An evaluation of on-line, interactive tutorials designed to teach practice concepts. Journal of Teaching in Social Work, 25(1), 103-115.

Seaman, J. (2009). Online Learning as a Strategic Asset. Volume II: The Paradox of Faculty Voices: Views and Experiences with Online Learning Results of a National Faculty Survey, Part of the Online Education Benchmarking Study Co ducted by the A♦P♦L♦U-Sloan National Commission on Online Learning. Association of Public and Land-grant Universities: Washington, D.C.

Siebert, D.C., Siebert, C.F., & Spaulding-Givens, J. (2006). Teaching clinical social work skills primarily online: An evaluation. Journal of Social Work Education, 42(2), 325-336.

Siegel, E., Jennings, J. G., Conklin, J., & Flynn, S.A.N. (1998). Distance learning in social work education: results and implications of a national survey. Journal of Social Work Education, 34(1), 71-80.

Thyer, B., Artlet, T., Markward, M., & Dozier, C. (1998). Evaluating distance learning in social work education: A replication study. Journal of Social Work Education, 34(2), 291-296.

U. S. Department of Education, Office of Planning, Evaluation, and Policy Development. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies [Electronic Version]. Retrieved from, http://www.ed.gov

Vernon, R., Vakalahi, H., Pierce, D., Pittman-Munke, P., & Adkins, L. F. (2009). Distance education programs in social work: current and emerging trends. Journal of Social Work Education, 45(2), 263-276.

Wilke, D., & Vinton, L. (2006). Evaluation of the first web-based advanced standing msw program. Journal of Social Work Education, 42(3), 607-620.

Woehle, R., & Quinn, A. (2009). An experiment comparing HBSE graduate social work classes: Face-to-face and at a distance. Journal of Teaching in Social Work, 29(4), 418-430.

York, R. (2008). Comparing three modes of instruction in a graduate social work program. Journal of Social Work Education, 44(2), 157-172.