Cluster 5 Spring 2005 Cluster 5 Spring 2005 Assessment ResultsAssessment Results

Sociocultural DomainSociocultural Domain

Cluster 5 Course RequirementCluster 5 Course Requirement

Complete 1 course in each domainComplete 1 course in each domain

Wellness DomainGHTH 100. Personal WellnessGKIN 100. Lifetime Fitness and WellnessGEIC 101. Individuals in the Human Community: The Individual

Perspective

Sociocultural DomainGPSYC 101. General PsychologyGPSYC 160. Life Span Human DevelopmentGSOCI 240. Individual in SocietyGEIC 102. Individuals in the Human Community: The Community

Perspective

Assessment Day Data Collection Assessment Day Data Collection Scheme: Scheme: Pre/PostPre/Post

Fall 2002Spring 2003

Fall 2003Spring 2004

Fall 2004Spring 2005

Fall 2005Spring 2006

COHORT 1

COHORT 2

COHORT 3

Students in each cohort Students in each cohort are tested are tested twicetwice on the on the same instrument – once as same instrument – once as incoming freshmen incoming freshmen (0 (0 credit hours)credit hours) and again in and again in after they’ve completed after they’ve completed 45-70 credit hours45-70 credit hours. .

Cohort 2: First time for Cluster 5 to have pre/post data since new objectives/instruments.

Sociocultural Domain InstrumentsSociocultural Domain Instruments Sociocultural Domain Assessment (SDA)Sociocultural Domain Assessment (SDA)

– 3 different versions of test3 different versions of test Scenario-based multiple-choice exam Scenario-based multiple-choice exam

created by facultycreated by faculty

Administration Sample # Items SDA Version Fall 2003 Incoming Freshmen 30 SDA1

Spring 2004 2nd semester Sophomores 42 SDA2 Fall 2004 Incoming Freshmen 40 SDA3

Spring 2005 2nd semester Sophomores 30/ 42 SDA1/ SDA2

Administration error led to SDA1 & SDA2 being administered in Spring 2005 –

because of this only 27 common items on both forms used to create total score

ResultsResults

First:First: examine results for 398 2 examine results for 398 2ndnd semester semester sophomores in Spring 2005sophomores in Spring 2005– Overall resultsOverall results– By JMU vs. Transfer/APBy JMU vs. Transfer/AP– By domain “completers” vs. “non-completers”By domain “completers” vs. “non-completers”– By which course completedBy which course completed– Rel. between course grade and SDA scoreRel. between course grade and SDA score

Second:Second: examine pre/post results for the 264 examine pre/post results for the 264 students who took the SDA in both Fall 2003 and students who took the SDA in both Fall 2003 and Spring 2005Spring 2005– Overall pre/post differenceOverall pre/post difference– pre/post difference by domain “completers” vs. “non-pre/post difference by domain “completers” vs. “non-

completers”completers”– pre/post difference by which course completedpre/post difference by which course completed– Pre/post difference by item for “completers”Pre/post difference by item for “completers”

Overall SDA Results: Overall SDA Results: 22ndnd Semester Sophomores Semester Sophomores (N = 398)(N = 398)

ReliabilityRaw % a

SDA Total Score 0 - 27 5 - 27 18.52 69% 3.82 0.67

Std. Dev.Score Possible Range

Earned Score Range

Average Score

On average, On average, students are students are answering answering almost 70% almost 70% of the items of the items correctly. correctly. Most scores Most scores are between are between 17 & 23.17 & 23.

JMU (n = 369)JMU (n = 369) vs. vs.Transfer/AP Students (n = 29)Transfer/AP Students (n = 29)

Are students who obtained transfer or AP Are students who obtained transfer or AP credit for their courses scoring differently credit for their courses scoring differently than students who completed their courses than students who completed their courses at JMU?at JMU?– No.No.

JMU students: M = 18.63, SD = 3.83JMU students: M = 18.63, SD = 3.83 AP/transfer students: M = 18.55, SD = 3.68AP/transfer students: M = 18.55, SD = 3.68 Difference was Difference was notnot statistically, statistically, tt(335)=.11, (335)=.11, pp

= .92, nor practically significant (= .92, nor practically significant (dd = .02). = .02).

Domain “Completers” vs. “Non-Completers”Domain “Completers” vs. “Non-Completers”SDA Total Score Means by the Number

of Sociocultural Courses Completed (Including AP and transfers)

18.8718.5717.92

0

3

6

9

12

15

18

21

24

27

0 1 2 or 3

Number of Sociocultural Courses Completed

SD

A T

ota

l S

core

N = 61 N = 282 N = 55

"Non-Completers" "Completers"

More courses completed, the higher the mean. More courses completed, the higher the mean. Differences among the means, however, are of Differences among the means, however, are of

small practical significance (small practical significance (dd’s between .08 - .25)’s between .08 - .25)

SDA Means by Course CompletedSDA Means by Course Completed“Completers” Only“Completers” Only

SDA Means by Type of Sociocultural Course Completed(Including AP and Transfer Students)

18.9619.0118.08

0

3

6

9

12

15

18

21

24

27

GPSYC101 GPSYC160 GSOCI240

Sociocultural Course

SD

A T

ota

l Sco

re

N = 221 N = 22N = 144

There were There were minor minor differences differences among the among the SDA means SDA means ((dd’s ~.01 ’s ~.01 - .24)- .24)

GEIC101 not reported due to small N

Relationship Between GSYC101 Course Grade and SDA Total Score

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 3 6 9 12 15 18 21 24 27

SDA Total Score

GP

SY

C10

1 C

ou

rse

Gra

de

Relationship Between GSYC160 Course Grade and SDA Total Score

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 3 6 9 12 15 18 21 24 27

SDA Total Score

GP

SY

C16

0 C

ou

rse

Gra

de

Relationship Between GSOCI240 Course Grade and SDA Total Score

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 3 6 9 12 15 18 21 24 27

SDA Total Score

GS

OC

I240

Co

urs

e G

rad

e

Relationship Between GEIC102 Course Grade and SDA Total Score

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 3 6 9 12 15 18 21 24 27

SDA Total Score

GE

IC10

2 C

ou

rse

Gra

de

Relationship between SDA and Relationship between SDA and Course GradeCourse Grade

r = .31 r = .23

r = .69 r = .36w/o outlier

Positive and Positive and moderate moderate

relationshipsrelationships…as …as

anticipated.anticipated.

Pre/Post Results (N = 264)Pre/Post Results (N = 264)

Spring 2005: Spring 2005: M = 18.78M = 18.78, , SD = 3.66SD = 3.66

Fall 2003: Fall 2003: M = 16.62M = 16.62, SD , SD = 3.38= 3.38

The difference between The difference between these means is of these means is of medium to large medium to large practical significance practical significance ((dd = .64) = .64)

Average Average increaseincrease of of

8% in 8% in sociocultural sociocultural

domain domain knowledgeknowledge

SDA Mean Total Scores from Fall 2003 and Spring 2005 by Number of Sociocultural Courses Completed

16.7816.7115.88

19.81

18.7517.94

0

3

6

9

12

15

18

21

24

27

0 1 2 - 3

Number of Sociocultural Courses Completed

SD

A T

ota

l Sco

re

Fall 2003 Spring 2005

Pre/Post for “Non-Completers” (n = 33) vs. Pre/Post for “Non-Completers” (n = 33) vs. “Completers” (n = 199) “Completers” (n = 199)

Non-completers Completers““Completers” Completers” start off with start off with higher means higher means than “Non-than “Non-completers” – completers” – but they both but they both make similar make similar gains over time gains over time (~2 pt. (~2 pt. increase, increase, dd ~.55) ~.55)

Larger gain (~3 pts, Larger gain (~3 pts, dd = 1.02) for the 32 students who = 1.02) for the 32 students who completed 2 – 3 sociocultural domain courses. completed 2 – 3 sociocultural domain courses.

Pre/Post by Sociocultural Course Completed Pre/Post by Sociocultural Course Completed

Spring 2005 average for GSOCI240 students may seem Spring 2005 average for GSOCI240 students may seem superior, keep in mind that these students were more superior, keep in mind that these students were more knowledgeable coming into the course and that the averages knowledgeable coming into the course and that the averages for GSOCI240 are based on only 10 students.for GSOCI240 are based on only 10 students.

SDA Mean Total Scores from Fall 2003 and Spring 2005 by Type of Sociocultural Courses Completed

17.9

16.37

16.86

20.00

18.8019.16

0

3

6

9

12

15

18

21

24

27

GPSYC101 GPSYC160 GSOCI240

Type of Sociocultural Courses Completed

SD

A T

ota

l Sco

re

Fall 2003 Spring 2005 The The magnitude magnitude of the of the change in change in knowledge knowledge is very is very similar for similar for the three the three different different courses courses ((dd ~ .75). ~ .75).

N = 153 N = 98 N = 10

Pre/Post Difference in Items for Pre/Post Difference in Items for “Completers”“Completers”

Differences in Proportion of Sample Obtaining Correct Answer for SDA Fall 2003 and Spring 2005 Administrations Sociocultural Domain "Completers" of One Course

s21

s18s11

s2

s9

s28

s3s25

s12s10

s4

s19

s5

s16

s13

s7s20s8

s1

s27

s15

s30

s17s26s29

s24

s23

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Pro

po

rtio

n C

orr

ect

(It

em

Dif

fic

ult

y)

Fall 2003

Spring 2005

7 7 items in items in Fall 2003 Fall 2003 where less where less than half of than half of the the students students are are obtained obtained the correct the correct answer; answer; there are there are only only 22 such such items in items in Spring 2005Spring 2005

Most of the gains are made for items of moderate to large difficulty.

Items with Large Pre/Post GainsItems with Large Pre/Post Gains

Item # Fall 2003 Spring 2005 Difference Item Content Goal/Objectives19 0.53 0.72 0.19 Generalization of results to other samples 3bs2 0.32 0.50 0.18 Study is so weak that it won't obtain IRB approval 2bs18 0.22 0.37 0.15 Peer review 3bs9 0.40 0.55 0.15 Which hypothesis not addressed by her research design 4as4 0.52 0.67 0.15 Reputable sources of information 3as13 0.66 0.80 0.14 Largest problem with how Marty asked about parental alcoholism 2as11 0.27 0.41 0.14 Least appropriate use of Marty's data 2as28 0.46 0.59 0.14 Nature-nuture debate 5cs23 0.64 0.76 0.13 Ethical treatment of subjects 4bs12 0.50 0.59 0.09 Multiple causes for behavior 1bs16 0.57 0.65 0.08 Reputable internet site 3as3 0.46 0.54 0.08 Potential bias in sources of information 3c

Proportion Correct

7% or more students answered these items correctly in 7% or more students answered these items correctly in Spring 2005 than in Fall 2003Spring 2005 than in Fall 2003

Majority of these items Majority of these items (67%) were written to (67%) were written to assess goals 2 and 3. assess goals 2 and 3.

Goal 2:Identify implications of taking action regarding social/behavioral issue

Goal 3:Use evidence to develop and evaluate positions regarding social/behavioral issue

Items with Small (or No) Items with Small (or No) Pre/Post GainsPre/Post Gains

Item # Fall 2003 Spring 2005 Difference Item Content Goal/Objectives15 0.77 0.74 -0.03 Improving the quality of her data 2cs8 0.76 0.78 0.02 Assumptions about Marty's perspective on alcohol use 1ds20 0.74 0.76 0.02 Reasonable interpretation of her hypothetical results 4cs7 0.73 0.75 0.02 Nature-nuture debate 5cs5 0.55 0.55 -0.01 Cause-effect relationships cannot be examined using survey data 1bs25 0.49 0.50 0.02 Changes made to her sample to make it more generalizable 4c

Proportion Correct

Items with no change, or change in the wrong direction Items with no change, or change in the wrong direction are shown below for items that are of medium to are shown below for items that are of medium to

extreme difficulty.extreme difficulty.

67% of these items are 67% of these items are assessing either goals 1 and 4, assessing either goals 1 and 4, which indicates that: 1) either which indicates that: 1) either more focus should be placed more focus should be placed these goals in sociocultural these goals in sociocultural domain courses or 2) that these domain courses or 2) that these items are of poor quality and items are of poor quality and either need to be revised or either need to be revised or replacedreplaced

Goal 1: Make plausible interpretations about behavior in social contexts

Goal 4: Discriminate between ethical and nonethical practices in the social/behavioral sciences

ConclusionsConclusions 22ndnd Semester Sophomore Data Semester Sophomore Data

22ndnd semester sophomores scoring on average semester sophomores scoring on average ~70%~70%

AP/Transfer students are scoring AP/Transfer students are scoring no differentlyno differently than JMU students than JMU students

““Non-completers” of the sociocultural domain Non-completers” of the sociocultural domain requirement are scoring requirement are scoring no differentlyno differently than than “Completers”“Completers”

Same average scoreSame average score regardless of what course regardless of what course student took to complete sociocultural domain student took to complete sociocultural domain requirementrequirement

There is a relationship between course grade and There is a relationship between course grade and score, indicating that score, indicating that test content and course test content and course content are relatedcontent are related

ConclusionsConclusions Pre/Post Data(1)Pre/Post Data(1)

Average Average increaseincrease of of 8%8% in sociocultural domain in sociocultural domain knowledge between 0 credit hours and 45-70 knowledge between 0 credit hours and 45-70 credit hours (large effect)credit hours (large effect)– 62% 62% 70% 70%

““Completers” and “Non-completers” Completers” and “Non-completers” both make both make similar gains over timesimilar gains over time (~2 pt. increase, (~2 pt. increase, dd ~.55) ~.55)– Might be evidence that score gains are a result of Might be evidence that score gains are a result of

maturation, not a result of taking sociocultural coursematuration, not a result of taking sociocultural course– Note that “completers” come in with more knowledge.Note that “completers” come in with more knowledge.– Larger gains (~3 pts, Larger gains (~3 pts, dd = 1.02)are seen for students = 1.02)are seen for students

completing 2 – 3 courses. completing 2 – 3 courses. – May need to collect larger sample of “non-completers” May need to collect larger sample of “non-completers”

in future assessmentsin future assessments

ConclusionsConclusions Pre/Post Data(2)Pre/Post Data(2)

Same gainsSame gains are being made on the test for are being made on the test for students who fulfilled their requirement by taking students who fulfilled their requirement by taking different sociocultural coursesdifferent sociocultural courses– Students who take SOCI240 may be more knowledge coming Students who take SOCI240 may be more knowledge coming

into the course than students who choose to take other coursesinto the course than students who choose to take other courses

77 items where <50% of students obtaining correct items where <50% of students obtaining correct response in Fall 2003 compared to response in Fall 2003 compared to 22 in Spring 2005 in Spring 2005

Not seeing gains on easy items Not seeing gains on easy items Most gainsMost gains are for items assessing are for items assessing Goals 2 & 3Goals 2 & 3 No gainsNo gains (or gains in the wrong direction) for items (or gains in the wrong direction) for items

assessing assessing Goals 1 & 4Goals 1 & 4

Next StepsNext Steps

Take a closer look at change in % of Take a closer look at change in % of students correctly answering items students correctly answering items over timeover time– Use information to make inferences Use information to make inferences

about program OR possibly to revise about program OR possibly to revise itemsitems

Set standard score on test in order to Set standard score on test in order to obtain richer informationobtain richer information

Setting StandardsSetting Standards May want to use standard setting procedures to establish “cut-offs” for proficiency on May want to use standard setting procedures to establish “cut-offs” for proficiency on

the testthe test Could be that students are gaining knowledge/skills over time, but are they gaining Could be that students are gaining knowledge/skills over time, but are they gaining

enough?enough?

30

55

0

10

20

30

40

50

60

70

80

90

100

Incoming Freshmen Core Curriculum "Completers"

Te

st

Sc

ore

STANDARD SET BY FACULTY