developing general education course assessment measures
DESCRIPTION
Developing General Education Course Assessment Measures. Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk County Community College http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/. Why Validity & Reliability ?. - PowerPoint PPT PresentationTRANSCRIPT
Developing General Developing General Education Course Education Course Assessment MeasuresAssessment Measures
Anthony R. Napoli, PhDAnthony R. Napoli, PhDLanette A. Raymond, MALanette A. Raymond, MA
Office of Institutional Research & AssessmentOffice of Institutional Research & AssessmentSuffolk County Community CollegeSuffolk County Community College
http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/
Why Validity & Reliability Why Validity & Reliability ?? Assessment results must Assessment results must
represent student achievement of represent student achievement of course learning objectivescourse learning objectives
Evaluation of the validity and Evaluation of the validity and reliability of the assessment reliability of the assessment measure provides the evidence measure provides the evidence that it does sothat it does so
Types of MeasuresTypes of Measures
‘‘Performance’ MeasuresPerformance’ Measures
‘‘Objective’ MeasuresObjective’ Measures
Validity for Validity for Performance MeasuresPerformance Measures
Identified learning outcomes represent the Identified learning outcomes represent the course (domain sampling)course (domain sampling)
The measure addresses the learning The measure addresses the learning outcomes (content validity)outcomes (content validity)
There is a match between the measure and There is a match between the measure and the rubric (criteria for evaluating the rubric (criteria for evaluating performance)performance)
Rubric scores can be linked to the learning Rubric scores can be linked to the learning outcomes, and indicate the degree of student outcomes, and indicate the degree of student achievement within the course achievement within the course
Validity for Validity for Objective MeasuresObjective Measures
Identified learning outcomes represent the Identified learning outcomes represent the course (domain sampling)course (domain sampling)
The items on the measure address specific The items on the measure address specific learning outcomes (content validity)learning outcomes (content validity)
Scores on the measure can be applied to Scores on the measure can be applied to the learning outcomes, and indicate the the learning outcomes, and indicate the degree of student achievement within the degree of student achievement within the course course
Content Validity (Content Validity (MA23MA23))
Objective Description 1. Apply basic probability principles 2. Demonstrate an understanding of basic statistical terms 3. Write and do basic analysis using binomial, normal, t, and Chi- square
distributions 4. Understand, construct, and interpret confidence intervals 5. Understand, conduct, and interpret hypothesis tests 6. Understand and apply the central limit theorem 7. Understand and compute measures of central tendency and variability 8. Organize and describe data mathematically and pictorially
Content Validity (Content Validity (MA23MA23))
Item
Apply basic probability principles;
Demonstrate an understanding of basic statistical terms;
Write and do basic analysis using binomial, normal, t, and chi square distributions;
Understand, construct and interpret confidence intervals.
Understand, conduct and interpret hypothesis tests;
Understand and apply the central limit theorem;
Understand and compute measures of central tendency and variability;
Organize and describe data, mathematically and pictorially;
1a
1b
1c
2a
2b
3a
3b
4a
4b
Learning Objectives
Instructions: Please match the items (in the left column) with the single most appropriate Learning Outcome (across the top of the grid) by placing an 'X' in the corresponding box.
Content Validity (Content Validity (MA23MA23))Item
Apply basic probability principles;
Demonstrate an understanding of basic statistical terms;
Write and do basic analysis using binomial, normal, t, and chi square distributions;
Understand, construct and interpret confidence intervals.
Understand, conduct and interpret hypothesis tests;
Understand and apply the central limit theorem;
Understand and compute measures of central tendency and variability;
Organize and describe data, mathematically and pictorially;
1a 18 82
1b 10 90
1c 9 91
2a 100
2b 100
3a 100
3b 100
4a 91
4b 18 9 73
4c 82
5a 100
5b 82
5c 100
5d 100
5e 100
6a 9 73 9 9
6b 82 9 9
7a 45.5 45.5
7b 70 30
7c 60 40
7d 50 50
7e 50 50Objective Rate 100 12 97 77.5 70 77.5 90.5 82
Overall Agreement Rate = 82% w/o objective 2
Learning Objectives
Content Validity (Content Validity (MA61MA61))Objective Description 1. Demonstrate an understanding of a mathematical function
including a) the range and domain of functions, b) symmetric functions, c) composite functions, and d) inverses of functions
2. a) Understand the zeros of quadratic functions b) Sketch graphs of quadratic functions
3. Comprehend the significance of the fundamental theorem of algebra and be able to a) solve a polynomial and b) Write a polynomial in its factored form
4. Sketch the graph of polynomial functions 5. Sketch the graph of rational functions 6. Sketch the graph of a) exponential and b) logarithmic functions 7. Solve a) exponential equations and b) logarithmic equations 8. a) Understand and graph the trigonometric functions
b) Solve applications using right triangle relationships
Content Validity (Content Validity (MA61MA61))2 3 4 7 11 12 14
a b a b c d a b c d e f g a b c d a b c a b c d a b c
1a 100 100 82 91 93
1b 100 100
1c 100 100
1d 100 100
2a 91 91
2b 73 73 91 79
3a 46 46 55 82 73 60
3b 91 91 91
4 91 91
5 91 91 100 94
6a 100 100 100 100
6b 82 100 100 100 96
7a 91 91
7b 91 91
8a 100 100 100 100
8b 100 10089
Obj rate
Overall agreement rate
Obj 1Problem Number (agreement rate - %)
1310965 8
Content Validity (Content Validity (SO11SO11))ObjectiveObjective
II
IIII
IIIIII
DescriptionDescriptionIdentify the basic methods of data Identify the basic methods of data
collectioncollectionDemonstrate an understanding of basic Demonstrate an understanding of basic
sociological concepts and social sociological concepts and social processes that shape human behaviorprocesses that shape human behavior
Apply sociological theories to current Apply sociological theories to current social issuessocial issues
A 30-item test measured students’ mastery of the objectives
Content Validity (Content Validity (SO11SO11))Test Question Number -objective 1
Rated OB1
Item Agreement % %
1 0 0.02 8 100.03 7 87.54 1 12.55 0 0.06 0 0.07 0 0.08 8 100.09 8 100.010 8 100.0
Objective-wide agreement rate 50.0Questions affectively linked to objective 1 5 Items 2,3,8,9,10
Content Validity (Content Validity (SO11SO11))Test Question Number -objective 2
Rated OB2
Item Agreement % %
11 5 62.512 5 62.513 4 50.014 5 62.515 7 87.516 6 75.017 6 75.018 5 62.519 8 100.020 8 100.0
Objective-wide agreement rate 73.6Questions affectively linked to objective 2 3 Items 15,19,20
Content Validity (Content Validity (SO11SO11))Test Question Number -objective 3
Rated OB3
Item Agreement % %
21 5 62.522 1 12.523 1 12.524 3 37.525 3 37.526 2 25.027 3 37.528 4 50.029 6 75.030 3 37.5
Objective-wide agreement rate 37.4Questions affectively linked to objective 3 1 Item 29
ReliabilityReliability Can it be done Can it be done consistentlyconsistently??
Can the rubric be applied Can the rubric be applied consistently across raters -- Inter-consistently across raters -- Inter-rater reliabilityrater reliability
Can each of the items act Can each of the items act consistently as a measure of the consistently as a measure of the construct -- Inter-item reliability construct -- Inter-item reliability
Inter-Rater Reliability Inter-Rater Reliability ((MA23MA23) – The Rubric) – The RubricItem 1A
Inter-Rater Reliability Inter-Rater Reliability ((MA23MA23) – The Rubric) – The RubricItem 1B
Inter-Rater Reliability Inter-Rater Reliability ((MA23MA23) – The Data Set) – The Data Set
Inter-Rater Reliability Inter-Rater Reliability ((MA23MA23) – Results) – Results
ItemItem LowLow HighHigh RangeRange1A1A .9837.9837 4.214.21 5.005.00 0.790.791B1B .9877.9877 3.423.42 4.144.14 0.720.721C1C .9715.9715 1.711.71 2.852.85 1.041.042A2A .9966.9966 3.573.57 3.783.78 0.210.212B2B .9974.9974 1.931.93 2.142.14 0.210.213A3A .9734.9734 4.004.00 4.574.57 0.570.573B3B .9963.9963 3.573.57 3.853.85 0.280.28
Inter-Item Reliability Inter-Item Reliability ((MA61MA61))
Objective Objective 11 Demonstrate an understanding of a mathematical function (range Demonstrate an understanding of a mathematical function (range
and domain, symmetric, composite, inverses). Items 1a, 1b, 2, 3,4, and domain, symmetric, composite, inverses). Items 1a, 1b, 2, 3,4, 8a,10a.8a,10a.
.61.61
22 Understand the zeros of quadratic functions and sketch graphs of Understand the zeros of quadratic functions and sketch graphs of quadratic functions. Items 5a-d.quadratic functions. Items 5a-d. .65.65
33 Comprehend the significance of the fundamental theorem of algebra Comprehend the significance of the fundamental theorem of algebra and be able to solve a polynomial and write a polynomial in its and be able to solve a polynomial and write a polynomial in its factored form. Items 6a-f, 7.factored form. Items 6a-f, 7.
.72.72
55 Sketch the graph of rational functions. Sketch the graph of rational functions. Items 8b-d.Items 8b-d.
.83.83
66 Sketch the graphs of exponential and logarithmic functions. Sketch the graphs of exponential and logarithmic functions. Items 9a-c, 10a-d.Items 9a-c, 10a-d.
.83.83
77 Solve exponential and logarithmic equations. Solve exponential and logarithmic equations. Items 11,12.Items 11,12.
.52.52
88 Understand and graph the trigonometric functions, and solve Understand and graph the trigonometric functions, and solve applications using right triangle relationships. Items 13a-c, 14.applications using right triangle relationships. Items 13a-c, 14. .68.68
Face Validity and Face Validity and Reliability… Is this Reliability… Is this
enough? enough? Measures with face validity & Measures with face validity &
adequate levels of reliability can adequate levels of reliability can produce misleading/inaccurate produce misleading/inaccurate results.results.
Even content valid measures cannot Even content valid measures cannot guarantee accurate estimates of guarantee accurate estimates of student achievement student achievement
MA23 – earlier pilotMA23 – earlier pilot
Criterion-Related Criterion-Related Validity Validity (MA23 – earlier (MA23 – earlier pilot) pilot)
Percent of students scoring 70 & above on assessment test
23% 21%34%
72%59%
39%
C C+ B B+ A overall
Course Grade
Criterion-Related Criterion-Related Validity Validity (MA23)(MA23)
(2) (3) (4)Assessment (1)+ 70.92 17.60 0.780 0.808 -0.644Course GPA (2) 2.86 1.13 0.967 -0.780
Term Average (3) 80.82 13.17 -0.678Absences (4) 1.32 1.36
*All correlations are significant at the p < .001 level+Coefficient alpha for the 21-item Assessment test = .80
Correlation Coefficients*Criteria Mean Std
Criterion-Related Criterion-Related Validity Validity (MA61)(MA61)
Assessment Test Grade 0.750 ** 0.692 ** 0.305 * 0.225 0.263 * 0.262 * 0.151 0.187
MA61 Course GPA 0.912 ** 0.388 ** 0.359 ** 0.366 ** 0.365 ** 0.055 0.304 **
Average of all tests 0.434 ** 0.482 ** 0.446 ** 0.343 ** 0.017 0.265 *
MA27GPA 0.097 0.232 0.025 -0.048 0.362 *
CPTR 0.702 ** 0.539 ** 0.021 -0.053CPTS 0.520 ** 0.159 0.293 *
CPTA 0.467 ** 0.126CPTAL 0.430 **
HSAVG*Correlation is significant at the 0.05 level (2-tailed).**Correlation is significant at the 0.01 level (2-tailed).
CPTS CPTA CPTAL HSAVGMA61
Course GPAAverage of
all testsMA27 GPA CPTR
Motivational Motivational Comparison (Comparison (PC11PC11))
2 Groups2 GroupsGraded Embedded QuestionsGraded Embedded QuestionsNon-Graded Form & Non-Graded Form &
Motivational Motivational SpeechSpeech Mundane RealismMundane Realism
Motivational Motivational Comparison (Comparison (PC11PC11))
Graded condition Graded condition produces higher produces higher scores (scores (tt(78) = (78) = 5.62, 5.62, pp < .001). < .001).
Large effect size Large effect size ((dd = 1.27). = 1.27).
8.6(43%)
12.5(62%)
0.02.04.06.08.0
10.012.014.0
Non-Graded Graded
Motivational Motivational Comparison (Comparison (PC11PC11))
Minimum Minimum competency competency
70% or better70% or better
Graded condition Graded condition produces greater produces greater competency (competency (ZZ = = 5.69, 5.69, pp < .001). < .001).
0%
41%
0%
10%
20%
30%
40%
50%
Non-Graded Graded
Motivational Motivational Comparison (Comparison (PC11PC11))
In the non-In the non-graded condition graded condition this measure is this measure is neither reliable neither reliable nor valid nor valid
KR-20KR-20N-gN-g = 0.29 = 0.290.29
0.71
0
0.2
0.4
0.6
0.8
Non-graded Graded
Motivational Motivational Comparison (Comparison (PC11PC11))
2 3 4 Mean Std n.85*** .79*** .45* 62.4 17.4 46
.95*** .48** 81.4 12.1 46.45* 2.9 0.9 46
CPT-Reading Comprehension test (4) 80.6 18.0 30* p < .05, ** p < .01, p < .0001
Correlation with
Correlations between assessment measure, course outcomes, and reading comprehension scores.
Assessment grade (1)Term Average (2)Course GPA (3)
Developing General Developing General Education Course Education Course Assessment MeasuresAssessment Measures
Anthony R. Napoli, PhDAnthony R. Napoli, PhDLanette A. Raymond, MALanette A. Raymond, MA
Office of Institutional Research & AssessmentOffice of Institutional Research & AssessmentSuffolk County Community CollegeSuffolk County Community College
http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/