assessing student achievement in the core curriculum office of assessment and program improvement...
Post on 21-Dec-2015
217 Views
Preview:
TRANSCRIPT
Assessing Student Assessing Student Achievement in theAchievement in the
Core CurriculumCore Curriculum
Office of Assessment and Program ImprovementOffice of Assessment and Program Improvement
Report to the University CommunityReport to the University Community
Fall 2003Fall 2003
2
Office of the ProvostOffice of the Provost
The Assessment Process
Define specific learning outcomes that should generally be evident in students who complete a program of study
Identify sources of evidence about how well students are typically achieving these outcomes
Set performance targets as criteria for program success
Conduct studies and use the results to improve achievement
3
Office of the ProvostOffice of the Provost
Assessing a Major vs.Assessing the Core
C O R E
What any AU student
should know
and be able to do
no matter what major
M A J O R
What an AU student
majoring in a given subject
should know
and be able to do
Defined by specialists
Defined Cooperatively
4
Office of the ProvostOffice of the Provost Working Assumptions
We can collect reasonably good evidence about how well students are achieving the desired “core outcomes” for an AU graduate, but it is difficult to know what led to this achievement
Assessing students’ core competencies is more feasible than ssessing students’ core competencies is more feasible than determining the exact contribution of the Core Curriculum itselfdetermining the exact contribution of the Core Curriculum itself
Using several methods and measurement points will provide fuller information than relying on any one “snapshot” method
The “core outcomes” are everybody’s businessThe “core outcomes” are everybody’s business
5
Office of the ProvostOffice of the Provost
Core Curriculum Oversight Committee
Created together with the Core Curriculum
• Original charge was to oversee implementation of Core
• Began assessment in 1995 with reviews of social science and Great Books courses
• Reviews focused on teaching – syllabi, assignments, textbooks, grade distributions
• Assessment best practice now focuses on student learning
6
Office of the ProvostOffice of the Provost Assessment Process
Define intended outcomes
7
Office of the ProvostOffice of the Provost
From Broad Purpose to Specific Plans
In 2001 the Core Curriculum Oversight Committee
Studied the broad purpose statement for the Core Curriculum
Formulated 12 more specific intended learning outcomes
Selected 4 outcomes for assessment in 2002 and 2003
8
Office of the ProvostOffice of the Provost
Starting Pointsfor Assessment
Read critically and analytically
Communicate in writing at a sophisticated level
Comprehend the basic concepts of algebra
Gather, interpret, and synthesize information in accordance with contemporary scholarly standards
Students who have completed the Core Curriculum will be able to
9
Office of the ProvostOffice of the Provost Future Assessment
Scientific Reasoning*
Art Appreciation
History, Culture, Values
Social Sciences
Oral Communication
Awareness of Issues
Critical Thinking
Problem Solving
Outcomes for Future Study
10
Office of the ProvostOffice of the Provost Assessment Process
Define intended outcomes
Identify means of assessment
11
Office of the ProvostOffice of the Provost Two Methods Used
Local methods to estimate the abilities of students who were still taking Core courses
Analysis of papers and exams Pre- and post-testing
ACT’s Collegiate Assessment of Academic Proficiency (CAAP) tests to estimate the abilities of students who had fulfilled all Core Curriculum requirements
Writing, Reading, Math, Critical Thinking, Science Reasoning
12
Office of the ProvostOffice of the Provost Assessment Methods
S T A N D A R D
ACT CAAP Tests
Project SAILS
L O C A L
Review of work samples
Pre- and Post-Testing
Taking Core Fulfilled Core
13
Office of the ProvostOffice of the Provost Assessment Process
Define intended outcomes
Identify means of assessment
Set targets for program success
14
Office of the ProvostOffice of the Provost
Standards Chosen:Local Methods
Critical Reading• 70% of portfolios reviewed should rate satisfactory or better
when evaluated by two independent judges
Written Communication• 70% of portfolios reviewed should rate satisfactory or better
when evaluated by two independent judges
College Algebra• Improvement should be evident on post-test
Information Literacy• Both standards – 70% satisfactory and post-test improvement
15
Office of the ProvostOffice of the Provost
Standards Chosen:CAAP Test
AU students who have completed the Core Curriculum should score at or above the 70th percentile nationally on each CAAP module
No more than 20% of AU students tested should perform below the 50th percentile nationally on any CAAP module
National comparison group was sophomores at about 80 public 4-year colleges
16
Office of the ProvostOffice of the Provost Assessment Process
Define intended outcomes
Identify means of assessment
Set targets for program success
Collect and analyze assessment data
Outcome 1Critical Reading
Students who have completed the Students who have completed the Core Curriculum will be able to read Core Curriculum will be able to read critically and analytically at a critically and analytically at a sophisticated levelsophisticated level
18
Office of the ProvostOffice of the Provost Critical Reading
Analysis of papers and exams from Great Books II showed
Students were able to find, remember, and interpret specific details from their readings
They were able to discuss themes found in their readings
Assessment could not determine whether students were able to discuss formal literary features of the works they had read or place them in their historical and social contexts
19
Office of the ProvostOffice of the Provost
Critical Reading
75
86
3833
90 88
49
61
0
100
Details Themes Forms Contexts
%
Papers Exams Can't Tell
Percent of Great Books II Portfolios Rated Excellent/Satisfactory
20
Office of the ProvostOffice of the Provost Critical Reading
In formal papers for Great Books II, students met but did not exceed the performance target for three aspects writing about texts that may imply critical reading ability
Formulating a thesis about a text they had read
Using textual evidence to support that thesis
Reasoning soundly to connect evidence and thesis
21
Office of the ProvostOffice of the Provost
Critical Reading
71 73 73
0
100
Thesis Evidence Reasoning
%
Percent Rated Excellent or Satisfactory
GB2 Essays rated Excellent or Satisfactory
22
Office of the ProvostOffice of the Provost Critical Reading
Results from the two CAAP tests for Critical Reading were mixed
Reading• Median score was at the 73rd percentile nationally• But 22% of those tested scored below the 50th percentile
Critical Thinking• Median score was at the 82nd percentile nationally• Only 11% scored below the 50th percentile
23
Office of the ProvostOffice of the Provost
Critical Reading
73
82
11
22
0
100
Reading Critical Thinking
%
Median Score as National Percentile Rank % Below 50th Percentile
Median Percentile Scores and “Failure Rates”
CAAP Reading and Critical Thinking Tests
Outcome 2Written Communication
Students who have completed the Students who have completed the Core Curriculum will be able to Core Curriculum will be able to communicate in writing at a communicate in writing at a sophisticated levelsophisticated level
25
Office of the ProvostOffice of the Provost Written Communication
Analysis of argument papers from Composition II showed
Students were clearly able
• to formulate a thesis• to use language that was appropriate to the writing situation
Students had more trouble
• Supporting their thesis with sound evidence• Organizing their arguments effectively
26
Office of the ProvostOffice of the Provost
Written Communication
90
73
89
66
0
100
Thesis Evidence Organization Language
%
Rated Satisfactory or Better
Four Aspects of Rhetorical Effectiveness
ENGL1120 Papers with Satisfactory Ratings
27
Office of the ProvostOffice of the Provost Written Communication
The two CAAP tests for writing were given in 2002 only
Students “maxed out” the Essay Writing test – nearly all scored at or above the 70th national percentile
Scores on the multiple-choice Writing Skills test were also strongly positive
It is not clear that there is good alignment between this test and the goals of Composition I and II
28
Office of the ProvostOffice of the Provost
Written Communication
86
97
167
0
100
Writing Skills (2002 only) Essay Writing (2002 only)
%
Median Score as National Percentile Rank % Below 50th Percentile
Median Percentile Scores and “Failure Rates”
CAAP Writing Skills and Essay Writing Tests
Outcome 3College Algebra
Students who have completed the Students who have completed the Core Curriculum will comprehend Core Curriculum will comprehend the basic concepts of algebrathe basic concepts of algebra
30
Office of the ProvostOffice of the Provost College Algebra
Local Assessment (2003 Only)
Students took Math Placement Exam at Camp War Eagle
Some items from this exam were embedded in the final exams for pre-calculus Core courses (not including MATH 1100)
On average, the 916 students tested scored 30% higher on these items on the final exam than they had done on the placement test
31
Office of the ProvostOffice of the Provost College Algebra
Results for the CAAP Mathematics test must be treated separately for 2002 and 2003 because of a change in the test design
2002 version (4 calculus items)
students met the performance targets for the algebra subscore
2003 version (no calculus items)
students met the performance targets for the whole test
For both years, students’ performance just cleared the bar of the 70th national percentile, but concerns remain
32
Office of the ProvostOffice of the Provost
College Algebra
7673
5
18
0
100
Algebra Subscore (2002) Mathematics Test (2003)
%
Median Score as National Percentile Rank % Below 50th Percentile
Median Percentile Scores for CAAP
Mathematics Test
33
Office of the ProvostOffice of the Provost College Algebra
The CAAP results are less positive when students are grouped
Those who had taken Math courses beyond the Core met performance targets
. . . But . . .
Those who had taken only one Math course failed to meet performance targets
34
Office of the ProvostOffice of the Provost
College Algebra
64
82
24
11
0
100
"More Math" (2003) "Core Math" (2003)
%
Median Score as National Percentile Rank % Below 50th Percentile
Median Percentile Scores and “Failure Rates”
“More Math” vs. “Core Math” Students
Experimental OutcomeScience Reasoning
36
Office of the ProvostOffice of the Provost Science Reasoning
The CAAP results for Science Reasoning (not formally assessed in either year) resemble those for Mathematics
Those who had taken only one Math course scored poorly in Science Reasoning
Those who had taken Math courses in addition to the Core did much better in Science Reasoning
37
Office of the ProvostOffice of the Provost
Science Reasoning
72.5
87
70
14
4
20
0
100
All Students "More Math" Students "Core Math" Students
%
Median Score as National Percentile Rank % Below 50th Percentile
Median Percentile Scores and “Failure Rates”
CAAP Science Reasoning Test
Outcome 4Information Literacy
Students who have completed the Students who have completed the Core Curriculum will be able to Core Curriculum will be able to gather, synthesize, and interpret gather, synthesize, and interpret information in accordance with information in accordance with contemporary scholarly standardscontemporary scholarly standards
39
Office of the ProvostOffice of the Provost Information Literacy
Before receiving bibliographic instruction (BI), ENGL 1120 students were tested on how to conduct library searches
Sample: “You wish to find current articles dealing with drug use on U.S. campuses. Your best “place” to begin?
Students were retested after BI and had higher scores
AU Bibliographic Instruction Quiz (Summer 2001)
48
61
0
100
Before BI After BI
% Correct
40
Office of the ProvostOffice of the Provost Information Literacy
165 UNIV1000 students used the online Texas Information Literacy Tutorial (TILT) and took the accompanying quizzes
The overall score on the TILT was 96%
Their lowest score was on an item that tested whether they thought the library’s whole collection was available online
Texas Information Literacy Test
96
86
0
100
Total Availability Question
% Correct
41
Office of the ProvostOffice of the Provost
Information LiteracyProject SAILS
In 2003, 247 ENGL1120 students took part in Project SAILS, a national information literacy test recognized by the Association of Research Libraries
Multiple choice questions yielded information about 4 key information literacy skills
On each skill, AU students did somewhat better than the average for all 10 participating institutions
Project SAILS
50
66
5057
0
100
AssessNeeds
FindInformation
EvaluateInformation
UnderstandIssues
% Correct
42
Office of the ProvostOffice of the Provost
Information LiteracyWork Samples
91
47
666863
0
100
Appropriateness Balance* Integration Citation* Reference List*
%
Rated Satisfactory or Better (* = 2003 only)
Aspects of Using Sources Effectively
ENGL1120 Papers with Satisfactory Ratings
43
Office of the ProvostOffice of the Provost Assessment Process
Define intended outcomes
Identify means of assessment
Set targets for program success
Collect and analyze assessment data
Act on the results to improve student achievement
44
Office of the ProvostOffice of the Provost Planned Actions
Remedy focused deficiencies by additional instruction and practice in appropriate courses – e.g., documenting sources
Guide instruction, assessment, and improvement by clarifying some broad intended outcomes – e.g., critical reading ability
Continue to get good evidence about our students’ core competencies through dialogue and collaboration
Benefit from experience with four outcomes to guide assessment planning for the intended learning outcomes that have not been studied so far
Further InformationFurther Information
Office of Assessment and Program ImprovementOffice of Assessment and Program ImprovementDrew Clark, DirectorDrew Clark, Director
209 Samford Hall209 Samford Hall844-5802844-5802
clarkj3@auburn.eduhttp://www.auburn.edu/academic/provost/assessment
Office of the ProvostOffice of the Provost
top related