evaluating academic challenge beyond the nsse jim purcell, associate vice chancellor oklahoma state...

Evaluating Academic Challenge Beyond the NSSE

Jim Purcell, Associate Vice Chancellor Oklahoma State Regents for Higher

Education

Legacy of Assessment

Letter from the Six Nations to William & Mary College

We are convinc'd, therefore, that you mean to do us Good by your Proposal; and we thank you heartily. But you, who are wise, must know that different Nations have different Conceptions of things; and you will therefore not take it amiss, if our Ideas of this kind of Education happen not to be the same with yours. We have had some Experience of it. Several of our Young People were formerly brought up at the Colleges of the Northern Provinces; they were instructed in all your Sciences; but, when they came back to us, they were bad Runners, ignorant of every means of living in the Woods, unable to bear either Cold or Hunger, knew neither how to build a Cabin, take a Deer, or kill an Enemy, spoke our Language imperfectly, were therefore neither fit for Hunters, Warriors, nor Counselors; they were totally good for nothing.

We are, however, not the less oblig'd by your kind Offer, tho' we decline accepting it; and, to show our grateful Sense of it, if the Gentlemen of Virginia will send us a Dozen of their Sons, we will take care of their Education; instruct them in all we know, and make Men of them.

Knowing is half the Battle GI Joe Cartoon Character and Action Figure

Know Thyself Socrates

“Every new adjustment is a crisis in self-esteem.”

Assessment is a Mirror

Assessment is a reflection of reality – not reality

Now we see but a poor reflection as in a mirror. Now I know in part; then I shall know fully . . . 1 Corinthians 13:12

JoHari Window

Known by Self Unknown by Self

Known by Others

Unknown by Others

Open Area

Hidden Area

Unknown Area

Blind Area

Joseph Luft and Harry Ingham

JoHari Window

Known by Self Unknown by Self

Known by Others

Unknown by Others Hidden Area

Unknown Area

Blind Area

Joseph Luft and Harry Ingham

Open Area

Georgia College & State University

Student Engagement

Notional value: We like the idea, but do not take action.

Cost for caring about student engagement: Investment of time, study, money, and risk.

What are we doing? Is it working?

Important to take risk: Pause to think day.

The NSSE first Reading

Asked the questions that were at the core of the institutional mission

Sense of Place Sense of Activity Sense of Interaction Provoked faculty interest Enabled institutional conversation

1= never 2= sometimes 3= often 4= very often

Asked questions in class or contributed to class discussions.

Selected NSSE findings

Seniors

3.06

2.75

1st-year

From NSSE 2000

2.81

1st-year

3.16 Seniors

3.001st-year

3.30 Seniors

National

Liberal Arts

Georgia College


Worked with other students on projects during class.


Seniors

2.49

2.42

1st-year

From NSSE 2000

2.28

1st-year

2.64 Seniors

2.441st-year

2.43 Seniors

National

Liberal Arts

Georgia College


Worked with other students on assignments outside of class.


Seniors

2.71

2.39

1st-year

From NSSE 2000

2.31

1st-year

2.83 Seniors

2.611st-year

2.76 Seniors

National

Liberal Arts

Georgia College

National Survey of Student EngagementSenior College Activities Character of Mental Activities

Red Indicates Significance at the .001 levelScale is 1=very little, 2=some, 3=quite a bit, 4=very much

2.59

3.04

2.69

2.62

3.06

2.63

3.07

2.78

2.68

3

2.57

3.1

2.82

2.69

3.01

2.35

3.25

3.07

2.88

3.09

Memorizing facts, ideas or methods from your cour

Analyzing the basic elements of an idea, experien

Synthesizing and organizing ideas, information, o

Making judgments about the value of information, a

Applying theories or concepts to practical proble

1 2 3

GCSU Master's I&II

National Liberal

National Survey of Student EngagementSenior College Activities

Weekly Activities

Red Indicates Significance at the .001 levelScale is 1 = 1 or fewer hours/week, 2=6-10 hours/week, 3=11-15 hours/week, 4=16-20 hours/week, 5=21-25, 6=26-30, 7=more than 30

3.3

1.48

3.62

1.52

2.63

2.79

3.29

1.54

3.65

1.65

2.75

2.36

3.43

1.65

3.18

1.77

2.87

2.04

3.78

1.79

2.14

2.25

3.07

1.42

Preparing for class (studying, reading, writing, l

Working for pay on campus

Working for pay off campus

Participating in co-curricular activities (organiz

Relaxing and socializing (watching TV, partying, e

Providing care for dependents living with you

1 2 3 4

GCSU

Master's I&II

National

Liberal

Faculty Response?

Bogus: The survey was not administered correctly

Bad Survey: The survey failed to measure student engagement

Bad Students: We deal with a different type of student than other schools

Bubba’s problem: The results do not apply to me and my colleagues in my department

A few months later . . .

Used e-mail to communicate with an instructor or other students. (Sig. .018)

A&S (2.80), Bus (2.90) , Ed (3.40), HS (3.40)

Made a class presentation. (Sig. .015)

A&S (2.87), Bus (2.69) , Ed (3.42), HS (3.07)

Worked with other students on projects during class. (Sig. .001)

A&S (2.33), Bus (2.55) , Ed (3.32), HS (2.60)

National Survey of Student Engagement Senior students –by GCSU Academic SchoolsStatistically significant responses at .05 level or higher

Participated in a community-based project as part of a regular class course. (Sig. .000)

A&S (1.47), Bus (1.25) , Ed (2.21), HS (2.53)

Used an electronic medium to complete or discuss a project. (Sig. .044)

A&S (2.27), Bus (2.17) , Ed (2.90), HS (2.53)

Discussed grades/assignment with instructor. (Sig. .014)

A&S (3.13), Bus (2.52) , Ed (3.00), HS (2.73)


Number of assigned textbooks, books, or book length packs of course reading. (Sig. .005)

A&S (3.24), Bus (2.90) , Ed (2.79), HS (3.80)

Number of papers over 20 pages. (Sig. .004)

A&S (1.50), Bus (1.50) , Ed (1.25), HS (2.00)

Number of papers under 20 pages. (Sig. .022)

A&S (2.72), Bus (2.62) , Ed (3.40), HS (3.20)


Course work emphasizes synthesizing and organizing ideas. (Sig. .005)

A&S (2.83), Bus (2.28) , Ed (2.70), HS (3.27)

Course work emphasizes making judgments about the value of information. (Sig. .027)

A&S (2.79), Bus (2.31) , Ed (2.55), HS (3.20)

Preparing for class (studying, reading, etc) (Sig. .017)

A&S (3.77), Bus (2.86) , Ed (2.95), HS (4.29)


Contributed to acquiring a work related skills. (Sig. .007)

A&S (3.13), Bus (2.83) , Ed (3.45), HS (3.73)

Contributed to analyzing quantitative problems. (Sig.033)

A&S (2.69), Bus (3.00) , Ed (2.40), HS (3.20)

Contributed to the welfare of your community. (Sig. .005)

A&S (2.07), Bus (2.21) , Ed (2.65), HS (3.07)


A few more months later . . .

Level of Academic ChallengeActive and Collaborative LearningStudent Interaction with FacultyEnriching Educational ExperienceSupportive Campus Environment

NSSE survey

Evaluating Academic Challenge Beyond the NSSE

Stephen L. Payne, Karynne L. M. Kleine, Jim Purcell, and Ginger Rudeseal Carter

http://www.springeronline.com/sgw/cda/frontpage/0,11855,4-40406-70-35612147-0,00.html

Kuh (2001) strongly encouraged institutional leaders to adopt multi-faceted, multi-year, data-collection strategies that capitalize on what contrasting assessment approaches do best.

Eleven items are incorporated in the academic challenge dimension of the NSSE.

Many items relate strongly to objectives on Bloom’s and other learning taxonomies and others involve the amount of time or effort that students must have on the course.

Academic Challenge

Do students at particular institutions discount the importance of aspects of academic challenge that the NSSE tries to measure?

Is the NSSE missing aspects of academic challenge that faculty and students at particular institutions believe would be helpful for improving academic challenge there?

How might an institution with a specific mission, academic culture, and set of characteristics examine its degree of academic challenge and move toward more comprehensive assessment and improvement of this academic challenge?

Focus Group Questions

Focus Groups Twenty-nine faculty Fifty-four students Main and branch campus students were interviewed. Transcriber Asked questions concerning:

conceptions of academic challenge, the appropriateness of the academic challenge items on the

NSSE, and what might be done to improve academic challenge at

this institution.

Academic Challenge Questions

Students Faculty

Amount of time spent preparing for class + +-Number of assigned textbooks, books, or book-length packs of course reading - +Number of written papers or reports of 20 pages or more between 5-19; of fewer than 5 pages. - -Analyzing the basic elements of an idea, experience, or theory such as examining a particular case or situation in depth considering its components +

Synthesizing and organizing ideas, information, or experiences new, more complex interpretations and relationships + +Making judgments about the value of information, arguments, or methods such as examining how others gathered and interpreted data and assessing the soundness of their conclusions +- +Applying theories or concepts to practical problems or in new situations + +About how often have you worked harder than you thought you could to meet an instructor’s standards or expectation? + -To what extent does your institution emphasize spending significant accounts of time studying and on academic work? + -

What is Academic challenge?

Initial reactions to the concept of academic challenge were frequently negative –

Emphasized the extra time and effort as well as the initial uncertainty or confusion associated with such courses

Students later mentioned rewards of academically challenging courses.

Synthesis and, application of learning content

Faculty teaching style and student interactions were a concern

Students

Off-campus students Stressed stronger concern for

career entry and success Reacted less favorably than other

students to academic challenge in courses, particular in core courses of those connected to their majors

Significant minority of faculty emphasized lack of student preparation to enable student engagement.

“Basics” limiting student opportunities for higher-order learning and academic challenge

Need for strategies to challenge students.

Need improved resources for faculty development

What is Academic challenge? Faculty

Academic challenge requires a recognition of different student learning styles and offering multiple paths

Learning outcomes become more important

Need more and improved assessment using the NSSE and other indicators

What is Academic challenge? Faculty

Proposal for expanded assessment

Accepted by the Academic Governance Committee of the University Senate of GC&SU (2004)

Ask to devote time to developing this internal assessment instrument

Used NSSE measures for benchmarks for quality enhancement plan

Critical thinking, reading, and writing are at the core of academic challenge, but these important concerns are not handled the existing NSSE items

Several academic challenge items stress student effort

Items are not linked to learning outcomes

Ideas for local research related to the NSSE discussions

Ideas for local research related to the NSSE discussions

Study relationships between student perceptions of learning and actual outcome measures of affective, creative, aesthetic, and leadership development particular programs

Development of items that permit distinction between experiences of courses

Distinguish between academic challenge and experiences that are merely highly competitive, demanding, or even intimidating.

Students have less interest and motivation for learning in core courses they perceive as having less direct career relevance.

What roles of responsibilities might faculty and students assume for improved learning outcomes?

Establishing both high academic rigor in their courses as well as a high level of faculty-student relationships out of class seems to be a contradiction to many faculty.

What other interesting tidbits from this study?

Students placed greater responsibility on faculty in their learning experience.

Faculty feel that assignments can result in more meaningful learning outcomes than students might realize at this point in their lives

What other interesting tidbits from this study?

Closing Remarks

We should design educational experiences that are empowering

Measure quality of encounters Ensure encounters are

transformational We should desire positive

restlessness

evaluating academic challenge beyond the nsse jim purcell, associate vice chancellor oklahoma state...

Documents

class discussions

class presentation

student engagementbad

different nations

different type of student

correctlybad survey

kind of education

faculty response