Seeing the Forest and the Trees

Seton Hall University

Technology, Assessment and Strategic Planning

Calvin Williams – Director of TLTC

Janet Easterling – Institutional Research Associate

Dr. Eric Fountain – Assessment Coordinator, IT

Dr. Peter Ahr – Professor, Religious Studies

Copyright Peter Ahr, Janet Easterling, Eric Fountain, Calvin Williams, 2005. This work is the intellectual property of the authors.

Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright

statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate

otherwise or to republish requires written permission from the authors.

• A Process for Transformation: University Strategic Planning

• Integrating an Information Technology (IT) Long-Range Plan

• Implementing Quality Academic Technology Initiatives to meet Strategic Goals

• Developing Institutional and IT Assessment measures to enhance programming

• Building on Assessment and shared growth through Multi-Campus Assessment

Questions and Answers: Major Presentation Themes

Seton Hall UniversityWho Are We?

• We Are A Mid Sized, Private, Catholic Affiliated University in Central New Jersey

• Carnegie Classification: Research / Doctoral II• Main Campus in South Orange, NJ

– 15 miles from New York City• 10,000 Students

– 4,400 Full Time Undergraduates (40% Residential)

• 350 FT Faculty (450 FTE Faculty)• FY’03 Annual Operating Budget Approx. $150 million

– FY’02 G&E for South Orange Campus Approx. $115 million

Pace Of Change at Seton Hall University

Before 1995:

– Inadequate Computer Labs

– Inadequate Local Networks/E-mail

– Lack of Integration of Technology in Teaching

– Disorganized Support/Allocation of Resources

Pace Of Change (Cont.)

Now:

Seton Hall University is doing IT “right”• 2003 FORBES “America's Most Connected

Campuses” - Ranked 16th• 2000 EDUCAUSE Award (Honorable Mention) for

Systemic Progress in T&L with Technology• 1999 EDUCAUSE Award for Campus Networking

Excellence• Growing National Reputation of our Mobile Computing

Initiative and other Teaching, Learning, and Technology Initiatives

• Alliance with IBM Corporation

A Vehicle for Change:University Strategic Planning

• Seton Hall University has accomplished heightened recognition among top-tier Catholic Universities nationally

• The strategic goals include a critical focus on the intellectual, personal and spiritual development of all students

• Students experience a rigorous, value-centered and learner-centered technologically enhanced environment

Advancing Strategic Goals with Technology

It was recommended that a strategic agenda include an Information Technology Strategic Planning process to build an Long-Range IT Plan

Advancing Strategic Goals with Technology

To advance the strategic goals, a strong focus was placed on communication, teaching and learning as well as support services for students using technology.

Core Curriculum Reform

• Faculty Senate Committee elected in 2000

• Request of the President

• Rounds of meetings with all faculty

• Analysis of possible models

• Focus on skills and literacies rather than content

• Emphasis on assessment and accountability

Core Curriculum Proposal

• Three courses developing institutional mission and student self-learning

• Systematic development of fundamental skills

• Integration of key literacies

• Electronic portfolios for all students

• Emphasis on assessment of outcomes of education

Core skills

• Critical thinking

• Writing

• Oral presentation

• Information fluency

• Specific metrics developed for integration into at least 50% of a student’s courses

• Conscious development of these habits of mind as part of coursework

Fundamental Literacies

• Religious literacy• Ethical literacy• Quantitative literacy• Scientific literacy• Aesthetic literacy• Historical/global literacy• All students must take at least four

courses which develop these literacies

E-portfolios

• Database of courses certified as fulfilling the Core requirements

• Ongoing checklist of student’s fulfilling Core requirements for advising

• Depository of objects attesting to student’s skill development

• Opportunity for student reflection on the learning process

Assessment via e-portfolios

• Assessment of student learning

• Faculty accountability for quality of work products

• Public accountability of effectiveness of programs

• Record for evaluation of success of Core

• Accountability of all parties

• System of institutional assessment

Value of technology to Core reform

• Based on mobile computing• Builds on existing initiatives• Uses our success in information technology as a

model for further innovation• E-portfolios as public record of student

development• Assessment of student learning as required by

Middle States

Division of Information TechnologyAssessment

History

Mobile Computing Assessment

Metrics Program

Metrics Program

• Internal Focus• Started in PC Support• Performance Measures Well Established• Facilitated by New Ticketing System• Focus on Organizing and Displaying Information

and Cost Metrics for ’04-’05

• University Level– Mobile Computing Assessment Team– Teaching, Learning, and Technology Roundtable– Institutional Research– Faculty Senate

• Program Level– ePortfolio

• Course Level– Large Course Redesign

• Multi-Campus

Mobile Computing Assessment Strategies

Narrowing the QuestionFantasy:a + b = c

a = teaching, b = technology, c = learningReality:

a + b + c + d + e + f…discipline, teaching style, technology proficiency, fit

of application to task, desired learning objectives...

Mobile Computing Assessment Team

Cross-Campus, Cross-DisciplineProvost’s OfficeInstitutional ResearchFacultyInstructional Design

Multiple ProjectsMobile Computing Assessment SurveyExtant Data AnalysisLarge-Course RedesignMulti-Campus

Mobile Computing Survey

R1. What is the effect of the Mobile Computing Program on student satisfaction?

R2. What is the effect of the Mobile Computing Program on technology use?

R3. What is the effect of the Mobile Computing Program on desired educational outcomes?

Research Questions

Technology Use Seton Hall Students 1998-2000

40.3

20.7

34.8

24

29.2

25.9

25.9

17.7

14.3

19.9

0

10

20

30

40

50

60

70

Per

cen

tag

e

Drill & Practice Computer-based training

Presentations Communicating(email, etc.)

Researching(Internet, etc.)

Technology Use Items - "TOP 5"

Technology Use - Items with highest proportions specifying 6 or more hours per week

6-10 hours per week

More than 10 hoursper week

61.058.8 55.1

43.6

34.2

Technology Use – Students1998-2000 2002 2003

6+ hours/week 6+ hours/week Daily,Weekly

• Communicating 44% 49% 48%,35%

• Researching 34% 34% 26%,47%

• Blackboard 14% 46%,18%

(43% 2+)

Satisfaction with Technology – Students

Satisfied or Highly Satisfied

1998-2000 2002 2003• Mobile Courses

56% 76% 93%• Pre-Loaded Software

73% 93%• Laptop Computer

68% 75% 86%

Impact of Technology on Learning Student Perceptions 1998-2000

54.5

23.7

50.7

23.4

53.6

19.2

52.6

19.5

54.4

17.7

0

10

20

30

40

50

60

70

80

Per

cen

tag

e

Contact w /Professors

Accessmaterialsrelated tocourse

Learntechnology

skills

Get promptfeedback from

Professors

Present w orkin many w ays

Impact of IT on Learning Items - "TOP 5"

Impact of IT on Learning - Items with highest proportions Positive or Very Positive responses

Very Positive

Positive

78.2 74.1 72.8 72.1 72.1

Impact of Technology on LearningStudent Perceptions

1998-2000 2002 2003Impact Positive or Very Positive Often+

• Contact with professors78% 95% 61v74%

• Working with other students

67% 85% 50v56% • Discuss/write/think in depth

61% 80% 64v64%

NSSE benchmarks for

Effective Educational Practices

Level of Academic Challenge

Active & Collaborative

Learning

EnrichingEducational Experiences

SupportiveCampus

Environment

StudentFaculty Interaction

What NSSE Measures

Student exposure to and participation in educational practices that promote engagement and student learning– Interaction with faculty in & out of the

classroom– Active & collaborative learning– High expectations and academic challenge– Enriching educational experiences

Student perception of a supportive campus environment helping them to succeed

NSSE Benchmarks and SHU

SHU strengths, according to NSSE?Connections to Technology?

• Active and Collaborative Learning • Interactions with Seton Hall Faculty

NSSE Benchmark Active/Collaborative Learning

In each NSSE year (’00, ’01, ’02, ‘03):

• SHU’s average Freshman score exceeded that of

80% of institutions in the same Carnegie classification category.

• SHU’s average Senior score exceeded that of 40-60%

NSSE Questions - Active & Collaborative Learning

NSSE 2003 - Active & Collaborative LearningSeton Hall Freshmen vs Peers & National

45.9

40.141.8

30

40

50

60

SHU Doctoral-Intensive

National

NSSE Benchmark Faculty-Student Interaction

In each NSSE year (’00, ’01, ’02, ‘03):

SHU’s average scores exceeded that of

80% - 90% of institutions in the same Carnegie classification category.

For Freshmen*, SHU exceeded 90-100%!

*In 2000, NSSE cited Seton Hall for this exemplary result

NSSE Questions - Faculty Student Interaction

NSSE 2003 – Faculty Student Interaction Seton Hall Freshmen vs Peers & National

42

35.337.2

30

40

50

60

SHU Doctoral-Intensive

National

Effective Learning Environments

…Connections to Technology?

Are positive NSSE outcomes evidence that SHU investments in IT help build SHU engaging educational environments?

NSSE experimental Technology Qs (2003)

NSSE Technology QsTechnology Use

SHU Students SHU Carnegie Peers

• Time online for academic - 6+ hours / week 55% 2003 SHU FR 40% Doct-Intensives FR 51% 2003 SHU SR 44% Doct-Intensives SR

• Time online for any reason - 6+ hours / week88% 2003 SHU FR 77% Doct-Intensives FR81% 2003 SHU SR 72% Doct-Intensives SR

NSSE Technology QsTechnology Use & Active Learning

SHU Students SHU Carnegie Peers

• How often required to use IT - % Very Often59% 2003 SHU FR 49% Doct-Intensives FR61% 2003 SHU SR 56% Doct-Intensives SR

• Used T to make presentations - % Very Often47% 2003 SHU FR 34% Doct-Intensives FR56% 2003 SHU SR 53% Doct-Intensives SR

NSSE Technology QsTechnology Use & Faculty Contact

SHU Students SHU Carnegie Peers

• Email ask instructor to clarify assign - % Very Often51% 2003 SHU FR 30% Doct-Intensives FR57% 2003 SHU SR 39% Doct-Intensives SR

• Email to express ideas to instructor - % Very Often22% 2003 SHU FR 13% Doct-Intensives FR28% 2003 SHU SR 16% Doct-Intensives SR

NSSE Technology QsTechnology Use & Active/Collaborative Learning

SHU Students SHU Carnegie Peers

• Work in teams in class using IT - % Very Often27 % 2003 SHU FR 14 % Doct-Intensives FR25 % 2003 SHU SR 19 % Doct-Intensives SR

• Communicate with classmates on-line to complete academic work - % Very Often51% 2003 SHU FR 27 % Doct-Intensives FR49 % 2003 SHU SR 39 % Doct-Intensives SR

NSSE Technology QsTechnology Use & Active Learning / Reflection

SHU Students SHU Carnegie Peers

• Gain new insights into course material from on-line discussions - % Quite a bit+43% 2003 SHU FR 37% Doct-Intensives FR31% 2003 SHU SR 32% Doct-Intensives SR

• Express ideas to instructor by email not comfortable saying in class - % Very Often22% 2003 SHU FR 13% Doct-Intensives FR28% 2003 SHU SR 16% Doct-Intensives SR

Conclusion/suggestion post review of NSSE

benchmarks & NSSE Technology Qs

SHU NSSE & technology results are consistent and together suggest that SHU investments in IT are helping build educational environments at SHU that are increasingly engaging, challenging, learning community oriented, reflective and “active”.

Mobile Computing Conclusions

• Attracted by the availability of technology at the University and the infusion of technology in the curriculum

• Satisfied with the Mobile Computing Program as well as with the laptop and support services

• Making good use of the technology available

• Perceiving a substantive positive impact of Seton Hall University’s Mobile Computing Program on the learning environment.

The Seton Hall University student is generally:

MCAT – Future Plans

• Expand Faculty Participation• Organize and Publicize Results• Approach Question of Classroom Use• FlashLight• Comparison Studies

Teaching, Learning, and Technology Roundtable

• Cross-Campus Representation

• Assessment Subcommittee– Studying faculty adoption/refusal rates and

patterns.

Other Campus-Wide Efforts

• Faculty Senate – IT Subcommittee

• Outcomes Assessment

• Accreditation Self-Study

• e-Portfolios

Seton Hall UniversityTeaching, Learning and Technology Center

Walsh Library, South Orange, NJ 07079973-275-2929

Thank You!

Calvin Williams – Director of TLTC

Janet Easterling – Institutional Research Associate

Dr. Eric Fountain – Assessment Coordinator, IT

Dr. Peter Ahr – Professor, Religious Studies