Mobile Computing and Mobile Computing and

Technology Assessment: Technology Assessment:

A Case Study with A Case Study with

ResultsResultsStephen G. Landry, Ph.D. Chief Information Officer

Heather Stewart, Ph.D. Assistant Vice President, Finance and Technology

Copyright Stephen G. Landry and Heather Stewart, 2003

Seton Hall UniversityWho Are We?

• We Are A Mid Sized, Private, Catholic Affiliated University in Central New Jersey

• Carnegie Classification: Research / Doctoral II• Main Campus in South Orange, NJ

– 15 miles from New York City• 10,000 Students

– 4,400 Full Time Undergraduates (50% Residential)

• 350 FT Faculty (450 FTE Faculty)• FY’01 Annual Operating Budget Approx. $150 million

– FY’00 G&E for South Orange Campus Approx. $115 million

Pace Of Change at Seton Hall University

• Before 1995:

– Inadequate Computer Labs

– Inadequate Local Networks/E-mail

– Lack of Integration of Technology in Teaching

– Disorganized Support/Allocation of Resources

Pace Of Change (Cont.)

Now:– Seton Hall University is doing IT “right”

• 2000 EDUCAUSE Award (Honorable Mention) for Systemic Progress in T&L with Technology

• 1999 EDUCAUSE Award for Campus Networking Excellence

• In Top 50 of Yahoo! Internet Life Survey of “Most Wired” Universities

• Growing National Reputation of our Mobile Computing Initiative and other Teaching, Learning, and Technology Initiatives

• Alliance with IBM Corporation

• Site Visits by National/International Universities

• Presentations at National Conferences/Workshop

A Vehicle for Change:University Strategic Planning

• Seton Hall University has accomplished heightened recognition among top-tier Catholic Universities nationally

• The strategic goals include a critical focus on the intellectual, personal and spiritual development of all students

• Students experience a rigorous, value-centered and technologically enhanced environment

Advancing Strategic Goals with Technology

• To advance the strategic goals, a strong focus was placed on communication, teaching and learning as well as support services for students using technology

• It was recommended that a strategic agenda include an Information Technology Strategic Planning process to build an IT Long-Range Plan

Seton Hall UniversityHow Did This Happen?

• Began With Development of IT Plan in 1995– Part of the University’s Strategic Planning Process– Cross Functional Team

• Included Faculty and Administrators• Given Half-Time Release to Work on IT Plan• Intensive Immersion in IT Issues, Trends, Directions

– Executive Sponsorship/Commitment to Implement• Chief Academic Officer/Chief Financial Officer

– Involvement of User Community• Focus Groups/Surveys• Involvement of Faculty and Administrator Constituency Groups

(e.g., Academic Computing Advisory Committee, Administrative Computing User Groups, etc.)

– Technology Planning Consultant from IBM Education

Seton Hall University’sStrategic Technology Plan

• Our Technology Strategy:

– Seton Hall University will develop and implement a• learner-centered, • network-centered, • distributed (mobile) learning environment

– The core of our distributed learning environment will be a robust set of digital information resources (e.g., a “digital library”) in support of teaching, learning, scholarship, and institutional transformation

SHU’s Strategic IT Plan

• Implications of the IT Plan:Focus on student experience– Students Want / Need:

• Ubiquitous Access to Technology• University-wide Hardware and Software Standards• University-wide Network• Central IT Support Services• Greater Curricular Integration of Technology • More Efficient Business Processes / Greater Access to

Institutional Information and Services / Bringing Together Academic and Administrative Computing

Implementing the Strategy

• In order to implement the strategy, the University undertook a number of teaching, learning, and technology initiatives:– Mobile (Ubiquitous) Computing Program– SetonWorldWide (Online Professional Programs)– Web-enabling Enrollment Services– Reorganization of Computing Services

• Created New User Services Organization, Including a Teaching, Learning, and Technology Center

– Major Technology Infrastructure Upgrade• Gigabit Ethernet Campus Backbone; Server Consolidation;

Campus-wide Email System; 802.11b Wireless; Internet Upgrade; etc.

Mobile Computing atSeton Hall University

• Seton Hall University’s Mobile Computing Program is an innovative academic program involving three components:

– Access: The University licenses the use of a laptop computer to all undergraduates as part of their tuition and fees

– Curricular Integration: The University provides support and incentives to faculty to use technology in innovative ways to enhance teaching and learning

– Network and Support Services: The University provides the infrastructure and support services that enable the effective use of technology in teaching and learning

Mobile Computing (cont.)• Current Model: IBM ThinkPad R-series computer

– 1.2 MHz PIII; 14” TFT Screen; 256 Mb RAM; 30 Gb HDD; CD-RW; Built-in 802.11b wireless networking; 10/100 Ethernet; 56Kb Modem; LiON Battery

– Computer is replaced every two years

• Current Technology Fee: $700 per semester for students in the program (“non mobile” students, e.g., many graduate students, pay a technology fee of $200 per semester)

• Bundled software includes: MS Windows ME, MS Office 2000, SPSS, Maple V, various utilities

• Bundled services include:– Technology Help Desk, PC Repairs, Loaner Computers– Network Services, including wireless network access from most

academic and public spaces

Mobile Computing (cont.)

• Program Phase-In

– 1995 – 1997: Pilot Programs

• Focus on Teaching and Learning

• Based on cohorts with common academic experiences (e.g., Business Honors Program in 1995, expanded to all 1st Year Business, Biology, and Honors students in Fall 1997)

• Began Teaching, Learning, and Technology Center and internal grant program to provide incentives to departments willing to integrate technology into large enrollment core courses

Mobile Computing (cont.)• Program Phase-In (cont.)

– 1998 – 1999: Full Roll Out of Program

• Program is Mandatory for ALL Incoming FT Undergraduates

• Focus on Infrastructure

• Major Network / Server Upgrades– Lotus Notes Email / Lotus LearningSpace Course

Management System– MS FrontPage Enabled Web Servers for Faculty and

Students– ATM Backbone / Campus Internet Upgrade /

Server Consolidation / Y2K Preparation

Mobile Computing (cont.)• Program Phase-In (cont.)

– 2000 – Present: Enterprise Perspective

• Focus on Enterprise Systems

– Implementation of Web services for students (e.g., online registration, course audit, grades, etc.)

– Automation of Mobile Computing processes (e.g., asset management, account generation, etc.)

– Implementation of BlackBoard course management / portal system; 24/7 support services through Eduprise

– Campus-wide wireless networking / Campus network upgraded to gigabit Ethernet backbone

Mobile Computing (cont.)

• Major investment on the part of the University– Central IT Budget (incl. laptop leases /student fees)

for FY’02 approx. $15 million or 13% SO Campus G&E(10% of the total University budget)• Almost equally divided between personnel, operating, and Mobile

Computing Program (technology fee) budgets• Most costs are operationalized / very little capital funding

• Seton Hall University is a different place because of the University’s strategic initiatives (enrollments up, SAT’s up, new programs launched, etc.)

• Mobile Computing Assessment Project indicates the program has had positive impact on learning / student recruitment

• Value of Seton Hall Results - DATA

• Underlying, Theory and Process

• Technology Assessment in Larger Institutional Context

Charge of theMobile Assessment Team

• Assess the impact of mobile computing on fall 1998 entering freshman

• This includes student satisfaction with the learning environment and student learning outcomes

Full Report and Resources

Available On-Line at:http://tltc.shu.edu/initiatives/assessment/index.html

Key Researchers and Co-Authors

Eric Fountain, Ed.D.

John Collins, Ed.D.

Janet Easterling

Circles of Assessment

Institution

External

Faculty

Students

When Assessment Occurs

Before

Determine Needs

During

Make Changes in Process

After

Evaluate Experience

Examples of Assessment Topics

Faculty Students Institution

• Technical Skills

• Instructional Goals

• Technical Skills

• Enrollment Decisions

• Community Engagement

• Resources

• Support

• Time Distribution

• Access

• General Satisfaction

• Bottlenecks

• Network Stability

• Training Needs

• Tool Fits Goal

• Info. Access• Communication• Technical Skills

• Delivery Change

• Evaluate Tech. Options

D

A

B

Types of Assessment

• Focus Groups• Interviews• Journals• Observations• Portfolios• Performances

• Standardized Tests• Surveys• Tracking Systems• Longitudinal Records• Document Reviews• Skills Application

Demonstrations

Narrowing the Question

Fantasy:a + b = c

a = teaching, b = technology, c = learning

Reality:a + b + c + d + e + f…

discipline, teaching style, technology proficiency, fit of application to task, desired learning objectives...

Faculty Journals (1997)

• “Cognitive, behavioral and affective sides of transition”

• “Technology has become a natural part of my teaching”

• “Be prepared to punt”

• “A year ago I couldn’t imagine using all this stuff: now I can.”

• “Technology is only, alas, a dumb machine that is no more creative, inventive, or productive than the human using it”

Technology Functions

• Edit• Collaborate• Create• Compose• Automate• Visualize• Simulate

• Communicate• Store• Structure Practice• Retrieve

• Contact between student and professor

• Cooperation among students

• Active learning• Prompt feedback• Time on Task• High Expectations• Respect diversity in

talents and learning

Seven Principles(Chickering and Gamson, 1989)

Nature of Learning(AAHE, NASPA, 1998)

• Strongly affected by educational climate

• Rapid, rich, frequent feedback

• Much is informal and incidental

• Requires transfer of specific knowledge

• Ability of learner to monitor own learning

• Making Connections

• Compelling Situation

• Active search for meaning by learner

• Developmental, cumulative process

• Individual learners are social beings

Evaluation of General Satisfaction, Technology Use, and Student

Perceptions of the Impact of Mobile Computing on the Learning Environment

at Seton Hall UniversitySpring 1998 – Fall 2000

Gender

R1a. Does a student’s gender have an effect on their perceptions regarding the satisfaction of the SHU Mobile Computing Program?

R2a. Does a student’s gender have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?

R3a. Does a student’s gender have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?

Independent Variables

Housing Status

R1b. Does a student’s housing status have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?

R2b. Does a student’s housing status have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?

R3b. Does a student’s housing status have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?

Independent Variables

Ethnicity

R1c. Does a student’s ethnic affiliation have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?

R2c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?

R3c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?

Independent Variables

• Structural

• Use of Mobile vs. Non-Mobile Construct

• Neutral Category

• Administrative

• Sampling

• Change from paper to online.

Limitations

RESULTS

Term Target Popn: #FR

Survey Responses#FR (FR Resp.Rate) 

TargetPopn:#Non-FR/U(returning students)

Survey Responses#Non-FR/U (Non-FR/U Resp. Rate)

Target Popn:

Comb. Total

Survey Responses(Comb. Resp.Rate)

Fall 1998 1226 324 (26%) 365(20+20+325)

90 (25%) 1591 414 (26%)

Spring 1999

1226 216 (18%) 365(20+20+325)

124 (34%) 1591 340 (27%)

Fall 1999 1070 311 (29%) 1591(365+1226)

453 (28%) 2661 764 (29%)

Spring 2000 1070 179 (17%) 1591(365+1226)

18 (01%) 2661 197 ( 7%)

Fall 2000 1125 235 (21%) 2661(1591+1070)

476 (18%) 3786 711 (19%)

Totals – Fall Terms Only

All 5 Terms

3421 870 (25%) 4617 1019 (22%) 12,290 2,426 (20%)

Response Rates

 

Gender Ethnicity Housing Status

Fall Terms Only FR Only N=870

46%

Male

35% Non-White or other/(missing)

39%

Commuters

SHU Freshmen(Fall ’98,’99,’00)

48%

Male

52% Non-White or other/(missing)

37%

Commuters

Response Demographics

46.5

21.9

45.7

18.2

39.1

17

41.4

14.1

40.5

12.7

0

10

20

30

40

50

60

70

Per

cen

tag

e

Satisfactionw / laptop

Satisfactionw / SHU

Value ofVisiting Help

Desk

Satisfactionw / MobileCourses

Satisfactionw / Non-Mobile

Courses

Satisfaction Items - "TOP 5"

Student Satisfaction - Items with highest proportions satisfied

Very HighSatisfaction

High Satisfaction

68.463.9

56.1 55.553.2

Top Five Responses – Satisfaction68.4 63.9 56.1 55.5 53.2

40.3

20.7

34.8

24

29.2

25.9

25.9

17.7

14.3

19.9

0

10

20

30

40

50

60

70

Pe

rce

nta

ge

Drill &

Pra

ctice

Compu

ter-b

ased

train

ing

Prese

ntat

ions

Comm

unicat

ing (e

mail

, etc.

)

Resea

rchi

ng (I

nter

net,

etc.

)

Technology Use Items - "TOP 5"

Technology Use - Items w ith highest proportions specifying 6 or more hours per week

6-10 hours per week

More than 10 hoursper week

61.0 58.855.1

43.6

34.2

Top Five Responses – Technology Use

54.5

23.7

50.7

23.4

53.6

19.2

52.6

19.5

54.4

17.7

0

10

20

30

40

50

60

70

80

Per

cen

tag

e

Contact w /Professors

Accessmaterialsrelated tocourse

Learntechnology

skills

Get promptfeedback

fromProfessors

Present w orkin manyw ays

Satisfaction Items - "TOP 5"

Student Satisfaction - Items with highest proportions satisfied

Very Positive

Positive

78.274.1 72.8 72.1 72.1

Top Five Responses – Outcomes

Examining the Research Questions

Gender

R1a. Does a student’s gender have an effect on their perceptions regarding the satisfaction of the SHU Mobile Computing Program?

R2a. Does a student’s gender have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?

R3a. Does a student’s gender have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?

Independent Samples Test - Gender

9.471 .002

-2.276 558.193 .023

.452 .501 -3.076 2186 .002

.190 .663 .760 1976 .447

Equal variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumed

Sum of SatisfactionVariables

Sum of Use Variables

Sum of OutcomesVariables

F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Gender

Housing Status

R1b. Does a student’s housing status have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?

R2b. Does a student’s housing status have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?

R3b. Does a student’s housing status have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?

Housing Status

Independent Samples Test - Housing Status

1.136 .287 -.420 592 .675

1.596 .207 -7.200 2175 .000

.681 .409 -.649 1969 .516

Equal variancesassumed

Equal variancesassumed

Equal variancesassumed

Sum of SatisfactionVariables

Sum of Use Variables

Sum of OutcomesVariables

F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Ethnicity

R1c. Does a student’s ethnic affiliation have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?

R2c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?

R3c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?

EthnicityANOVA - Racial/Ethnic Identification

129.872 4 32.468 .964 .426

19694.6 585 33.666

19824.5 589

449.155 4 112.289 1.541 .188

157751 2165 72.864

158200 2169

3072.575 4 768.144 3.332 .010

452080 1961 230.535

455153 1965

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

Sum of SatisfactionVariables

Sum of Use Variables

Sum of OutcomesVariables

Sum ofSquares df Mean Square F Sig.

Changes in MC 2002 Results

• 81% of the respondents said they had used email to communicate on class-related information on a daily or weekly basis.

• 68% of the respondents used the internet for research on a daily or weekly basis.

• 56% submitted work electronically.

Changes in MC 2002, cont.• 70% had contact with their professors often or very

often. An additional 20% reported occasional contact.• 68% reported that it helped them take control of their

own learning.• 70% used the internet to access class materials online.• While 66% of students felt their laptop use did not

detract from their ability to pay attention in class, 45% felt that laptop use was distracting other students.

Conclusions

•Attracted to Seton Hall University by the availability of technology at the University and the infusion of technology in the curriculum, as suggested by student reported positive influences on the decision to attend the University,

• Satisfied with the Mobile Computing Program as well as with the laptop and support services once enrolled at the University,

• Making good use of the technology available to all students, at least in terms of certain types of technology use, and.

• Perceiving a substantive positive impact of Seton Hall University’s Mobile Computing Program on the learning environment.

The typical Seton Hall University student, whether representing all respondents, or particular groups of students (groups identified by gender, race/ethnicity, and residence status), is generally:

Technology Assessment Projects

Recurring Program Assessment

University Assessment, Planning and Budget

Technology Assessment

Initiatives

Programs for Coordinating Assessment

Major University Assessment,

Budgeting and Planning

Mobile Computing Outcomes Assessment Team

Sesquicentennial Strategic Plan

Tablet PC Pilot TLTR Subcommittee

IT 2

Large Course Redesign

Middle States Association

Questions?