dr. stephen landry, cio dr. eric fountain, it assessment coordinator danielle mirliss, instructional...
TRANSCRIPT
Dr. Stephen Landry, CIODr. Eric Fountain, IT Assessment CoordinatorDanielle Mirliss, Instructional Designer
Using External Data Sources to Increase Confidence in Assessment Conclusions
Who We Are …
Seton Hall University
Mid sized, Catholic affiliated university in suburban NJ, 15 miles from New York City.
4,400 Undergraduates, 50% residential.
Two campuses: Main Campus in South Orange, NJ and Law School in Newark, NJ.
Approx. 350 Full Time Faculty on the South Orange Campus.
Research Intensive (Research II) Institution.
Institutional Change @ SHUSHU made a major commitment to institutional change
in its 1996 Strategic Plan
Seton Hall became a ubiquitous computing campus, providing laptop computers to the learning community as part of tuition and fees
Seton Hall increased enrollment while increasing academic requirements for admission
Seton Hall added new programs and faculty
Seton Hall increased emphasis on assessment and accountability
Ubiquitous Computing @ SHU
Seton Hall University’s Mobile Computing Program is designed to ensure all members of the learning community have access to learning resources
which are increasingly online.
Program has three interdependent components:
Access: The University licenses students the use of a laptop as part of their tuition and fees;
Curricular Integration: The University provides support and incentives to faculty to use technology in T&L;
Services and Support: Network (Wired and Wireless), Help Desk, Laptop Repairs, Insurance, Training, etc.
Implementing Ubiquitous Computing1994 – 1996: Development of SHU Strategic Plan
1995 – 1997: Mobile Computing Pilot Project with small cohorts of students in Business School
1997: Mobile Computing Pilot Project expands to 300 incoming Freshman
1998: Mobile Computing Program launched for all incoming Freshman
2001: Mobile Computing Program expanded to select graduate programs; campus wireless network implemented
2002: SHU graduates first mobile class
2004: Tablet PC Pilot Project
Mobile Computing and Curriculum Development
Seton Hall recognized that access to technology is necessary for the changes envisioned in the learning environment, but not a
sufficient condition.
Teaching, Learning and Technology Center
Provides support and training for faculty in their use of technology in their teaching
Internal Grant Programs:
Curriculum Development Initiative
Faculty Innovation Grants
Implementation of Classroom and Network Technology
Teaching, Learning, and Technology Roundtable
Mobile Computingand Assessment
In 1997 Seton Hall recognized it had a unique assessment opportunity
Mobile Computing Assessment Program has two objectives:
To assess the impact of ubiquitous computing on the learning environment
To provide prompt feedback to the Mobile Computing Program planning team to address potential implementation issues with the program
Assessment Methodology and Resources
Foundations of the Assessment Movement
Research Traditions Contributing to the Assessment Movement:
• Student Learning in College
• Retention and Student Behavior
• Evaluation and Scientific Management
• Mastery Learning
A B
CD
A. Reflect
B. Plan
C.Act
D.Observe
Assessment Process Cycle
SkillDemonstration
Performance Evaluations
Tracking Systems
Faculty Fellows
Surveys
ePortfolios
Focus Groups
Types of Assessment
National Studies
Studies Focused on Students:
• National Survey of Student Engagement (http://www.iub.edu/~nsse/)
• Higher Education Research Institute (http://www.gseis.ucla.edu/heri/heri.html
•CIRP Freshman Survey: profile of entering freshman class. •CSS College Student Survey: follow up of CIRP cohorts. •YFCY Your First College Year
• Flashlight (http://www.tltgroup.org/programs/flashlight.html)
Studies Focused on Faculty:
• Faculty Survey of Student Engagement (http://www.indiana.edu/~nsse/fsse/index.htm)
National Studies
Studies Focused on Technology:
• EDUCAUSE Core Data Survey (http://www.educause.edu/apps/coredata/index.asp?bhcp=1)
• Campus Computing Project (http://www.campuscomputing.net/)
• Cost of Supporting Technology Services (http://www.costsproject.org/)
Assessment in the Teaching, Learning and Technology Center
Current Initiatives
• Mobile Computing Assessment Team
• Project Assessment
• TLTR-Assessment Committee
Mobile Computing Assessment Team
Cross-Campus, Cross-Discipline• Provost’s Office• Institutional Research• Faculty• TLTC
Multiple Projects• Mobile Computing Assessment Survey• Extant Data Analysis• Large-Course Redesign• Classroom Study• Multi-Campus
Mobile Computing Survey
R1. What is the effect of the Mobile Computing Program on student satisfaction?
R2. What is the effect of the Mobile Computing Program on technology use?
R3. What is the effect of the Mobile Computing Program on desired educational outcomes?
Research Questions
Results
•Attracted to Seton Hall University by the availability of technology at the University and the infusion of technology in the curriculum, as suggested by student reported positive influences on the decision to attend the University,
• Satisfied with the Mobile Computing Program as well as with the laptop and support services once enrolled at the University,
• Making good use of the technology available to all students, at least in terms of certain types of technology use, and.
• Perceiving a substantive positive impact of Seton Hall University’s Mobile Computing Program on the learning environment.
The typical Seton Hall University student, whether representing all respondents, or particular groups of students (groups identified by gender, race/ethnicity, and residence status), is generally:
Web-based Online Courses
Mixed mode Hybrid Courses
Enhanced Courses with electronic resources and exercises
Course Level Assessment Strategies
• Collaborative effort between Provost’s Office, faculty, and TLTC
• Student and Faculty focus groups
• Development and implementation of pilot survey
Hybrid Course Assessment
Tablet PC Highlights
MCAT – Future Plans
Expand Faculty Participation
Expand Membership to Include Student Affairs and Student Government
Organize and Publicize Results
Approach Question of Classroom Use
FlashLight
Comparison Studies
Using External Data Sources to Increase Confidence in Assessment Conclusions
An Example from Seton Hall University using results from the 2003 National Survey of Student Engagement
What NSSE Measures
Student exposure to and participation in educational practices that promote engagement and student
learningInteraction with faculty in & out of the classroom
Active & collaborative learningHigh expectations and academic challenge
Enriching educational experiences
Student perception of a supportive campus environment helping them to succeed
NSSE Questions - Active & Collaborative Learning
NSSE 2003 - Active & Collaborative LearningSeton Hall Freshmen vs Peers & National
45.9
40.141.8
30
40
50
60
SHU Doctoral-Intensive
National
NSSE Questions - Faculty Student Interaction
NSSE 2003 – Faculty Student Interaction Seton Hall Freshmen vs Peers & National
42
35.337.2
30
40
50
60
SHU Doctoral-Intensive
National
NSSE Technology QsTechnology Use
SHU Students SHU Carnegie Peers
Time online for academic - 6+ hours / week 55% 2003 SHU FR 40% Doct-Intensives FR 51% 2003 SHU SR 44% Doct-Intensives SR
Time online for any reason - 6+ hours / week88% 2003 SHU FR 77% Doct-Intensives FR81% 2003 SHU SR 72% Doct-Intensives SR
NSSE Technology Qs
Technology Use & Active LearningSHU Students SHU Carnegie Peers
How often required to use IT - % Very Often59% 2003 SHU FR 49% Doct-Intensives FR61% 2003 SHU SR 56% Doct-Intensives SR
Used T to make presentations - % Very Often47% 2003 SHU FR 34% Doct-Intensives FR56% 2003 SHU SR 53% Doct-Intensives SR
NSSE Technology Qs
Technology Use & Faculty ContactSHU Students SHU Carnegie Peers
Email ask instructor to clarify assign - % Very Often51% 2003 SHU FR 30% Doct-Intensives FR57% 2003 SHU SR 39% Doct-Intensives SR
Email to express ideas to instructor - % Very Often22% 2003 SHU FR 13% Doct-Intensives FR28% 2003 SHU SR 16% Doct-Intensives SR
NSSE Technology Qs
Technology Use & Active/Collaborative LearningSHU Students SHU Carnegie Peers
Work in teams in class using IT - % Very Often27 % 2003 SHU FR 14 % Doct-Intensives FR25 % 2003 SHU SR 19 % Doct-Intensives SR
Communicate with classmates on-line to complete academic work - % Very Often
51% 2003 SHU FR 27 % Doct-Intensives FR49 % 2003 SHU SR 39 % Doct-Intensives SR
NSSE Technology QsTechnology Use & Active Learning / Reflection
SHU Students SHU Carnegie Peers
Gain new insights into course material from on-line discussions - % Quite a bit+
43% 2003 SHU FR 37% Doct-Intensives FR31% 2003 SHU SR 32% Doct-Intensives SR
Express ideas to instructor by email not comfortable saying in class - % Very Often
22% 2003 SHU FR 13% Doct-Intensives FR28% 2003 SHU SR 16% Doct-Intensives SR
Conclusions
Questions?
Contacts:Dr. Stephen Landry, CIO – [email protected]
Dr. Eric Fountain, IT Assessment – [email protected]
Danielle Mirliss, Instructional Designer – [email protected]
ResourcesInternet Resources for Higher Education Outcomes Assessment - http://www2.acs.ncsu.edu/UPA/assmt/resource.htm#nsse
AAHE Assessment Forum - http://www.aahe.org/assessment/assess_links.htm
Assessment in Higher Education - http://ahe.cqu.edu.au/