CBU Online CBU Online Survey SystemSurvey System

MASECMASEC

March 31, 2006March 31, 2006

John VenturaJohn Ventura

Teacher EvaluationsTeacher Evaluations

Project DescriptionProject Description

Current methods at CBUCurrent methods at CBU Paper surveyPaper survey

Count results by hand or scantronCount results by hand or scantron

I.T.S. worker creates a web pageI.T.S. worker creates a web page Takes a lot of time and effortTakes a lot of time and effort

Project DescriptionProject Description

Easy to use web-based evaluation tools:Easy to use web-based evaluation tools: Create, edit and administer surveys from a Create, edit and administer surveys from a

web siteweb site Export and import information to and from Export and import information to and from

a personal computera personal computer Manage users and groups, with usernames Manage users and groups, with usernames

and passwordsand passwords Analyze results with different types of Analyze results with different types of

table and graphs, and compare information table and graphs, and compare information for a survey over several terms.for a survey over several terms.

ABETABET

Accreditation Board for Engineering Accreditation Board for Engineering and Technology (ABET)and Technology (ABET)

Web-based Evaluation ProcessWeb-based Evaluation Process Data base for developing trendsData base for developing trends

Project GoalsProject Goals

Analyze available products on the Analyze available products on the market.market.

Design a software system to create Design a software system to create and manipulate data in a survey.and manipulate data in a survey.

Implement software system suitable Implement software system suitable for the Christian Brothers University for the Christian Brothers University environment.environment.

Important PointsImportant Points

Levels of user accessLevels of user access Department Head and AdministrationDepartment Head and Administration

Faculty and StaffFaculty and Staff

StudentsStudents

Alumni and GuestsAlumni and Guests

Important PointsImportant Points

SecuritySecurity Access based on user login and Access based on user login and

password similar to CBU Web mailpassword similar to CBU Web mail Import a text file with student usernamesImport a text file with student usernames Create students one at a timeCreate students one at a time

User identification used only for login User identification used only for login and not to indicate which surveys have and not to indicate which surveys have been takenbeen taken

Additional FunctionalityAdditional Functionality

User login verificationUser login verification

Import / export dataImport / export data

Adding, editing and deleting Adding, editing and deleting response setsresponse sets

Help pages and tutorialsHelp pages and tutorials

Research MethodResearch Method (a) establish an ECE Curriculum Committee, (a) establish an ECE Curriculum Committee,

(b) expand the responsibilities of the ECE (b) expand the responsibilities of the ECE Advisory Board, Advisory Board,

(c) develop Web-based instruments to (c) develop Web-based instruments to measure the achievement of program measure the achievement of program objectives of graduates and learning objectives of graduates and learning outcomes of students, and outcomes of students, and

(d) provide a model that integrates input from (d) provide a model that integrates input from constituents, assessment committees, and constituents, assessment committees, and results formulated from the results of results formulated from the results of measuring achievement of students and measuring achievement of students and graduates into an evaluation process. graduates into an evaluation process.

Procedures Procedures Employee surveys/questionnaires to measure the Employee surveys/questionnaires to measure the

achievement of program objectives using Web-based achievement of program objectives using Web-based technologiestechnologies

Establish an ECE Curriculum Committee that Establish an ECE Curriculum Committee that contains members from local engineering societies contains members from local engineering societies and organizations, the student chapter of IEEE, and and organizations, the student chapter of IEEE, and facultyfaculty

Expand the responsibilities of the ECE Advisory Expand the responsibilities of the ECE Advisory Board to include the examination and validation of Board to include the examination and validation of the evaluation processesthe evaluation processes

Provide constituents with instruments that measure Provide constituents with instruments that measure the level of achievement of program objectivesthe level of achievement of program objectives

Provide results of surveys/questionnaires to the ECE Provide results of surveys/questionnaires to the ECE Advisory Board and ECE Curriculum Committee that Advisory Board and ECE Curriculum Committee that enabled them to validate the survey instruments and enabled them to validate the survey instruments and the model for evaluationthe model for evaluation

ECE Curriculum ECE Curriculum Committee Committee

A member of the executive board of the Memphis A member of the executive board of the Memphis Chapter of IEEEChapter of IEEE

A member of the executive board of the Memphis A member of the executive board of the Memphis Chapter of the Tennessee Society of Professional Chapter of the Tennessee Society of Professional Engineers (TSPE)Engineers (TSPE)

A member of the Organizing Committee of the A member of the Organizing Committee of the Memphis Area Engineering and Sciences Memphis Area Engineering and Sciences Conference Conference

Chair of the IEEE Student Chapter at CBUChair of the IEEE Student Chapter at CBU ECE facultyECE faculty Dean of EngineeringDean of Engineering Chair of the Master of Engineering ManagementChair of the Master of Engineering Management A faculty member from the School of Sciences A faculty member from the School of Sciences

1 Strongly Disagree

Strongly Not Confident

Poor Not at All None Not Applicable

2 Disagree Not Confident Fair Very Little

Very Little

Unimportant

3 Neutral Neutral Good Some Some Important

4 Agree Confident Very good A Lot A Lot Very Important

5 Strongly Agree

Strongly Confident

Excellent A Great Deal

A Great Deal

Essential

Rating Scale

Grading Criteria for ABET Learning Outcomes for Grading Criteria for ABET Learning Outcomes for Industry SurveyIndustry Survey

Grade Criteria Grade Criteria A+ >= 80% of responses in categories 3 and 4; > = 50% rated as 3A+ >= 80% of responses in categories 3 and 4; > = 50% rated as 3 A >= 80% of responses in categories 3 and 4; > = 37.5% rated as 4 A >= 80% of responses in categories 3 and 4; > = 37.5% rated as 4 A– >= 80% of responses in categories 3 and 4; < 37.5% rated as 4A– >= 80% of responses in categories 3 and 4; < 37.5% rated as 4 B+ 60 to < 80% in categories 3 and 4; >= 37.5% rated as 4B+ 60 to < 80% in categories 3 and 4; >= 37.5% rated as 4 B 60 to < 80% in categories 3 and 4; >= 25% rated as 4B 60 to < 80% in categories 3 and 4; >= 25% rated as 4 B–60 to < 80% in categories 3 and 4; < 25% rated as 4B–60 to < 80% in categories 3 and 4; < 25% rated as 4 C+ Highest frequency of ratings for category 2 and 3 but < =60% in C+ Highest frequency of ratings for category 2 and 3 but < =60% in

category 2 and 3; number of (3+4) > number of (1+2) category 2 and 3; number of (3+4) > number of (1+2) C 60 to < 80% in category 2 and 3 C 60 to < 80% in category 2 and 3 C– Highest frequency of ratings for category 2 and 3 but < =60% in C– Highest frequency of ratings for category 2 and 3 but < =60% in

category 2 and 3; number of (1+2) > number of (3+4)category 2 and 3; number of (1+2) > number of (3+4) D+ < 90% to >= 70% in categories 1 and 2; < 25% are in category 1D+ < 90% to >= 70% in categories 1 and 2; < 25% are in category 1 D < 90% to >= 70% in categories 1 and 2; >= 25% to < 37.5% are in D < 90% to >= 70% in categories 1 and 2; >= 25% to < 37.5% are in

category 1category 1 D– < 90% to >= 70% in categories 1 and 2; >= 37.5% are in category D– < 90% to >= 70% in categories 1 and 2; >= 37.5% are in category

11 F >= 90% are in categories 1 and 2F >= 90% are in categories 1 and 2

ECE Advisory Board and ECE Advisory Board and ECE Curriculum Committee ECE Curriculum Committee Validate instruments that contain Validate instruments that contain

the expectations of constituents, the expectations of constituents, especially graduates and employers.especially graduates and employers.

Assess the quality and performance Assess the quality and performance of the evaluation process.of the evaluation process.

Assess the methods used for Assess the methods used for gathering information in a Web-gathering information in a Web-based environment.based environment.

EvaluationEvaluation

ECE Advisory Board evaluates the ECE Advisory Board evaluates the level of achievement of program level of achievement of program objectives and evaluation processesobjectives and evaluation processes

ECE Curriculum Committee ECE Curriculum Committee evaluates survey instruments and evaluates survey instruments and evaluation processesevaluation processes

ECE Curriculum Committee ECE Curriculum Committee evaluates the level of achievement evaluates the level of achievement of learning outcomes of learning outcomes

Formative EvaluationFormative Evaluation

Formative evaluation processes should be Formative evaluation processes should be

undertaken due to the unfavorable factors undertaken due to the unfavorable factors inherent in information obtained from student inherent in information obtained from student self-assessmentself-assessment

Summative processes based on self-Summative processes based on self-assessment is questionable assessment is questionable

Improvements in programs should be based Improvements in programs should be based on trends developed from data obtained over on trends developed from data obtained over several semesters rather than a summative several semesters rather than a summative assessment of asingle cycle of an evaluation assessment of asingle cycle of an evaluation process process

Online Survey Online Survey System System

DemonstrationDemonstration

Any Questions?Any Questions?