slide 1 assessments and culminating experiences institutional workshop mr. bill kohlruss-dr. mark...

www.aabi.aerowww.aabi.aero

ASSESSMENTS ANDCULMINATING EXPERIENCES

INSTITUTIONAL WORKSHOP

Mr. Bill Kohlruss-Dr. Mark ShermanCO-CHAIRS, GUIDANCE COMMITTEE

FEBRUARY 21, 2013


Where Have We Gone Astray?

Based on Feedback From Accreditation Visits…


COMPILATION OF RECOMMENDATIONSFebruary 2009 to July 2012

Recommendations related to:• Program Educational Objectives – 48• Culminating Experience – 23• Assessment – 19• Outcomes – 14• Safety – 11• Industry Advisory Boards/Committees – 10• Curriculum – 8• Faculty – 6• Facilities, Equipment & Services – 4

Thanks

Ceci &

Vic


Number ONE on the list (48 Hits) is Program Educational Objectives !

Program Education Objectives was addressed in depth at the Workshop last February and we are addressing that weak area through a change in wording in the Criteria…

Now called “Program Mission and Educational Goals”.

New terminology, perhaps easier to understand ?


So… Number 2 and 3 are on the agenda for today.

#2 Culminating Experiences.

#3 Assessments.


To put a POSITIVE SPIN on today’s workshop…

We have asked Dr. Beth Bjerke and Dr. Mark Dusenburyto do a presentation on “How They Assessed the Programs atUND”.

This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.

Beth Bjerke and Mark Dusenbury

ASSESSMENT

“Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development” (Palomba and Banta, 1999, pg. 4)

Assessment Process

Establishing your Program Mission Defining your Program Educational Goals Defining your Student Learning Outcomes (AABI

General, Core and Program) Identifying your Techniques and Target Groups Data Collection ‘Closing the Loop’ process Implementing changes to improve student learning Assessing the Assessment Process!

Techniques and Target Groups Techniques

Direct Assessment of Student Learning Written Exams Performance Evaluations Assignments

Target Group – Students Indirect Assessment of Student Learning and Program Quality

Surveys Target Group - Students, Alumni, Industry

Interviews/Focus Groups Target Group - Students, Alumni, Industry

Survey Data

Designing Surveys

• Online Survey Tools• Easy to use and abuse• Set up a timeline and stick to it• Have a purpose for the survey• Share results with faculty and students• Use to improve the learning experience

Gather Indirect Perception Data(AABI a-k from Students)

Gather Indirect Perception Data(AABI a-k from Alumni)

AABI General Outcomes2007 2009 2011

An ability to apply knowledge of mathematics, science and applied sciences 2.86 2.94 3.05An ability to analyze and interpret data 3.15 3.18 3.27An ability to function on multi-disciplinary teams 3.04 3.09 3.21An understanding of professional and ethical responsibilities 3.17 3.25 3.35An ability to communicate effectively, including both written and verbal communication skills 3.09 3.14 3.24A recognition of the need for, and an ability to engage in, life long learning 3.10 3.18 3.33A knowledge of contemporary issues 2.87 2.99 3.12An ability to use the techniques, skills and modern technology necessary for professional practice 3.39 3.35 3.42An understanding of the national and international aviation environment 3.08 3.06 3.23An ability to apply pertinent knowledge in identifying and solving problems. 3.17 3.22 3.35An ability to apply knowledge of business sustainability to aviation 3.10

Closing the Loop with Survey Data Take the time to have the conversation with faculty

Make recommendations and action items Document the conversation!

Interviews and Focus Groups

Students Have a lot to say, and many great ideas Focus Group Discussions Graduating Senior Interviews

Interviews and Focus Groups

Industry/Alumni Use of Advisory Board Other Sources

Campus Visitors Document!

Closing the Loop with Interviews/Focus Groups

Take the time to have the conversation with faculty Make recommendations and action items Document the conversation!

Direct Learning

Course Prime Reports Stage Check Analysis Capstone

Course Mapping

Course Prime Reports

Stage Check Analysis

Capstone

Writing analysis between majors in capstone courses Infant stages – small sample size – will get better with time Food for thought at this time

Techniques Timeline

Faculty Outcome Discussions

MOST IMPORTANT ASPECT TO ASSESSMENT…

Using what is learned through the process to improve the student

learning experience in your program!


To address our second topic we have Ms. Andrea Georgiou and Mr. Paul Carlson doing a presentation on “Our Capstone/Culmination Experience” at MTSU.

This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.

CULMINATING EXPERIENCESCULMINATING EXPERIENCES

Middle Tennessee State UniversityMiddle Tennessee State University

Andrea M. GeorgiouAndrea M. Georgiou

Paul R. CarlsonPaul R. Carlson

FOCUS Lab ProjectFOCUS Lab Project

Middle Tennessee State UniversityMiddle Tennessee State University

Andrea M. GeorgiouAndrea M. Georgiou

Paul R. CarlsonPaul R. Carlson

FOCUS Lab ProjectFOCUS Lab Project


• FOCUS concept first proposed in 2009 by Dr. Paul A. CraigFOCUS concept first proposed in 2009 by Dr. Paul A. Craig

• Received NASA grant in 2010 to study team decision-makingReceived NASA grant in 2010 to study team decision-making

• Project would address best practices for communication, Project would address best practices for communication, disruption management, teamwork, and group situational-disruption management, teamwork, and group situational-awarenessawareness

• Conceptualization and construction began in 2010 with a small Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate studentsresearch team comprised of faculty and graduate students

• Students began with tabletop discussions and now has evolved to Students began with tabletop discussions and now has evolved to include:include:

• Complex and real-world scenariosComplex and real-world scenarios

• Immediate feedback and ramifications to decisionsImmediate feedback and ramifications to decisions

• Real-time performance analysisReal-time performance analysis

• In-depth after action reviewsIn-depth after action reviews

• FOCUS concept first proposed in 2009 by Dr. Paul A. CraigFOCUS concept first proposed in 2009 by Dr. Paul A. Craig

• Received NASA grant in 2010 to study team decision-makingReceived NASA grant in 2010 to study team decision-making

• Project would address best practices for communication, Project would address best practices for communication, disruption management, teamwork, and group situational-disruption management, teamwork, and group situational-awarenessawareness

• Conceptualization and construction began in 2010 with a small Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate studentsresearch team comprised of faculty and graduate students

• Students began with tabletop discussions and now has evolved to Students began with tabletop discussions and now has evolved to include:include:

• Complex and real-world scenariosComplex and real-world scenarios

• Immediate feedback and ramifications to decisionsImmediate feedback and ramifications to decisions

• Real-time performance analysisReal-time performance analysis

• In-depth after action reviewsIn-depth after action reviews

PROJECT PROJECT BACKGROUNDBACKGROUND


Aerospace CoreAerospace CoreAerospace CoreAerospace Core

TechnologyTechnologyTechnologyTechnologyFlight Flight DispatchDispatch

Flight Flight DispatchDispatch

Pro PilotPro PilotPro PilotPro PilotMaintenance Maintenance ManagementManagementMaintenance Maintenance ManagementManagement

AdminAdminAdminAdmin

Seminar + SimulationSeminar + SimulationSeminar + SimulationSeminar + Simulation• MTSU AEROSPACE program MTSU AEROSPACE program comprised of 5 concentrations:comprised of 5 concentrations:

• PROFESSIONAL PILOTPROFESSIONAL PILOT

• MAINTENANCE MANAGEMENTMAINTENANCE MANAGEMENT

• AEROSPACE ADMINISTRATIONAEROSPACE ADMINISTRATION

• FLIGHT DISPATCHFLIGHT DISPATCH

• TECHNOLOGYTECHNOLOGY

• MTSU AEROSPACE program MTSU AEROSPACE program comprised of 5 concentrations:comprised of 5 concentrations:

• PROFESSIONAL PILOTPROFESSIONAL PILOT

• MAINTENANCE MANAGEMENTMAINTENANCE MANAGEMENT

• AEROSPACE ADMINISTRATIONAEROSPACE ADMINISTRATION

• FLIGHT DISPATCHFLIGHT DISPATCH

• TECHNOLOGYTECHNOLOGY

•All concentrations take aerospace All concentrations take aerospace core classescore classes•All concentrations take aerospace All concentrations take aerospace core classescore classes

•After specializing in their chosen After specializing in their chosen concentrations, students return to concentrations, students return to take a capstone coursetake a capstone course

• Prior to FOCUS lab, capstone course Prior to FOCUS lab, capstone course only addressed industry topics, only addressed industry topics, résumé, and interview preparationrésumé, and interview preparation

•After specializing in their chosen After specializing in their chosen concentrations, students return to concentrations, students return to take a capstone coursetake a capstone course

• Prior to FOCUS lab, capstone course Prior to FOCUS lab, capstone course only addressed industry topics, only addressed industry topics, résumé, and interview preparationrésumé, and interview preparation

PROJECT CONCEPTPROJECT CONCEPT

PROJECT CONCEPTPROJECT CONCEPTCULMINATING EXPERIENCESCULMINATING EXPERIENCES

• Surveys of industry experts revealed that newly-minted Surveys of industry experts revealed that newly-minted aviation professionals required 5-10 years to fully understand aviation professionals required 5-10 years to fully understand their role and impact within an organization.their role and impact within an organization.

• Surveys of industry experts revealed that newly-minted Surveys of industry experts revealed that newly-minted aviation professionals required 5-10 years to fully understand aviation professionals required 5-10 years to fully understand their role and impact within an organization.their role and impact within an organization.

•Research revealed that new aviation professionals lacked:Research revealed that new aviation professionals lacked:

• operational understanding of others’ roles in the operational understanding of others’ roles in the organizationorganization

• basic understanding of how their decisions and job basic understanding of how their decisions and job performance impacts othersperformance impacts others

• the ability to communicate and coordinate effectively with the ability to communicate and coordinate effectively with others to meet organizational objectivesothers to meet organizational objectives

• the use of foresight, planning, and time management in the use of foresight, planning, and time management in making decisions and performing tasksmaking decisions and performing tasks

•Research revealed that new aviation professionals lacked:Research revealed that new aviation professionals lacked:

• operational understanding of others’ roles in the operational understanding of others’ roles in the organizationorganization

• basic understanding of how their decisions and job basic understanding of how their decisions and job performance impacts othersperformance impacts others

• the ability to communicate and coordinate effectively with the ability to communicate and coordinate effectively with others to meet organizational objectivesothers to meet organizational objectives

• the use of foresight, planning, and time management in the use of foresight, planning, and time management in making decisions and performing tasksmaking decisions and performing tasks

““On average, it usually takes our employees 10 years On average, it usually takes our employees 10 years to fully understand the big picture; how the decisions to fully understand the big picture; how the decisions they make impact others in the company.”they make impact others in the company.”


• The FOCUS project is specifically designed to address The FOCUS project is specifically designed to address

teamwork deficiencies of new employees in the aviation teamwork deficiencies of new employees in the aviation

industryindustry

• Students are placed in a realistic simulated airline flight Students are placed in a realistic simulated airline flight

operations control centeroperations control center

• Here, students from all aviation disciplines are placed in Here, students from all aviation disciplines are placed in

teams to practice working with othersteams to practice working with others

• The FOCUS project is specifically designed to address The FOCUS project is specifically designed to address

teamwork deficiencies of new employees in the aviation teamwork deficiencies of new employees in the aviation

industryindustry

• Students are placed in a realistic simulated airline flight Students are placed in a realistic simulated airline flight

operations control centeroperations control center

• Here, students from all aviation disciplines are placed in Here, students from all aviation disciplines are placed in

teams to practice working with othersteams to practice working with others



•Teams work to meet organizational goals like safety, on-time Teams work to meet organizational goals like safety, on-time

performance, customer satisfaction, and disruption performance, customer satisfaction, and disruption

managementmanagement

• Teams are given various scenarios (or Teams are given various scenarios (or triggerstriggers) that require ) that require

the students to work together to resolve issues quickly and the students to work together to resolve issues quickly and

effectivelyeffectively

• Teams are monitored and evaluated by graduate student and Teams are monitored and evaluated by graduate student and

faculty observersfaculty observers

•Teams work to meet organizational goals like safety, on-time Teams work to meet organizational goals like safety, on-time

performance, customer satisfaction, and disruption performance, customer satisfaction, and disruption

managementmanagement

• Teams are given various scenarios (or Teams are given various scenarios (or triggerstriggers) that require ) that require

the students to work together to resolve issues quickly and the students to work together to resolve issues quickly and

effectivelyeffectively

• Teams are monitored and evaluated by graduate student and Teams are monitored and evaluated by graduate student and

faculty observersfaculty observers


FF OO CC UU SSlightlight

perationsperations

enterenter

nifiednified

imulationimulation


•FOCUS is a high-fidelity replication of a FOCUS is a high-fidelity replication of a part 121 regional airline operations part 121 regional airline operations control centercontrol center

• Students participate in a 3 hour Students participate in a 3 hour simulated work shift working for a virtual simulated work shift working for a virtual airline we call “Universal E-Lines”airline we call “Universal E-Lines”

• Universal E-Lines operates 30 CRJ-200 Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout aircraft on 16 flight routes throughout the southeastern United States.the southeastern United States.

• Utilizing a hub and spoke system, Utilizing a hub and spoke system, Nashville and Jacksonville act as the Nashville and Jacksonville act as the airline’s hubsairline’s hubs

• Students manage up to 80 flight events Students manage up to 80 flight events in a 3 hour simulation sessionin a 3 hour simulation session

•FOCUS is a high-fidelity replication of a FOCUS is a high-fidelity replication of a part 121 regional airline operations part 121 regional airline operations control centercontrol center

• Students participate in a 3 hour Students participate in a 3 hour simulated work shift working for a virtual simulated work shift working for a virtual airline we call “Universal E-Lines”airline we call “Universal E-Lines”

• Universal E-Lines operates 30 CRJ-200 Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout aircraft on 16 flight routes throughout the southeastern United States.the southeastern United States.

• Utilizing a hub and spoke system, Utilizing a hub and spoke system, Nashville and Jacksonville act as the Nashville and Jacksonville act as the airline’s hubsairline’s hubs

• Students manage up to 80 flight events Students manage up to 80 flight events in a 3 hour simulation sessionin a 3 hour simulation session

WHAT IS FOCUS?WHAT IS FOCUS?

CULMINATING EXPERIENCESCULMINATING EXPERIENCESPROJECT CONCEPTPROJECT CONCEPT

CULMINATING EXPERIENCESCULMINATING EXPERIENCESPHOTOSPHOTOS

CULMINATING EXPERIENCESCULMINATING EXPERIENCESThere are 10 distinct There are 10 distinct positions in the FOCUS Lab. positions in the FOCUS Lab. They are:They are:

• Flight Ops. CoordinatorFlight Ops. Coordinator

• Weather & ForecastingWeather & Forecasting

• Crew SchedulingCrew Scheduling

• Fuel/Cargo Management*Fuel/Cargo Management*

• Flight Tracking & Schedule Flight Tracking & Schedule

Management* Management*

• Maintenance PlanningMaintenance Planning

• Maintenance ControlMaintenance Control

• Ramp TowerRamp Tower

• Pseudo PilotPseudo Pilot

• CRJ Simulator CrewCRJ Simulator Crew

WX & WX & ForecastingForecasting

Crew Crew SchedulingScheduling

MX MX ControlControl

MX MX Planning & Planning & SchedulingScheduling



The MTSU and its The MTSU and its industry partners industry partners pioneered the ability to pioneered the ability to electronically link to a electronically link to a CRJ-200 simulator CRJ-200 simulator located at the flight located at the flight school. Pilots in this school. Pilots in this simulator can simulator can communicate with communicate with anyone in the operations anyone in the operations center. The CRJ simulator center. The CRJ simulator even appears on flight even appears on flight tracking monitors and is tracking monitors and is displayed in the ramp displayed in the ramp tower as a photo realistic tower as a photo realistic aircraft moving about the aircraft moving about the airport environment. airport environment.

PHOTOSPHOTOS

CULMINATING EXPERIENCESCULMINATING EXPERIENCESSCENARIO DESIGN & CONTROLSCENARIO DESIGN & CONTROL

Scenario Design & Control:Scenario Design & Control:

• Graduate StudentsGraduate Students

• Monitoring team performance Monitoring team performance and status of the operationand status of the operation

• Provide real-world feedback Provide real-world feedback and consequencesand consequences

• Issue disruptive ‘triggers’Issue disruptive ‘triggers’

• LibraryLibrary

• TaxonomyTaxonomy

• Disruptive potential scaleDisruptive potential scale

• Monitor response to triggersMonitor response to triggers

• Evaluate effectivenessEvaluate effectiveness

CULMINATING EXPERIENCESCULMINATING EXPERIENCESAFTER-ACTION AFTER-ACTION

REVIEWREVIEWThe Aerospace Department The Aerospace Department collaborates with collaborates with the Psychology Departmentthe Psychology Department on the FOCUS on the FOCUS project. Studies include:project. Studies include:

• Occupational stereotypes & attitudesOccupational stereotypes & attitudes

• Group dynamicsGroup dynamics

• Individual & team cognitionIndividual & team cognition

After- Action Review:After- Action Review:

• Engages students to discuss events of Engages students to discuss events of simulationsimulation

• Feedback from team members and observersFeedback from team members and observers

• Discussion of unfavorable outcomesDiscussion of unfavorable outcomes

• How to prevent in the future?How to prevent in the future?

• Anticipating problemsAnticipating problems

• Communication & time managementCommunication & time management

• Reinforces positive behaviors and solutionsReinforces positive behaviors and solutions

• Teams establish goals and new strategiesTeams establish goals and new strategies

CULMINATING EXPERIENCESCULMINATING EXPERIENCESADVANTAGESADVANTAGES

• Active learning experienceActive learning experience

• Students provide performance feedback to peersStudents provide performance feedback to peers

• Has the potential to reveal holes in curriculum Has the potential to reveal holes in curriculum

• Encourages faculty to incorporate a greater ‘real-world’ Encourages faculty to incorporate a greater ‘real-world’ perspective in the classroomperspective in the classroom

• Allows students to use knowledge gained from their Allows students to use knowledge gained from their coursework to creatively solve problemscoursework to creatively solve problems

• High motivation to succeed and learn from High motivation to succeed and learn from mistakesmistakes

• Students learn to consider multiple solutions Students learn to consider multiple solutions to a problem, weighing likelihood, impact, and to a problem, weighing likelihood, impact, and other downstream implicationsother downstream implications

• Gives students a sense of time sensitivity Gives students a sense of time sensitivity while problem solvingwhile problem solving

• Internship & job opportunitiesInternship & job opportunities

• Students enjoy it!Students enjoy it!

CULMINATING EXPERIENCESCULMINATING EXPERIENCESQUESTIONSQUESTIONS

t h a n k y o ut h a n k y o u

Dr. Paul A. CraigDr. Paul A. Craig

Ms. Andrea GeorgiouMs. Andrea Georgiou

Mr. Paul R. CarlsonMr. Paul R. Carlson

withwith

Dr. Glenn LittlepageDr. Glenn Littlepage

Dr. Mike HeinDr. Mike Hein

Dr. Richard MoffettDr. Richard Moffett

Mr. Gerald HillMr. Gerald Hill

[email protected]@mtsu.edu


[email protected]@mtmail.mtsu.ed

uu

[email protected]@mtsu.ed

uu





Thank You UND & MTSU Culminating Experience Assessment

Guidance Workshop July 2013 Kansas City

slide 1 assessments and culminating experiences institutional workshop mr. bill kohlruss-dr. mark...

Documents

program educational

program missiondefining

program education objectives

educational programs

student learningassessing

assessment processestablishing

beth bjerke

understanding of professional