Accreditation Visit to<name of HEI><date range>

Name of PresenterTitle

Welcome!• Outline of this presentation

• Background• General notes on accreditation • Goals of the CEAB• Objectives of the visit• Team’s responsibilities• Tasks and tools• Visit schedule• Accreditation criteria highlights• NEW! Graduate attribute criteria• Potential issues• Post-Visit Activities• Visit-Specific Issues

Background• History and current situation

• <when HEI opened>• <current student complement>• <any major contextual factors>

• Purpose of this visit• <list programs being visited>• <indicate when these programs were most recently visited and what the CEAB

decisions was>

General notes about accreditation• Accreditation applies only to programs, not to departments or faculties

• Undertaken only at the invitation of the HEI and with the consent of the appropriate regulator

• Accreditation constitutes:• Quantitative and qualitative evaluation of the curriculum• Qualitative evaluation of the program environment

• Accreditation is granted for a period of time up to and normally not exceeding six years

Goals of the CEAB

• Engineering programs offered by Canadian institutions will meet or exceed minimum educational standards acceptable for professional engineering licensure in Canada

• The quality and relevance of engineering education will continuously improve

• The Engineers Canada Board of Directors will be provided with advice and recommendations on international matters relating to engineering accreditation and education

Objectives of the visiting team• Conduct fact-finding on behalf of the Accreditation Board

• Review, validate and/or add to the information provided by the host institution

• Review of materials, meetings, and facility tours to corroborate program strengths and weaknesses and bring forward issues to the CEAB

• Describe progress toward use of graduate attributes in program assessment and improvement

• Collaborate in preparing a report of the team’s findings

• The visiting team or its members do not make any recommendations ~ accreditation decisions are made by the CEAB

Team’s responsibilities• Thoroughly read the institution’s completed questionnaire

• Identify issues for investigation during the visit

• Attend the visit

• Participate in team discussions

• Complete your report (before the exit meeting!)

• Be available to answer questions after the visit and before the CEAB’s decision meeting

Team’s responsibilities ~ cont’d.• The confidential nature of the process

• Check everything you question with the program coordinator or other responsible person ~ don’t assume!

• Be sure to get the names of all the people that you interview. Circulate an attendance list.

• Visitors should not get into giving suggestions

• All issues that are to appear in the final report must be brought up in the exit meeting ~ i.e. there must not be any surprises in the final report.

• We must agree on all issues to be raised and I must have a clear understanding of them.

• Do not use the terms “concern”, “weakness” or “deficiency”. In the accreditation process, these terms have very specific meanings (accreditation jargon!).

Tasks and tools

• Interviews with appropriate senior administrative officers, including the president, the dean of engineering and the chairs of the departments responsible for the programs

• Interviews with individuals and groups of faculty members to evaluate:• professional attitudes• motivations• morale• the balance of opinions concerning theoretical and practical elements of the curriculum

• Interviews with individuals and groups of students. Ask open-ended questions to get them talking

• Examine compliance with graduate attribute criteria

Tasks and tools ~ cont’d.• Tours of physical facilities such as laboratories, libraries, and computing

facilities, to evaluate their effectiveness

• Note that the Accreditation Board does not require any Faculty to spend money - the question is whether the equipment, supplies, etc. are adequate

• A review of recent examination papers, laboratory instruction sheets, student transcripts, student reports and theses, models or equipment constructed by students and other evidence of student performance

• Are performance expectations and grading standards appropriate?

Tasks and tools ~ cont’d.• Attendance form ~ for meetings and interviews

• AU re-allocation form

• Sample questions for faculty and staff

• Sample questions for students• Interpretive Statements and Regulations

• Interpretive statement on natural sciences• Interpretive statement on licensure expectations and requirements• Regulations for granting transfer credits (effective January 2012)

Visit SchedulePre-visit Activities• All team members review the written material submitted by the host institution, and

spend time reviewing graduate attributes

• Team members prepare a list of potential issues and a preliminary draft of their contribution to the report.

• Initial team meeting. Sometimes the Visiting Team Chair schedule a Saturday Team meeting pre-visit.

• A detailed schedule for Sunday to Tuesday has been prepared for each team member

Visit Schedule ~ cont’d.Visit – Day 1 (Sunday)• Lunch with institution officials• In the afternoon, the team visits the institution to review course

materials including consideration of graduate attribute compliance• Team dines together• Team meets in the evening ~ agenda includes discussion of:

• observations and findings of the day• potential issues and how to investigate further• previous decision issues and areas to be re-examined

Visit Schedule ~ cont’d.Visit – Day 2 (Monday)• Plenary Session with the Team, Dean, Program Chairs / Co-ordinators:

• Introductions and purpose (fact-finding for the CEAB)• Process and time lines

• Conduct visit

• Lunch with administrators and faculty members

• Continuation of visit

• Team dinner and meeting• Discussion and preliminary consensus regarding issues• List areas of strength and list issues that require further investigation

Visit Schedule ~ cont’d.Visit – Day 3 (Tuesday)• Update information with Dean, Program Coordinators• Revise visit schedule as necessary• In camera Team Working Lunch:

• Complete draft reports• Review reports, arrive at consensus on final conclusions• Draft copies of each report provided to the chair

Visit Schedule ~ cont’d.Visit – Day 3 (Tuesday) ~ cont’d.• Exit meeting with Dean, Program Chairs, Faculty (verbal

presentation by the Team Chair)– Restate - Visiting Team’s role is fact-finding– Accreditation decisions are made only by the Accreditation Board– Repeat time-line of the process to follow– Emphasize confidentiality– Summarize all issues and state strengths– Thank the Dean and staff for visit arrangements and their hospitality

Timeline after visit- Chair submits report to CEAB Secretariat- Report is edited, formatted and returned with any questions to chair- Chair may contact team members with questions- Report finalized, sent to institution- Institution responds and sends update- Accreditation decision made (June or Sept mtg)- Institution and Team members notified of decision (within month)

If you see an issue with a program- Visit Team is on a fact-finding mission- Institution’s documentation will emphasize the positive but your direct

observation may differ- You need to verify documentation and identify discrepancies if any, to inform

CEAB decision- Add something about editing process.- If there is an issue, the institution still has multiple opportunities to address it

and improve- Do not hesitate to dig for the full picture and describe it accurately in your report

Accreditation Decisions2014-2015 Cycle

There are now 279 accredited programs at 43 Higher Education Institutions in CanadaAnd 14 substantially equivalent programs in 3 HEIs outside of Canada

0

5

10

15

20

25

30

35

40

6V 3R 3V 3T 2R 2V AE

Accreditation Criteria Highlights• The processes of accreditation place emphasis on the quality of the

curriculum, the students, the academic staff, support staff, facilities, and resources

• The accreditation criteria:• Reflect the need for engineers to be adaptive, creative, resourceful, and responsive

• Ensure that the graduates understand the role and responsibilities of professional engineers to society

• Reflect the need for the professional engineer to function as an effective member of a team and to communicate effectively

Accreditation Criteria Highlights cont’d.Qualitative Evaluation – Curriculum Considerations

• Curriculum must include the application of computers and appropriate laboratory experience and safety procedures

• Students must be exposed to material dealing with professionalism, ethics, equity, public and worker safety and health considerations, concepts of sustainable development, environmental stewardship

• The Curriculum must prepare students to learn independently and to work as an effective member of a team

Accreditation Criteria Highlights cont’d.Qualitative Evaluation – Curriculum Considerations

• Curriculum must include studies in:– communication skills– engineering economics – impact of technology on society – subject matter that deals with central issues, methodologies and thought

processes of humanities and social sciences, and; – must culminate in a significant design experience

Accreditation Criteria Highlights cont’d.Qualitative Evaluation – Curriculum Considerations

• Engineering Design:– integration of curriculum elements – creative, iterative and open-ended– subject to constraints imposed by legislation or standards– to satisfy specification using optimization– economics should be part of the design experience– to be supervised by licensed engineers

Every program must culminate in a significant design experience

Accreditation Criteria Highlights cont’d.Quantitative Evaluation

Accreditation units (AU) • one hour of lecture (corresponding to 50 minutes of activity) = 1 AU • one hour of laboratory or scheduled tutorial = 0.5 AU

Accreditation Criteria Highlights cont’d.

Accreditation Criteria Highlights cont’d.• In reviewing course information and course materials, check

reasonableness of AU allocations ~ not an exact science!• AU re-allocations should be team decisions, after discussion

• We will discuss re-allocations, if any, on Sunday evening, and again on Monday evening

• Discuss allocations with responsible faculty member, but no need to argue

• Agree to disagree

Accreditation Criteria Highlights cont’d.Professional Licensure

• Dean, Department Chairs, and faculty members teaching courses that are primarily engineering science and engineering design are expected to be licensed to practice engineering in Canada

• minimum of 225AU of ED to be instructed by P.Eng./ing.• minimum of 600 AU of ES+ED to be instructed by P.Eng./ing. or 'Applied’• Interpretive Statement as guidance

• Curriculum development and control should be in the hands of persons licensed to practice engineering in Canada

Accreditation Criteria Highlights cont’d. – Graduate Attributes

Criterion 3.1 Graduate Attributes• Two components• Attributes:

– Interpreted at time of graduation– Recognized that achievement does not end there

• Continuous Improvement:– Ongoing evolution of engineering programs – Processes needed

> Assessment of attribute achievement> Results used to improve program

Graduate Attributes

Accreditation Criteria Highlights cont’d. – Graduate Attributes

Criterion 3.1 Graduate AttributesTimeframe:• “While programs are expected to provide evidence to demonstrate compliance

with this criterion, a transition and development period will be allowed.

Graduate Attributes

Accreditation Criteria Highlights cont’d. – Graduate Attributes

1. Knowledge Base

2. Problem Analysis

3. Investigation

4. Design

5. Use of Engineering Tools

6. Individual and Team work

7. Communication

8. Professionalism

9. Impact on Society and the Environment

10. Ethics and Equity

11. Economics and Project Management

12. Life-Long Learning

Graduate Attributes

Graduate Attributes ~ ExamplesThe institution must demonstrate that the graduates of a program possess the attributes under the following headings:

3.1.1 A Knowledge Base for Engineering: Demonstrated competence in university level mathematics, natural sciences, engineering fundamentals, and specialized engineering knowledge appropriate to the program.

3.1.8 Professionalism: An understanding of the roles and responsibilities of the professional engineer in society, especially the primary role of protection of the public and the public interest.

Graduate Attributes

Graduate Attributes progress ~ Evaluation by Program Visitors

The Program Visitor must evaluate progress toward each graduate attribute to fill out the report template:Program Visitor’s Observations on Implementation Evaluate the evidence and actions (either seen on-site or in the questionnaire) proposed to demonstrate the level of achievement of each graduate attribute

Program Visitor’s Observations on Implementation Evaluate the evidence and actions (either seen on-site or in the questionnaire) proposed to demonstrate the level of continual improvement achievement

Graduate Attributes

Graduate Attributes progress ~ Evaluation by Program Visitors

• Things the Program Visitor will need to see:– Graduate Attributes (Accreditation Criteria)– Learning outcomes that support Graduate Attributes– Indicators – Acceptable levels– Feedback mechanism

Graduate Attributes

Graduate Attributes: Evaluation by HEI

The program is assessed, not the students

Continuous improvement process

Not required to assess every student

Not required to assess in every course

Not required to assess every year

Graduate Attributes

Continuous Improvement – The big picture

HEIs evaluate:• Are students meeting expectations?

– In what areas are they successful

– What areas require improvement• What data would help institution improve their program?

Graduate Attributes

What to look for: Program background

• Is the program clearly described?– Is there a curriculum map?

• Is the context of the program clear?

Graduate Attributes

What to look for:Curriculum Mapping

• Information in the curriculum map is – Accurate, with some depth– Identifies intended outcomes from learning experiences– Not simply a list of topics “covered”

• Map provides information for each attribute– Can include curricular and other experiences

Graduate Attributes

Methodology: Data Collection Plan • On what does the program propose collecting data (i.e. indicators)?

• What methods are proposed for collecting data?

• Is the data collection plan good?

Graduate Attributes

Terminology for Data Collection (1)•Valid Indicators• measure what they are supposed to measure

•Reliable Indicators• the results are consistent; the measurements are the same when repeated with

the same subjects under the same conditions

Graduate Attributes

Terminology for Data Collection (2)Direct measures • directly observable or measurable assessments of student learning

Indirect measures

• opinion or self-reports of student learning or educational experiences

Use both direct and indirect measures if possible.

Graduate Attributes

Data Collection - Indicators • An indicator is like a sensor: what indicators has the program chosen? • Where have they placed their indicators? Where are the data collection points?• Does the proposed data collection plan make sense?

Graduate Attributes

What to look for: Overall - data collection plan

• Integrity:– How good is the quality of the data collection plan

• Are Indicators well chosen?• Are assessment points well chosen?

– Is valid, reliable data collection proposed?

– Is plan cyclic, continuous?

• will results be useful for informing curriculum change? Ask the question: “why are you collecting this data?”

Graduate Attributes

What to look for:Indicators in data collection

• Indicators align with attributes and questions

• Indicators are “leading indicators”: central to attribute; indicate competency

• Enough indicators defined to identify strength areas and weak areas within an attribute

• Not too many indicators – resulting in reams of data but little deep information

• Indicators are clearly articulated and measurable

Graduate Attributes

Selecting Assessment Points

• Learning is generally demonstrated through:– Artifacts, e.g. written test, report, built project– Performances, e.g. oral presentation, observed

practice

Graduate Attributes

What to look for: Assessment Points

• Indicators are well aligned to the proposed assessment points

• Enough assessment points are utilized• Expectations of performance quality are clear,

i.e. the scale is defined

Graduate Attributes

What to look for: Triangulation in improvement process• Are opportunities included for informal assessment, students’ self-reports of

learning, and even unsolicited data from placement supervisors or employers?

• Are more than one type of assessment used when analyzing data?

• Are all assessments valued, not just major events?

• Are the data gained from assessment used to answer questions about authentic learning?

• Are data across time intervals looked at?

Graduate Attributes

Measurement Tools ~ Illustrative Examples

• Examinations– Final– Mid-term– Entry and exit– Standardized (PPE, FE)– Oral– Embedded questions

• Portfolios– Culminating design experience– Projects– Laboratories– Internship/stage– Co-op

• Surveys– Exit– Alumni– Employers– Self– Course Evaluations– Advisory Board

• Student Work– Reports– Peer Reviews– Reviews/critiques– Presentations– Posters

Graduate Attributes

Quality Improvement Loop• 3.1 Graduate attributes

– Engineering programs are expected to continually improve – There must be processes in place that demonstrate that program outcomes are

being assessed in the context of the attributes– And, that the results are applied to the further development of the program.

Thus,If observed outcomes are not consistent with expected attributes, then system inputs and/or process must be adjusted

Graduate Attributes

What to look for: Evidence of Feedback Loop

• Results are consolidated for each learning outcome.• Determination is made regarding whether learning outcome is met.• Results of assessment are used to determine if changes need to be made in curriculum, courses, prerequisites, performance criteria or metrics.• Change is implemented• Assessment is repeated to determine effect of change.

Graduate Attributes

Continuous ImprovementProgram

Objectives and

Expected Outcomes

Curriculum

Assessment Processes

andMetrics

DataCollection

andEvaluation

Graduate Attributes

Criterion 3.1 ~ 2009 Findings• Most programs at beginning stages of developing GA

measurement tools and data management systems• Some examples:

• Mapping of GAs to curriculum• Structuring activities to address specific GA aspects

Graduate Attributes

Criterion 3.1 ~ 2011-2012 Cycle Findings• Accreditation site visits in 2011-2012 assessed state of

progress toward Graduate Attribute compliance.• Out of 17 institutions, based on present progress:

– 7 were rated “will fail to reach compliance by 2014”– 3 were rated “likely to fail to reach compliance by 2014”– 7 were rated “on track to comply by 2014”

Graduate Attributes

Criterion 3.1 and Criterion 3.2 ~ 2012-2013 Cycle Findings

• Accreditation site visits in 2012-2013 assessed state of progress toward Graduate Attribute compliance.

• Out of 15 institutions, based on present progress:– 6 were rated “not likely to be in compliance with Criteria 3.1 and 3.2 by 2014”– 2 were rated “likely to be in compliance with Criteria 3.1 and 3.2 by 2014”– 7 were rated “on track to comply with Criteria 3.1 and 3.2 by 2014”

Criterion 3.1 and Criterion 3.2 ~ 2013-2014 Cycle Findings

• Accreditation site visits in 2014-2015 assessed state of progress toward Graduate Attribute compliance.

• Out of 14 institutions, based on present progress:– 6 were rated “not likely to be in compliance with Criteria 3.1 and 3.2 by 2014”– 7 were rated “on track to comply with Criteria 3.1 and 3.2 by 2014”

2015 Decision Meetings Summary

– 13 higher education institutions (HEI)– 57 program decisions

2015 Decision SummaryDecision Description Number

6V Visit after 6 years 39 68%3R Report after 3 years 12 21%3V Visit after 3 years 1 2%3T Termination after 3 years 1 2%2R Report after 2 years 2 4%2V Visit after 2 years 1 2%AE Accreditation extended for 1 year 1 2%

57

Percentage

89%

100%

11%

2015 Summary of Issues

• Comments were used to clarify CEAB expectations

Deficiencies Weaknesses Concerns Comments Total Notes

Issues raised before meeting 26 73 97 53 249Issues raised by visiting teams and AB lead reviewers

Issues resolved 4 27 29 0 60

Issues considered resolved by AB following HEI feedback

Issues unresolved 22 46 68 53 189 Comments are for information

Percentage issues resolved 15% 37% 30% 0% 24%

2015 Analysis of Issues

• Comments were used to clarify CEAB expectations

Criterion # Short Name Deficiencies Weaknesses Concerns Comments Total Notes

3.1 Graduate Attributes 3 5 1 0 9 Few issues overall good compliance

3.2 Continual Improvement 3 10 45 53 111Comments and concerns mostly formative feedback

3.3 Students 0 7 4 0 11 Few issues overall good compliance

3.4 Curriculum 2 12 1 0 15 Few issues overall good compliance

3.5 Program Environment 13 11 17 0 41Deficiencies mostly related to licensure criterion 3.5.5

3.6 Procedures and Processes 1 1 0 0 2 Few issues overall good compliance

Total 22 46 68 53 189

Questionnaire ChangesFor the 2015 – 2016 cycle the major changes to the questionnaire were:• Major revision of the course information sheet (CIS)• Minor revision of Table 3.1.1. Graduate Attribute Map• Elimination of most exhibits • Replacement of Tables 3.1.3 Assessment tools 3.1.4

Assessment results and 3.2.1 Continual improvement with new “free-format” Exhibit 1

59 of X

Course Information SheetMajor changes for the 2015-2016 cycle were:• Web link for calendar description added

- This is now the reference used for content• Course type flags added

- X required - E elective -M minimum-path elective• Content codes reduced

- To I or D or A (from I or D or A or IA or ID or DA or IDA)

• Required professor/instructor details expanded- Ph.D., academic rank, licensure, etc.

• Instructional hours details modified- Instructions unclear – will be improved for the next cycle

• Lecture and lab content topics replaced by learning outcomes*- Lecture/Lab learning outcomes to be merged next cycle

60 of X

Potential Issues that visit can reveal• Large classes• Faculty numbers ~ “faculty who are not faculty”

• Long-term leaves and long sabbaticals (counted at institution, but not available to teach)• Soft-funded faculty• Teaching loads ~ critical dependence on a single individual

• Course failure rates• Check the course information sheet to see if the numbers make sense (tough courses should

have higher failure rates)

• Students pass while failing• Attrition rate

Potential Issues cont’d.• Grading of major written reports• Appropriateness of AU allocations ~ especially for Engineering Design and Complementary

Studies• Plans for renewal of equipment• Design experience• Access to Dean’s office, Program Director's office, etc.• Notional contact hours• Admissions:

• Practices followed for granting advanced standing or transfer credits? • Who authorizes exceptions? • Is control rigorous?

Report Expectations• All issues on the visit

• Must be tied to criteria

• Finalized by Monday night

• Complete report to extent possible by Tuesday p.m.

• Will be used to make exit statement

• Recall ~ all issues need to be raised at the exit meeting

• Your reports transferred to my computer before you go

Post-Visit Activities• Chair prepares a draft team report

• Incorporates Program Visitors' reports• Target ~ within 2 weeks of Visit

• Distributed by e-mail to all team members for review and comment • Target ~ comments returned within 4 weeks of Visit

• Report edited by CEAB Executive Committee member• Edited report sent to HEI for review and comment• CEAB relies on report + HEI’s response to make decision

Post-Visit Activities cont’d.• All team members will be advised of decision• Opportunity to be acknowledged by CEAB for your contribution• Evaluation

• HEI evaluates the visit process• Team Chair evaluates team members• Team members evaluate Team Chair and process (currently this is a pilot project.

Not all teams do this)• General Visitor provides report on visit process to association of jurisdiction

where institution is located

Visit-Specific Issues

• Insert per pre-visit summary of issues list

For more information:300-55 Metcalfe Street, Ottawa, Ontario K1P 6L5

Tel. 613-232-2474 / Fax. 613-230-5759engineerscanada.ca

The road to a P.Eng. begins with the right education

P. ENG.*