university of xyz
DESCRIPTION
University of XYZ. ?? Programs to be evaluated ?? member team ?? observers, ?? co-chairs, ?? PEV’s We’ll need to stay focused!. Important Team Expectations. Evaluators represent the EAC of ABET We are accrediting programs, certifying that they satisfy the criteria - PowerPoint PPT PresentationTRANSCRIPT
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
1
June 10, 2009
University of XYZ
• ?? Programs to be evaluated
• ?? member team– ?? observers, ?? co-chairs, ?? PEV’s
• We’ll need to stay focused!
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
2
June 10, 2009
Important Team Expectations• Evaluators represent the EAC of ABET• We are accrediting programs, certifying that
they satisfy the criteria• Team effort—team decisions• Confidentiality• Conflict of interest—every visitor should have
signed a conflict-of-interest statement• Observers
– no evaluative statements– exit interview—thank-you only
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
3
June 10, 2009
Confidentiality• Do not discuss conclusions with faculty,
students, and others• Keep all materials until the July 2010 EAC
meeting. At conclusion of accreditation process (July 2010) materials are to be destroyed
• Information specific to the institution is to remain confidential without time limit
• Institutional data are confidential except with written authorization of institution
• ABET materials only released by ABET staff
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
4
June 10, 2009
Communication
• Maintain open line of communication with the department head
• Identify deficiencies as soon as possible• Discuss all issues with the department head at
the debriefing • Do not discuss the recommended accreditation
action with anyone except team members
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
5
June 10, 2009
General Visit Outline• Today –
– Visit campus, review curriculum materials– Tonight share what we’ve learned, plan
• Tomorrow – – AM: Dean’s overview, PEV’s in departments– Lunch with institutional officials– PM: Support areas, PEV’s in departments– Evening: Share what we’ve learned, tentative conclusions;
draft exit statements• Tuesday –
– Tie up loose ends– Draft exit statements by late morning– Debrief department heads before lunch– Exit meeting at 2PM
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
6
June 10, 2009
Support Program AssignmentsBiology 1:30 Mon
Chemistry 1:30 Mon
Comp Sci 1:30 Mon
English 1:30 Mon
Math 1:30 Mon
Physics 1:30 Mon
Statistics 1:30 Mon
Gen Ed 1:30 Mon
Library 1:30 Mon
Office of Info Tech 1:30 Mon
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
7
June 10, 2009
Visit details• Important forms – done by Tues AM
– Transcript & curriculum analyses – Program Audit Form + Explanation of Shortcoming (two
hard copies to Team Chair—one will be left with Dean after Summary Meeting with Administration—one soft copy to Team Chair)
– Exit Statement—soft copy to Team Chair– Program Evaluator Worksheet—soft copy to Team Chair– Form A2a - Short form—recommended actions—
prepared by Team Chair on basis of information from evaluators
– Program Evaluator Report—soft copy to Team Chair
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
8
June 10, 2009
Terminology• Deficiency – criterion, policy, or procedure is
NOT satisfied.• Weakness – criterion, policy, or procedure is
satisfied, but lacks strength of compliance to assure the quality of the program will not be compromised prior to next general review.
• Concern – criterion policy, or procedure is satisfied, but potential exists for non-satisfaction in the near future.
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
9
June 10, 2009
Working Definition of Key Terms
• Deficiency: assigned to any criterion, policy, or procedure that is totally or largely unmet
• Weakness: criterion, policy, or procedure is met to some meaningful extent, but compliance is insufficient to fully satisfy requirements
• Concern: criterion, policy, or procedure is fully met, but there is potential for non-compliance in the near future
• Observation: general commentary possibly, but not necessarily, related to criteria
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
10
June 10, 2009
Limit Use of Key Terms• Use Key Term only in reference to overall
evaluation of each criterion
• The Key Term (defined in the previous slides) is the overall assessment for the criterion as a whole, not the worst finding among the sub-areas on the worksheet. Do not, for example, give a deficiency to a program that lacks only a measurement for outcome 3(e)
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
11
June 10, 2009
Consistency Issues for the Team
• Depth and completeness of the evaluation from program to program
• Consistency across all programs in an institution
• Assignment of appropriate key terms (deficiency, weakness, concern) to describe shortcomings
• For weaknesses, consistency on interim recommendations—IR vs. IV
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
12
June 10, 2009
Applicable Criteria• Use Criteria for 2009-2010 Accreditation
Cycle. These are the same as the renumbered criteria used in 2008-2009
• If this is an interim visit, the criteria in effect at the time the prior shortcomings were identified should be used, unless it is to the program’s benefit to use the currently applicable ones, and it is the program’s choice.
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
13
June 10, 2009
Application & Interpretation of Criteria
In 2008-09, EAC editors observed that:
• Issues arose related to the linkage between Criterion 2, Criterion 3, and Criterion 4
• Criterion 2 and new programs
• Criterion 4 and new programs
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
14
June 10, 2009
What Does Criterion 2 Say?
The program must have in place:• Published PEO’s consistent with mission and
these Criteria • Process that periodically documents and
demonstrates that the PEO’s are based on the needs of the program’s constituencies
• An assessment and evaluation process that periodically documents and demonstrates the extent to which these objectives are attained
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
15
June 10, 2009
Criterion 2 Highlights1. The process needs to document and
demonstrate that the PEO’s are based on constituent needs – NOT “a process based on the constituents needs in which PEO’s are determined and evaluated” (the old language)
2. Requirement for assessment and evaluation is the same, but there is no longer language in Criterion 2 that requires that the results of the assessment and evaluation process be used for program improvement. (Program improvement is now in Criterion 4.)
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
16
June 10, 2009
Consistency IssuesCriterion 2:
– Do the published PEO’s meet the definition?– Does the program convince the team that the PEO’s
are based on constituent needs?• Notice that there is no language that insists on constituent
approval or involvement! • The program does need to convince the team that it has a
way of determining what the needs of its constituencies are.
– Is there an assessment and evaluation process in place that gives info about the extent to which PEO’s are attained by grads
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
17
June 10, 2009
Criterion 2 Problem in 08-09• Several instances in which programs were being
required to show how results of C2 or C3 assessment and evaluation processes were being used to improve the program.
• This reflects application of old criteria.
• That's not in C2 or C3 any more. C4 refers to results of C2 and C3 processes as possible sources of information upon which continuous improvement could be based.
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
18
June 10, 2009
Criterion 2 FAQ’s• What if the PEO’s really sound like outcomes
(instead of objectives?– If PEO’s are not PEO’s, there is a C2 shortcoming.
• What if PEO’s are ambiguous or reflect outcomes retooled to apply after graduation?– Team judgment – do they meet the intent of the
Criterion?• Is an assessment process for PEO’s that
considers predominately data based on accomplishments of current undergraduates adequate? – Probably not
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
19
June 10, 2009
What Does Criterion 3 Say?
• The program must demonstrate that (a) – (k) are attained
• Program outcomes are defined as (a) – (k) plus any additional ones articulated by the program
• Program outcomes must foster attainment of the PEO’s
• There must be an assessment and evaluation process that periodically documents and demonstrates the degree to which outcomes are attained
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
20
June 10, 2009
Important – for Criterion 3• The definition of program outcomes as being (a) – (k)
plus locally articulated ones– The program may not have its outcomes expressed as (a) – (k)
plus others. It may have just identified a set of outcomes. As long as the program has demonstrated attainment of (a) – (k) and its own outcomes, this element of the criterion is met.
• Requirement for assessment and evaluation is the same, but there is no language in Criterion 3 that results of the assessment process be applied to further development of the program. (Program improvement is now in Criterion 4.)
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
21
June 10, 2009
Criterion 3 Consistency Issues• Be sure to apply this criterion in a holistic sense• The process of assessment and evaluation needs to
demonstrate the degree to which outcomes are attained, but …– There is no language that says all outcomes must be
attained to the same degree– There is no language that says anything about a numeric
scale measuring degree of attainment– There is no language that says the outcomes must be
measured– There is nothing in Criterion 3 that says anything about
use of the assessment and evaluation information for program improvement
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
22
June 10, 2009
Criterion 3 FAQ’s
• What about assessment data? What is adequate data? – Does it all have to be objective/direct? (NO)– Can it be subjective? (Some of it may be; nothing
says it has to be)– Is the observation or conclusion of course instructor
adequate? (What was his or her basis for the observation?)
– Does evidence for each outcome have to be in the form of work the student has produced? (No, but the team needs to be convinced that outcome attainment has been demonstrated.)
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
23
June 10, 2009
What Does Criterion 4 Say?
“Each program must show evidence of actions to improve the program. These actions should be based on available information, such as results from Criterion 2 and Criterion 3 processes.”
• The improvements can be based on any available information!
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
24
June 10, 2009
Consistency Issues• The language of Criterion 4 simply insists on
evidence of action to improve the program.– Such actions could be stimulated by results of the C2
and C3 assessment and evaluation processes– But they could also be stimulated by other information
• The language of this Criterion does not require that the C2 and C3 information be used as the basis for program improvement. It suggests use of the results of C2 and C3 processes as sources of information for program improvement.
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
25
June 10, 2009
Deliverables
• From each PEV, the team chair must have:– A recommended action relative to the program under
review that is consistent with the team’s conclusions (remember, it is a TEAM recommendation)
– A PAF that accurately reflects the team findings– An Exit Statement that will:
• Be read verbatim in the Exit Meeting• Be used as the basis for construction of the draft statement
to the institution
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
26
June 10, 2009
Program Audit Form (for GR or New)(We provide a copy to the institution at the Exit Meeting)
If heading doesn’t have the current visit year, you are using an old form!
Do not use old forms!!
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
27
June 10, 2009
Program Audit Form (for Interim Report Evaluations)
We provide a copy to the institution at the Exit Meeting for IV only
If the heading does not say
INTERIM REPORT
do not use it for an IR evaluation
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
28
June 10, 2009
Program Audit Form (for Interim Visit Evaluations)
We provide a copy to the institution at the Exit Meeting for IV only
If the heading does not say
INTERIM VISIT
do not use it for an IV evaluation
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
29
June 10, 2009
Exit Statement Format• INTRODUCTION—USEFUL
PROGRAM STATISTICS• PROGRAM ISSUES
– Strengths (special, unique or particularly conspicuous strengths standing above the norm)
1.
2.
– Deficiencies (In order, only for those criteria where deficiencies exist)
1. XXX
2. etc.
– Weaknesses (In order, only for those criteria where weaknesses exist)
1. YYY
2. etc.
– Concerns (In order, where concerns exist)
1. ZZZ
2. etc.
– Observations (do not have to relate to criteria)
1. etc.
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
30
June 10, 2009
Writing the Exit Statement• For each shortcoming – use the following
structure for your statement of finding:– What is required:
• Describe what criterion or policy is applicable• Describe what that criterion or policy requires
– What was observed:• Describe what was observed• Describe how your finding violates criterion or policy
– Negative Impact• Describe the negative impact it has on the program (for D
or W) or the current or potential future impact on the program (C)
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
31
June 10, 2009
What does your TC need in the exit statement you write?
• The language must be clear:– “There is inadequate assessment of outcome 3(j).”
• How does the program know why its assessment of student knowledge of contemporary issues is inadequate?
– “There is incomplete evidence that students attain outcome 3(i).”• What is it that is incomplete about the evidence that students
have a recognition of the need for and the ability to engage in life-long learning.
– “It appears that outcome 3(h) is not fully assessed.”• Why did the team find that the broad education necessary to
understand the impact of engineering solutions in a global, economic, environmental, and societal context is not fully assessed? What does “full assessment” mean?
• The program has to understand what the shortcoming really is in order to fix it, the TC has to be able to explain the team recommendation, and later teams have to understand the issues
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
32
June 10, 2009
What Would Make it Clearer?• Instead of “There is inadequate assessment of outcome 3(j).”
– The only mechanism used for assessment of student knowledge of contemporary issues is through administration of a survey instrument asking students whether they have knowledge of contemporary issues. This mechanism does not provide adequate information for determining the degree to which this program outcome is attained.
• Instead of “There is incomplete evidence that students attain outcome 3(i).”– Although student grades in the “Introduction to the Profession” course
were claimed as documentation and demonstration that students have recognition of the need for and an ability to engage in life-long learning, no clear links between the student grades and this outcome were established.
• Instead of “It appears that outcome 3(h) is not fully assessed.”– Even though some anecdotal assessment of whether students do have
the requisite broad general education was described in the self-study report, evidence shows only preliminary plans for the development of a systematic process for assessment and evaluation of this program outcome.
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
33
June 10, 2009
The language must support the team recommendation
• If it is a D, the words have to clearly say that the criterion is totally or largely unmet
• “there is no evidence that”• “there is no assessment and evaluation process”• “not all students are required to engage in a major
design experience”
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
34
June 10, 2009
The language must support the team recommendation
• If it is a W, the words have to clearly say that the criterion is met but that strength of compliance is lacking, and how it is lacking needs to be clear– “Assessment of outcomes 3(d), 3(f), 3(g), 3(h), and 3(i) appears
to be ad hoc. Systematic assessment and evaluation of these outcomes would strengthen compliance with this criterion.” [supporting a recommended citation of a Criterion 3 weakness relative to strength of assessment and evaluation process]
– “While the program has demonstrated that graduates are able to engage in most of the activities required by the Civil Engineering program criteria, the program has not clearly demonstrated that graduates are able to apply knowledge of physics, chemistry, and at least one additional area of science.” [supporting a recommended citation of a Criterion 9 weakness for a Civil program]
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
35
June 10, 2009
The language must support the team recommendation
• If it is a C, the words have to clearly say that the criterion is met and indicate exactly what leads to the concern about the potential for future noncompliance– “It is clear that all of the criterion 3 outcomes are being assessed,
but evaluation of some of the assessment data appears to be inconsistent. Unless evaluation of the extent to which outcomes are attained is carried out on a consistent basis, future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 3 concern]
– “Although all of the transcripts examined provided evidence that students meet all graduation requirements, the processes by which graduation requirements are audited prior to graduation appears to be ad hoc. This leads to concern that future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 1 concern]
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
36
June 10, 2009
The PEV Competency Modeland its Application
• Your performance as a PEV will be evaluated against the PEV Competency Model. You are expected to be:– Technically current– Effective at communicating– Interpersonally skilled– Team-oriented– Professional– Organized
• The evaluation forms are found on the ABET web sitehttp://www.abet.org/pev.shtml
Cop
yrig
ht ©
200
9 by
AB
ET
, In
c.
37
June 10, 2009
Last Words
• Remember – this team is “the face of ABET”
• The 4 big C’s go a long way in promoting a productive evaluation:– Courtesy – Consistency– Clarity– Confidentiality