1

SUCCESS CRITERIA

QUALITY ASSURANCE OFFICE

WHAT ARE SUCCESS CRITERIA?

• The standards/levels by which to judge whether an objective/goal/ target/outcome has been achieved/successful.

• Success criteria are linked to intended outcomes and targets e.g. of action plans, of strategic plans.

• Success criteria are often linked to metrics.

• Success criteria can be discrete: Yes/No: We did or did not do/achieve something.

• Success criteria can be continuous: measurable on a scale.• Success criteria need not be numerical.

WHY DO WE NEED SUCCESS CRITERIA?• Tracking: they can indicate how well something is working.• Achievement: they can identify and recognize achievement.• Standards: they set standards and indicate how well these have been met.• Progress: they can chart/measure progress: trends in set areas.• Performance: they can chart performance against benchmarks and baseline

assessments.• Development and improvement: they can identify areas for improvement and

development.• Decision making: they can support evidence-informed evaluation, decision-

making, strategic direction, policy formation and management of the University/ unit: navigational and decision-making tools.

• Focus: they give focus.• Relevance: they can be linked to target setting and action planning.• Monitoring: they can contribute to keeping the University/unit(s) on track and

goal-focused: monitoring and direction. 3

WHY DO WE NEED SUCCESS CRITERIA?• Accreditation and quality audit: they can meet external accreditation/

quality audit requirements (and externally set success criteria), e.g. to meet the understanding of ‘success’ used by other universities.

• Overall picture: they can contribute to the overall picture of what the University is doing.

• Impact: they can inform judgements about the consequences of interventions.

• Strengths and weaknesses: they can provide evidence of strengths and weaknesses.

• Management: they can support performance management.• Efficacy: they can provide evidence of efficacy.• Morale(???): they can acknowledge improvements made.• Transparency: they contribute to transparency.• Accountability: they can be used for accountability. 4

TERMINOLOGY1. GOALS: Targets; intended outcomes.

Example: To raise student attainment through collaborative learning in the University.

2. SUCCESS CRITERIA: How will you know if the goals/intended outcomes have been achieved? What standards/criteria will you use to judge whether or not it has been successful (i.e. what defines the success of the project)

Example: All modules include collaborative learning as part of assessment.90% of our students have GPAs of 18/20 or above.

3. KEY PERFORMANCE INDICATORS: Statements of the area of achievement of a goal, i.e. operationalizing the goal and success criteria, and the metrics used to track the success towards a goal (outcome measures of/in the success criteria).

Examples: (a) Rise in student attainment in comparison to the previous year.(b) Increase in collaborative learning in all modules.

4. EVIDENCE: Data required to inform the indicator.Examples: (a) Students’ semester grades over two years.

(b) Classroom observation data over two years.(c) Stakeholder satisfaction data.

CLARIFYING TERMINOLOGYQUESTION: What is the difference between success criteria and indicators?ANSWER: Success criteria are what define the success of the project: levels, standards and cut-off points. Indicators are statements of areas focused on in order to informthe success criteria: the tools and /or metrics to measure or judge the achievement of the success criteria.

Success criterion: Every session in every module uses collaborative learning.Indicator: Collaborative learning by groups of students.

QUESTION: What is the difference between indicators and evidence?ANSWER: Indicators are statements or measures being used in relation to a field. Evidence is the data used to inform the indicator.

Indicator: Student involvement in the University.Evidence: Number of students attending a University meeting.

EXAMPLE

EXAMPLE

2

SOME RUDIMENTARY SCs ARE CATEGORICAL, E.G. YES/NO

• Administrative staff turnover rates below 10% each academic year.• 75% of postgraduates have IELTS 6.5 or above. • An average of 3 high-impact journal papers per year for each staff Faculty member.

7

SOME SCs ARE QUALITATIVE• Increased student willingness to participate in the university’s extra-curricular

activities.• Clarity and suitability of instructor’s expectations, requirements, guidance and

directions for student performance.

SOME SC’s ARE NUMBERS ONLY

• USJ has Standard Operating Procedures for all its administrative units.• All University academic and administrative staff attend a minimum of 30 hours

of formal training each academic year.

USING SUCCESS CRITERIA• Success criteria are often linked to changes, developments, interventions,

innovations, plans, i.e. they should assess the move from ‘where we are’ to ‘where we want to be’ (rather than maintaining where we currently are).

• Success criteria should set a standard/level of achievement, i.e. a cut-off point at or above which success is assumed and below which success has not been achieved, e.g.:

90% of students will achieve a GPA of 3.5 or above.

Average student evaluations of teaching for every academic staff member are 4 out of 5.

• Success criteria should be achievable/deliverable.

USING SUCCESS CRITERIA

• Justify your standard/level – where does it come from, and why?

• Who decides whether or not to use the metrics, and what for?

• Who decides the cut-off points, levels and standards?

• Who decides when not to use metrics, and what for?

• How to avoid having different people/modules/units etc. setting criteria that are very different in type or ‘level’ from those set in other areas, and, if not, does this matter? Who decides?

How to justify the

success criteria as

being ‘reasonable’?

BEING REALISTIC ABOUT SUCCESS CRITERIA (SCs)• Why choose this criterion?• What does ‘success’ really mean in your context/unit?• What really constitutes ‘success’?• How many times must the success criterion be achieved for it to be judged

‘successful’?• How do you decide the ‘cut-off’ point – the standard or level – for success? Who

decides?• How to match internal success criteria with those of external agencies?• What do the numbers really ‘mean’ and show in using metrics?

10

BEING REALISTIC ABOUT SUCCESS CRITERIA (SCs)• SCs indicate; they do not tell the whole story.• SCs inform; humans decide.• SCs provide an incomplete, partial and selective coverage of work and activities:

not everything is covered in a SC or a series of SCs.• SCs are tools, not ends in themselves.• SCs must be taken within the context of the overall University/unit; using them in

isolation risks distorting the overall work, focus and goals of the University/unit.• They may focus on what rather than why.• Numbers are useful but not everything is susceptible to/suitable for metrics.• Numbers can contribute to/complement qualitative data, judgements, evaluation

and policy. • SCs should not be misused: consequential validity is important; there are dangers

in using them punitively rather than for improvement and development.• SCs risk focusing on short-term and narrow achievements: only performance. 11

Success Criteria in Project ManagementCost, Time, Performance, Satisfaction, Use, EffectivenessTechnical performance, Efficiency of project execution, Managerial and organizational implications, Personal growth, Project termination, Technical innovativeness, Manufacturability and business performance.Time Performance, Cost Performance, Quality Performance, Health, Safety and Environment (HSE), Client Satisfaction.Cost, Time, Meeting the technical specification, Customers' satisfaction, Stakeholders, satisfaction.Cost, Time, Quality, Scope, Customer Satisfaction, Safety, Team Satisfaction, Shareholder Satisfaction.Cost, Quality, Time, Customer Satisfaction, Technical Specifications, and Functional Requirements, Revenue and Profits, Competitive Advantage, Market Share, Reputation.Cost, Time, Technical Requirements, Customer Satisfaction, Objectives Achievement.Client’s satisfaction, Project completed on time, Project completed to specified quality standard, Absence of disputes, safety, Completion within budget.Quality, Time, Cost, Health, Safety and Environment (HSE), Scope, Customer Satisfaction, Efficiency of use resource, Effectiveness Productivity, Profitability, Shareholder satisfaction, Experience gain from the project, Achievement of project’s objectives, Sustainability, Reliability

Frefer AA, Mahmoud M, Haleema H, Almamlook R (2018) Overview Success Criteria and Critical Success Factors in Project Management. Ind Eng Manage 7: 244. DOI: 10.4172/2169-0316.1000244

3

https://www.girlsguidetopm.com/the-definitive-guide-to-project-success-criteria/

1. You must define what success looks like for your project or you won’t know if you have achieved it.

2. Success criteria measure what’s important to your stakeholders.3. Document success criteria and get everyone to agree to them.4. Use continuous measurements where possible.5. Baseline today’s performance so you know where you are starting from.6. Track as appropriate and report on your progress.

Each list item should include:

1. Name of success criterion.2. How it is going to be measured.3. How often it is going to be measured.4. Who is responsible for measuring it.

PROJECT SUCCESS CRITERIA HOW TO DEFINE OR MEASURE SUCCESS OF A PROJECT

‘Success criteria should include hard metrics such as delivering the project on time [1] and within budget [2], achieving the project scope [3], meeting milestone dates [4], cost targets [5], project quality requirements [6], project risks [7], safety, health, environmental and security [8] requirements. You don’t have to include all the above metrics in the success criteria to evaluate the success of your project. It would be wise to use just three or four metrics.’

https://www.orchestra-ppm.com/en/2016/11/09/project-performance-how-to-measure-or-define-success-in-project-management/

A possible task: Identify a project that your unit has for the coming semester. Write [1], [3], [4], [6] and [7] for your project.

DON’T FORGET INTANGIBLES• ‘Hard facts and metrics are good, but don’t underestimate intangible

things such as behavior and attitude of project managers or project teams, team satisfaction, quality of daily work, communication and collaboration among team members. It is equally important to evaluate the human side of project management.’

(https://www.orchestra-ppm.com/en/2016/11/09/project-performance-how-to-measure-or-define-success-in-project-management)

TASK ONE: INTANGIBLES• Write a success criterion for ‘collaboration among team members’. • This can be in the form of a metric or a qualitative statement.

TASK TWO: A METRIC FOR A SUCCESS CRITERION

Step One: Write one strategic objective or target for your unit.Step Two: Write one Key Performance Indicator (KPI) for this strategic

objective/target.Step Three: Write one source of evidence for this KPI.Step Four: Write one Success Criterion for this strategic objective/target,

in the form of a metric, indicating the level or standard to be reached. What is your scale measuring?

STRATEGIC OBJECTIVE 2: “To provide the highest international academic standards.”

Actionable measure: Ensure the international character of its academy by the use of English as the regular medium of internal communication.

Task Three: Write two success criteria for this Actionable Measure.

LINK SUCCESS CRITERIA TO ACTION PLANS

The achievement of:

1. the goals, targets and SMART objectives of the Action Plan2. the content areas of the Action Plan3. the intended outcomes and outputs of the Action Plan

SCs must address those parts of the Action Plan’s goals, targets, outcomes and contents which are amendable to SCs; they must be sufficiently comprehensive and fair: valid and reliable.

Success criteria can refer to:

• Input• Process • Outcome

4

ADVICE FOR USING SUCCESS CRITERIA (SCs)• Make sure that your SCs are valid and reliable – that they measure/set standards

for what they are intended to address, and do so reliably and credibly.• Make sure your SCs are realistic and achievable (with some effort, i.e. are

developmental).• Make sure your SCs are easy to understand.• Make sure your SCs are connected to goals/targets.• Do not have too many SCs: as few as necessary but no fewer.• Make sure that your SCs are not too easy but not so demanding as to be

unachievable.• Keep the wording of your SCs as brief as possible but no briefer.• Make sure that the SCs fairly cover and represent the Action Plan’s goals and

targets, i.e. they must be valid and reliable.• Be VERY careful what you infer from SCs; don’t over-interpret!

19

RESPONSIBLE METRICSResponsible metrics can be understood in terms of number of dimensions:• Robustness: basing metrics on the best possible data in terms of accuracy

and scope;• Humility: recognising that quantitative evaluation should support – but

not supplant – qualitative, expert assessment;• Transparency: keeping data collection and analytical processes open and

transparent, so that those being evaluated can test and verify the results;• Diversity: accounting for variation by field, and using a range of indicators

to reflect and support a plurality of research . . . across the system;• Reflexivity: recognising and anticipating the systemic and potential effects

of indicators, and updating them in response.(Wilsdon, J,. (2016) The Metric Tide. London: Sage, p. 138)

20

In the right hand column, indicate whether the statement is a Success Criterion (SC), Key Performance Indicator (KPI) or evidence (E).

STATEMENT SC, KPI or E

1. Students’ on-task behaviour and interaction in classroom sessions.

2. Student engagement in learning, cognitively, affectively, motivationally and socially.

3. Students’ evaluations of teaching.

4. Every staff member attends a minimum of 30 hours’ staff development a year.

5. Class observations of, and feedback on, laboratory sessions.

6. Ninety per cent of students attain GPAs of 18/20 or above.

7. Every non-academic unit has Standard Operating Procedures for all its main operations.

FOLLOWING UP SUCCESS CRITERIA

Two questions:

1. What do the SCs really tell you about the achievement of the goals and targets set out in the Action Plan, i.e. what do you want to do with the SCs?

2. What will you do now, as a consequence of (1)?

WHAT DOES IT MEAN FOR ME?1. You must know the SCs very well.2. You must be prepared to create and write SCs, and to participate

constructively in discussions on the creation and feedback on draft documents.

3. You must be prepared to collect, collate, provide, analyse and report evidence of using SCs for improvement and development.

4. You must be prepared to respond to QAE-related requests from the QA Office.

5. You must be prepared to defend the SCs and to know their limitations.

23 24THANK YOUTHANK YOU