Open ECBCheck

Methods for Quality Development

Rafael García Rodríguez

University of Augsburg, 2010

Introduction

Open ECBCheck label makes use of five central methods for quality evaluation and validation that all have distinct characteristics and potential advantages and disadvantages.

Benchmarking

Qualitative Weighting and Summation

Benchlearning

Community of Practice

Quality Assessment

Self-assessment

Peer-review

Benchmarking

Definition

Benchmarking is a systematic and continuous process to evaluate products, services or processes of an organization with the recognized market leaders (“best in class").

The target, in addition to improve himself, would be to get in long term market leadership or excellence.

Benchmarking

Benchmarking is divided into four main stages with a concluding fifth phase maturity:

Maturity

Implementation

Planning

Integration

Analysis

Identification of object, organisations and methods

Identification of gaps and performance levels

Communication of results and setting of targets

Development, execution and check

Leading position and integration of benchmarking

Self-assessment

Benchmarking

There are, however, a number of restrictions associated with benchmarking:

First, learning from benchmarking is mostly concerned with the past as the competitor has already achieved this level before.

There is a high uncertainty in identifying the “best in class” organisation or best practice. It's necessary a step by step approach.

Benchmarking only focuses on current best practices and is not a source for innovation and possible future best practice.

The best practice cases are not specific, they need to be generalisable to be transferable to one organisation.

Benchlearning

Definition

Benchlearning is the process of learning from the "best in class" with the purpose of integrating these best practices in all organizational levels of the company.

Benchlearning

Benchlearning is connected to benchmarking.

These two processes search the way of how the best practices can be transferred to the own organisation and how skills and processes can be improved, and they distinguish benchaction as the actual implementation of all changes that have been set as targets.

(Foundation)

(Learning process)Benchlearning

Benchmarking =

=

Self-assessment

Definition

Self-assessment is a regular and systematic analysis of strengths and weaknesses of a company or organisation to determine one position, to identify areas for improvement and to transfer these insights into implementation.

The initiative is supposed to come from the organisation itself and the organisation that conducts the assessment is also responsible for the process. Usually, the self-assessment is conducted against a set of criteria.

Self-assessment

The European Foundation For Quality Management (EFQM) offers the EFQM Excellence Model as a reference point for a self-assessment.

EFQM Excellence Model

Self-assessment

The EFQM manual discusses data collection methods that could be used for self-assessments that differ in results, time or resources required.

Questionnaires

Assessment workshop

“Award Simulation”

Easy to use

Don’t consume many resources

Used as foundation for other methods

Data collection

Presentation and assessment

Reach a consensus

Full documentation for “European Quality Award”

Results from organisations itself or outside

Definition

Peer-review is a external evaluation of an organisation or individual by experts that belong to a different organisation or may also be colleagues within the same organisation for a Quality Assurance.

Peer-review doesn’t focus on data collection but on a competent review of existing data to derive core issues, however, peer-reviewers may collect additional data by a peer-review visit or interviews.

Peer-review

Peer-review

Design review as the subject of interest:

Review of the self-assessment report/results.

Review of the conclusions in the self-assessment report/ site visits and interviews with stakeholders.

Review of the subject of interest itself

Basic design of the Peer-review process

Self-evaluation

Final review report

External evaluation

(+ on-site visit)

Even so, there are numerous possible designs for peer-reviews

Peer-reviews

Advantages Disadvantages

• SRCIVEN (1991, p. 255)

- Halo-effects

- Secret-contract bias

- Fear of retaliatory action

• Gutknecht Gmeiner (2008, p. 23)

- Lack of qualification

- Lack of objectivity

(refers to the reviewers)

• In a formative way:

- Mutual learning possibility

• Relatively good cost-benefit ratio

• More acceptable perception of colleagues or experts in the same field of study than external evaluators.

Qualitative Weighting and Summation

Definition

Qualitative Weighting and Summation is a method for evaluation based on criteria catalogues. Is described in the context of the evaluation of learning management systems. This method is proposed as an alternative to the Numerical Weighting and Summation method to avoid some disadvantages of this latter.

In Numerical Weighting and Summation method:

1. Each criterion is weighted according to importance.

2. The evaluand is rated in all criteria

3. The products of all ratings with the corresponding weighing are summed up to

one final score that indicates the relative rank of product, organisation or service)

Qualitative Weighting and Summation

Qualitative Weighting and Summation Process

All criteria are weighted based on non numeric characters (Essential / Very Valuable / Valuable / Marginally Valuable / Zero). All criteria that are rated with “Zero” are excluded.

Only criteria that have been weighted as essential are assessed.

All remaining products are rated only with help of the remaining criteria (“Very Valuable” to “Marginally Valuable”)

The ratings in the three different importance levels are then summarised resulting in three aggregated values for each product.

Qualitative Weighting and Summation

Disadvantages

Qualitative Weighting and Summation Numerical Weighting and Summation

Method relatively complex.

It doesn’t offer a definite decision algorithm.

Sometimes needs to be applied several times in an iterative process to final conclusion.

Existence of minimum criteria.

Existence of interconnections between criteria.

The assumption of linear scales.

How to weigh and consider the relevance of the criteria.

The procedure of weighting is unreproducible.

• Ehlers, Ulf-Daniel (2009). Selected Methods for Quality Development in Open

• http://www.qualityfoundation.org/openecbcheck/

• http://www.efqm.org/en/

More information about Open ECB-Check in: http://www.qualityfoundation.org/openecbcheck/

References

ECBCheck. University Duisburg-Essen, Germany.