EC&MOS.ppt 001

Presentation

File Name:

ASSCOMF.ppt

Based on EC&MOS.ppt

Version 5 March 2007

EC&MOS.ppt 071

HOW ARE HIGH-LEVEL

COMPETENCIES

TO BE ASSESSED?

EC&MOS.ppt 072

New Forms of Assessment Are Essential

If We Are To Be Able To:• Help students and employees to clarify their motives and areas of

competence.

• Help supervisors, managers, and teachers to perform their crucial function of thinking about the talents of subordinates, and how best to place, develop, utilise and reward them.

• Mount meaningful - rather than misleading and damaging – research – and especially evaluations of educational programmes and social policies.

• Select appropriate people for senior management positions in society.

• Enable public servants to get credit in staff-appraisal systems for taking initiative, seeking out and acting on information in a forward-looking manner and inventing better ways of meeting their clients' needs.

• Build teams made up of people with different concerns and patterns of competence.

/cont.

EC&MOS.ppt 075

First have to note that competencies of

the kind we have been discussing

(ie creativity, initiative, etc.)

are to be understood as

Motivational Dispositions.

Let us take Initiativeas an example.

To repeat, the first important feature of initiative which needs to be noted is that it is self-motivated.

It does not make sense to describe as ‘initiative’ any behaviour which the individual concerned has been told to carry out.

Second, if an individual is to take a successful initiative, he or she has to devote a great deal of time, thought, and effort to the activity.

•It is necessary to initiate action, monitor the effects of that action, and learn from them more about the problem that has to be tackled and the effectiveness of the strategies initially adopted.

•It is necessary to wake up at night in an effort to seize on flickering glimmerings of understanding on the fringe of consciousness and bring them to the centre of attention so that they become fully conscious and usable.

•It is necessary to anticipate obstacles that are likely to be encountered and invent ways of circumventing them.

•It is necessary to beg, bully, persuade or cajole other people to help.

•It is necessary to build up a unique set of idiosyncratic specialist knowledge of the problem and how it is to be tackled. 076

No one is going to do any of these things unless they care very strongly indeed about the goal in relation to which they are attempting to take the initiative.

The value placed on goal or activity is therefore of crucial importance.

It follows that one must identify the individual’s values or intentions before one tries to assess his or her abilities.

Important abilities or components of competence will only be displayed while the individual is undertaking activities that he or she values.

In other words, it does not make sense to attempt to assess separately the cognitive, affective, and conative components of competence independently of the goal in which they might be displayed.

Someone who is extremely creative at finding ways of putting people at ease is unlikely to display his or her creativity (or internal locus of control) in relation to a bag of Lego bricks.

Furthermore, for an initiative to be successful, these components must work together and interpenetrate.

Neither determination exercised without specific and appropriate thought nor thought without feeling (and persistence in making meaning out of fleeting feelings) is likely to make for success.

077

More precisely, not only can these components of competence not be assessed except in relation to goals the individual cares about, not only do they interpenetrate, they are also cumulative and substitutable.

The more of these cumulative and substitutable, difficult and demanding, activities that an individual brings to bear to undertake an activity the more successful he or she is likely to be.

In other words, not only is there no hope of assessing e.g. “the ability to think” except in relation to a particular kind of activity that might be “thought” about.

There is no hope of getting an “internally consistent” measure of e.g. “initiative”.

Thus the question is not “How much initiative does this person display?”, still less “How well can this person think?” or “How creative is he?”

It is “In relation to what does this person display initiative, think, or show creativity?”

Everyone is a genius at something. Everyone is creative in relation to something. The question is: “At what?”

This way of thinking can be made more concrete in the following diagram. 078

EC&MOS.ppt 079

Examples of Potentially Valued Styles of Behaviour

Achievement Affiliation Power

Examples of componentsof effective behaviour.

Doing things w

hich have not been done before.

Inventing things.

Doing things m

ore efficiently than they have been done before.

Developing new

formal scientific theories.

Providing support and facilitation for som

eone concerned with achievem

ent.

Establishing w

arm, convivial relationships w

ith others.

Ensuring that a group w

orks together without conflict.

Establishing effective group discussion procedures.

Ensuring that group m

embers share their know

ledge so that good decisions can be taken.

Articulating group goals and releasing the energies of others in pursuit of them

.

Setting up dom

ino-like chains of influence to get people to do as one wishes w

ithout having to contact them

directly.

CognitiveThinking (by opening one's mind to experience, dreaming, and using other sub-conscious process) about what is to be achieved and how it is to be achieved.

Anticipating obstacles to achievement and taking steps to avoid them.

Analysing the effects of one's actions to discover what they have to tell one about the nature of the situation one is dealing with.

Making one's value conflicts explicit and trying to resolve them.

Consequence anticipated:Personal: e.g. "I know there will be difficulties, but I know from my previous experience that I can find ways round them.

Personal normative beliefs: e.g. "I would have to be more devious and manipulative than I would like to be to do that."

Social normative beliefs: e.g. "My friends would approve if I did that": "It would not be appropriate for someone in my position to do that."

AffectiveTurning one's emotions into the task:Admitting and harnessing feelings of delight and frustration:using the unpleasantness of tasks one needs to complete as an incentive to get on with them rather than as an excuse to avoid them.

Anticipating the delights of success and the misery of failure.

Using one's feelings to initiate action, monitor its effects, and change one's behaviour.

ConativePutting in extra effort to reduce the likelihood of failure.

Persisting over a long period, alternatively striving and relaxing.

Habits and experienceConfidence, based on experience, that one can adventure into the unknown and overcome difficulties, (This involves knowledge that one will be able to do it plus a stockpile of relevant habits).

A range of appropriate routineised, but flexibly contingent behaviours, each triggered by cues which one may not be able to articulate and which may be imperceptible to others.

Experience of the satisfactions which have come from having accomplished similar tasks in the past.

EC&MOS.ppt 080

An Internal Consistency

or a

Multiple Regression

Model?

EC&MOS.ppt 081

A succinct way of capturing the fact that we first need to elicit the individual’s concerns and thereafter uncover the components of

competence he or she brings to bear in his or her efforts to undertake those things (as distinct from asking him or her to perform tasks which

someone else thinks are important [ike solving RPM problems or demonstrating creativity with Leggo bricks])

is to say that:

We need to develop

Operant

(rather than Respondant)

Measures

EC&MOS.ppt 082

Some notesThe completed grid leads to what amounts to a DESCRIPTIVE STATEMENT about the individual.

This is akin to a statement about what kind of animal, plant, or chemical one is dealing with.

Such a statement is very different from a profile of scores on “variables”.

It would not make sense to try to “score” all animals on scales of, for example, “dogginess”.

Effectivness depends on how many of a series of cumulative and substitutable components of competence an individual brings to bear to undertake a particular activity.

He or she will normally only exercise and display these components of competence whilst undertaking an activity he or she is intrinsically motivated to carry out.

Just as a chemical will only reveal its elements in the course of an interactive chemical analysis.

And, as in chemistry, what happens depends on the environment in which the individual is placed.

An alternative analogy is an animal in its habitat.

EC&MOS.ppt 083

Physicists model:S = ut + ½ ut2

Chemists model:Cu + 2H2SO4 = CuSO4 + 2H2O + SO2

Ecological model:Linnaeus/Darwinian Descriptive Classification

Using Branching Framework+

Maps of networks of complex interactions and feedbacks re the habitat.

EC&MOS.ppt 084

To summarise what we have done: - 1

1. We first argued that competencies are:– Value-based– Internally heterogeneous

2. This way of thinking could be more easily grasped if it were presented as an extended version of Grid 1.

3. The completed grid – with ticks (check marks) in the cells – provided more useful information than a profile of heterogeneous scores.

4. This information could be more succinctly conveyed by a procedure analogous to a chemical formula, but consisting of a description of the behaviours which someone is strongly internally motivated to undertake and the competencies displayed while undertaking them.

/contd

EC&MOS.ppt 085

To summarise what we have done: - 2

5. Unless the environments in which people find themselves engage with their motives they will neither develop nor display the competencies of which they are capable.

6. The environments in which people find themselves can also be described in a manner analogous to that used in chemistry.

7. More than that, these descriptions of the situation in which observations are made must become an integral part of the assessment of the individual

8. This way of thinking allows us, for the first time, to model the transformational processes which take place in homes, schools, and workplaces. Non linear, but, equally, cannot transform people into others for which they do not have the potential.

9. It remains to add that, just as chemists require thorough familiary with the conceptual framework they are using and with a range of interventionist analytical procedures to get behind surface characteristics, the same is going to be true in psychology.

EC&MOS.ppt 086

Ways of getting the information needed to complete the grid or make meaningful

descriptive statements1.Observation.

2.Projective methods and Behavioral Event Interviewing.

3.Portfolios of work; Records of personal experience.

4.Value-expectancy methods.

EC&MOS.ppt 087

1. Observation

EC&MOS.ppt 088

Competencies Possessed

Competencies Displayed

(but only observable through value-tinted spectacles)

Values / values for task

Environment: (1) Whether previously tapped values so that informant discovered & developed relevant competencies.

(2) Whether task currently engages informant’s values.

EC&MOS.ppt 090

It follows that we cannot expect people to exercise, never mind display, the high-level competencies they possess unless they are in a situation that engages their motives.

So an assessment of the context in which the observations were made must form an integral part of the statement.

EC&MOS.ppt 091

Hence problem with Assessment Centres: May not be interested in getting people to work together to build bridges, but may well be very interested in developing building materials.

EC&MOS.ppt 092

Nevertheless one can use direct observation with reasonable confidence if the context is a “developmental environment”.

And one can easily find out whether this is the case by using appropriate Questionnaires.

So the topic shifts from one of accrediting student outcomes to accrediting courses.

And this means studying whether real thought has been given to individual students’ developmental programmes.

The network of personnel needed to do this – i.e. to accredit courses as well as outcomes - contributes to the introduction of an innovatory educational system because it promotes discussion of goals and how to achieve them.

EC&MOS.ppt 094

Assessment: Part II

Projective Methods and Behavioral Event

Interviewing

EC&MOS.ppt 095

Projective Methods

EC&MOS.ppt 095

Display File: TAT PICS.ppt

Instructions for Story Composition

(filename: unknown)

Work Through Scoring System

(filename: unknown)

Practice Stories(filename: unknown)

Score Own Stories(filename: unknown)

EC&MOS.ppt 097

Examples of Potentially Valued Styles of Behaviour

Achievement Affiliation Power

Examples of componentsof effective behaviour.

Doing things w

hich have not been done before.

Inventing things.

Doing things m

ore efficiently than they have been done before.

Developing new

formal scientific theories.

Providing support and facilitation for som

eone concerned with achievem

ent.

Establishing w

arm, convivial relationships w

ith others.

Ensuring that a group w

orks together without conflict.

Establishing effective group discussion procedures.

Ensuring that group m

embers share their know

ledge so that good decisions can be taken.

Articulating group goals and releasing the energies of others in pursuit of them

.

Setting up dom

ino-like chains of influence to get people to do as one wishes w

ithout having to contact them

directly.

CognitiveThinking (by opening one's mind to experience, dreaming, and using other sub-conscious process) about what is to be achieved and how it is to be achieved.

Anticipating obstacles to achievement and taking steps to avoid them.

Analysing the effects of one's actions to discover what they have to tell one about the nature of the situation one is dealing with.

Making one's value conflicts explicit and trying to resolve them.

Consequence anticipated:Personal: e.g. "I know there will be difficulties, but I know from my previous experience that I can find ways round them.

Personal normative beliefs: e.g. "I would have to be more devious and manipulative than I would like to be to do that."

Social normative beliefs: e.g. "My friends would approve if I did that": "It would not be appropriate for someone in my position to do that."

AffectiveTurning one's emotions into the task:Admitting and harnessing feelings of delight and frustration:using the unpleasantness of tasks one needs to complete as an incentive to get on with them rather than as an excuse to avoid them.

Anticipating the delights of success and the misery of failure.

Using one's feelings to initiate action, monitor its effects, and change one's behaviour.

ConativePutting in extra effort to reduce the likelihood of failure.

Persisting over a long period, alternatively striving and relaxing.

Habits and experienceConfidence, based on experience, that one can adventure into the unknown and overcome difficulties, (This involves knowledge that one will be able to do it plus a stockpile of relevant habits).

A range of appropriate routineised, but flexibly contingent behaviours, each triggered by cues which one may not be able to articulate and which may be imperceptible to others.

Experience of the satisfactions which have come from having accomplished similar tasks in the past.

EC&MOS.ppt 099

Behavioral Event Interviewing

EC&MOS.ppt 101

Assessment: Part III

Folios of Work; Reports of Personal

Experience

EC&MOS.ppt 103

Assessment: Part IV

Value Expectancy Methods

EC&MOS.ppt 104

p(B) = [∑PiIi]w0 + [∑NBpMCp]w1 + [∑NBsMCs]w1

EC&MOS.ppt 105

Value-Expectancy Methods

as Implemented in

The Edinburgh Questionnaires

i

Consequences

If you have said that it is very important to you to work in a clean environment and that you are dissatisfied with the current state of affairs; what would happen if you tried to get something done about it? How likely is it that each of the following would happen?

Very Likel

y

Unlikely

Personal Reactions

I would enjoy trying to get something done about this.

Self Image

I would have to be devious and manipulative.

Reference Groups’ Reactions

My boss would promote me.

Satisfaction

How satisfied are you with:

Hi Low

1. The cleanliness of your work environment.

2. Your opportunity to do new things which have not been done before.

Figure 28.2 The Assessment of the Components of Competence: An illustration from The Edinburgh Questionnaires. Part A The Process. Part B (Flow Chart) will be found on the next page. Note: this is a schematic representation only and does not bear a direct relationship to the Questionnaires. 106

Importance

How important is it to you to:

Hi Low

1.Work in a clean environment

2. Be able to do new things which have not been done before.

EC&MOS.ppt 106B

Importance

Hi Low

Satisfaction

Hi Low

Consequences

Yes

Personal

Self Image

Ref Group

Sum of products gives probability that will do something about it Johnsons “resultant”

x

x

xCompareMost Important Source of Dissatisfaction

Use “importance ratings to weight consequences by importance attached to them

Cumulate over workforce data to form the basis for discussion.

Cumulate over workforce data to form the basis for discussion.

Cumulate over workforce data to form the basis for discussion.

1. Organisational Development2. Monitoring of Effectiveness of O.D.3. Accreditation of Agents

Inspection of individual items leads to guidance and placement.

Inspection of individual items gives personal development needs and hence placement.

Figure 28.2 (Continued) The Assessment of the Components of Competence: An illustration from The Edinburgh Questionnaires. Part B Flow Chart. Note: The Figure has been prepared for illustrative purposes. The flows are over simplified and schematic.

EC&MOS.ppt 107

The main demonstrations of the value of the model have so far been in

programme evaluation:Lothian Region Educational Home

Visiting Projectand

Value of Environmentally Based, Competency-Oriented, Project Work.

alsoCross-Cultural Comparisons:

Germany, Japan, US, UK etc. (Graham).

EC&MOS.ppt 108

To Summarise what I have said:

1. If we want to:• Develop and capitalize upon and reward more

of the human resources available.• Run ethical schools.• Mount genuinely scientific evaluations to

guide policy.• Stem the destruction of the planet.

We need to evolve new ways of thinking about competence and how it is to be assessed.

/cont

EC&MOS.ppt 109

Summary (contd.)

2. We need to move toward a measurement model which:

• Seeks to be value-based rather than value free.• Utilizes a two-stage model.• Builds on a multiple-regression sum across

independent predictors, rather than an internal consistency model.

• Embodies multiple talents.• Includes beliefs about how society works and

one’s role in it./contd

EC&MOS.ppt 110

Summary (contd.)

3. We need to abandon our preoccupation with variables and:

• Adopt a descriptive paradigm.• Discriminate between people mainly in terms

of the activities they value and the competencies they bring to bear whilst undertaking them.

• Include in our descriptions a statement about those aspects of the environment which engage with the motives or values of the person being assessed and thus lead him or her to display the talents they possess.

EC&MOS.ppt 111

Conclusion

I have shown that an alternative model is:

• Operationalisable.

• Yields more valid results.

• Yields results which are much more morally justifiable.

• Yields results which lead to greater advances in understanding.

• Yields more useful results.

EC&MOS.ppt 112

In comparison, traditional forms of academic assessment:• Have zero predictive validity outside the

educational system.• Are more dependent on the examiner than the

student (60% of the variance being so accounted for).

• Have little discriminative power: The difference between an A and a D grade typically being 8 raw score points.

• Depend mainly on presentation.• Relate to material which was out-of-date when it

was taught, does not relate to people’s problems, and will be forgotten by the time it is needed.

• /

contd

EC&MOS.ppt 113

Traditional forms of academic assessment, contd. • Do not identify the unique combinations of up-to-

date, largely tacit, knowledge that people require to tackle their problems.

• Assess temporary knowledge of poorly sampled sub domains of the vast amount of explicit knowledge that exists.

• Make no assessment of the competencies actually required to perform well in the domains of competence they are said to identify: i.e. they lack construct as well as predictive validity.

• Help to legitimize a divided society which compels most people to contribute in ways they do not like to processes that are heading our species toward its own extinction, carrying the destruction of the planet with it.

EC&MOS.ppt 114

In fact, it has now been recognized even by the American Psychological Association that failure to make broadly-based, comprehensive, evaluations of both people and programmes represents the most widespread and most serious misuse of tests.