ec&mos.ppt 001 presentation file name: asscomf.ppt based on ec&mos.ppt version 5 march 2007
Post on 28-Dec-2015
237 Views
Preview:
TRANSCRIPT
EC&MOS.ppt 001
Presentation
File Name:
ASSCOMF.ppt
Based on EC&MOS.ppt
Version 5 March 2007
EC&MOS.ppt 071
HOW ARE HIGH-LEVEL
COMPETENCIES
TO BE ASSESSED?
EC&MOS.ppt 072
New Forms of Assessment Are Essential
If We Are To Be Able To:• Help students and employees to clarify their motives and areas of
competence.
• Help supervisors, managers, and teachers to perform their crucial function of thinking about the talents of subordinates, and how best to place, develop, utilise and reward them.
• Mount meaningful - rather than misleading and damaging – research – and especially evaluations of educational programmes and social policies.
• Select appropriate people for senior management positions in society.
• Enable public servants to get credit in staff-appraisal systems for taking initiative, seeking out and acting on information in a forward-looking manner and inventing better ways of meeting their clients' needs.
• Build teams made up of people with different concerns and patterns of competence.
/cont.
EC&MOS.ppt 075
First have to note that competencies of
the kind we have been discussing
(ie creativity, initiative, etc.)
are to be understood as
Motivational Dispositions.
Let us take Initiativeas an example.
To repeat, the first important feature of initiative which needs to be noted is that it is self-motivated.
It does not make sense to describe as ‘initiative’ any behaviour which the individual concerned has been told to carry out.
Second, if an individual is to take a successful initiative, he or she has to devote a great deal of time, thought, and effort to the activity.
•It is necessary to initiate action, monitor the effects of that action, and learn from them more about the problem that has to be tackled and the effectiveness of the strategies initially adopted.
•It is necessary to wake up at night in an effort to seize on flickering glimmerings of understanding on the fringe of consciousness and bring them to the centre of attention so that they become fully conscious and usable.
•It is necessary to anticipate obstacles that are likely to be encountered and invent ways of circumventing them.
•It is necessary to beg, bully, persuade or cajole other people to help.
•It is necessary to build up a unique set of idiosyncratic specialist knowledge of the problem and how it is to be tackled. 076
No one is going to do any of these things unless they care very strongly indeed about the goal in relation to which they are attempting to take the initiative.
The value placed on goal or activity is therefore of crucial importance.
It follows that one must identify the individual’s values or intentions before one tries to assess his or her abilities.
Important abilities or components of competence will only be displayed while the individual is undertaking activities that he or she values.
In other words, it does not make sense to attempt to assess separately the cognitive, affective, and conative components of competence independently of the goal in which they might be displayed.
Someone who is extremely creative at finding ways of putting people at ease is unlikely to display his or her creativity (or internal locus of control) in relation to a bag of Lego bricks.
Furthermore, for an initiative to be successful, these components must work together and interpenetrate.
Neither determination exercised without specific and appropriate thought nor thought without feeling (and persistence in making meaning out of fleeting feelings) is likely to make for success.
077
More precisely, not only can these components of competence not be assessed except in relation to goals the individual cares about, not only do they interpenetrate, they are also cumulative and substitutable.
The more of these cumulative and substitutable, difficult and demanding, activities that an individual brings to bear to undertake an activity the more successful he or she is likely to be.
In other words, not only is there no hope of assessing e.g. “the ability to think” except in relation to a particular kind of activity that might be “thought” about.
There is no hope of getting an “internally consistent” measure of e.g. “initiative”.
Thus the question is not “How much initiative does this person display?”, still less “How well can this person think?” or “How creative is he?”
It is “In relation to what does this person display initiative, think, or show creativity?”
Everyone is a genius at something. Everyone is creative in relation to something. The question is: “At what?”
This way of thinking can be made more concrete in the following diagram. 078
EC&MOS.ppt 079
Examples of Potentially Valued Styles of Behaviour
Achievement Affiliation Power
Examples of componentsof effective behaviour.
Doing things w
hich have not been done before.
Inventing things.
Doing things m
ore efficiently than they have been done before.
Developing new
formal scientific theories.
Providing support and facilitation for som
eone concerned with achievem
ent.
Establishing w
arm, convivial relationships w
ith others.
Ensuring that a group w
orks together without conflict.
Establishing effective group discussion procedures.
Ensuring that group m
embers share their know
ledge so that good decisions can be taken.
Articulating group goals and releasing the energies of others in pursuit of them
.
Setting up dom
ino-like chains of influence to get people to do as one wishes w
ithout having to contact them
directly.
CognitiveThinking (by opening one's mind to experience, dreaming, and using other sub-conscious process) about what is to be achieved and how it is to be achieved.
Anticipating obstacles to achievement and taking steps to avoid them.
Analysing the effects of one's actions to discover what they have to tell one about the nature of the situation one is dealing with.
Making one's value conflicts explicit and trying to resolve them.
Consequence anticipated:Personal: e.g. "I know there will be difficulties, but I know from my previous experience that I can find ways round them.
Personal normative beliefs: e.g. "I would have to be more devious and manipulative than I would like to be to do that."
Social normative beliefs: e.g. "My friends would approve if I did that": "It would not be appropriate for someone in my position to do that."
AffectiveTurning one's emotions into the task:Admitting and harnessing feelings of delight and frustration:using the unpleasantness of tasks one needs to complete as an incentive to get on with them rather than as an excuse to avoid them.
Anticipating the delights of success and the misery of failure.
Using one's feelings to initiate action, monitor its effects, and change one's behaviour.
ConativePutting in extra effort to reduce the likelihood of failure.
Persisting over a long period, alternatively striving and relaxing.
Habits and experienceConfidence, based on experience, that one can adventure into the unknown and overcome difficulties, (This involves knowledge that one will be able to do it plus a stockpile of relevant habits).
A range of appropriate routineised, but flexibly contingent behaviours, each triggered by cues which one may not be able to articulate and which may be imperceptible to others.
Experience of the satisfactions which have come from having accomplished similar tasks in the past.
EC&MOS.ppt 080
An Internal Consistency
or a
Multiple Regression
Model?
EC&MOS.ppt 081
A succinct way of capturing the fact that we first need to elicit the individual’s concerns and thereafter uncover the components of
competence he or she brings to bear in his or her efforts to undertake those things (as distinct from asking him or her to perform tasks which
someone else thinks are important [ike solving RPM problems or demonstrating creativity with Leggo bricks])
is to say that:
We need to develop
Operant
(rather than Respondant)
Measures
EC&MOS.ppt 082
Some notesThe completed grid leads to what amounts to a DESCRIPTIVE STATEMENT about the individual.
This is akin to a statement about what kind of animal, plant, or chemical one is dealing with.
Such a statement is very different from a profile of scores on “variables”.
It would not make sense to try to “score” all animals on scales of, for example, “dogginess”.
Effectivness depends on how many of a series of cumulative and substitutable components of competence an individual brings to bear to undertake a particular activity.
He or she will normally only exercise and display these components of competence whilst undertaking an activity he or she is intrinsically motivated to carry out.
Just as a chemical will only reveal its elements in the course of an interactive chemical analysis.
And, as in chemistry, what happens depends on the environment in which the individual is placed.
An alternative analogy is an animal in its habitat.
EC&MOS.ppt 083
Physicists model:S = ut + ½ ut2
Chemists model:Cu + 2H2SO4 = CuSO4 + 2H2O + SO2
Ecological model:Linnaeus/Darwinian Descriptive Classification
Using Branching Framework+
Maps of networks of complex interactions and feedbacks re the habitat.
EC&MOS.ppt 084
To summarise what we have done: - 1
1. We first argued that competencies are:– Value-based– Internally heterogeneous
2. This way of thinking could be more easily grasped if it were presented as an extended version of Grid 1.
3. The completed grid – with ticks (check marks) in the cells – provided more useful information than a profile of heterogeneous scores.
4. This information could be more succinctly conveyed by a procedure analogous to a chemical formula, but consisting of a description of the behaviours which someone is strongly internally motivated to undertake and the competencies displayed while undertaking them.
/contd
EC&MOS.ppt 085
To summarise what we have done: - 2
5. Unless the environments in which people find themselves engage with their motives they will neither develop nor display the competencies of which they are capable.
6. The environments in which people find themselves can also be described in a manner analogous to that used in chemistry.
7. More than that, these descriptions of the situation in which observations are made must become an integral part of the assessment of the individual
8. This way of thinking allows us, for the first time, to model the transformational processes which take place in homes, schools, and workplaces. Non linear, but, equally, cannot transform people into others for which they do not have the potential.
9. It remains to add that, just as chemists require thorough familiary with the conceptual framework they are using and with a range of interventionist analytical procedures to get behind surface characteristics, the same is going to be true in psychology.
EC&MOS.ppt 086
Ways of getting the information needed to complete the grid or make meaningful
descriptive statements1.Observation.
2.Projective methods and Behavioral Event Interviewing.
3.Portfolios of work; Records of personal experience.
4.Value-expectancy methods.
EC&MOS.ppt 087
1. Observation
EC&MOS.ppt 088
Competencies Possessed
Competencies Displayed
(but only observable through value-tinted spectacles)
Values / values for task
Environment: (1) Whether previously tapped values so that informant discovered & developed relevant competencies.
(2) Whether task currently engages informant’s values.
EC&MOS.ppt 090
It follows that we cannot expect people to exercise, never mind display, the high-level competencies they possess unless they are in a situation that engages their motives.
So an assessment of the context in which the observations were made must form an integral part of the statement.
EC&MOS.ppt 091
Hence problem with Assessment Centres: May not be interested in getting people to work together to build bridges, but may well be very interested in developing building materials.
EC&MOS.ppt 092
Nevertheless one can use direct observation with reasonable confidence if the context is a “developmental environment”.
And one can easily find out whether this is the case by using appropriate Questionnaires.
So the topic shifts from one of accrediting student outcomes to accrediting courses.
And this means studying whether real thought has been given to individual students’ developmental programmes.
The network of personnel needed to do this – i.e. to accredit courses as well as outcomes - contributes to the introduction of an innovatory educational system because it promotes discussion of goals and how to achieve them.
EC&MOS.ppt 094
Assessment: Part II
Projective Methods and Behavioral Event
Interviewing
EC&MOS.ppt 095
Projective Methods
EC&MOS.ppt 095
Display File: TAT PICS.ppt
Instructions for Story Composition
(filename: unknown)
Work Through Scoring System
(filename: unknown)
Practice Stories(filename: unknown)
Score Own Stories(filename: unknown)
EC&MOS.ppt 097
Examples of Potentially Valued Styles of Behaviour
Achievement Affiliation Power
Examples of componentsof effective behaviour.
Doing things w
hich have not been done before.
Inventing things.
Doing things m
ore efficiently than they have been done before.
Developing new
formal scientific theories.
Providing support and facilitation for som
eone concerned with achievem
ent.
Establishing w
arm, convivial relationships w
ith others.
Ensuring that a group w
orks together without conflict.
Establishing effective group discussion procedures.
Ensuring that group m
embers share their know
ledge so that good decisions can be taken.
Articulating group goals and releasing the energies of others in pursuit of them
.
Setting up dom
ino-like chains of influence to get people to do as one wishes w
ithout having to contact them
directly.
CognitiveThinking (by opening one's mind to experience, dreaming, and using other sub-conscious process) about what is to be achieved and how it is to be achieved.
Anticipating obstacles to achievement and taking steps to avoid them.
Analysing the effects of one's actions to discover what they have to tell one about the nature of the situation one is dealing with.
Making one's value conflicts explicit and trying to resolve them.
Consequence anticipated:Personal: e.g. "I know there will be difficulties, but I know from my previous experience that I can find ways round them.
Personal normative beliefs: e.g. "I would have to be more devious and manipulative than I would like to be to do that."
Social normative beliefs: e.g. "My friends would approve if I did that": "It would not be appropriate for someone in my position to do that."
AffectiveTurning one's emotions into the task:Admitting and harnessing feelings of delight and frustration:using the unpleasantness of tasks one needs to complete as an incentive to get on with them rather than as an excuse to avoid them.
Anticipating the delights of success and the misery of failure.
Using one's feelings to initiate action, monitor its effects, and change one's behaviour.
ConativePutting in extra effort to reduce the likelihood of failure.
Persisting over a long period, alternatively striving and relaxing.
Habits and experienceConfidence, based on experience, that one can adventure into the unknown and overcome difficulties, (This involves knowledge that one will be able to do it plus a stockpile of relevant habits).
A range of appropriate routineised, but flexibly contingent behaviours, each triggered by cues which one may not be able to articulate and which may be imperceptible to others.
Experience of the satisfactions which have come from having accomplished similar tasks in the past.
EC&MOS.ppt 099
Behavioral Event Interviewing
EC&MOS.ppt 101
Assessment: Part III
Folios of Work; Reports of Personal
Experience
EC&MOS.ppt 103
Assessment: Part IV
Value Expectancy Methods
EC&MOS.ppt 104
p(B) = [∑PiIi]w0 + [∑NBpMCp]w1 + [∑NBsMCs]w1
EC&MOS.ppt 105
Value-Expectancy Methods
as Implemented in
The Edinburgh Questionnaires
i
Consequences
If you have said that it is very important to you to work in a clean environment and that you are dissatisfied with the current state of affairs; what would happen if you tried to get something done about it? How likely is it that each of the following would happen?
Very Likel
y
Unlikely
Personal Reactions
I would enjoy trying to get something done about this.
Self Image
I would have to be devious and manipulative.
Reference Groups’ Reactions
My boss would promote me.
Satisfaction
How satisfied are you with:
Hi Low
1. The cleanliness of your work environment.
2. Your opportunity to do new things which have not been done before.
Figure 28.2 The Assessment of the Components of Competence: An illustration from The Edinburgh Questionnaires. Part A The Process. Part B (Flow Chart) will be found on the next page. Note: this is a schematic representation only and does not bear a direct relationship to the Questionnaires. 106
Importance
How important is it to you to:
Hi Low
1.Work in a clean environment
2. Be able to do new things which have not been done before.
EC&MOS.ppt 106B
Importance
Hi Low
Satisfaction
Hi Low
Consequences
Yes
Personal
Self Image
Ref Group
Sum of products gives probability that will do something about it Johnsons “resultant”
x
x
xCompareMost Important Source of Dissatisfaction
Use “importance ratings to weight consequences by importance attached to them
Cumulate over workforce data to form the basis for discussion.
Cumulate over workforce data to form the basis for discussion.
Cumulate over workforce data to form the basis for discussion.
1. Organisational Development2. Monitoring of Effectiveness of O.D.3. Accreditation of Agents
Inspection of individual items leads to guidance and placement.
Inspection of individual items gives personal development needs and hence placement.
Figure 28.2 (Continued) The Assessment of the Components of Competence: An illustration from The Edinburgh Questionnaires. Part B Flow Chart. Note: The Figure has been prepared for illustrative purposes. The flows are over simplified and schematic.
EC&MOS.ppt 107
The main demonstrations of the value of the model have so far been in
programme evaluation:Lothian Region Educational Home
Visiting Projectand
Value of Environmentally Based, Competency-Oriented, Project Work.
alsoCross-Cultural Comparisons:
Germany, Japan, US, UK etc. (Graham).
EC&MOS.ppt 108
To Summarise what I have said:
1. If we want to:• Develop and capitalize upon and reward more
of the human resources available.• Run ethical schools.• Mount genuinely scientific evaluations to
guide policy.• Stem the destruction of the planet.
We need to evolve new ways of thinking about competence and how it is to be assessed.
/cont
EC&MOS.ppt 109
Summary (contd.)
2. We need to move toward a measurement model which:
• Seeks to be value-based rather than value free.• Utilizes a two-stage model.• Builds on a multiple-regression sum across
independent predictors, rather than an internal consistency model.
• Embodies multiple talents.• Includes beliefs about how society works and
one’s role in it./contd
EC&MOS.ppt 110
Summary (contd.)
3. We need to abandon our preoccupation with variables and:
• Adopt a descriptive paradigm.• Discriminate between people mainly in terms
of the activities they value and the competencies they bring to bear whilst undertaking them.
• Include in our descriptions a statement about those aspects of the environment which engage with the motives or values of the person being assessed and thus lead him or her to display the talents they possess.
EC&MOS.ppt 111
Conclusion
I have shown that an alternative model is:
• Operationalisable.
• Yields more valid results.
• Yields results which are much more morally justifiable.
• Yields results which lead to greater advances in understanding.
• Yields more useful results.
EC&MOS.ppt 112
In comparison, traditional forms of academic assessment:• Have zero predictive validity outside the
educational system.• Are more dependent on the examiner than the
student (60% of the variance being so accounted for).
• Have little discriminative power: The difference between an A and a D grade typically being 8 raw score points.
• Depend mainly on presentation.• Relate to material which was out-of-date when it
was taught, does not relate to people’s problems, and will be forgotten by the time it is needed.
• /
contd
EC&MOS.ppt 113
Traditional forms of academic assessment, contd. • Do not identify the unique combinations of up-to-
date, largely tacit, knowledge that people require to tackle their problems.
• Assess temporary knowledge of poorly sampled sub domains of the vast amount of explicit knowledge that exists.
• Make no assessment of the competencies actually required to perform well in the domains of competence they are said to identify: i.e. they lack construct as well as predictive validity.
• Help to legitimize a divided society which compels most people to contribute in ways they do not like to processes that are heading our species toward its own extinction, carrying the destruction of the planet with it.
EC&MOS.ppt 114
In fact, it has now been recognized even by the American Psychological Association that failure to make broadly-based, comprehensive, evaluations of both people and programmes represents the most widespread and most serious misuse of tests.
top related